PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

arXiv:2604.08340v1 [cs.CV] 9 Apr 2026

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models Ruizhi Zhang

Ye Huang∗

Yuangang Pan

Chuanfu Shen

Zhilin Liu

SIAS, UESTC Shenzhen, China

SIAS, UESTC Shenzhen, China

CFAR/IHPC A*STAR Singapore

SIAS, UESTC Shenzhen, China

SIAS, UESTC Shenzhen, China

Ting Xie

Wen Li

Lixin Duan

SIAS, UESTC Shenzhen, China

SIAS, UESTC Shenzhen, China

SIAS, UESTC Shenzhen, China

Prior Works

State: XYZ = (239, 125, 32) Front_Voxel = Wood Block ID = 3

PokeGym O1 2D Tiles

3D World

Fixed Views

Variable Views

Simplified Vision

Complex Elements

State-based Input

Vision-only Input

Symbolic Format

Human-like View

Degrades to LLM Task

VLM Real Challenge

Human Eval

Automated Eval

Costly

Scalable

Human Bias

Objectivity

Human Dependent

Liberate Human

Task is successful/failed.

Task is successful/failed.

Figure 1: Advancing prior works, PokeGym features complex 3D environments, raw pixels, and scalable automated evaluation.

Abstract While Vision-Language Models (VLMs) have achieved remarkable progress in static visual understanding, their deployment in complex 3D embodied environments remains severely limited. Existing benchmarks suffer from four critical deficiencies: (1) passive perception tasks circumvent interactive dynamics; (2) simplified 2D environments fail to assess depth perception; (3) privileged state leakage bypasses genuine visual processing; and (4) human evaluation is prohibitively expensive and unscalable. We introduce PokeGym, a visually-driven long-horizon benchmark instantiated within Pokémon Legends: Z-A, a visually complex 3D open-world Role-Playing Game. PokeGym enforces strict code-level isolation: agents operate solely on raw RGB observations while an independent evaluator verifies success via memory scanning, ensuring pure vision-based decision-making and automated, scalable assessment. ∗ Corresponding author

The benchmark comprises 30 tasks (30–220 steps) spanning navigation, interaction, and mixed scenarios, with three instruction granularities (Visual-Guided, Step-Guided, Goal-Only) to systematically deconstruct visual grounding, semantic reasoning, and autonomous exploration capabilities. Our evaluation reveals a key limitation of current VLMs: physical deadlock recovery, rather than high-level planning, constitutes the primary bottleneck, with deadlocks showing a strong negative correlation with task success. Furthermore, we uncover a metacognitive divergence: weaker models predominantly suffer from Unaware Deadlocks (oblivious to entrapment), whereas advanced models exhibit Aware Deadlocks (recognizing entrapment yet failing to recover). These findings highlight the need to integrate explicit spatial intuition into VLM architectures. The code and benchmark will be available on GitHub.

1

Introduction

Recent Vision-Language Models (VLMs) have achieved impressive progress in static visual understanding and instruction following

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

[11, 14, 37, 58]. Yet it remains unclear to what extent these capabilities translate into autonomous behavior in visually rich 3D environments [12, 22, 82], where agents must perceive from pixels, act under partial observability, and pursue long-horizon goals through continuous interaction [32, 70, 75, 80]. A central obstacle is the lack of benchmarks that can evaluate it faithfully and at scale. An effective benchmark for embodied VLM agents should jointly enable at least four properties: long-horizon interaction, realistic 3D visual reasoning, decision-making from pure visual observations, and scalable automated evaluation. However, existing protocols typically trade away one or more of these properties: (1) Static image benchmarks and single-turn tasks, such as visual question answering (VQA) or image captioning [2, 17, 34, 41, 76], reduce evaluation to momentary recognition and bypass the challenges of persistent planning and control [50, 73]. (2) Interactive benchmarks in 2D games or grid worlds [21, 49] introduce sequential decision-making, but their simplified visuals do not match the complexity of real-world scenes, failing to capture depth perception and 3D spatial reasoning. (3) More realistic 3D environments often expose privileged internal states, such as coordinates or symbolic world representations [10, 15, 33, 38, 88], allowing agents to bypass the perceptual burden that real-world visual agents must solve. (4) Conversely, game benchmarks that restrict agents to pure visual inputs frequently rely on human evaluation [5, 61, 62, 65], limiting scalability, reproducibility, and objectivity. As a result, strong performance on existing benchmarks may not reflect robust embodied competence. To bridge this gap, as illustrated in Figure 1, we introduce PokeGym, a visually-driven, long-horizon benchmark instantiated in a 3D open-world Role-Playing Game (RPG), Pokémon Legends: ZA. This game serves as an ideal testbed because its mechanics mirror the core challenges of real-world embodiment: partial observability forces agents to build spatial memory, navigation and diverse object interactions test fine-grained visual-action grounding, while intricate quest structures and extended temporal dependencies demand robust long-horizon planning and error recovery. PokeGym resolves the tension between pure visual realism and automated evaluation: the agent acts solely from raw RGB observations, while task success is verified independently through state extraction using Array of Bytes (AOB) memory scanning. PokeGym contains 30 tasks derived from 10 quests, with trajectories ranging from 30 to 220 environment steps and covering navigation, interaction, and mixed long-horizon scenarios. Each task is instantiated under three instruction granularities: VisualGuided, Step-Guided, and Goal-Only. These granularities create a controlled setting for disentangling embodied capabilities: visual grounding under explicit cues, semantic reasoning under procedural guidance, and autonomous exploration under sparse goals. Beyond success rates, PokeGym also supports fine-grained diagnosis of embodied failures, highlighting the value of PokeGym not only as an evaluation suite, but also as a diagnostic testbed for embodied VLM research. Our primary contributions are summarized as follows: (1) We introduce PokeGym, a visually-driven, long-horizon benchmark for embodied VLMs in a 3D open-world game. Its mechanics capture core challenges of real-world embodiment.

Table 1: Comparison of VLM Benchmarks. Open World reflects whether the environment permits unconstrained, nonlinear exploration. Interactivity differentiates closed-loop multi-turn embodied dynamics from passive single-turn responses. Long-Horizon indicates the necessity for multi-step sequential planning. Evaluation Benchmark

Open InterLong Env Only Eval World activity Horizon Domain Vision Method

MVP-Bench [28] LVLM-eHub [76] VLMbench [84] VisGym [72] NetHack [26] StarDojo [60] MINEDOJO [15] Cradle [62] Lumine [61]

× × × × ✓ ✓ ✓ ✓ ✓

Single Single Multi Multi Multi Multi Multi Multi Multi

× × × ✓ ✓ ✓ ✓ ✓ ✓

VQA VQA Robotics Mixed 2D RPG 2D RPG 3D RPG 3D RPG 3D RPG

✓ ✓ × × × × × ✓ ✓

QA Acc QA Acc Auto Auto Auto Auto Auto Human Human

PokeGym

✓

Multi

✓

3D RPG

✓

Auto

(2) We present a rigorous and scalable evaluation pipeline in the complex game environment. It restricts agents to pure-pixel observations by eliminating privileged state leakage, and features an independent evaluator that extracts game states via AOB memory scanning for automated, objective verification. (3) We establish a controlled diagnostic framework for disentangling key embodied capabilities in VLMs. Specifically, we design 30 long-horizon tasks across three instructional granularities to independently assess visual grounding, semantic understanding, and autonomous exploration. (4) We provide a comprehensive analysis of VLM failures, revealing that physical deadlock recovery—rather than high-level planning—is the primary bottleneck. We further uncover a metacognitive divide between weaker and stronger models when trapped.

2 Related Work 2.1 Benchmarks for VLMs The growth of Vision-Language Models (VLMs) has shifted evaluation from static perception to dynamic interaction [7, 18, 57]. Early benchmarks typically evaluate VLMs on passive visual understanding tasks, such as Visual Question Answering (VQA) [28, 76], image captioning [9, 27, 35, 87], and visual grounding [8, 56, 77, 86]. While some benchmarks have utilized videos for semantic and spatial reasoning [28, 79], they treat perception as a passive task, overlooking the interactive dynamics of closed-loop environments, where an agent’s actions continuously alter future observations. To address this, recent efforts have introduced interactive and embodied benchmarks [16, 23, 36, 43, 59, 68]. Frameworks such as VLMbench [84] focus on tabletop manipulation, whereas VisGym [72] and EMemBench [31] evaluate multi-step visual interactions and episodic memory. Despite these advancements, existing interactive benchmarks rely on constrained state spaces or short episodes, reducing the need for long-range planning. In contrast, PokeGym plunges VLMs into a visually complex, unconstrained 3D open world, demanding sustained visual interaction and long-horizon spatial planning.

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

2.2

Game-based Evaluation Environments

Games have served as ideal testbeds because they provide rich visual and diverse gameplay [4, 42, 48, 51, 81]. Traditional game benchmarks such as NetHack [26], DOOM [25], and 2D grid-worlds like Pokémon Red [49], have been used for reinforcement learning [47, 67, 74]. With the rise of foundational agents, recent works have shifted towards open-ended simulations and RPGs [19, 40, 55, 69, 78, 85]. For instance, StarDojo [60] evaluates agents in productionliving simulations Stardew Valley, while MineDojo [15] assesses agents across open-ended crafting and exploration tasks in the 3D voxel world of Minecraft. More recently, many agents interact with complex 3D worlds through screen pixels, keyboard and mouse actions [30, 53]. Some works have demonstrated that VLM agents can complete long missions in AAA games [61, 62]. Additionally, foundation models like NitroGen [39] have shown impressive crossgame generalization. However, evaluating these agents reveals critical flaws: 2D games lack spatial realism, 3D simulators leak game states, and pixel-only AAA games demand unscalable human assessment. PokeGym resolves this by combining a complex 3D world and pure-pixel inputs with an memory-based evaluator, ensuring scalable, automated, and objective success verification. The qualitative comparison of VLM benchmarks is summaried in Table 1.

3 PokeGym Benchmark 3.1 Game Environment PokeGym is a visual-centric, long-horizon evaluation benchmark built upon the 3D open-world game Pokémon Legends: Z-A. Unlike traditional 2D grid-world benchmarks or sandbox-style 3D environments (e.g., Pokémon Red [49] or Minecraft [15]), this game provides a richer and more challenging setting for VLM-based agents, mainly due to three distinctive properties: (1) Freely controllable camera with changing viewpoints. The game camera can be rotated to view the world from different angles. This makes the observation space highly viewpointdependent: key targets may be outside the screen, partially blocked, or only recognizable from specific angles. As a result, the agent must actively look for useful information by turning the camera, checking nearby areas, and adjusting its distance to objects rather than passively reacting to a fixed view. (2) Visually complex 3D scenes with dense, diverse elements. The open world contains cluttered geometry (buildings, vegetation), dynamic actors (NPCs, wild Pokémon), interactive props, UI overlays, and multiple depth layers. To act correctly, the agent needs to disambiguate similar-looking objects, read small text, and reason over spatial relations under lighting changes and occlusion. (3) Structured progression beyond sandbox-style planning. In contrast to Minecraft [10, 15, 33], where long-term planning is often centered on resource gathering, crafting and construction, Pokémon Legends: Z-A features progression that is tied to quests, encounters, and event triggers. Agents must coordinate exploration, object interaction, battle, and goal completion under delayed and context-specific consequences, making success depend not only on open-ended planning but also on understanding task structure and scripted progression.

3.2

Task Definitions and Budgets

PokeGym contains 30 long-horizon tasks spanning three categories: navigation, interaction, and mixed tasks. These categories broadly cover movement to target locations, interaction with objects, and multi-stage tasks that combine multiple gameplay skills. Further details are provided in the supplementary material. To eliminate ambiguity, every task is formalized with 4 components. Initial State: Each task is initialized from a corresponding preconfigured save file to equalize starting conditions for all agents. Success Criteria: Task completion is threshold-verified using memory variables (e.g., a navigation goal is complete when the coordinates fall within a predefined bounding box). Fixed Step Budget: Each task is assigned a fixed budget of environment steps. Based on heuristic human demonstrations, the budgets range from 180 to 360 environment steps. Termination: An episode terminates under two conditions: (1) Success criteria met; (2) Step budget exhausted. The relevant information of the tasks is displayed on Figure 2.

3.3

Instruction Granularity & Cognitive Probes

To diagnose the specific bottlenecks of VLM agents, the 30 tasks are derived from 10 distinct quests. We map these tasks across three levels of instruction granularity, varying the information density to probe distinct cognitive capabilities, as illustrated in Figure 2. Visual-Guided: The prompt provides a multi-stage procedural plan with visual anchors (e.g., "Approach and enter the door of the house, locate and talk to the hotel owner behind the reception desk"). This setup evaluates the model’s visual grounding capability and the ability to map linguistic descriptions to pixel-level features. Step-Guided: The prompt retains the procedural sub-goals but removes the visual anchors (e.g., "Approach and enter the door of the house, locate and talk to the hotel owner"). Without specific visual features, the agent must rely on semantic understanding and common sense to identify generic objects. Goal-Only: The prompt provides only the ultimate objective (e.g., "Locate and talk to the hotel owner"). The agent must autonomously decompose the goal, explore the space, and deduce the intermediate steps. This setting tests long-horizon planning and autonomous exploration capabilities. By comparing performance across above tiers, we can systematically probe an agent’s specific cognitive strengths and bottlenecks.

3.4

System Architecture

The architecture of PokeGym is illustrated in Figure 3. At a high level, the framework consists of four parts: (i) an observation interface that provides visual inputs from the environment, (ii) a VLMbased decision module, optionally augmented with a self-reflection mechanism, (iii) an action interface that translates model outputs into executable controls, and (iv) an evaluation interface for automated progress tracking and success verification. The environment is built on the Ryujinx emulator implemented in C#. Observation Interface. PokeGym models the agent as a pure visual learner. At each decision step, the agent receives configurable visual observations, with the current front-view frame serving as the default input across all settings. To provide richer spatial and temporal context, the observation space can be extended with:

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

PokeGym

Nav

Int

···

···

···

··· Mix

Instruction Granularity Approach and enter the door of the house, locate and talk Visual-Guided to the hotel owner behind the remove visual reception desk. anchor Approach and enter the door Step-Guided of the house, locate and talk remove step to the hotel owner. guide Locate and talk to the hotel Goal-Only owner.

Figure 2: Overview of the tasks of PokeGym. The Top 3 Rows: Sample visual trajectories representing Navigation (Nav), Interaction (Int), and Mixed (Mix) tasks. Bottom-Left: Illustration of the three instruction granularities. Bottom-Right: Environment step budgets and distribution of the 10 quests evaluated in the benchmark. VLM

Input

Responses VLM Decision Module

Observation Interface Current Frame

Left View

Previous Frame

Right View

Optional: Self-Reflection - Reflect on the strategy - Update short-term memory - Analyze and refine - Update experiences

Action Interface Defined High-level Actions PressA

RotateLeft

Parametric Control X: -0.6 Type: LeftStick

GPU Textures Auxiliary Design: adaptive pause mechanism

··· MoveForward Y:1.0 Duration: 500ms

Ryujinx Emulator

Automated Evaluation Interface

AOB Scanning

- Map ID; - X/Y/Z Coordinates; - Quest Flags;

Figure 3: Overview Architecture of the proposed PokeGym. • Previous frame: the frame before the last executed action, enabling reflection on action outcomes and temporal feedback; • Left and right (L/R) views: peripheral images that expand the agent’s spatial awareness.

Rather than relying on OS-level screen capture, these RGB observations are directly extracted from GPU textures. This design reduces

visual acquisition latency, avoids rendering bottlenecks, and eliminates window occlusion issues. To ensure fairness, no internal game state is exposed to the agent. VLM Decision Module. Given the visual observations, the VLM produces action decisions based solely on the provided image context and interaction history. To further support long-horizon adaptation, we provide an optional self-reflection module. When enabled, every 𝑘 steps (default 𝑘=5), a summarization routine prompts the VLM to analyze recent response history and evaluate the effectiveness of its current strategy. The resulting reflection updates the short-term memory M𝑡 , while distilled actionable insights are written into the persistent experience library E𝑡 through (ADD, DEL, MOD, KEEP) operations. This design keeps the context concise while allowing the model to iteratively revise its strategy online, despite the lack of explicit external feedback. Action Interface. PokeGym has two action execution paradigms: • Defined high-level actions: the agent outputs discrete commands (e.g., MoveForward, RotateRight), which are mapped to fixed execution durations in the environment wrapper (e.g., 500 ms for moving and 200 ms for rotating); • Parametric control: the agent directly specifies the maneuver type, execution duration, and continuous joystick values (e.g., 𝑋, 𝑌 ∈ [−1.0, 1.0]). To support different planning granularities, we decouple decision steps from environment steps. A decision step corresponds to one model query, whereas an environment step corresponds to

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

one physically executed action in the emulator. Accordingly, the VLM may output either a single action (1 environment step) or an ordered sequence of actions (3 environment steps) per query. For fair comparison, the total budget of environment steps is kept constant across settings. Evaluation Interface. For automated progress tracking and success verification, the environment performs Array of Bytes (AOB) memory scanning at initialization to locate memory addresses associated with map IDs, character coordinates, and quest flags via signature patterns. These values are only used by the evaluator and are never exposed to the agent prompt. This mechanism enables scalable and cross-machine automatic evaluation under the same game version, removing the need for manual checking. Auxiliary Design. For combat tasks that require high-frequency reactions, we introduce an adaptive pause mechanism that pauses the environment during the reasoning phase and resumes during action execution. This prevents differences in VLM inference latency from introducing confounding bias in time-sensitive scenarios.

3.5

Compliance and Reproducibility

PokeGym does not distribute game ROMs, decryption keys, firmware, or any proprietary assets. Researchers must legally acquire and dump their own game copies to use the benchmark. Given a legally obtained ROM and the specified game version, PokeGym can be reproduced by combining an open-sourced emulator framework, pre-configured initial save files for each task, and an automatic evaluator that verifies success through signature patterns. These components will be released as non-proprietary resources.

4 Experiments 4.1 Experimental Design Overview The proposed benchmark can differentiate models across distinct embodied capabilities (capability coverage), offers interpretable diagnosis of both cognitive and physical failure modes (diagnosticity), and supports controlled analysis of interventions and design choices (actionability). It is designed not only to report model rankings but also to serve as a useful evaluation instrument. Capability coverage. We evaluate a diverse set of VLMs under three instruction granularities. This design enables our benchmark to distinguish models along multiple embodied capabilities, including visual grounding, semantic reasoning, and long-horizon planning. Rather than collapsing these abilities into a single undifferentiated score, our benchmark reveals fine-grained performance differences across models. Diagnosticity. Beyond final task success, we analyze the execution process through trajectory-level physical metrics and detailed failure categories. It reveals why agents fail, rather than merely indicating failure outcomes. This diagnostic value enables systematic failure decomposition across models and task settings. Actionability. Finally, we perform intervention and ablation studies, including deadlock interventions, visual-context ablations, action-execution strategies, and self-reflection analysis. These support flexible combinations of diverse configurations and enables close inspection of model behaviors. This modular design yields actionable insights by pinpointing bottlenecks and providing targeted guidance for improving model and agent architectures.

Table 2: Performance comparison across 3 granularity levels. Success Rate (SR, %) measures the percentage of episodes that successfully complete the task. Average Environment Steps (Stp) denote the average number of environment steps in successful episodes. Bold indicates the best performance. Model

Navigation Interaction SR↑

Mixed Stp↓

Average

Stp↓

SR↑

Stp↓ SR↑

SR↑

GLM-4.6V 25.00 123.20 Qwen3.5-35B 45.00 124.67 Qwen3.5-122B 60.00 124.92 55.00 81.73 Visual Qwen3.5-Plus Guided Qwen3-VL-30B 50.00 89.10 Claude-Sonnet-4.6 55.00 124.45 Gemini-3-Pro 20.00 120.00 GPT-5.2 25.00 147.00

46.67 80.00 66.67 66.67 66.67 80.00 66.67 93.33

58.14 60.00 74.33 43.89 85.22 61.75 26.67 84.50 50.56 90.31 67.10 53.33 101.38 60.00 97.80 65.30 26.67 153.00 49.45 100.01 50.20 53.33 142.25 56.67 93.85 81.00 46.67 131.14 60.56 112.20 61.60 46.67 134.14 44.45 105.25 41.50 60.00 86.22 59.44 91.57

Stp↓

GLM-4.6V 25.00 136.80 Qwen3.5-35B 45.00 85.56 Qwen3.5-122B 25.00 79.40 Step Qwen3.5-Plus 50.00 75.70 Guided Qwen3-VL-30B 40.00 73.50 Claude-Sonnet-4.6 55.00 81.73 Gemini-3-Pro 70.00 101.86 GPT-5.2 30.00 96.00

53.33 60.00 66.67 53.33 60.00 60.00 93.33 86.67

42.13 46.67 66.29 41.67 81.74 77.56 33.33 89.40 46.11 84.17 37.10 20.00 162.33 37.22 92.94 42.75 26.67 125.50 43.33 81.32 60.11 46.67 115.43 48.89 83.01 91.22 60.00 155.33 58.33 109.43 85.29 60.00 104.89 74.44 97.34 74.62 53.33 94.00 56.67 88.21

GLM-4.6V 25.00 211.40 73.33 46.73 26.67 166.00 41.67 141.38 Qwen3.5-35B 45.00 111.56 80.00 77.92 13.33 125.00 46.11 104.82 Qwen3.5-122B 25.00 126.20 73.33 39.64 40.00 126.17 46.11 97.33 50.00 66.60 46.67 79.00 20.00 100.33 38.89 81.98 Goal Qwen3.5-Plus Only Qwen3-VL-30B 45.00 90.78 73.33 92.45 33.33 147.80 50.55 110.34 Claude-Sonnet-4.6 55.00 99.73 60.00 59.78 6.67 125.00 40.56 94.84 Gemini-3-Pro 45.00 108.22 100.00 79.00 26.67 115.75 57.22 100.99 GPT-5.2 40.00 76.25 100.00 89.07 40.00 145.33 60.00 103.55

4.2

Implementation Details

We evaluate diverse VLMs, encompassing both open-weight models (GLM-4.6V [66], Qwen 3/3.5 series [3, 52, 64]) and closed-source proprietary models (GPT-5.2 [44], Gemini-3-Pro [13], and ClaudeSonnet-4.6 [1]). Each setting is evaluated with 5 trials. All models share the identical initial state, prompt template, and budget accounting within the same task. An episode terminates when the task is successfully completed or the step budget is exhausted.

4.3

Cognitive Capability Coverage

Table 2 presents a comparison of model performance across the three instruction granularity levels. For the experiments in this section, the observation space includes all four images. For the action space, all models employ the defined high-level actions paradigm, and each decision step outputs an ordered sequence of three actions, equating to three environment steps. Visual Grounding. In the Visual-Guided tasks, the prompt provides procedural steps with visual anchors. Claude-Sonnet-4.6 achieves the highest average Success Rate (SR 60.56%), closely followed by Qwen3.5-122B (60.00%) and GPT-5.2 (59.44%), indicating strong grounding from visual cues to actionable decisions. Qwen3.5122B achieves the best Navigation SR (60.00%), highlighting its visual grounding capability in spatial traversal, enabling it to leverage visual references for navigation and movement decisions. Semantic Reasoning. In the Step-Guided tasks, visual references are removed and the procedural sub-goals are retained, forcing agents to rely on semantic understanding to identify generic objects within the 3D environment. Gemini-3-Pro experiences a performance leap, surging from an average SR of 44.45% to a leading 74.44%, while dominating Navigation (70.00%), Interaction (93.33%)

6XFFHVV5DWH

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

9LVXDO*XLGHG

6WHS*XLGHG

SHDUVRQU  S

 

*RDO2QO\

SHDUVRQU  S

SHDUVRQU  S

    









,QHIIHFWLYH0RYHV,0 &ODXGH6RQQHW */09











,QHIIHFWLYH0RYHV,0

*37 *HPLQL3UR











,QHIIHFWLYH0RYHV,0

4ZHQ9/% 4ZHQ%

4ZHQ% 4ZHQ3OXV

Figure 4: Correlation between Success Rate and Ineffective Moves. Each model has 10 points per subplot, representing its average performance across the 10 distinct tasks (5 trials each). The dashed lines indicate linear regression trends, and shaded areas represent the 95% confidence intervals. and Mixed (60.00%). This indicates that it can leverage its pretrained world knowledge and common sense to infer the visual appearance of generic targets, demonstrating robust semantic reasoning and reliable instruction following. In contrast, the openweight models, including the Qwen models and GLM-4.6V, lag behind in this setting, with average SRs clustered between 37.22% and 48.89%, substantially below Gemini-3-Pro, Claude-Sonnet-4.6, and GPT-5.2. The results reveal a gap between open-source and closed-source models in semantic reasoning. Long-term Planning and Autonomous Exploration. The Goal-Only setting strips away procedural sub-goals. Under this sparsity, both GPT-5.2 and Gemini-3-Pro achieve a 100.00% SR in Interaction tasks, indicating robust goal alignment and physical manipulation capabilities once a target is identified. However, performance generally degrades on Mixed tasks, highlighting the need for longhorizon planning and exploration. For instance, Gemini-3-Pro’s Mixed SR collapses from 60.00% (Step-Guided) to 26.67%, ClaudeSonnet-4.6’s Mixed SR plummets to a mere 6.67%, and Qwen3.5-Plus manages only 20.00%. Cross-Granularity Analysis. Comparing performance across instruction granularities reveals opposite responses to the removal of visual guidance. Gemini-3-Pro improves markedly once visual anchors are removed, with average SR rising from 44.45% under Visual-Guided to 74.44% under Step-Guided, with Navigation increasing from 20.00% to 70.00% and Mixed from 46.67% to 60.00%. This suggests that dense visual cues may over-constrain Gemini’s reasoning, acting as distractors rather than useful grounding signals. In contrast, Qwen models deteriorate when visual references are removed. Qwen3.5-122B drops from 60.00% to 37.22% average SR from Visual-Guided to Step-Guided, indicating stronger dependence on explicit visual anchors for object grounding and trajectory alignment in 3D scenes.

4.4

Physical Bottlenecks Diagnosis

While the previous analysis highlights macro-level cognitive bottlenecks, empirical observations reveal that low-level physical deadlocks are a prominent characteristic of failed episodes. To quantify this embodied friction, we monitor Ineffective Moves (IM), which measure decision steps where movement actions result in zero spatial displacement due to collisions with the environment. Figure 4 shows a significant negative Pearson correlation between Success Rate (SR) and IM across all three instruction granularities, with correlation coefficients of 𝑟 = −0.57, 𝑟 = −0.65, and

𝑟 = −0.52, respectively. Moreover, all correlations are statistically significant with 𝑝 < 0.001, confirming that the observed negative association is highly unlikely to arise by chance. These results underscore that failures in collision handling are not incidental but an important bottleneck limiting successful task completion. Divergence in Recovery Behaviors. We disaggregate the performance metrics by successful and failed episodes in Table 3. The data reveals a distinct divergence in error handling dynamics. First, while successful episodes exhibit non-zero IM%, the high Recovery Rate (e.g., Gemini-3-Pro’s 100% Rec% in Mixed tasks) indicates a bump-and-recover behavior where errors are transient. Conversely, when agents fail to recover immediately, errors cascade, evidenced by the increase in Maximum Consecutive Ineffective Moves (MaxIM) across all models in failed runs. Analyzing Action Entropy (Ent) reveals the underlying behavioral collapse. Successful episodes exhibit near-zero entropy, indicating deliberate, deterministic recovery actions. In contrast, failed episodes show a notable Ent increase (e.g., GPT-5.2 in Mixed tasks jumps from 0.00 to 1.11). This demonstrates that rather than employing systematic spatial reasoning and recovery actions, trapped agents tend to exhibit erratic, high-entropy flailing. Ultimately, the gap between success and failure is defined not by the absence of errors, but by whether errors evolve into persistent, high-entropy stagnation. Efficiency in Successful Trajectories. We further evaluate the execution performance of successful episodes using processoriented metrics to assess not only task completion, but also the quality of their performance during execution. Closed-source models generally show lower IM%, MaxIM, Ent and higher Rec%, indicating smoother and more stable control. For example, Gemini-3-Pro performs best: in Navigation it records 2.12 IM% and 0.14 Ent, and in Mixed tasks nearly zero collision (0.47 IM%, 0.50 MaxIM). Conversely, open models like Qwen3.5-122B suffer high physical friction (14.40% IM%) despite ultimately completing Navigation tasks, exposing a gap in fine-grained control. This suggests stronger embodied competence means not just reaching goals, but doing so efficiently and reliably.

4.5

Failure Causes Diagnosis

Physical metrics in preceding analysis cannot explain the underlying cognitive breakdown: does the agent struggle because it is oblivious to the collision, or because it lacks the spatial intuition to escape an acknowledged trap? To answer this, we bridge the gap between the agent’s macrolevel semantic reasoning and its micro-level physical execution using a granular diagnosis. We categorize the root causes of episode failures into four types, which can almost cover the failures in our tasks. These categories are defined by contrasting the agent’s subjective internal reasoning against its objective physical state: • Unaware Deadlock: The agent is physically stuck, yet it suffers from hallucinated progress. Its internal reasoning claims that the path is clear or the strategy is effective, completely oblivious to the collision. • Aware Deadlock: The agent’s reasoning explicitly recognizes the physical deadlock. Yet, its chosen recovery actions fail to resolve the spatial trap, keeping it oscillating in a small area.

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

Table 3: Behavioral Analysis of Successful and Failed Episodes. Metrics: Ineffective Move Rate (IM%: the percentage of decision steps with movement actions that resulted in no spatial displacement), Recovery Rate (Rec%: percentage of non-IMs immediately following an IM), Maximum Consecutive Ineffective Moves (MaxIM), and Action Entropy (Ent: the Shannon entropy of actions during ≥ 3 consecutive IMs). Model

Successful Episodes

Failed Episodes

34.78 26.08 22.80 54.44 35.27 52.58 54.10 42.98

7.47 5.78 9.55 1.84 4.93 2.58 1.15 3.00

0.91 0.75 0.59 0.31 0.95 0.71 0.14 0.37

19.76 22.16 18.91 6.33 15.85 6.76 5.54 8.04

23.39 11.19 17.57 54.13 25.64 60.04 41.32 40.84

16.24 29.52 17.92 3.59 12.21 3.78 3.76 5.05

1.01 1.03 0.97 1.28 1.35 1.16 0.60 0.80

GLM-4.6V 9.22 Qwen3.5-35B 14.20 Qwen3.5-122B 12.38 Qwen3.5-Plus 9.75 Interaction Qwen3-VL-30B 12.38 Claude-Sonnet-4.6 7.85 Gemini-3-Pro 1.27 GPT-5.2 2.15

34.48 42.14 43.72 59.60 43.53 60.11 84.21 80.65

3.54 3.94 3.32 2.28 3.63 1.93 0.62 0.86

0.25 0.77 0.62 0.42 0.75 0.45 0.04 0.00

21.25 21.28 22.25 12.02 19.25 9.90 3.78 11.13

27.15 29.76 24.69 49.67 33.86 51.52 51.35 50.68

16.00 13.50 16.57 4.90 10.80 4.40 1.50 4.00

1.26 1.44 1.28 1.44 1.48 1.27 0.22 1.00

8.79 45.57 8.99 51.61 9.00 44.32 4.03 73.68 7.19 52.33 4.83 68.64 0.47 100.00 1.63 92.31

3.85 3.09 4.82 1.73 3.40 2.59 0.50 0.74

0.78 0.66 0.81 0.09 0.65 0.86 0.00 0.00

18.89 13.47 17.91 7.71 11.96 10.74 2.41 9.33

17.37 31.26 21.23 51.47 31.24 26.98 74.80 39.58

19.40 11.38 16.71 3.56 11.72 10.11 1.44 5.14

1.13 1.17 1.14 0.97 1.18 1.39 0.35 1.11

Mixed

Navigation

Interaction

SR↑

SR↑

Baseline

31.67 101.11 93.33 68.74 51.11 104.35 58.70 91.40

Stp↓

Stp↓

Mixed SR↑

Stp↓

Average SR↑

Stp↓

Textual Feedback 30.00 95.00 66.67 72.20 33.33 110.00 43.33 92.40 Forced Back 40.00 61.88 100.00 83.40 46.67 110.86 62.22 85.38 Forced Back + Rotate 30.00 61.00 86.67 89.23 33.33 103.40 50.00 84.54

IM%↓ Rec%↑ MaxIM↓ Ent↓ IM%↓ Rec%↑ MaxIM↓ Ent↓

GLM-4.6V 13.66 Qwen3.5-35B 12.85 Qwen3.5-122B 14.40 Qwen3.5-Plus 3.88 Navigation Qwen3-VL-30B 9.02 Claude-Sonnet-4.6 6.33 Gemini-3-Pro 2.12 GPT-5.2 5.93

GLM-4.6V Qwen3.5-35B Qwen3.5-122B Qwen3.5-Plus Qwen3-VL-30B Claude-Sonnet-4.6 Gemini-3-Pro GPT-5.2

Intervention Strategy

• Lost: The agent makes physical progress and continuously updates its coordinates, but fails to reach the goal within the step limit. The reasoning log confirms that the target is not visible, indicating aimless wandering. • Execution Failure: The agent successfully explores and states it sees the target in reasoning. However, it struggles with execution, stuck on adjacent micro-geometry during approach or spamming the interaction button from out of range. We utilize GPT-5.2 to automatically diagnose all failed trajectories across the five models. We feed the judge the entire episode history including the task, the agent’s internal reasoning, the chosen actions, and the objective physical states. By forcing the judge to compare the agent’s subjective text generation against the groundtruth physical trajectory, GPT-5.2 determines the failure categories, avoiding the high cost and subjective bias in manual analysis. We randomly sample 20 episodes per model (100 in total, covering 24%–31% of each model’s failures) for human annotation, approximately preserving the class distribution. GPT-5.2 judgments achieve a Micro-F1 of 0.7368 and a sample-wise Jaccard similarity of 0.6425 against human labels. These results indicate that GPTbased classification is reasonably reliable and that our four failure categories can be stably identified by human annotators. The percentage of failure categories across VLMs is shown in Figure 5. Execution Failure emerges as a universal bottleneck across all VLMs, highlighting a gap in translating 2D semantic recognition into precise 3D spatial control. Beyond this, Open-weight Qwen models are dominated by Unaware Deadlocks, suffering from cognitive errors where they persistently hallucinate progress while physically trapped. Conversely, GPT-5.2 predominantly experiences Aware Deadlocks. It correctly identifies its collision state

but fails to execute valid recovery maneuvers. This contrast suggests a metacognitive difference: weaker models more often fail to recognize that they are trapped, whereas stronger proprietary VLMs more often recognize the deadlock but still struggle to execute effective recovery maneuvers. One possible explanation is that the former lack the 3D geometric state estimation, while the latter are limited in micro-level physical control. 100

Percentage of Failure Categories (%)

Category

Table 4: Ablation Study on Deadlock Intervention Strategies. Baseline indicates no intervention.

80 60 40

29.8%

33.6%

14.0%

11.2%

45.5%

20 0

26.4%

28.8%

39.7%

38.8%

41.0%

10.9% 5.1%

15.9%

11.5%

12.4%

17.3%

32.9%

30.2%

44.2%

10.7%

GPT-5.2 Gemini-3-Pro Qwen3-VL Qwen3.5-Plus Claude

Unaware Deadlock

Aware Deadlock

Lost

Execution Failure

Figure 5: Percentage of Failure Categories across VLMs.

4.6

Interventions on Deadlocks

We conduct an intervention study on GPT-5.2 to test whether deadlocks are merely correlated with failure or are a direct cause of it. The intervention is triggered whenever the agent accumulates 3 consecutive ineffective moves. We compare three strategies: (1) Textual Feedback, which informs the model that it is stuck; (2) Forced Back, which executes 3 backward steps; and (3) Forced Back + Rotate, which executes 2 backward steps and 1 viewpoint rotation. All forced actions are counted in the action budget to ensure fair comparison. The results are shown in Table 4. Merely informing the model about the deadlock is not sufficient. Textual feedback reduces the average Success Rate (SR) from 58.70% to 43.33%, with declines across all three task types. This indicates that GPT-5.2 can often recognize that it is blocked, yet still fails to convert that awareness into an effective recovery action, consistent with our earlier diagnosis of Aware Deadlocks. Physically resolving the deadlock is more effective than textual guidance alone. Forced Back improves average SR to 62.22% while reducing average steps to 85.38, with the largest gain on Navigation tasks (SR 31.67% to 40.00%, Steps 101.11 to 61.88). Moreover, Forced Back + Rotate also achieves a higher average SR than Textual Feedback. This shows that simple deterministic recovery strategy can reliably break local traps than merely providing textual awareness of the deadlock.

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

Table 5: Analysis of Visual Inputs. Configuration

Image Navigation Interaction

L/R Views Vis. Refl. Count SR↑ × × ✓ ✓

× ✓ × ✓

Stp↓

SR↑

Stp↓

Mixed SR↑

Table 7: Comparison of model performance with (w/) and without (w/o) the Self-Reflection (Self-Refl.) module.

Stp↓

Model

30.00 79.00 46.67 104.14 33.33 70.60 35.00 138.00 46.67 64.14 73.33 107.64 20.00 131.00 86.67 50.85 46.67 83.14 31.67 101.11 93.33 68.74 51.11 104.35

1 2 3 4

Navigation

Interaction

Mixed

Paradigm Act/Q

SR↑ IM%↓ Rec%↑

SR↑ IM%↓ Rec%↑

SR↑ IM%↓ Rec%↑

High-level Actions

1 3

25.00 25.52 25.66 40.00 31.67 7.74 41.08 93.33

0.65 3.81

Parametric Control

1 3

50.00 45.48 15.00 12.93

1.09 82.61 33.33 18.11 4.23 71.11 40.00 10.19

4.7

Max

22.02 66.67 27.42 80.00

60.00 40.00 64.44 51.11

9.36 12.19 6.88 43.55 22.02 37.50

Impact of Visual Context

We analyze how different forms of visual context affect GPT-5.2, with the goal of distinguishing the value of temporal information from that of spatial information. Specifically, we compare four input settings: using only the front-view image; adding temporal visual reflection from previous frames (Vis. Refl.); adding left/right views (L/R Views) to expand spatial perception; and providing all four images together. The results are summarized in Table 5. Temporal visual reflection is the most reliable contributor to success rate. Under the same spatial-view setting, adding Vis. Refl. consistently improves SR or keeps it unchanged. Without L/R Views, it increases Navigation from 30.00% to 35.00% and Mixed from 33.33% to 73.33%. With L/R Views, it further improves all 3 tasks. These results show temporal reflection yields the most stable, consistently positive gains across settings, likely because recent visual history helps the model verify action outcomes and maintain cross-step consistency in partially observable environments. Spatial views have a task-specific effect: they help Interaction, but tend to hurt Navigation. Holding temporal reflection fixed, adding L/R Views always produces a large SR gain on Interaction: from 46.67% to 86.67% without temporal reflection, and from 46.67% to 93.33% with temporal reflection. In contrast, Navigation drops in both settings. This suggests that spatial views are better aligned with Interaction than with Navigation, likely because they reveal useful object relations for Interaction but may distract from forward-looking cues in Navigation.

4.8

Effect of Action Execution Strategies

We study how action paradigms and execution frequency affect GPT-5.2. Specifically, we compare High-level Actions and Parametric Control under different execution frequencies, controlled by the maximum number of actions predicted per query (Max Act/Q). The results are shown in Table 6. High-level actions benefit from multi-step execution. Under High-level Actions, increasing Max Act/Q from 1 to 3 generally improves Success Rate (SR) and Recovery Rate (Rec%) while reducing Ineffective Move Rate (IM%). This suggests that predictive macro-actions allow the model to plan over short horizons more effectively and reduce getting trapped in physical deadlocks.

Interaction SR↑

Stp↓

Mixed SR↑

Stp↓

Average SR↑

Stp↓

w/o 51.67 74.90 55.56 61.92 24.44 128.64 43.89 88.49 Qwen3.5-Plus w/ 41.67 119.84 66.67 61.87 24.44 126.18 44.26 102.63

Table 6: Analysis on Action Execution Paradigms. The environment step budget remains constant across all settings in the same task. Action

Self- Navigation Refl SR↑ Stp↓

Qwen3-VL

w/o 45.00 85.04 66.67 68.67 44.44 134.25 52.04 95.98 w/ 55.00 110.70 68.89 75.55 28.89 111.77 50.93 99.34

GPT-5.2

w/o 31.67 101.11 93.33 68.74 51.11 104.35 58.70 91.40 w/ 33.33 109.95 93.33 84.02 48.89 83.05 58.52 92.34

Gemini-3-Pro

w/o 45.00 106.67 86.67 76.79 44.44 117.30 58.70 100.25 w/ 60.00 96.22 93.33 94.40 44.44 103.50 65.93 98.04

Parametric control is less robust under multi-step execution. When executing three actions per query, Parametric Control yields lower SR and higher IM% than High-level Actions across all task types. This suggests that fine-grained control poses greater challenges when predicting multiple future actions, as small lowlevel errors can accumulate across steps and are harder to correct without frequent replanning.

4.9

Efficacy and Limitations of Self-Reflection

We evaluate self-reflection across multiple models to examine under what conditions reflecting on recent history is beneficial for online decision making, as presented in Table 7. The effectiveness of self-reflection depends on model capability. For strong proprietary models such as Gemini-3-Pro, reflection substantially improves the average SR, from 58.70% to 65.93%, while also reducing steps. In contrast, weaker models do not benefit consistently: Qwen3-VL drops sharply on Mixed tasks (44.44% to 28.89%), and Qwen3.5-Plus also declines on Navigation. This suggests that self-reflection is useful only when the model can reliably revise its own strategy rather than amplify earlier mistakes. Self-reflection is consistently ineffective for Mixed tasks. Across all evaluated models, reflection fails to improve SR on Mixed tasks. This indicates a limitation of history-based reflection in environments with drastic context shifts, such as switching from navigation to combat. In such cases, recent history may become outdated or misleading, causing reflection to reinforce irrelevant strategies instead of supporting adaptation.

5

Conclusion

We present PokeGym, a rigorous benchmark that resolves the fundamental tension between environmental realism and scalable evaluation in embodied game VLM research. By enforcing strict code-level isolation, agents operate solely on raw RGB observations while an independent evaluator verifies success via AOB memory scanning. This design enables the first automated assessment of long-horizon, visually-driven decision-making in complex 3D open-world games. Our analysis across 8 VLMs reveals that physical deadlock recovery, rather than high-level planning, constitutes the primary bottleneck, highlighting the need to integrate explicit spatial intuition into VLM architectures. Furthermore, the identified metacognitive divergence between model tiers, where weaker models suffer from Unaware Deadlocks while stronger models exhibit Aware Deadlocks, suggests that failure mitigation strategies must be capability-specific.

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

References [1] Anthropic. 2026. Claude Sonnet 4.6. https://www.anthropic.com/news/claudesonnet-4-6. [2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. 2425–2433. [3] Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. 2025. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631 (2025). [4] Fuqing Bie, Shiyu Huang, Xijia Tao, Zhiqin Fang, Leyi Pan, Junzhe Chen, Min Ren, Liuyu Xiang, and Zhaofeng He. 2025. OmniPlay: Benchmarking OmniModal Models on Omni-Modal Game Playing. arXiv preprint arXiv:2508.04361 (2025). [5] Adrian Bolton, Alexander Lerchner, Alexandra Cordell, Alexandre Moufarek, Andrew Bolt, Andrew Lampinen, Anna Mitenkova, Arne Olav Hallingstad, Bojan Vujatovic, Bonnie Li, et al. 2025. Sima 2: A generalist embodied agent for virtual worlds. arXiv preprint arXiv:2512.04797 (2025). [6] Center for AI Safety, Scale AI, and HLE Contributors Consortium. 2026. A benchmark of expert-level academic questions to assess AI capabilities. Nature 649 (2026), 1139–1146. arXiv:2501.14249 [cs.LG] doi:10.1038/s41586-025-09962-4 [7] Yandu Chen, Kefan Gu, Yuqing Wen, Yucheng Zhao, Tiancai Wang, and Liqiang Nie. 2025. IntentionVLA: Generalizable and Efficient Embodied Intention Reasoning for Human-Robot Interaction. arXiv:2510.07778 [cs.RO] https: //arxiv.org/abs/2510.07778 [8] Zhihong Chen, Ruifei Zhang, Yibing Song, Xiang Wan, and Guanbin Li. 2023. Advancing visual grounding with scene knowledge: Benchmark and method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15039–15049. [9] Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen. 2025. Caparena: Benchmarking and analyzing detailed image captioning in the llm era. In Findings of the Association for Computational Linguistics: ACL 2025. 14077–14094. [10] Gautier Dagan, Frank Keller, and Alex Lascarides. 2024. Plancraft: an evaluation dataset for planning with LLM agents. arXiv preprint arXiv:2412.21033 (2024). [11] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36 (2023), 49250–49267. [12] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1–10. [13] Google DeepMind. 2025. Gemini 3 Pro. https://deepmind.google/models/gemini/ pro/. [14] Ning Ding, Yehui Tang, Zhongqian Fu, Chao Xu, Kai Han, and Yunhe Wang. 2025. GPT4Image: Large Pre-trained Models Help Vision Models Learn Better on Perception Task. In Companion Proceedings of the ACM on Web Conference 2025. 2056–2065. [15] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. 2022. MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 18343–18362. https://proceedings.neurips.cc/paper_files/paper/2022/ file/74a67268c5cc5910f64938cac4526a90-Paper-Datasets_and_Benchmarks.pdf [16] Qiaozi Gao, Govind Thattai, Suhaila Shakiah, Xiaofeng Gao, Shreyas Pansare, Vasu Sharma, Gaurav Sukhatme, Hangjie Shi, Bofei Yang, Desheng Zhang, et al. 2023. Alexa arena: A user-centric interactive platform for embodied ai. Advances in Neural Information Processing Systems 36 (2023), 19170–19194. [17] Simon Ging, María A Bravo, and Thomas Brox. 2024. Open-ended VQA benchmarking of vision-language models by exploiting classification datasets and their semantic hierarchy. arXiv preprint arXiv:2402.07270 (2024). [18] Jinghan He, Junfeng Fang, Feng Xiong, Zijun Yao, Fei Shen, Haiyun Guo, Jinqiao Wang, and Tat-Seng Chua. 2026. Active Zero: Self-Evolving Vision-Language Models through Active Environment Exploration. arXiv:2602.11241 [cs.CV] https://arxiv.org/abs/2602.11241 [19] Daniel P Hogan and Andrea Brennen. 2024. Open-ended wargames with large language models. arXiv preprint arXiv:2404.11446 (2024). [20] Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. 2025. Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos. (2025). https://arxiv.org/abs/2501.13826 [21] Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P. Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. 2026. lmgame-Bench: How Good are LLMs at Playing Games?. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=qeziG97WUZ [22] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. 2023. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871 (2023). [23] Zixia Jia, Mengmeng Wang, Baichen Tong, Song-Chun Zhu, and Zilong Zheng.

2024. LangSuit· E: Planning, controlling and interacting with large language models in embodied text environments. In Findings of the Association for Computational Linguistics: ACL 2024. 14778–14814. [24] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66 [25] Michał Kempka, Marek Wydmuch, Grzegorz Runc, et al. 2016. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE conference on computational intelligence and games (CIG). IEEE, 1–8. [26] Heinrich Küttler, Nantas Nardelli, Alexander Miller, et al. 2020. The nethack learning environment. Advances in Neural Information Processing Systems 33 (2020), 7671–7684. [27] Tony Lee, Haoqin Tu, Chi H Wong, et al. 2024. Vhelm: A holistic evaluation of vision language models. Advances in Neural Information Processing Systems 37 (2024), 140632–140666. [28] Guanzhen Li, Yuxi Xie, and Min-Yen Kan. 2024. MVP-Bench: Can Large VisionLanguage Models Conduct Multi-level Visual Perception Like Humans?. In Findings of the Association for Computational Linguistics: EMNLP 2024. 13505–13527. [29] Kaixin Li, Meng Ziyang, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. In Workshop on Reasoning and Planning for Large Language Models. https://openreview.net/forum?id=XaKNDIAHas [30] Muyao Li, Zihao Wang, Kaichen He, Xiaojian Ma, and Yitao Liang. 2025. Jarvisvla: Post-training large-scale vision language models to play visual games with keyboards and mouse. In Findings of the Association for Computational Linguistics: ACL 2025. 17878–17899. [31] Xinze Li, Ziyue Zhu, Siyuan Liu, Yubo Ma, Yuhang Zang, Yixin Cao, and Aixin Sun. 2026. EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents. arXiv preprint arXiv:2601.16690 (2026). [32] Mingxian Lin, Wei Huang, Yitang Li, Chengjie Jiang, Kui Wu, Fangwei Zhong, Shengju Qian, Xin Wang, and Xiaojuan Qi. 2025. Embrace-3k: Embodied reasoning and action in complex environments. arXiv preprint arXiv:2507.10548 (2025). [33] Shunyu Liu, Yaoru Li, Kongcheng Zhang, Zhenyu Cui, Wenkai Fang, Yuxuan Zheng, Tongya Zheng, and Mingli Song. 2024. Odyssey: Empowering minecraft agents with open-world skills. arXiv preprint arXiv:2407.15325 (2024). [34] Fan Lu, Wei Wu, Kecheng Zheng, Shuailei Ma, Biao Gong, Jiawei Liu, Wei Zhai, Yang Cao, Yujun Shen, and Zheng-Jun Zha. 2025. Benchmarking large visionlanguage models via directed scene graph for comprehensive image captioning. In Proceedings of the Computer Vision and Pattern Recognition Conference. 19618– 19627. [35] Fan Lu, Wei Wu, Kecheng Zheng, Shuailei Ma, Biao Gong, Jiawei Liu, Wei Zhai, Yang Cao, Yujun Shen, and Zheng-Jun Zha. 2025. Benchmarking large visionlanguage models via directed scene graph for comprehensive image captioning. In Proceedings of the Computer Vision and Pattern Recognition Conference. 19618– 19627. [36] Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. 2025. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025. 1160–1183. [37] Feipeng Ma, Hongwei Xue, Yizhou Zhou, Guangting Wang, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. 2024. Visual perception by large language model’s weights. Advances in Neural Information Processing Systems 37 (2024), 28615–28635. [38] Chris Madge and Massimo Poesio. 2024. Large language models as minecraft agents. arXiv preprint arXiv:2402.08392 (2024). [39] Loïc Magne, Anas Awadalla, Guanzhi Wang, Yinzhen Xu, Joshua Belofsky, Fengyuan Hu, Joohwan Kim, Ludwig Schmidt, Georgia Gkioxari, Jan Kautz, Yisong Yue, Yejin Choi, Yuke Zhu, and Linxi "Jim" Fan. 2026. NitroGen: An Open Foundation Model for Generalist Gaming Agents. arXiv:2601.02427 [cs.CV] https://arxiv.org/abs/2601.02427 [40] Glenn Matlin, Parv Mahajan, Isaac Song, Yixiong Hao, Ryan Bard, Stu Topp, Evan Montoya, M Rehan Parwani, Soham Shetty, and Mark Riedl. 2025. Shall We Play a Game? Language Models for Open-ended Wargames. arXiv preprint arXiv:2509.17192 (2025). [41] Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. 2023. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3113– 3124. [42] Filippo Momentè, Alessandro Suglia, Mario Giulianelli, Ambra Ferrari, Alexander Koller, Oliver Lemon, David Schlangen, Raquel Fernández, and Raffaella Bernardi. 2025. Triangulating llm progress through benchmarks, games, and cognitive tests. arXiv preprint arXiv:2502.14359 (2025). [43] Muhammad Umair Nasir, Steven James, and Julian Togelius. 2024. Gametraversalbenchmark: Evaluating planning abilities of large language models through

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

traversing 2d game maps. Advances in Neural Information Processing Systems 37 (2024), 31813–31827. [44] OpenAI. 2025. GPT-5.2. https://openai.com/index/introducing-gpt-5-2/. [45] OpenAI. 2026. GPT-5.4. https://openai.com/index/introducing-gpt-5-4/. [46] OpenAI. 2026. GPT-5.4 mini and nano. https://openai.com/index/introducinggpt-5-4-mini-and-nano/. [47] Davide Paglieri, Bartłomiej Cupiał, Sam Coward, Ulyana Piterbarg, Maciej Wołczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. 2024. BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games. arXiv preprint arXiv:2411.13543 (2024). [48] Dongmin Park, Minkyu Kim, Beongjun Choi, Junhyuck Kim, Keon Lee, Jonghyun Lee, Inkyu Park, Byeong-Uk Lee, Jaeyoung Hwang, Jaewoo Ahn, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Pritam Biswas, Yoshi Suhara, Kangwook Lee, and Jaewoong Cho. 2026. Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum? id=H1ncX6O6Yh [49] Marco Pleines, Daniel Addis, David Rubinstein, Frank Zimmer, Mike Preuss, and Peter Whidden. 2025. Pokémon Red via Reinforcement Learning. In 2025 IEEE Conference on Games (CoG). 1–8. doi:10.1109/CoG64752.2025.11114399 Efficient Long[50] Weikang Qiu, Tinglin Huang, and Rex Ying. 2026. Horizon Vision-Language-Action Models via Static-Dynamic Disentanglement. arXiv:2602.03983 [cs.RO] https://arxiv.org/abs/2602.03983 [51] Yun Qu, Boyuan Wang, Jianzhun Shao, Yuhang Jiang, Chen Chen, Zhenbin Ye, Liu Linc, Yang Feng, Lin Lai, Hongyang Qin, et al. 2023. Hokoff: Real game dataset from honor of kings and its offline reinforcement learning benchmarks. Advances in Neural Information Processing Systems 36 (2023), 22166–22190. [52] Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen. ai/blog?id=qwen3.5 [53] Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, et al. 2024. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404.10179 (2024). [54] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling. https://openreview.net/forum?id=Ti67584b98 [55] Mikayel Samvelyan. 2025. Robust Agents in Open-Ended Worlds. arXiv preprint arXiv:2512.08139 (2025). [56] Burak Satar, Zhixin Ma, Patrick Amadeus Irawan, Wilfried Ariel Mulyawan, Jing Jiang, Ee-Peng Lim, and Chong-Wah Ngo. 2025. Seeing culture: A benchmark for visual reasoning and grounding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 22238–22254. [57] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10737– 10746. doi:10.1109/CVPR42600.2020.01075 [58] Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Wayne Xin Zhao, Fuzheng Zhang, Di Zhang, and Kun Gai. 2024. Parrot: Enhancing multiturn instruction following for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 9729–9750. [59] Sinan Tan, Weilai Xiang, Huaping Liu, Di Guo, and Fuchun Sun. 2020. Multiagent embodied question answering in interactive environments. In European Conference on Computer Vision. Springer, 663–678. [60] Weihao Tan, Changjiu Jiang, Yu Duan, Mingcong Lei, Li JiaGeng, Yitian Hong, Xinrun Wang, and Bo An. 2025. StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production–Living Simulations with Stardew Valley. In First Workshop on Multi-Turn Interactions in Large Language Models. https://openreview.net/forum?id=R0mmX6BEau [61] Weihao Tan, Xiangyang Li, Yunhao Fang, Heyuan Yao, Shi Yan, Hao Luo, Tenglong Ao, Huihui Li, Hongbin Ren, Bairen Yi, Yujia Qin, Bo An, Libin Liu, and Guang Shi. 2025. Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds. arXiv:2511.08892 [cs.AI] https://arxiv.org/abs/2511.08892 [62] Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. 2025. Cradle: Empowering Foundation Agents towards General Computer Control. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 58658–58725. https://proceedings.mlr.press/v267/tan25h.html [63] Arena Team. 2026. Arena Leaderboard Dataset. https://arena.ai/blog/arenaleaderboard-dataset/

[64] Qwen Team. 2026. Qwen3.5: Accelerating Productivity with Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5 [65] SIMA Team, Maria Abi Raad, Arun Ahuja, et al. 2024. Scaling Instructable Agents Across Many Simulated Worlds. arXiv:2404.10179 [cs.RO] https://arxiv.org/abs/ 2404.10179 [66] V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. 2025. GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning. arXiv:2507.01006 [cs.CV] https://arxiv.org/abs/2507.01006 [67] Tristan Tomilin, Meng Fang, Yudi Zhang, and Mykola Pechenizkiy. 2023. Coom: A game benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems 36 (2023), 67794–67832. [68] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 16022–16076. [69] Xinyu Wang, Bohan Zhuang, and Qi Wu. 2025. Are large vision language models good game players? arXiv preprint arXiv:2503.02358 (2025). [70] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. 2024. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. IEEE Transactions on Pattern Analysis and Machine Intelligence 47, 3 (2024), 1894–1907. [71] Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. 2024. CharXiv: charting gaps in realistic chart understanding in multimodal LLMs. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 3609, 129 pages. [72] Zirui Wang, Junyi Zhang, Jiaxin Ge, Long Lian, Letian Fu, Lisa Dunlap, Ken Goldberg, Xudong Wang, Ion Stoica, David M. Chan, Sewon Min, and Joseph E. Gonzalez. 2026. VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents. arXiv preprint arXiv:2601.16973 (2026). https://arxiv.org/ abs/2601.16973 [73] Azmine Toushik Wasi, Wahid Faisal, Abdur Rahman, Mahfuz Ahmed Anik, Munem Shahriar, Mohsin Mahmud Topu, Sadia Tasnim Meem, Rahatun Nesa Priti, Sabrina Afroz Mitu, Md. Iqramul Hoque, Shahriyar Zaman Ridoy, Mohammed Eunus Ali, Majd Hawasly, Mohammad Raza, and Md Rizwan Parvez. 2026. SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild? arXiv:2602.03916 [cs.CV] https://arxiv.org/abs/2602.03916 [74] Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. 2023. Smartplay: A benchmark for llms as intelligent agents. arXiv preprint arXiv:2310.01557 (2023). [75] Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, et al. 2025. Agentgym: Evaluating and training large language model-based agents across diverse environments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 27914–27961. [76] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2024. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence 47, 3 (2024), 1877–1893. [77] Yunqiu Xu, Linchao Zhu, and Yi Yang. 2025. Mc-bench: A benchmark for multicontext visual grounding in the era of mllms. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 17675–17687. [78] Ming Yan, Ruihao Li, Hao Zhang, Hao Wang, Zhilan Yang, and Ji Yan. 2023. Larp: Language-agent role play for open-world games. arXiv preprint arXiv:2312.17653 (2023). [79] Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. 2025. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10632–10643. doi:10.1109/CVPR52734.2025.00994 [80] Yi Yang, Jiaxuan Sun, Siqi Kou, Yihan Wang, and Zhijie Deng. 2025. Lohovla: A unified vision-language-action model for long-horizon embodied tasks. arXiv preprint arXiv:2506.00411 (2025).

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

[81] Pengfei Yu, Dongming Shen, Silin Meng, Jaewon Lee, Weisu Yin, Andrea Yaoyun Cui, Zhenlin Xu, Yi Zhu, Xingjian Shi, Mu Li, et al. 2025. Rpgbench: Evaluating large language models as role-playing game engines. arXiv preprint arXiv:2502.00595 (2025). [82] Shu Yu and Chaochao Lu. 2024. Adam: An embodied causal agent in open-world environments. arXiv preprint arXiv:2410.22194 (2024). [83] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2025. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 15134–15186. doi:10. 18653/v1/2025.acl-long.736 [84] Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. 2022. Vlmbench: A compositional benchmark for vision-and-language manipulation. Advances in Neural Information Processing Systems 35 (2022), 665–678. [85] Xinyue Zheng, Haowei Lin, Kaichen He, Zihao Wang, Qiang Fu, Haobo Fu, Zilong Zheng, and Yitao Liang. 2025. MCU: An evaluation framework for open-ended game agents. In Forty-second International Conference on Machine Learning. [86] Chunlin Zhong, Shuang Hao, Junhua Wu, Xiaona Chang, Jiwei Jiang, Xiu Nie, He Tang, and Xiang Bai. 2025. PathVG: A New Benchmark and Dataset for Pathology Visual Grounding. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 454–463. [87] Qing Zhou, Tao Yang, Junyu Gao, Weiping Ni, Junzheng Wu, and Qi Wang. 2025. A benchmark for multi-lingual vision-language learning in remote sensing image captioning. arXiv preprint arXiv:2503.04592 (2025). [88] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. 2023. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144 (2023).

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

Appendix A

Detailed Comparison with Benchmarks

We conduct a detailed comparison with three representative benchmarks in Table 8. Specifically, we consider: (1) ALFRED [57], a classic embodied benchmark in a realistic indoor environment. (2) Minecraft-based, a widely adopted game benchmark. (3) Cradle-based [62], an agent benchmark in general AAA games. Through these comparisons, we aim to clarify how PokeGym differs from prior environments in terms of environmental realism, observation conditions, task design, and evaluation protocol. Overall, these benchmarks together highlight the distinctive position of PokeGym as a scalable and visually grounded testbed for embodied agents.

B

Quantitative Complexity Analysis

To mathematically illustrate the challenge PokeGym poses to VisionLanguage Models (VLMs), we quantify the environment’s complexity across three fundamental dimensions: state space, action space, and decision horizon.

(e.g., MoveForward, RotateLeft). Since three actions are executed per query, the size of the discrete action space per decision step is: |𝐴𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒 | = |𝐴𝑏𝑎𝑠𝑒 |𝑘 = 73 = 343.

(4)

2. Parametric Control (Continuous). This paradigm enables fine-grained manipulations where an individual action consists of an Action Type (Left Stick, Right Stick, or Button A) and corresponding continuous parameters. Specifically, joystick actions require 𝑋 and 𝑌 coordinates (−1.0 to 1.0) along with a hold duration 𝑡, whereas Button A only requires the duration 𝑡. To quantify this space, we discretize 𝑋 and 𝑌 with a step of 0.1 (yielding 21 possible values per axis), and the duration 𝑡 ∈ [0, 2000ms] with a step of 100ms (yielding 21 possible values). The size of a single parametric action space |𝐴𝑠𝑖𝑛𝑔𝑙𝑒_𝑝𝑎𝑟𝑎 | is the sum of all joystick and button combinations. Given that the agent outputs a sequence of 3 actions per query, the total parametric action space per decision step |𝐴𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 | is calculated as follows: |𝐴𝑠𝑖𝑛𝑔𝑙𝑒_𝑝𝑎𝑟𝑎 | = (2 × 21 × 21 × 21) + 21 = 18, 543, | {z } |{z} Left & Right Sticks

Button A

3

|𝐴𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 | = (18, 543) ≈ 6.38 × 1012 .

B.1

State Space Complexity

We simplify the analysis by omitting the environmental states (e.g., dynamic NPCs) and focus on the spatial state. This state can be represented as 𝑠 = (𝑥, 𝑧, 𝜃 ), where (𝑥, 𝑧) denotes the horizontal position and 𝜃 represents the camera yaw angle. We explicitly omit the vertical coordinate 𝑦 and the camera pitch angle, as they remain nearly constant in our evaluated tasks. To estimate the size of the state space |𝑆 |, we discretize the map with a spatial step size of Δ𝑑 = 1 unit and the viewing direction with an angular step size of Δ𝜃 = 1◦ . Let 𝐴 denote the map area. The resulting state space size can be approximated as:     𝐴 360◦ |𝑆 | ≈ × . (1) Δ𝑑 2 Δ𝜃

(2)

|𝑆 max | ≈ ⌈2418.12⌉ × 360 = 2419 × 360 = 870, 840.

(3)

This demonstrates that even under a highly simplified assumption with coarse discretization, the agent still faces a massive state space relying purely on visual observations.

B.2

Action Space Complexity

We analyze the action space complexity under two control paradigms: 1. Defined High-level Actions (Discrete). In this paradigm, the agent selects from a pre-defined set of 7 discrete macro-actions

(6)

Such an enormous action space requires VLMs to possess an extremely high level of physical intuition and precise multi-step execution capability.

B.3

Decision Horizon Complexity

We evaluate the game tree complexity O (𝑏𝑑 ), where 𝑏 represents the effective branching factor per environment step and 𝑑 is the maximum decision depth. According to our task budgets, the maximum effective horizon reaches up to 𝑑 = 360 environment steps. For the discrete high-level action paradigm, the effective branching factor is 𝑏 discrete = 7. For the continuous parametric control paradigm, based on our prior discretization in Section B.2, the branching factor expands to 𝑏 parametric = 18, 543. Then the sizes of the decision trees for the two paradigms are calculated as:

Since map sizes vary across tasks in PokeGym, we further estimate the state space range using the smallest map with area 𝐴min = 186.65 and the largest map with area 𝐴max = 2418.12: |𝑆 min | ≈ ⌈186.65⌉ × 360 = 187 × 360 = 67, 320,

(5)

Game Tree SizeDiscrete ≈ O (7360 ) ≈ 10304, Game Tree SizeParametric ≈ O (18, 543

360

(7) 1536

) ≈ 10

.

(8)

This explosion highlights that brute-force exploration or shortsighted planning is intractable in PokeGym. To succeed, the VLM must maintain a coherent, long-term semantic plan and robust error-recovery strategies.

C

Qualitative Complexity Analysis

Unlike traditional grid-worlds or simplified voxel-based simulators, PokeGym is built upon a modern game engine and presents a diverse set of realistic physical and visual challenges. As illustrated in Figure 11, agents in PokeGym must handle partial observability, visual ambiguity, lighting variability, topological complexity and element density. Figure 12 showcases qualitative trajectories of our

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

Table 8: Detailed Comparison between Classic Benchmarks (ALFRED, Minecraft-based, Cradle-based) and PokeGym.

Dimension

Benchmark

Characteristics

Environment (Visual & Physics)

ALFRED

Confined 3D indoor household scenes, fixed object placements, limited interactivity, constrained environmental physics

Minecraft-based

Voxel-based visuals, orthogonal geometry, uniform textures, predictable topology

PokeGym (Ours)

Unconstrained 3D open world, complex topology (slopes, stairs, invisible walls), diverse biomes, dynamic lighting and shadows, dense elements (crowds, wildlife)

Minecraft-based

Privileged-state observations, including explicit (𝑥, 𝑦, 𝑧) coordinates, text-based inventory lists, block IDs

PokeGym (Ours)

Pure RGB observations, zero state leakage, no privileged API access

ALFRED

Linear subgoal progression, household chore tasks, short-to-medium horizons, step-by-step instructions

Minecraft-based

Self-driven progression, sandbox exploration, resource gathering, recipe-based crafting, open-ended building

Cradle-based

Main-story missions, combat scenarios, open-ended tasks such as NPC following

PokeGym (Ours)

Quest-driven narrative progression, long-horizon spatial planning, structured navigation, specific NPC interactions, combat requirements

Cradle-based

Human evaluation, manual task-success verification, high annotation cost, limited scalability, potential human bias

PokeGym (Ours)

Automated AOB memory scanning, threshold-based verification, fast evaluation, objective judgment, scalable assessment

ALFRED

Visual grounding in confined domains, step-by-step instruction following, basic object manipulation

Minecraft-based

Long-horizon planning, recipe-logic reasoning, open-world survival strategies

Cradle-based

General computer control, UI interaction, zero-shot and few-shot adaptation to new software

PokeGym (Ours)

Autonomous exploration, long-horizon planning, fine-grained visual grounding, semantic reasoning, depth perception, 3D spatial collision recovery, multimodal integration, narrative instruction following

Observation Space

Task Structure & Progression

Evaluation Methodology

Evaluated Capabilities

tasks, highlighting the prolonged decision horizon required.

D

Details of the Automatic Evaluation Pipeline

To enable scalable and reproducible evaluation, PokeGym uses an automatic memory-based verifier instead of manual inspection. We explain this process using the player coordinate 𝑦 as an example in the following sections. In Pokémon Legends: Z-A, the 𝑦 coordinate corresponds to the vertical direction in the game world: moving upward increases 𝑦, while moving downward decreases 𝑦. The pipeline includes feature signatures extraction and Array of Bytes (AOB) memory scanning.

D.1

Feature Signatures Extraction

Because raw memory addresses are not stable across restarts, we extract feature signatures, which are stable byte patterns around the target variable, so that the variable can be relocated later. This process consists of four steps: (1) initial unknown-value scan, (2) motion-based filtering, (3) binary elimination through value locking,

and (4) repeated runs for stable signature discovery. Step 1: Initial unknown-value scan. We attach a memoryediting tool (e.g., Cheat Engine) to the emulator process and perform an initial scan with unknown values under the single-precision float type (the value type of 𝑦 coordinate). This yields a large candidate set of memory addresses. Step 2: Motion-based filtering. We then reduce the candidate set by repeatedly moving the player character and filtering according to how the value should change: • move the character up or down and keep only changed values; • keep the character stationary and keep only unchanged values; • move upward and keep only values that increase; • move downward and keep only values that decrease. These filters are applied iteratively until the number of remaining candidates stabilizes and cannot be reduced further by simple motion-based constraints. Step 3: Binary elimination through value locking. The remaining candidates still typically contain many correlated values,

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

including derived variables, cached copies, or unrelated states that happen to co-vary with movement. To isolate the memory address that actually controls the player position, we perform a binary elimination procedure. Specifically, we split the candidate addresses into two halves and use the memory-editing tool to lock one half, preventing those values from changing. We then move the character vertically: • if the character becomes stuck or cannot move smoothly in the vertical direction, then the true controlling address is among the locked half; • if the character can still move freely, then the true address is among the unlocked half. We recursively repeat this halving procedure until a single address or a very small set of addresses remains. We then record the target address and the bytes in a local neighborhood around it. This step distinguishes values that merely reflect position from the variable that can causally control it. Step 4: Repeated runs for stable signature discovery. To derive a relocatable signature, we repeat Step 3 multiple times (typically three to four independent repetitions), each time after restarting or reloading the game state and rediscovering the same target variable. For each repetition, we record the memory bytes in the surrounding region. We then compare these local byte regions across repetitions and search for identical byte subsequences that consistently appear before or after the target variable. These repeated, stable byte sequences are used as feature signatures. These feature signatures are robust across game restarts and different machines, enabling reliable relocation of the corresponding states.

D.2

AOB-Based Memory Scanning

After discovering stable signatures offline, the evaluator uses AOB scanning at runtime to relocate the corresponding memory addresses at the beginning of each episode. Signature definitions. Wildcard tokens (XX) indicate bytes that may vary across runs and should be ignored during matching, while their actual stored values represent the target game state. In our implementation, the map signature is defined as a fixed 8-byte header followed by 32 wildcard bytes (the map string): mapSignature = header || XX XX . . . XX, | {z }

(9)

32 bytes

where || denotes concatenation. The other signatures are similarly defined as short fixed byte patterns with wildcard gaps. Memory-region filtering. Rather than scanning every memory page indiscriminately, the scanner filters regions using Windows memory metadata obtained via VirtualQuery. Only regions satisfying all of the following conditions are scanned: • MEM_COMMIT: the memory page is committed; • PAGE_READWRITE: the page is readable and writable; • MEM_MAPPED: the page is mapped memory. This reduces unnecessary scanning and focuses the search on regions where emulator-managed game state is most likely to reside. Wildcard matching. The matcher then performs byte-wise comparison between candidate memory locations and the signature. An address is considered a match if every non-wildcard byte agrees

Algorithm 1 Feature Signature Extraction Require: Emulator memory space M, Value type 𝜏 (e.g., float), Neighborhood size Δ, Repetitions 𝑁 Ensure: Set of stable feature signatures S 1: B ← ∅ ⊲ Stores local byte regions across different restarts 2: for 𝑖 = 1 to 𝑁 do 3: 𝐶 ← InitialScan(M, 𝜏) ⊲ Step 1: Unknown-value scan 4: repeat ⊲ Step 2: Motion-based filtering 5: 𝐿 ← |𝐶 | 6: 𝐶 ← {𝑐 ∈ 𝐶 | Δval(𝑐) ≠ 0 on Move} 7: 𝐶 ← {𝑐 ∈ 𝐶 | Δval(𝑐) = 0 on Idle} 8: 𝐶 ← {𝑐 ∈ 𝐶 | Δval(𝑐) > 0 on MoveUp} 9: 𝐶 ← {𝑐 ∈ 𝐶 | Δval(𝑐) < 0 on MoveDown} 10: until |𝐶 | = 𝐿 ⊲ Iterate until candidate set size stabilizes 11: while |𝐶 | > 1 do ⊲ Step 3: Binary elimination 12: Split 𝐶 into two disjoint subsets: 𝐶 lock and 𝐶 free 13: LockMemoryValues(𝐶 lock ) 14: AttemptVerticalMovement 15: if character becomes stuck then 16: 𝐶 ← 𝐶 lock ⊲ Target is locked 17: else 18: 𝐶 ← 𝐶 free ⊲ Target is free 19: end if 20: UnlockMemoryValues(𝐶 lock ) 21: end while 22: 𝑎𝑑𝑑𝑟 ∗ ← the single remaining element in 𝐶 23: 𝐵𝑖 ← ExtractByteRegion(M, 𝑎𝑑𝑑𝑟 ∗, Δ) ⊲ Store bytes 24: B ← B ∪ {𝐵𝑖 } 25: RestartGame 26: end for ⊲ Step 4: Repeated runs for stable discovery 27: S ← FindCommonSubseqences(B) ⊲ Extract signatures 28: return S with the corresponding memory byte. Success checking during episode execution. Once all addresses are found, the evaluator stores them and uses them for automatic progress tracking. At the end of each executed action sequence, the environment reads the relevant in-memory values and checks whether the success condition is satisfied. If the success condition is met, the episode terminates immediately as successful. Otherwise, execution continues until the step budget is exhausted.

D.3

Robustness and Practicality

The proposed automatic evaluation pipeline has two practical advantages. First, it removes the need for manual annotation or human judgment during benchmark evaluation. Second, because it relies on byte signatures rather than hard-coded raw addresses, it remains stable across repeated runs and different machines under the same game version. At the same time, these memory values are used strictly for evaluation and are never provided to the agent, preserving the visual-only nature of the benchmark.

E Extended Experiments and Leaderboard E.1 Extended Experiments We further extend our main experiments by evaluating GPT-5.4 [45], GPT-5.4-mini [46], and GPT-5.4-nano [46] in Table 9.

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

Table 9: Performance comparison across 3 granularity levels in extended experiments. Success Rate (SR, %) measures the percentage of episodes that successfully complete the task. Average Environment Steps (Stp) denote the average number of environment steps in successful episodes. Bold indicates the best performance. Navigation Interaction

Model

SR↑

Stp↓

Mixed

SR↑ Stp↓ SR↑

Average

Stp↓

SR↑

Stp↓

GPT-5.4-nano 50.00 101.20 66.67 93.80 33.33 110.00 50.00 101.67 Visual-Guided GPT-5.4-mini 30.00 165.67 73.33 39.73 46.67 96.71 50.00 100.70 GPT-5.4 40.00 103.88 93.33 53.71 46.67 102.57 60.00 86.72 GPT-5.4-nano 40.00 89.38 33.33 80.00 40.00 95.50 37.78 88.29 Step-Guided GPT-5.4-mini 10.00 67.50 93.33 55.43 40.00 109.83 47.78 77.59 GPT-5.4 30.00 79.83 93.33 49.29 40.00 97.50 54.44 75.54 Goal-Only

GPT-5.4-nano 20.00 90.75 46.67 85.14 0.00 22.22 87.95 GPT-5.4-mini 10.00 125.50 86.67 59.62 26.67 111.75 41.11 98.96 GPT-5.4 50.00 114.60 73.33 92.45 13.33 125.00 45.56 110.68

Table 10: PokeGym Leaderboard. Models are ranked by their overall success rate (average of SR across all 9 task configurations), with a random baseline included for reference. The three instruction granularities are abbreviated as Vis-G (Visual-Guided), Stp-G (Step-Guided), and Goal-O (Goal-Only). Rank Model

Navigation

Interaction

Mixed

Overall SR

Vis-G Stp-G Goal-O Vis-G Stp-G Goal-O Vis-G Stp-G Goal-O #1

Gemini-3-Pro

20.00 70.00

45.00

66.67 93.33 100.00 46.67 60.00

26.67

58.70

#2

GPT-5.2

25.00 30.00

40.00

93.33 86.67 100.00 60.00 53.33

40.00

58.70

#3

GPT-5.4

40.00 30.00

50.00

93.33 93.33

73.33

46.67 40.00

13.33

53.33

#4

Claude-Sonnet-4.6 55.00 55.00

55.00

80.00 60.00

60.00

46.67 60.00

6.67

53.15

#5

Qwen3-VL-30B

50.00 40.00

45.00

66.67 60.00

73.33

53.33 46.67

33.33

52.04

#6

Qwen3.5-122B

60.00 25.00

25.00

66.67 66.67

73.33

53.33 20.00

40.00

47.78

#7

Qwen3.5-35B

45.00 45.00

45.00

80.00 60.00

80.00

26.67 33.33

13.33

47.59

#8

GPT-5.4-mini

30.00 10.00

10.00

73.33 93.33

86.67

46.67 40.00

26.67

46.30

#9

Qwen3.5-Plus

55.00 50.00

50.00

66.67 53.33

46.67

26.67 26.67

20.00

43.89

#10 GLM-4.6V

25.00 25.00

25.00

46.67 53.33

73.33

60.00 46.67

26.67

42.41

#11 GPT-5.4-nano

50.00 40.00

20.00

66.67 33.33

46.67

33.33 40.00

0.00

36.67

-

Random

0.00

The GPT-5.4 series exhibits a capability-scaling law that correlates with model size. The flagship model, GPT-5.4, demonstrates particularly strong performance, achieving an average Success Rate (SR) of 60.00% under the Visual-Guided setting and 54.44% under the Step-Guided setting, outperforming its smaller counterparts. Conversely, the lightweight model, GPT-5.4-nano, struggles in complex scenarios, failing completely (0.00% SR) in the GoalOnly Mixed tasks. Overall, these results highlight a pronounced performance gap within the GPT-5.4 family and further confirm the importance of model scale for embodied game-playing agents.

E.2

PokeGym Leaderboard

We aggregate the results of all 11 evaluated models into the PokeGym leaderboard, together with a random baseline that randomly

0.00

6.67

2.22

selects from the available actions and is evaluated with 5 runs per task, as shown in Table 10. Its near-zero overall success rate (2.22%) suggests that the benchmark cannot be solved by chance and requires non-trivial planning and instruction grounding. The leaderboard shows that proprietary models occupy the top tier, with Gemini-3-Pro (58.70%), GPT-5.2 (58.70%), and GPT-5.4 (53.33%) ranking among the strongest performers. In particular, Gemini-3-Pro and GPT-5.2 share first place, reflecting their superior adaptability to complex 3D open-world scenarios, ranging from long-horizon spatial navigation to dynamic interactions based on pure-pixel inputs. Meanwhile, the leading open-weight model, Qwen3-VL-30B, achieves a highly competitive 52.04% Overall SR, securing the 5th place and closely following the top proprietary models. Consequently, this leaderboard offers a comprehensive

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

Aware Deadlock

Unaware Deadlock

Reasoning: In the current screen, the character has successfully moved forward as the character is no longer obstructed by the pillar. (Unaware) State: Not changed. (Deadlock)

Reasoning: The previous and current screens look nearly identical with the red barrier grid in front. The last action did not advance us toward the alley. (Aware) State: Not changed. (Deadlock)

Lost

Execution Failure

Reasoning: There is no visible Pokémon in this view, but the open space suggests potential movement or interaction opportunities. (Target Missing) State: Changed. (Ultimately Failed)

Reasoning: The front view shows the ticket gates and the exit beyond them. The path is straight and unobstructed. (Target Visible) State: Changed. (Ultimately Failed)

Figure 6: Four Failure Type Case Studies. reference for future research in generalist embodied agents.

F Qualitative Analysis of Failures F.1 Case Studies of the Four Failure Types To bridge the gap between the agent’s semantic reasoning and its micro-level physical execution, we classify episode failures into four distinct types, with representative case studies shown in Figure 6: (1) Unaware Deadlock. This failure occurs when a physically trapped agent hallucinates progress, completely oblivious to the collision. (2) Aware Deadlock. In this scenario, the agent explicitly recognizes the barrier but lacks the spatial intuition to execute a valid escape maneuver. (3) Lost. This category describes aimless wandering where the agent makes physical movement but fails to spot the target. (4) Execution Failure. This failure emerges when the agent successfully spots the target but struggles with precise finalstep operations, such as getting snagged by adjacent microgeometry, failing to trigger the correct interactive prompt or

spamming the interaction button from slightly outside the valid trigger range.

F.2

Obstacles of Unaware Deadlocks

We collect the locations where unaware deadlocks occur, count their frequencies, and select the positions with relatively high occurrence rates. Based on the structural characteristics of the obstacle scenes, we group them into three representative categories in Figure 7. Visually permeable barriers refer to cases where the visible background appears traversable, but the actual physical boundary blocks the agent. In such scenes, the agent tends to infer navigability from distant open space, such as grass, trees, houses, or other visible regions beyond the barrier, while neglecting the rigid collision constraints imposed by pillars, fences, or similar structures. As a result, the agent repeatedly attempts to move toward an apparently open direction and becomes stuck. Irregular micro-geometries describe situations where the agent can correctly avoid large, salient walls at the macro level, but fails to account for the collision boundaries of small adjacent

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

Visually Permeable Barriers The agent treats visible background space as an open path but ignores the existing physical boundaries. Claude-Sonnet-4.6: The left view shows more open space with a grassy area and trees, suggesting an open path.

GLM-4.6V: The left view confirms the passage is between two red pillars, with a house visible beyond it.

Irregular Micro-Geometries The agent avoids macro-walls but ignores the rigid collision boundaries of adjacent micro-props (e.g., plants, NPCs). Qwen3.5-35B: The left view shows a dead end against a building wall. The right view shows the NPC and the path forward.

Qwen3.5-122B: The left view shows a close-up of a wall that is not an option. The right view reveals the path forward and a plant.

Misleading Interactive Elements The agent persistently interacts with task-irrelevant objects (e.g., doors, elevators). Gemini-3-Pro: The right view shows a door. The next logical step is to interact with the door.

Qwen3-VL-30B: The left view shows the elevator. The next logical step is to interact with the elevator.

Figure 7: Representative Obstacle Patterns behind Unaware Deadlocks. The figure presents three distinct categories of obstacles, overlaying actual physical collision boundaries that cause errors (red planes), boundaries on the other side of the dead corner (white planes), and agent positions (red ellipses) alongside the models’ flawed internal reasoning. Table 11: Performance comparison between external benchmarks and our proposed PokeGym. External benchmark scores (e.g., MMMU-Pro, GPQA) and PokeGym results are reported as accuracy or success rates from 0 to 1, except for “Text Arena”, which uses absolute Elo rating. For compactness, some benchmark names are abbreviated: VidMMMU (VideoMMMU), ScrSpot (ScreenSpot-Pro), CharXiv (CharXiv-R), HLE (Humanity’s Last Exam), and SWE-V (SWE-Bench Verified). PokeGym is evaluated under different instruction granularities: Vis-G (Visual-Guided), Stp-G (Step-Guided), and Goal-O (Goal-Only), as well as task categories: Nav (Navigation), Int (Interaction), and Mix (Mixed).

External Benchmarks

Model

PokeGym (Ours)

MMMU-Pro VidMMMU ScrSpot CharXiv HLE GPQA SWE-V Arena Vis-G Stp-G Goal-O Nav Int Mix Gemini-3-Pro GPT-5.2 Qwen3-VL-30B Qwen3.5-122B Qwen3.5-35B

0.81 0.80 0.60 0.77 0.75

0.88 0.86 0.69 0.82 0.80

0.73 0.86 0.61 0.70 0.69

0.81 0.82 0.49 0.77 0.78

objects, such as plants or NPCs. Although the global path appears identifiable, these local micro-props create narrow or blocked passages that the agent does not model properly, which causes repeated failed movement attempts and deadlocks. Misleading interactive elements correspond to scenes containing task-irrelevant interactive objects, such as doors or elevators. In these cases, the agent over-attributes affordance to the interactive object and persistently chooses interaction as the next action, even when the object is irrelevant to task completion or cannot resolve the current navigation state. This leads to cyclical, unproductive behaviors and eventually unaware deadlocks. Overall, these examples show that unaware deadlocks are not

0.46 0.35 0.10 0.48 0.47

0.92 0.92 0.70 0.87 0.84

0.76 0.80 0.12 0.72 0.69

1486 1440 1383 1416 1400

0.42 0.56 0.56 0.60 0.50

0.74 0.54 0.48 0.36 0.46

0.56 0.58 0.50 0.44 0.46

0.45 0.87 0.44 0.32 0.93 0.51 0.45 0.67 0.44 0.37 0.69 0.38 0.45 0.73 0.24

randomly distributed, but are strongly associated with recurring obstacle patterns that exploit failures in traversability estimation, fine-grained collision reasoning, and relevance judgment. This suggests that current VLMs still over-rely on appearance-level semantics and affordance priors, while lacking robust grounded reasoning about local physical constraints.

G

Correlation Analysis with Benchmarks

To better understand what aspects of VLM-agent capability are captured by PokeGym, we further analyze how model performance on PokeGym correlates with a diverse set of established external benchmarks. We conduct the analysis on the five frontier VLMs

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

%HQFKPDUN&RUUHODWLRQ0DWUL[ 9LGHR0008               6FUHHQ6SRW3UR               &KDU;LY5               +XPDQLW\VB/DVWB([DP               *34$               6:(%HQFK               7H[W$UHQD               9LVXDO*XLGHG               6WHS*XLGHG               *RDO2QO\               1DYLJDWLRQ              

     

3HDUVRQ&RUUHODWLRQU

00083UR              

 

,QWHUDFWLRQ              



00

08 9LG 3UR HR 6F 00 UHH 08 Q6 SR W3 +X PD &K UR D QLW \VB U;LY /D 5 VWB ([ DP *3 6: 4$ (% HQ 7H FK [W 9LV $UHQ D XD O* XLG 6WH HG S* XLG *R HG DO 2 1D QO\ YLJ DWL ,QW RQ HUD FWLR Q 0L [H G

0L[HG              

Figure 8: Cross-Benchmark Pearson Correlation Matrix. (Gemini-3-Pro, GPT-5.2, Qwen3-VL-30B, Qwen3.5-122B, Qwen3.535B). We include 8 external benchmarks that cover complementary capability regimes: • General multimodal reasoning: MMMU-Pro [83], VideoMMMU [20], CharXiv-R [71]. • Scientific / expert knowledge reasoning: GPQA [54], Humanity’s Last Exam (HLE) [6]. • GUI / grounded understanding: ScreenSpot-Pro [29]. • Agentic software task solving: SWE-Bench Verified [24]. • Interactive agent benchmark: Text-Arena [63]. For each model, we compare its PokeGym success rates against its scores on external benchmarks (reported on Table 11), and compute Pearson correlation coefficients across models. A preliminary overview of the Pearson correlation matrix (Figure 8) reveals that while some PokeGym task categories exhibit strong alignments with specific external benchmarks, others show near-zero or negative correlations. These results suggest that, rather than acting as a monolithic score, PokeGym decomposes VLM-agent ability into multiple strata, some partially reflected by existing evaluations and others largely orthogonal to them. This supports our design

goal of using instruction granularity and task type not only as difficulty controls, but also as diagnostic probes of different embodied cognitive bottlenecks.

G.1

Scatter Plots Analysis

Figure 9 presents scatter plots between each PokeGym setting (StepGuided, Goal-Only, Mixed) and the benchmark that shows the strongest or most relevant association. Step-Guided primarily probes structured multi-step instruction following with semantic grounding. Step-Guided is most strongly associated with Text-Arena (Pearson 𝑟 = 0.81). This is notable because the two evaluation settings differ substantially: Text-Arena is a text-based interactive benchmark, whereas PokeGym is fully visual and embodied. The transfer therefore likely does not come from low-level perception, but from the ability to execute coherent multi-step action sequences under partially specified instructions. At the same time, the correlation is not perfect, indicating that Step-Guided in PokeGym still requires additional embodied abilities beyond those captured by a text-based benchmark. Goal-Only draws on a combination of autonomous task

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

*RDO2QO\YV7H[W$UHQD 3HDUVRQU 

6WHS*XLGHGYV7H[W$UHQD 3HDUVRQU 

*RDO2QO\

6WHS*XLGHG

   

   











7H[W$UHQD *RDO2QO\YV6FUHHQ6SRW3UR 3HDUVRQU 

























7H[W$UHQD 0L[HGYV6FUHHQ6SRW3UR 3HDUVRQU 







0L[HG

*RDO2QO\



 

  

 



*HPLQL3UR





6FUHHQ6SRW3UR *37





4ZHQ9/%

6FUHHQ6SRW3UR

4ZHQ%

4ZHQ%

Figure 9: Cross-Domain Pearson Correlation Analysis. Scatter plots displaying the relationship between specific PokeGym tasks and external benchmarks. Each data point represents an evaluated VLM. The dashed lines indicate the linear regression fit with 95% confidence intervals (shaded regions). decomposition, semantic grounding, and long-horizon embodied control. Goal-Only is moderately associated with both Text-Arena (Pearson 𝑟 = 0.66) and ScreenSpot-Pro (Pearson 𝑟 = 0.63). Text-Arena focuses on text-based interactive decision-making, ScreenSpot-Pro focuses on visual grounding, whereas Goal-Only in PokeGym requires acting in a fully visual and embodied environment without procedural scaffolding. The transfer therefore likely does not come from any single ability in isolation, but from the combination of grounding underspecified goals, decomposing them into executable subgoals, and carrying out interactive actions over long horizons. At the same time, the correlations are only moderate, indicating that Goal-Only in PokeGym still requires additional embodied abilities beyond those captured by either a text-based interactive benchmark or a visual grounding benchmark alone. Mixed draws in part on visual grounding, but also depends heavily on additional sequential and embodied skills. Mixed is moderately associated with ScreenSpot-Pro (Pearson 𝑟 = 0.45). ScreenSpot-Pro focuses on visual target localization in screen-like observations, whereas Mixed in PokeGym requires acting across interleaved phases of navigation, interaction, and battle transitions under drastically changing visual contexts. What transfers across

the two benchmarks is therefore more plausibly the ability to recognize task-relevant objects and interface cues, rather than the full set of competencies required by Mixed. To perform well in Mixed, an agent must additionally preserve behavioral consistency through phase changes and sustain effective actions over long horizons. Accordingly, the modest correlation suggests that Mixed in PokeGym depends on a broader range of embodied and sequential capabilities not captured by a visual grounding benchmark alone, such as robustness over extended trajectories, adaptation to changing task regimes, and resilience to irreversible compounding errors.

G.2

Trend Lines Analysis

Figure 10 summarizes how three representative PokeGym dimensions (Interaction, Navigation, and Visual-Guided) correlate with each external benchmark. Interaction is the dimension most consistently aligned with external benchmarks. Its correlations are uniformly positive and relatively high, including 𝑟 = 0.69 with MMMU-Pro, 0.77 with VideoMMMU, 0.88 with ScreenSpot-Pro, 0.78 with GPQA, and 0.78 with Text-Arena, suggesting that stronger general frontier-model capability usually translates into better interaction performance. This pattern is also intuitive from the task structure: Interaction

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

&RUUHODWLRQ7UHQGVDFURVV([WHUQDO%HQFKPDUNV

3HDUVRQ&RUUHODWLRQU



,QWHUDFWLRQ  

1DYLJDWLRQ

 

9LVXDOJXLGHG

3UR

8 00

0

R 8 W3U 00 R 0 S R 6 HHQ 9LGH 6FU

5 [DP ;LY ( U B D W DV &K VB/ \ W L DQ +XP

4$

*3

K

HQF

6

% :(

D UHQ

W$ 7H[

([WHUQDO%HQFKPDUNV

Figure 10: Correlation Trends across External Benchmarks. The line chart traces the Pearson correlation coefficients (𝑟 ) of selected PokeGym categories across diverse external benchmarks. requires identifying semantically meaningful entities and acting at the correct location, which overlaps with multimodal understanding, visual grounding, and action execution. The especially strong correlation with ScreenSpot-Pro indicates that grounded target localization is central, while the substantial correlations with reasoning-oriented benchmarks further show that successful interaction depends not only on perception but also on semantic interpretation. Navigation is the least covered by existing benchmarks. This is evidenced by its weak or negative correlations with nearly all of these benchmarks, including 𝑟 = −0.42 with MMMU-Pro, 𝑟 = −0.41 with VideoMMMU, 𝑟 = −0.80 with ScreenSpot-Pro, 𝑟 = −0.50 with GPQA, and 𝑟 = −0.13 with Text-Arena. This suggests that strong performance on mainstream multimodal, grounding, or knowledge benchmarks does not predict embodied navigation ability. The reason is that Navigation relies on persistent spatial memory, path planning, obstacle avoidance, and stable long-horizon control, which are only weakly captured by mostly static or shorthorizon evaluations. Visual-Guided probes a distinct and poorly transferred capability. Its correlations with other benchmarks are mostly negative, ranging from −0.41 to nearly 0.00, and dropping to −0.66 with Text-Arena. Although this setting provides the most prompt information, that information mainly comes as fine-grained visual anchors, making the task less about abstract reasoning and more about precise language-to-pixel grounding during execution. The strong negative correlation with Text-Arena further shows that textual interactive competence transfers poorly to this visually anchored embodied setting.

H

Token Consumption and API Cost

Table 12 reports the average token consumption per episode across three prompting settings. For all models, we set the reasoning or thinking effort to the minimum level allowed by each model or API, in order to make token usage and cost comparisons as fair and consistent as possible. In the following, we examine the results from two perspectives: comparisons among closed-source models and comparisons among open-source models.

H.1

Comparisons Among Closed-Source Models

The large disparity in token consumption and cost. Gemini-3Pro is the most token-intensive and costly model under all three settings, reaching 388k, 277k, and 341k total tokens per run, with the highest per-run cost of $1.246 in the Visual-Guided setting. At the other extreme, GPT-5.4-nano is by far the cheapest closedsource option, costing only $0.036, $0.038, and $0.043 per run across the three settings, despite using 85k to 102k total tokens. Among the GPT models, GPT-5.4 is more token-efficient than GPT-5.2, requiring 64k–78k total tokens compared with 82k for GPT-5.2. However, its cost per run is higher, at $0.285–$0.345, compared with $0.266 for GPT-5.2. These results indicate a substantial efficiency gap. The difference in sensitivity to instruction granularity. GPT-5.2 remains perfectly stable across all three settings, with identical input, output, total token counts, and cost, indicating minimal sensitivity to instruction granularity. Claude-Sonnet-4.6 is also relatively stable, varying only slightly from 191k to 203k total tokens and from $0.840 to $0.888 in cost. In contrast, Gemini-3-Pro shows the largest variation, ranging from a minimum of 277k total tokens in Step-Guided to a maximum of 388k in Visual-Guided. The GPT-5.4 series exhibits moderate variation: GPT-5.4 increases from

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

Table 12: Token Consumption and API Cost per Run. The token metrics (Input, Output, and Total) represent the average token count for a single episode. The cost per run is calculated for proprietary closed-source models. Type

Visual-Guided

Model

Step-Guided

Goal-Only

Input Output Total Cost Input Output Total Cost Input Output Total Cost Gemini-3-Pro 341k Claude-Sonnet-4.6 170k GPT-5.4 54k Closed-Source GPT-5.2 72k GPT-5.4-mini 66k GPT-5.4-nano 67k

Open-Source

GLM-4.6V Qwen3.5-Plus Qwen3-VL-30B Qwen3.5-122B Qwen3.5-35B

69k 75k 67k 49k 52k

47k 22k 10k 10k 12k 18k

388k $1.246 238k 191k $0.840 174k 64k $0.285 57k 82k $0.266 72k 78k $0.104 67k 85k $0.036 72k

39k 22k 10k 10k 13k 19k

277k $0.944 296k 196k $0.852 181k 68k $0.293 66k 82k $0.266 72k 80k $0.109 71k 91k $0.038 82k

46k 23k 12k 10k 14k 21k

341k $1.144 203k $0.888 78k $0.345 82k $0.266 85k $0.116 102k $0.043

68k 28k 21k 24k 29k

137k 103k 88k 73k 81k

63k 27k 22k 29k 30k

132k 103k 92k 87k 84k

59k 29k 22k 28k 28k

132k 111k 95k 84k 82k

64k to 78k total tokens from Visual-Guided to Goal-Only, while GPT-5.4-mini and GPT-5.4-nano show similar but smaller upward trends. Overall, GPT-5.2 and Claude-Sonnet-4.6 are the most stable across instruction granularities, the GPT-5.4 family shows moderate sensitivity, and Gemini-3-Pro is the most sensitive.

H.2

Comparisons Among Open-Source Models

The noticeably higher token overhead of GLM-4.6V. GLM-4.6V consistently produces the highest total token usage among opensource models: 137k under Visual-Guided and 132k under both Step-Guided and Goal-Only. This is mainly due to its exceptionally large output token counts (68k, 63k, and 59k), which are more than double those of most other open-source models. This suggests that GLM-4.6V tends to generate substantially more verbose responses, making it the least token-efficient option within the open-source set. The relatively stable response to instruction granularity. Most open-source models remain fairly stable across prompting settings. For example, Qwen3.5-35B varies only from 81k to 84k, and Qwen3-VL-30B from 88k to 95k. Qwen3.5-Plus also remains reasonably stable, with only an 8k spread across settings. Overall, the open-source group demonstrates tighter token control than the more variable closed-source models such as Gemini-3-Pro, while still showing clear differences in efficiency across model families.

I

Limitations and Future Work

PokeGym currently focuses on pure-pixel RGB observations, which provides a clean testbed for visual grounding and spatial reasoning but omits other important sensory modalities. In both real-world settings and complex interactive environments, auditory perception is a fundamental channel for decision-making, often conveying information that is unavailable or less salient in vision alone. In games, for example, audio cues can signal dialogue, environmental events, nearby threats, and changes in state that are crucial for timely and effective action. A natural future direction is therefore to augment the benchmark with real-time audio input, enabling the evaluation of genuinely multi-modal embodied agents and bringing the setting closer to how humans perceive and act in the world.

– – – – –

69k 75k 70k 58k 54k

– – – – –

73k 82k 73k 56k 54k

– – – – –

PokeGym presently functions primarily as a zero-shot and fewshot evaluation benchmark. While this design is suitable for capability assessment, it does not yet support large-scale agent training. Given that our AOB memory-scanning framework can be adapted to produce dense automated rewards, an important next step is to release PokeGym as an interactive environment for reinforcement learning and imitation learning. This would broaden its utility from evaluation to training, and support the development of generalist agents for long-horizon decision-making in open-world settings.

J

Prompts for PokeGym

The prompts used for agent planning are shown in Figure 13, Figure 14, and the prompts for self-reflection are shown in Figure 15, Figure 16, Figure 17.

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

Target Visible

Target Invisible

Partial observability Distractor

True Target

True Target

Distractor

Visual Ambiguity

Nighttime

Daytime

Lighting Variability Path C

Path C

Path B

Path A

Path A

Path B

Topological Complexity

NPCs

NPCs

Element Density

Figure 11: Environmental Complexity in PokeGym. Qualitative examples of diverse challenges across five key dimensions.

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

Long-horizon Trajectory 1

Navigation

Navigation

Long-horizon Trajectory 2

Combat

Navigation Figure 12: Qualitative Examples of Long-Horizon Trajectories in PokeGym.

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

Prompt 1: Planning (Defined High-level Actions) You are an experienced player of Pokémon Legends ZA. Your goal is to complete the main quests one by one. Task: {task} === ACTION SPACE === Movement: MoveForward, MoveBackward, MoveLeft, MoveRight (Move the character a short distance forward or backward, left or right in the current direction, to reach or approach the target location.) Camera: RotateLeft, RotateRight (Turn the camera view to the left or right to find something or to provide a better perspective.) Interaction: PressA (Confirm the dialog box/Talk/Pick up) === VISUAL INPUT DEFINITION === You will receive 4 images representing the agent’s status: 1. Previous Screen: The state before your last action. 2. Current Screen (Front): The current main view. 3. Left View: The current visual information to your left. 4. Right View: The current visual information to your right. Provide the step-by-step reasoning: 1. describe the differences between the previous and current screens if available and verify the effectiveness of the previous action execution, {history} 2. analyze the key information from the left view frame and the right frame view 3. plan the next 3 actions based on the reasoning of the first and second steps After the step-by-step reasoning, you will finish by returning in this JSON format as follows: { "actions":["action1", "action2", ""] } If no action is needed (waiting/idling), use an empty string "". Figure 13: Prompt for Agent Planning (Defined High-level Actions).

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

Prompt 2: Planning (Parametric Control) You are an experienced player of Pokémon Legends ZA. Your goal is to complete the main quests one by one. Task: {task} === ACTION SPACE === Instead of high-level actions, you must explicitly control the gamepad by defining the exact action type, the duration to hold the input (in milliseconds), and the parameters (if applicable). You can output an array of UP TO 3 actions to be executed sequentially. Available types: 1. "Left" (Left Stick - Movement): - Requires duration (e.g., 500 ms). - Requires parameters: A float array [X, Y] ranging from -1.0 to 1.0. - X-axis: -1.0 (Left) to 1.0 (Right). - Y-axis: -1.0 (Backward) to 1.0 (Forward). 2. "Right" (Right Stick - Camera): - Requires duration (e.g., 200 ms). - Requires parameters: A float array [X, Y] ranging from -1.0 to 1.0. - X-axis: -1.0 (Look Left) to 1.0 (Look Right). - Y-axis: -1.0 (Look Down) to 1.0 (Look Up). 3. Buttons: - Does NOT require parameters. Only requires duration (e.g., 200 ms). - "A": General interaction (confirm, talk, pick up). 4. "Idle": - Wait/Do nothing. Requires duration. No parameters. === VISUAL INPUT DEFINITION === You will receive 4 images representing the agent’s status: 1. Previous Screen: The state before your last action. 2. Current Screen (Front): The current main view. 3. Left View: The current visual information to your left. 4. Right View: The current visual information to your right. Provide the step-by-step reasoning: 1. Describe the differences between the previous and current screens if available, and verify the effectiveness of the previous action execution, {history}. 2. Analyze the key information from the left view frame and the right frame view. 3. Plan a sequence of controller inputs based on your reasoning. After the reasoning, return in this strictly formatted JSON containing UP TO 3 actions: { "actions":[ { "type": "Left", "duration": 500, "parameters":[0.0, 1.0] }, { "type": "Right", "duration": 200, "parameters":[0.5, 0.0] }, { "type": "A", "duration": 200 } ] } (Note: If the type is a button or skill, simply omit the parameters field). Figure 14: Prompt for Agent Planning (Parametric Control).

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

Prompt 3: Trajectory Summarization An agent in the Pokémon game produced the following sequence of thoughts and actions. Summarize the history trajectory based on the following instructions. <trajectory> {trajectory} </trajectory> Constraint: Do NOT list specific action commands (e.g., "RotateLeft", "PressA"). Use descriptive language only. Analysis: 1. Critical Visual Cues: Describe the key environmental features that triggered the agent’s decisions. 2. Strategy Effectiveness: Analyze the effectiveness of the agent’s apparent goal or strategy. 3. Current Status: Briefly state where the agent is now compared to the start of this trajectory. Figure 15: Prompt for Trajectory Summarization in Self-reflection.

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

Prompt 4: Experience Refinement You are an expert Game Mechanics Analyst and Strategy Coach. Your goal is to refine the agent’s long-term memory (Experience Library) by analyzing its recent performance trajectory summary. Your Task: Identify gaps, conflicts, or inefficiencies in the Experience Library based on the Trajectory Summary. You must distill universal game rules or high-probability heuristics that will help the agent in future unseen scenarios. Step-by-Step Reasoning Process: 1. Gap Analysis (Summary vs. Library): - Does the Trajectory Summary show a success that isn’t covered by the current Experience Library? -> Signal to ADD. - Does the Trajectory Summary show a failure where the agent followed an existing rule? -> Signal to MODIFY. - Is the current rule too vague? 2. Formulate Generalizable Rules and Update Existing Experiences: - You have two options: [modify, add] * modify: You can modify current experiences to make them helpful * add: You can introduce new experiences that may be needed - Do create rules based on visual patterns and game logic - Every experience must follow a Condition -> Action -> Reason logic. - You can update at most 2 experiences for this case. 3. Requirements for each new or modified experience: - Prioritize correcting fatal errors (getting stuck, loops) over minor optimizations. - The content must be concise, instructional, and general. - Constraint: Do NOT list specific action commands (e.g., "RotateLeft", "PressA"). Use descriptive language only. Please provide reasoning in details under the guidance of the above 3 steps. After the step-by-step reasoning, you will finish by returning in this JSON format as follows: [ { "option": "modify", "experience": "the modified experience", "modified_from": "G17" # specify the ID of experience that is modified }, { "option": "add", "experience": "the added experience" }, ... ] Note that your updated experiences may not need to cover both options. Only using one type of updates is also acceptable. <summary> {summary} </summary> <experiences> {experiences} </experiences> Figure 16: Prompt for Experience Refinement in Self-reflection.

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan

Prompt 5: Experience Revision You are an Expert AI Strategist responsible for curating and refining the master experience library for an agent in the Pokémon game. From the reflections, some suggestions on the existing experiences have been posed. Your task is to collect and think for the final experience revision plan. Each final experience must satisfy the following requirements. 1. It must be clear, generalizable lessons for this case, with no more than 32 words. 2. Follows the Condition -> Action -> Reason logic established in the analysis. 3. Avoid repeating saying similar experiences in multiple different experiences. 4. Constraint: Do NOT list specific action commands (e.g., "RotateLeft", "PressA"). Use descriptive language only. <existing_experiences> {existing_experiences} </existing_experiences> <suggested_updates> {suggested_updates} </suggested_updates> Please provide reasoning in each of the suggestions, and think about how to update existing experiences. You have two update options: [modify, merge] * modify: You can modify current experiences to make them helpful * merge: You can merge some similar experiences into more general forms to reduce duplication After generating the step-by-step reasoning, you need to give the final experience revision details by returning in this JSON format as follows: [ { "option": "modify", "experience": "the modified experience", "modified_from": "C1" # specify the str ID of experience that is modified }, { "option": "merge", "experience": "the merged experience", "merged_from":["C1", "C3", "S4", ...]

# specify the str IDs to merge from

}, ... ] Figure 17: Prompt for Experience Revision in Self-reflection.

Related documents

• Unveiling the Visual Counting Bottleneck in Vision-Language Models#238626
• CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models#175446
• IntroConformal: Conformal Factuality Guarantees for Large Vision-Language Models via Introspective Signals#821315
• Aligning Vision and Language: Harnessing Language Semantics for Efficient Vision Models#995758
• Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputs#1012404