arXiv:2604.08340v1 [cs.CV] 9 Apr 2026
PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models Ruizhi Zhang
Ye Huang∗
Yuangang Pan
Chuanfu Shen
Zhilin Liu
SIAS, UESTC Shenzhen, China
SIAS, UESTC Shenzhen, China
CFAR/IHPC A*STAR Singapore
SIAS, UESTC Shenzhen, China
SIAS, UESTC Shenzhen, China
Ting Xie
Wen Li
Lixin Duan
SIAS, UESTC Shenzhen, China
SIAS, UESTC Shenzhen, China
SIAS, UESTC Shenzhen, China
Prior Works
State: XYZ = (239, 125, 32) Front_Voxel = Wood Block ID = 3
PokeGym O1 2D Tiles
3D World
Fixed Views
Variable Views
Simplified Vision
Complex Elements
State-based Input
Vision-only Input
Symbolic Format
Human-like View
Degrades to LLM Task
VLM Real Challenge
Human Eval
Automated Eval
Costly
Scalable
Human Bias
Objectivity
Human Dependent
Liberate Human
Task is successful/failed.
Task is successful/failed.
Figure 1: Advancing prior works, PokeGym features complex 3D environments, raw pixels, and scalable automated evaluation.
Abstract While Vision-Language Models (VLMs) have achieved remarkable progress in static visual understanding, their deployment in complex 3D embodied environments remains severely limited. Existing benchmarks suffer from four critical deficiencies: (1) passive perception tasks circumvent interactive dynamics; (2) simplified 2D environments fail to assess depth perception; (3) privileged state leakage bypasses genuine visual processing; and (4) human evaluation is prohibitively expensive and unscalable. We introduce PokeGym, a visually-driven long-horizon benchmark instantiated within Pokémon Legends: Z-A, a visually complex 3D open-world Role-Playing Game. PokeGym enforces strict code-level isolation: agents operate solely on raw RGB observations while an independent evaluator verifies success via memory scanning, ensuring pure vision-based decision-making and automated, scalable assessment. ∗ Corresponding author
The benchmark comprises 30 tasks (30–220 steps) spanning navigation, interaction, and mixed scenarios, with three instruction granularities (Visual-Guided, Step-Guided, Goal-Only) to systematically deconstruct visual grounding, semantic reasoning, and autonomous exploration capabilities. Our evaluation reveals a key limitation of current VLMs: physical deadlock recovery, rather than high-level planning, constitutes the primary bottleneck, with deadlocks showing a strong negative correlation with task success. Furthermore, we uncover a metacognitive divergence: weaker models predominantly suffer from Unaware Deadlocks (oblivious to entrapment), whereas advanced models exhibit Aware Deadlocks (recognizing entrapment yet failing to recover). These findings highlight the need to integrate explicit spatial intuition into VLM architectures. The code and benchmark will be available on GitHub.
1
Introduction
Recent Vision-Language Models (VLMs) have achieved impressive progress in static visual understanding and instruction following
Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan
[11, 14, 37, 58]. Yet it remains unclear to what extent these capabilities translate into autonomous behavior in visually rich 3D environments [12, 22, 82], where agents must perceive from pixels, act under partial observability, and pursue long-horizon goals through continuous interaction [32, 70, 75, 80]. A central obstacle is the lack of benchmarks that can evaluate it faithfully and at scale. An effective benchmark for embodied VLM agents should jointly enable at least four properties: long-horizon interaction, realistic 3D visual reasoning, decision-making from pure visual observations, and scalable automated evaluation. However, existing protocols typically trade away one or more of these properties: (1) Static image benchmarks and single-turn tasks, such as visual question answering (VQA) or image captioning [2, 17, 34, 41, 76], reduce evaluation to momentary recognition and bypass the challenges of persistent planning and control [50, 73]. (2) Interactive benchmarks in 2D games or grid worlds [21, 49] introduce sequential decision-making, but their simplified visuals do not match the complexity of real-world scenes, failing to capture depth perception and 3D spatial reasoning. (3) More realistic 3D environments often expose privileged internal states, such as coordinates or symbolic world representations [10, 15, 33, 38, 88], allowing agents to bypass the perceptual burden that real-world visual agents must solve. (4) Conversely, game benchmarks that restrict agents to pure visual inputs frequently rely on human evaluation [5, 61, 62, 65], limiting scalability, reproducibility, and objectivity. As a result, strong performance on existing benchmarks may not reflect robust embodied competence. To bridge this gap, as illustrated in Figure 1, we introduce PokeGym, a visually-driven, long-horizon benchmark instantiated in a 3D open-world Role-Playing Game (RPG), Pokémon Legends: ZA. This game serves as an ideal testbed because its mechanics mirror the core challenges of real-world embodiment: partial observability forces agents to build spatial memory, navigation and diverse object interactions test fine-grained visual-action grounding, while intricate quest structures and extended temporal dependencies demand robust long-horizon planning and error recovery. PokeGym resolves the tension between pure visual realism and automated evaluation: the agent acts solely from raw RGB observations, while task success is verified independently through state extraction using Array of Bytes (AOB) memory scanning. PokeGym contains 30 tasks derived from 10 quests, with trajectories ranging from 30 to 220 environment steps and covering navigation, interaction, and mixed long-horizon scenarios. Each task is instantiated under three instruction granularities: VisualGuided, Step-Guided, and Goal-Only. These granularities create a controlled setting for disentangling embodied capabilities: visual grounding under explicit cues, semantic reasoning under procedural guidance, and autonomous exploration under sparse goals. Beyond success rates, PokeGym also supports fine-grained diagnosis of embodied failures, highlighting the value of PokeGym not only as an evaluation suite, but also as a diagnostic testbed for embodied VLM research. Our primary contributions are summarized as follows: (1) We introduce PokeGym, a visually-driven, long-horizon benchmark for embodied VLMs in a 3D open-world game. Its mechanics capture core challenges of real-world embodiment.
Table 1: Comparison of VLM Benchmarks. Open World reflects whether the environment permits unconstrained, nonlinear exploration. Interactivity differentiates closed-loop multi-turn embodied dynamics from passive single-turn responses. Long-Horizon indicates the necessity for multi-step sequential planning. Evaluation Benchmark
Open InterLong Env Only Eval World activity Horizon Domain Vision Method
MVP-Bench [28] LVLM-eHub [76] VLMbench [84] VisGym [72] NetHack [26] StarDojo [60] MINEDOJO [15] Cradle [62] Lumine [61]
× × × × ✓ ✓ ✓ ✓ ✓
Single Single Multi Multi Multi Multi Multi Multi Multi
× × × ✓ ✓ ✓ ✓ ✓ ✓
VQA VQA Robotics Mixed 2D RPG 2D RPG 3D RPG 3D RPG 3D RPG
✓ ✓ × × × × × ✓ ✓
QA Acc QA Acc Auto Auto Auto Auto Auto Human Human
PokeGym
✓
Multi
✓
3D RPG
✓
Auto
(2) We present a rigorous and scalable evaluation pipeline in the complex game environment. It restricts agents to pure-pixel observations by eliminating privileged state leakage, and features an independent evaluator that extracts game states via AOB memory scanning for automated, objective verification. (3) We establish a controlled diagnostic framework for disentangling key embodied capabilities in VLMs. Specifically, we design 30 long-horizon tasks across three instructional granularities to independently assess visual grounding, semantic understanding, and autonomous exploration. (4) We provide a comprehensive analysis of VLM failures, revealing that physical deadlock recovery—rather than high-level planning—is the primary bottleneck. We further uncover a metacognitive divide between weaker and stronger models when trapped.
2 Related Work 2.1 Benchmarks for VLMs The growth of Vision-Language Models (VLMs) has shifted evaluation from static perception to dynamic interaction [7, 18, 57]. Early benchmarks typically evaluate VLMs on passive visual understanding tasks, such as Visual Question Answering (VQA) [28, 76], image captioning [9, 27, 35, 87], and visual grounding [8, 56, 77, 86]. While some benchmarks have utilized videos for semantic and spatial reasoning [28, 79], they treat perception as a passive task, overlooking the interactive dynamics of closed-loop environments, where an agent’s actions continuously alter future observations. To address this, recent efforts have introduced interactive and embodied benchmarks [16, 23, 36, 43, 59, 68]. Frameworks such as VLMbench [84] focus on tabletop manipulation, whereas VisGym [72] and EMemBench [31] evaluate multi-step visual interactions and episodic memory. Despite these advancements, existing interactive benchmarks rely on constrained state spaces or short episodes, reducing the need for long-range planning. In contrast, PokeGym plunges VLMs into a visually complex, unconstrained 3D open world, demanding sustained visual interaction and long-horizon spatial planning.
PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models
2.2
Game-based Evaluation Environments
Games have served as ideal testbeds because they provide rich visual and diverse gameplay [4, 42, 48, 51, 81]. Traditional game benchmarks such as NetHack [26], DOOM [25], and 2D grid-worlds like Pokémon Red [49], have been used for reinforcement learning [47, 67, 74]. With the rise of foundational agents, recent works have shifted towards open-ended simulations and RPGs [19, 40, 55, 69, 78, 85]. For instance, StarDojo [60] evaluates agents in productionliving simulations Stardew Valley, while MineDojo [15] assesses agents across open-ended crafting and exploration tasks in the 3D voxel world of Minecraft. More recently, many agents interact with complex 3D worlds through screen pixels, keyboard and mouse actions [30, 53]. Some works have demonstrated that VLM agents can complete long missions in AAA games [61, 62]. Additionally, foundation models like NitroGen [39] have shown impressive crossgame generalization. However, evaluating these agents reveals critical flaws: 2D games lack spatial realism, 3D simulators leak game states, and pixel-only AAA games demand unscalable human assessment. PokeGym resolves this by combining a complex 3D world and pure-pixel inputs with an memory-based evaluator, ensuring scalable, automated, and objective success verification. The qualitative comparison of VLM benchmarks is summaried in Table 1.
3 PokeGym Benchmark 3.1 Game Environment PokeGym is a visual-centric, long-horizon evaluation benchmark built upon the 3D open-world game Pokémon Legends: Z-A. Unlike traditional 2D grid-world benchmarks or sandbox-style 3D environments (e.g., Pokémon Red [49] or Minecraft [15]), this game provides a richer and more challenging setting for VLM-based agents, mainly due to three distinctive properties: (1) Freely controllable camera with changing viewpoints. The game camera can be rotated to view the world from different angles. This makes the observation space highly viewpointdependent: key targets may be outside the screen, partially blocked, or only recognizable from specific angles. As a result, the agent must actively look for useful information by turning the camera, checking nearby areas, and adjusting its distance to objects rather than passively reacting to a fixed view. (2) Visually complex 3D scenes with dense, diverse elements. The open world contains cluttered geometry (buildings, vegetation), dynamic actors (NPCs, wild Pokémon), interactive props, UI overlays, and multiple depth layers. To act correctly, the agent needs to disambiguate similar-looking objects, read small text, and reason over spatial relations under lighting changes and occlusion. (3) Structured progression beyond sandbox-style planning. In contrast to Minecraft [10, 15, 33], where long-term planning is often centered on resource gathering, crafting and construction, Pokémon Legends: Z-A features progression that is tied to quests, encounters, and event triggers. Agents must coordinate exploration, object interaction, battle, and goal completion under delayed and context-specific consequences, making success depend not only on open-ended planning but also on understanding task structure and scripted progression.
3.2
Task Definitions and Budgets
PokeGym contains 30 long-horizon tasks spanning three categories: navigation, interaction, and mixed tasks. These categories broadly cover movement to target locations, interaction with objects, and multi-stage tasks that combine multiple gameplay skills. Further details are provided in the supplementary material. To eliminate ambiguity, every task is formalized with 4 components. Initial State: Each task is initialized from a corresponding preconfigured save file to equalize starting conditions for all agents. Success Criteria: Task completion is threshold-verified using memory variables (e.g., a navigation goal is complete when the coordinates fall within a predefined bounding box). Fixed Step Budget: Each task is assigned a fixed budget of environment steps. Based on heuristic human demonstrations, the budgets range from 180 to 360 environment steps. Termination: An episode terminates under two conditions: (1) Success criteria met; (2) Step budget exhausted. The relevant information of the tasks is displayed on Figure 2.
3.3
Instruction Granularity & Cognitive Probes
To diagnose the specific bottlenecks of VLM agents, the 30 tasks are derived from 10 distinct quests. We map these tasks across three levels of instruction granularity, varying the information density to probe distinct cognitive capabilities, as illustrated in Figure 2. Visual-Guided: The prompt provides a multi-stage procedural plan with visual anchors (e.g., "Approach and enter the door of the house, locate and talk to the hotel owner behind the reception desk"). This setup evaluates the model’s visual grounding capability and the ability to map linguistic descriptions to pixel-level features. Step-Guided: The prompt retains the procedural sub-goals but removes the visual anchors (e.g., "Approach and enter the door of the house, locate and talk to the hotel owner"). Without specific visual features, the agent must rely on semantic understanding and common sense to identify generic objects. Goal-Only: The prompt provides only the ultimate objective (e.g., "Locate and talk to the hotel owner"). The agent must autonomously decompose the goal, explore the space, and deduce the intermediate steps. This setting tests long-horizon planning and autonomous exploration capabilities. By comparing performance across above tiers, we can systematically probe an agent’s specific cognitive strengths and bottlenecks.
3.4
System Architecture
The architecture of PokeGym is illustrated in Figure 3. At a high level, the framework consists of four parts: (i) an observation interface that provides visual inputs from the environment, (ii) a VLMbased decision module, optionally augmented with a self-reflection mechanism, (iii) an action interface that translates model outputs into executable controls, and (iv) an evaluation interface for automated progress tracking and success verification. The environment is built on the Ryujinx emulator implemented in C#. Observation Interface. PokeGym models the agent as a pure visual learner. At each decision step, the agent receives configurable visual observations, with the current front-view frame serving as the default input across all settings. To provide richer spatial and temporal context, the observation space can be extended with:
Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Wen Li, and Lixin Duan
PokeGym
Nav
Int
···
···
···
··· Mix
Instruction Granularity Approach and enter the door of the house, locate and talk Visual-Guided to the hotel owner behind the remove visual reception desk. anchor Approach and enter the door Step-Guided of the house, locate and talk remove step to the hotel owner. guide Locate and talk to the hotel Goal-Only owner.
Figure 2: Overview of the tasks of PokeGym. The Top 3 Rows: Sample visual trajectories representing Navigation (Nav), Interaction (Int), and Mixed (Mix) tasks. Bottom-Left: Illustration of the three instruction granularities. Bottom-Right: Environment step budgets and distribution of the 10 quests evaluated in the benchmark. VLM
Input
Responses VLM Decision Module
Observation Interface Current Frame
Left View
Previous Frame
Right View
Optional: Self-Reflection - Reflect on the strategy - Update short-term memory - Analyze and refine - Update experiences
Action Interface Defined High-level Actions PressA
RotateLeft
Parametric Control X: -0.6 Type: LeftStick
GPU Textures Auxiliary Design: adaptive pause mechanism
··· MoveForward Y:1.0 Duration: 500ms
Ryujinx Emulator
Automated Evaluation Interface
AOB Scanning
- Map ID; - X/Y/Z Coordinates; - Quest Flags;
Figure 3: Overview Architecture of the proposed PokeGym. • Previous frame: the frame before the last executed action, enabling reflection on action outcomes and temporal feedback; • Left and right (L/R) views: peripheral images that expand the agent’s spatial awareness.
Rather than relying on OS-level screen capture, these RGB observations are directly extracted from GPU textures. This design reduces
visual acquisition latency, avoids rendering bottlenecks, and eliminates window occlusion issues. To ensure fairness, no internal game state is exposed to the agent. VLM Decision Module. Given the visual observations, the VLM produces action decisions based solely on the provided image context and interaction history. To further support long-horizon adaptation, we provide an optional self-reflection module. When enabled, every 𝑘 steps (default 𝑘=5), a summarization routine prompts the VLM to analyze recent response history and evaluate the effectiveness of its current strategy. The resulting reflection updates the short-term memory M𝑡 , while distilled actionable insights are written into the persistent experience library E𝑡 through (ADD, DEL, MOD, KEEP) operations. This design keeps the context concise while allowing the model to iteratively revise its strategy online, despite the lack of explicit external feedback. Action Interface. PokeGym has two action execution paradigms: • Defined high-level actions: the agent outputs discrete commands (e.g., MoveForward, RotateRight), which are mapped to fixed execution durations in the environment wrapper (e.g., 500 ms for moving and 200 ms for rotating); • Parametric control: the agent directly specifies the maneuver type, execution duration, and continuous joystick values (e.g., 𝑋, 𝑌 ∈ [−1.0, 1.0]). To support different planning granularities, we decouple decision steps from environment steps. A decision step corresponds to one model query, whereas an environment step corresponds to
PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models
one physically executed action in the emulator. Accordingly, the VLM may output either a single action (1 environment step) or an ordered sequence of actions (3 environment steps) per query. For fair comparison, the total budget of environment steps is kept constant across settings. Evaluation Interface. For automated progress tracking and success verification, the environment performs Array of Bytes (AOB) memory scanning at initialization to locate memory addresses associated with map IDs, character coordinates, and quest flags via signature patterns. These values are only used by the evaluator and are never exposed to the agent prompt. This mechanism enables scalable and cross-machine automatic evaluation under the same game version, removing the need for manual checking. Auxiliary Design. For combat tasks that require high-frequency reactions, we introduce an adaptive pause mechanism that pauses the environment during the reasoning phase and resumes during action execution. This prevents differences in VLM inference latency from introducing confounding bias in time-sensitive scenarios.
3.5
Compliance and Reproducibility
PokeGym does not distribute game ROMs, decryption keys, firmware, or any proprietary assets. Researchers must legally acquire and dump their own game copies to use the benchmark. Given a legally obtained ROM and the specified game version, PokeGym can be reproduced by combining an open-sourced emulator framework, pre-configured initial save files for each task, and an automatic evaluator that verifies success through signature patterns. These components will be released as non-proprietary resources.
4 Experiments 4.1 Experimental Design Overview The proposed benchmark can differentiate models across distinct embodied capabilities (capability coverage), offers interpretable diagnosis of both cognitive and physical failure modes (diagnosticity), and supports controlled analysis of interventions and design choices (actionability). It is designed not only to report model rankings but also to serve as a useful evaluation instrument. Capability coverage. We evaluate a diverse set of VLMs under three instruction granularities. This design enables our benchmark to distinguish models along multiple embodied capabilities, including visual grounding, semantic reasoning, and long-horizon planning. Rather than collapsing these abilities into a single undifferentiated score, our benchmark reveals fine-grained performance differences across models. Diagnosticity. Beyond final task success, we analyze the execution process through trajectory-level physical metrics and detailed failure categories. It reveals why agents fail, rather than merely indicating failure outcomes. This diagnostic value enables systematic failure decomposition across models and task settings. Actionability. Finally, we perform intervention and ablation studies, including deadlock interventions, visual-context ablations, action-execution strategies, and self-reflection analysis. These support flexible combinations of diverse configurations and enables close inspection of model behaviors. This modular design yields actionable insights by pinpointing bottlenecks and providing targeted guidance for improving model and agent architectures.
Table 2: Performance comparison across 3 granularity levels. Success Rate (SR, %) measures the percentage of episodes that successfully complete the task. Average Environment Steps (Stp) denote the average number of environment steps in successful episodes. Bold indicates the best performance. Model
Navigation Interaction SR↑
Mixed Stp↓
Average
Stp↓
SR↑
Stp↓ SR↑
SR↑
GLM-4.6V 25.00 123.20 Qwen3.5-35B 45.00 124.67 Qwen3.5-122B 60.00 124.92 55.00 81.73 Visual Qwen3.5-Plus Guided Qwen3-VL-30B 50.00 89.10 Claude-Sonnet-4.6 55.00 124.45 Gemini-3-Pro 20.00 120.00 GPT-5.2 25.00 147.00
46.67 80.00 66.67 66.67 66.67 80.00 66.67 93.33
58.14 60.00 74.33 43.89 85.22 61.75 26.67 84.50 50.56 90.31 67.10 53.33 101.38 60.00 97.80 65.30 26.67 153.00 49.45 100.01 50.20 53.33 142.25 56.67 93.85 81.00 46.67 131.14 60.56 112.20 61.60 46.67 134.14 44.45 105.25 41.50 60.00 86.22 59.44 91.57
Stp↓
GLM-4.6V 25.00 136.80 Qwen3.5-35B 45.00 85.56 Qwen3.5-122B 25.00 79.40 Step Qwen3.5-Plus 50.00 75.70 Guided Qwen3-VL-30B 40.00 73.50 Claude-Sonnet-4.6 55.00 81.73 Gemini-3-Pro 70.00 101.86 GPT-5.2 30.00 96.00
53.33 60.00 66.67 53.33 60.00 60.00 93.33 86.67
42.13 46.67 66.29 41.67 81.74 77.56 33.33 89.40 46.11 84.17 37.10 20.00 162.33 37.22 92.94 42.75 26.67 125.50 43.33 81.32 60.11 46.67 115.43 48.89 83.01 91.22 60.00 155.33 58.33 109.43 85.29 60.00 104.89 74.44 97.34 74.62 53.33 94.00 56.67 88.21
GLM-4.6V 25.00 211.40 73.33 46.73 26.67 166.00 41.67 141.38 Qwen3.5-35B 45.00 111.56 80.00 77.92 13.33 125.00 46.11 104.82 Qwen3.5-122B 25.00 126.20 73.33 39.64 40.00 126.17 46.11 97.33 50.00 66.60 46.67 79.00 20.00 100.33 38.89 81.98 Goal Qwen3.5-Plus Only Qwen3-VL-30B 45.00 90.78 73.33 92.45 33.33 147.80 50.55 110.34 Claude-Sonnet-4.6 55.00 99.73 60.00 59.78 6.67 125.00 40.56 94.84 Gemini-3-Pro 45.00 108.22 100.00 79.00 26.67 115.75 57.22 100.99 GPT-5.2 40.00 76.25 100.00 89.07 40.00 145.33 60.00 103.55
4.2
Implementation Details
We evaluate diverse VLMs, encompassing both open-weight models (GLM-4.6V [66], Qwen 3/3.5 series [3, 52, 64]) and closed-source proprietary models (GPT-5.2 [44], Gemini-3-Pro [13], and ClaudeSonnet-4.6 [1]). Each setting is evaluated with 5 trials. All models share the identical initial state, prompt template, and budget accounting within the same task. An episode terminates when the task is successfully completed or the step budget is exhausted.
4.3
Cognitive Capability Coverage
Table 2 presents a comparison of model performance across the three instruction granularity levels. For the experiments in this section, the observation space includes all four images. For the action space, all models employ the defined high-level actions paradigm, and each decision step outputs an ordered sequence of three actions, equating to three environment steps. Visual Grounding. In the Visual-Guided tasks, the prompt provides procedural steps with visual anchors. Claude-Sonnet-4.6 achieves the highest average Success Rate (SR 60.56%), closely followed by Qwen3.5-122B (60.00%) and GPT-5.2 (59.44%), indicating strong grounding from visual cues to actionable decisions. Qwen3.5122B achieves the best Navigation SR (60.00%), highlighting its visual grounding capability in spatial traversal, enabling it to leverage visual references for navigation and movement decisions. Semantic Reasoning. In the Step-Guided tasks, visual references are removed and the procedural sub-goals are retained, forcing agents to rely on semantic understanding to identify generic objects within the 3D environment. Gemini-3-Pro experiences a performance leap, surging from an average SR of 44.45% to a leading 74.44%, while dominating Navigation (70.00%), Interaction (93.33%)