Executable World Models for ARC-AGI-3 in the Era of Coding Agents ∗
arXiv:2605.05138v1 [cs.AI] 6 May 2026
Sergey Rodionov SingularityNET [email protected]
Abstract We evaluate an initial coding-agent system for ARC-AGI-3 in which the agent maintains an executable Python world model, verifies it against previous observations, refactors it toward simpler abstractions as a practical proxy for an MDL-like simplicity bias, and plans through the model before acting. The system is intentionally direct: it uses a scripted controller, predefined world-model interfaces, verifier programs, and a plan executor, but no hand-coded game-specific logic. We report results on the 25 public ARC-AGI-3 games. Each recorded playthrough uses a fresh agent instance with no access to previous playthrough-specific files or conversation state. Most games have a single recorded playthrough; for a few games, we report multiple independent fresh-agent playthroughs to expose run-to-run variability. The agent fully solved 7 games, achieved a Relative Human Action Efficiency greater than 75%, on 6 games, and obtained a mean per-game RHAE of 32.58%. Because the system uses no game-specific code, it can serve as a game-general baseline for ARC-AGI-3. Performance on the private validation set remains to be tested. Overall, the results provide preliminary evidence that verifier-driven executable world models are a promising approach for ARC-AGI-3 agents.
1
Introduction
Large language models are most reliable when they are not used as final authorities, but as proposal mechanisms inside systems that can check their outputs. This pattern appears in recent successes such as AlphaCode, where generated programs are filtered by execution [8], FunSearch, where proposed programs are selected by automated evaluators [10], and AlphaGeometry, where a neural model proposes auxiliary geometric constructions and an exact symbolic engine derives and verifies their consequences [12]. In each case, the language model supplies approximate search, while reliability comes from an external verification process. This generate-and-verify pattern is powerful in domains such as programming, algorithm discovery, and formal mathematics because verification is relatively cheap. A program can be run against tests; a candidate algorithm can be scored by an evaluator; a formal proof can be checked by a proof assistant. General interactive agents face a different problem. In an unfamiliar environment, each action may consume time, change the state irreversibly, or reduce the final score. The agent therefore cannot rely on trial and error in the environment as its main source of verification. Instead, it needs an internal model in which possible actions can be simulated, rejected, revised, and planned before they are executed. We use the term world model in its standard broad sense: an internal model that allows an agent to predict consequences, evaluate hypotheses, and plan before acting. In this paper, the world ∗
Project code will be released publicly after acceptance.
1
model is not a latent neural state or an opaque learned simulator. It is an executable world model : a Python codebase whose functions encode the agent’s current hypothesis about the environment. The model can be run, tested, edited, and used for planning. It can also be refactored as new observations arrive, so that accidental special cases are replaced by simpler abstractions. Our approach builds on prior work on world models and model-based agents. World models are central to model-based reinforcement learning and agentic intelligence more generally [6]. Recent work has also studied world models represented explicitly as programs. WorldCoder builds a Python program representing an agent’s knowledge of the world from interaction data [11]. Code World Models study world models generated by large language models in the form of Python code for model-based reinforcement learning [2]. The contribution of this work lies in adapting this programmatic world-modeling perspective to ARC-AGI-3 and studying how a coding agent can maintain, verify, refactor, and use such a model under strict interaction constraints. A second principle is simplicity. When an agent observes only a small number of transitions, many world models may be consistent with the data. The useful model is not merely one that fits previous observations, but one that captures the underlying regularities compactly enough to support future planning. This is closely related to the Minimum Description Length perspective, which treats good explanations as those that compress the observations while accounting for the complexity of the model itself [5]. In our system, however, we do not implement a formal MDL objective. Instead, we use a practical proxy suited to coding agents: after new observations are incorporated, the agent is repeatedly asked to refactor its executable model, replacing special cases with simpler abstractions while preserving verifier correctness. This connects to prior work on library learning and code refactoring, where reusable abstractions and MDL-like compression have been used to guide or evaluate better code organization [3, 4, 7]. ARC-AGI-3 is a natural testbed for this approach. Unlike static puzzle benchmarks, ARCAGI-3 places agents in novel, abstract, turn-based environments where they must explore, infer goals, build models of environment dynamics, and plan action sequences without explicit naturallanguage instructions [1]. The benchmark is explicitly framed around adaptive efficiency: humans can solve the environments, while frontier AI systems scored below 1% as of March 2026 [1]. This makes ARC-AGI-3 a controlled version of a broader AGI problem. The environments are simple enough that explicit computational models are plausible, but direct experimentation is still costly enough that blind trial and error is inadequate. To perform well, an agent must learn from limited interaction, compress observations into a useful model, and use that model to decide which actions are worth spending in the real environment. In this paper, we study an initial agent based on this principle. The agent is instructed to maintain a Python codebase representing its current model of the game, including functions for state representation, transition prediction, goal checking, and planning. After each modification of the world model, the agent is instructed to run verifiers that test consistency with previous observations. The agent is also prompted to refactor the model as evidence accumulates, replacing special cases with simpler executable abstractions. We evaluate the agent under strict ARC-AGI-3 interaction constraints. The unit of evaluation is a recorded playthrough: for each playthrough, a fresh agent process and clean workspace are launched, and the agent receives only one exposure to the target game. Within that playthrough, the agent may not restart the whole game to obtain a better trajectory, may not return to previously completed levels, may not use information from previous playthroughs of the same game, and is not given hand-coded game-specific logic. Some public games were evaluated in more than one independent playthrough; these rows are fresh launches of the agent from scratch, not replays by the same agent instance. We report the performance of our agent on all 25 public games. The agent and harness are 2
designed to be game-general within ARC-AGI-3 rather than tailored to individual games. In practice, most of the development of this first version was performed using the public game ls20. However, only evaluation on the private validation set can directly test how well the approach generalizes. We report this first implementation as a baseline for studying verifier-driven executable world models in ARC-AGI-3.
2
Implementation
2.1
ARC-AGI-3 interaction protocol
ARC-AGI-3 games are interactive, level-based environments. At each step, the agent observes the current game state and submits one action from the available action set. A level attempt may end in either LEVEL COMPLETED or GAME OVER. If GAME OVER is reached, the benchmark-provided RESET action can be used to restart the current level attempt within the same game run. In our evaluation, RESET is treated as an ordinary environment action and counts against the action budget. The agent cannot restart the whole game to obtain a better run, and the agent cannot return to previously completed levels.
2.2
Agent architecture
Our architecture is intentionally simple. The system consists of a coding agent controlled by a scripted external controller. The external controller does not solve levels directly. Instead, it starts the game, passes observations to the coding agent, monitors whether the current level is still running, completed, or in GAME OVER, and sends predefined prompts in these situations. During normal play, the coding agent is instructed to continue working until it either completes the current level or reaches GAME OVER. Before normal continuation, after terminal failure, and when progress appears to stall, the controller asks the coding agent to simplify and refactor its world model before continuing. If GAME OVER is reached, the controller issues RESET and returns the new attempt observation to the coding agent. The coding agent works inside an initialized Python workspace. The workspace provides template components for three functions: reconstructing and rendering game states from observations, implementing an executable world model of the transition dynamics, and planning actions inside that model. These templates expose predefined interfaces but initially contain no task-specific logic. The coding agent is responsible for filling in and maintaining these components as new levels and observations are encountered. The coding agent works inside an initialized Python workspace. This workspace contains templates for the executable world model and planner: world_model_engine.py for transition dynamics, world_model_state_io.py for state reconstruction and rendering, and world_model_main_ planner.py for planning in the learned model. These files initially contain only predefined interfaces; the coding agent is responsible for filling them in and maintaining them as new levels and observations are encountered.
2.3
Verification and execution helpers
The workspace also provides general helper programs. The most important are the world-model verifier, the planner verifier, and the plan executor. The world-model verifier checks that the executable model reproduces the recorded observations from previous attempts. The planner verifier checks that the main planner can produce plans that reach LEVEL COMPLETED inside the learned
3
model for solved levels. Planner-running utilities allow the agent to test plans from the current state, from the initial state of a level, or from an intermediate point in a previous attempt. The plan executor is the main interface between planning and real environment actions. Given a proposed sequence of actions, it simulates the sequence in the world model and executes the same actions in the real game. After each non-terminal step, it compares the predicted settled ASCII frame with the observed settled ASCII frame. If the prediction diverges from the observation, the executor stops immediately and records mismatch artifacts for inspection. It also stops on LEVEL COMPLETED or GAME OVER. Thus, successful plan execution is not merely action replay: it is an online test of the current world model. In the current implementation, the coding agent still has direct access to the game client and can therefore bypass the plan executor. The prompt instructs the agent to use the plan executor as early as possible and to treat any mismatch as a blocking modeling error.
2.4
Refactoring loop
The coding agent is repeatedly prompted to keep the world model compact and general. In particular, it is asked to replace special cases with shared rules, simplify state reconstruction, remove ad hoc rendering overrides, and keep the planner expressed in terms of the world-model engine. This refactoring loop is our practical proxy for an MDL-like bias: the model should not merely fit the observed transitions, but should explain them through a simpler executable structure that remains valid as new levels are encountered.
3
Results on 25 Public Games
We evaluated the agent on all 25 public ARC-AGI-3 games. Each reported row is a single playthrough attempt: the run is started from scratch, is not manually resumed after interruption, and is not restarted from the beginning to obtain a better trajectory. Some games were run more than once using the same fixed agent and harness, either to characterize run-to-run variability or because an earlier run was interrupted. Each repeated playthrough used a fresh agent launch with no access to previous playthrough-specific files, logs, or conversation state. We report these runs separately rather than selecting the best run for each game.
3.1
Evaluation setup
The agent plays each game without cross-game learning. At the start of a game, it receives only the initialized workspace, the general ARC-AGI-3 interaction tools, and the current game observations. It cannot return to previously completed levels. If the current level reaches GAME OVER, the benchmark-provided RESET action may be used to restart that level attempt within the same game run; this is counted as an ordinary environment action. Restarting the entire game run to obtain a better trajectory is not allowed. Agent runtime. All public-game runs reported in this section used Codex CLI version 0.122.0 as the coding-agent runtime and GPT-5.4 as the underlying language model. Codex CLI is an OpenAI coding agent that can read, edit, and execute code in a local working directory [9]. The exact model and runtime version are reported because the system depends not only on the prompt and helper programs, but also on the behavior of the coding agent used to edit, test, and refactor the world model.
4
Table 1: Results on public ARC-AGI-3 games. Each row is one recorded playthrough by a fresh agent instance. Repeated rows for the same game denote independent fresh launches, not replay by the same agent. Interrupted runs are reported as-is and are not resumed manually. Game
Run Index
Levels solved
RHAE
Est. API cost
ar25 bp35 cd82 cn04 cn04 dc22 dc22 ft09 g50t g50t ka59 ka59 lf52 lp85 ls20 m0r0 r11l re86 s5i5 sb26 sc25 sk48 sp80 su15 tn36 tr87 tu93 vc33 wa30
01 01 01 01 02 01 02 01 01 02 01 02 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01
8/8 1/9 6/6 5/6 1/6 4/6 4/6 6/6 3/7 4/7 0/7 1/7 4/10 8/8 6/7 1/6 2/6 6/8 5/8 8/8 0/6 2/8 1/6 2/9 1/7 6/6 9/9 2/7 0/9
100.00% 0.61% 86.51% 62.15% 0.01% 24.60% 34.23% 51.86% 21.43% 34.03% 0.00% 0.01% 14.65% 100.00% 27.11% 1.05% 14.29% 33.23% 8.21% 92.70% 0.00% 0.75% 4.76% 3.58% 0.01% 100.00% 78.33% 8.59% 0.00%
$76.83 $343.40 $104.68 $258.62 $282.34 $313.67 $274.62 $68.38 $37.07 $339.50 $42.98 $243.81 $248.13 $52.93 $374.53 $124.45 $224.99 $419.86 $395.20 $34.08 $97.34 $620.33 $86.81 $378.11 $192.04 $44.49 $102.51 $369.47 $88.18
Status normal termination normal termination normal termination interrupted, 1041 steps interrupted, 1998 steps interrupted, 632 steps interrupted, 2156 steps normal termination interrupted, 318 steps normal termination interrupted, 481 steps normal termination interrupted, 713 steps normal termination interrupted, 2846 steps normal termination normal termination interrupted, 1644 steps interrupted, 2495 steps normal termination interrupted, 708 steps normal termination interrupted, 305 steps normal termination normal termination normal termination normal termination normal termination normal termination
Because the coding agent has broad access to its local workspace, we took additional steps to reduce possible data leakage. The ARC-AGI-3 library itself is isolated behind a server-client interface: the coding agent interacts with the game through our client, which does not expose game source code or hidden mechanics. Runs were also performed in fresh isolated agent environments to avoid access to previous conversations or previous run-specific files.
3.2
Interruption policy
This first implementation does not support robust continuation after technical failures, such as model-capacity errors or process interruptions. Some runs therefore ended early because of such failures. We report these runs as-is and do not manually resume them. Manual continuation would make performance depend on the number, timing, and handling of interruptions, making the evaluation less reproducible.
3.3
Public-game results
Table 1 reports all recorded public-game runs. The Relative Human Action Efficiency (RHAE) is the official efficiency-based score for the run. Estimated API cost is not part of the benchmark score; we include it only to characterize the computational cost of the current implementation. Because several games have more than one recorded run, we compute aggregate RHAE statistics in two stages: first averaging RHAE across runs of the same game, and then averaging these per-
5
game means across the 25 public games. Under this convention, the mean RHAE was 32.58% and the median per-game RHAE was 14.65%. The agent fully solved 7 of 25 games and achieved high human-normalized score, defined here as mean game RHAE greater than 75%, on 6 of 25 games. It obtained mean game RHAE below 5% on 9 of 25 games. Across all 29 recorded runs, including repeated games, the agent solved 106 of 209 attempted levels. These results show a heterogeneous performance profile. The agent solved several games completely, including ar25, lp85, and tr87 with 100% RHAE. At the same time, it failed almost completely on several games. The repeated runs illustrate that performance can vary substantially between playthroughs. For example, cn04 produced RHAE values of 62.15% and 0.01%, while g50t produced 21.43% and 34.03%. This variance is expected for the current implementation: the coding agent makes many open-ended modeling and planning decisions, and small differences in early hypotheses can lead to very different later trajectories.
4
Discussion
The public-game results are encouraging for a first implementation. The agent was assembled with a relatively simple architecture: a coding agent, a fixed external controller, executable world-model templates, verifiers, and a plan executor. It does not yet include a library of reusable planning or reasoning skills. Nevertheless, it fully solved 7 of the 25 public games and reached RHAE above 75% on 6 games. This suggests that verifier-driven executable world models are a promising approach for ARC-AGI-3 agents. At the same time, the results also show that executable world models are not sufficient by themselves. The agent’s performance is highly uneven. Some environments appear to be modeled and exploited effectively, while others lead to near-total failure. From inspection of runs, one common failure mode occurs early in a game: the agent forms an initial hypothesis about the objects or goal and then continues to elaborate that hypothesis instead of actively considering alternatives. This kind of tunnel vision is especially damaging on first levels, where the agent has little evidence and a wrong ontology can contaminate the rest of the model. Better exploration prompts, explicit competing-hypothesis tracking, or verifier tests designed to falsify the current model may reduce this failure mode. A second failure mode appears on later levels. In several runs, the agent seemed to construct a partially correct world model, but failed to build an effective planner over that model. This suggests a separation between model learning and model use. A correct or nearly correct transition model is not enough if the planner cannot search the induced state space efficiently. This is a natural place to add reusable agentic skills: breadth-first search, A* search, symbolic constraint solving, backtracking, subgoal decomposition, or planner-selection routines. An alternative is to separate the system into a world-modeling agent and a planner agent, with the planner agent specialized for search over the current executable model. From the ARC-AGI-3 perspective, the main lesson is that a coding agent can sometimes use a compact executable model as an internal testbed for planning. This is different from directly trialand-erroring in the environment. The agent spends unscored computation editing, testing, and simulating the world model, and spends scored environment actions only on selected plans. The current system is still inefficient and brittle, but the successes indicate that the basic loop—observe, model, verify, refactor, plan, execute—is viable on a nontrivial subset of public games. From the AGI perspective, these results should be treated as a limited but useful case study. ARC-AGI-3 is far simpler than the real world: its environments are discrete, deterministic, and
6
visually abstract. However, it captures an important structural problem for general agents. Direct interaction is costly, and the agent must decide which hypotheses and plans are worth testing. In such settings, world models play a role analogous to verifiers in programming or formal reasoning: they provide an internal space in which candidate actions can be evaluated before acting. The open question is how to scale this idea from small, hand-initialized executable models to richer, hierarchical models of real-world domains. Overall, the present system should be viewed as a baseline rather than a finished architecture. The most immediate improvements are better hypothesis management and stronger planner skills (or a separate planner agent). The private validation set is the decisive test of whether the current ARC-AGI-3-specific tools are genuinely game-general within the benchmark or whether they have indirectly adapted to the public games.
5
Conclusion
We presented an initial ARC-AGI-3 agent that uses a coding agent to build an executable Python world model, verify it against observed transitions, refactor it toward simpler abstractions, and plan through it before spending environment actions. On the 25 public ARC-AGI-3 games, the system fully solved 7 games and achieved a mean per-game human-normalized score of 32.58%, but performance was highly uneven. These results provide preliminary evidence that verifier-driven executable world models are a promising approach for ARC-AGI-3 agents. The main next steps are to enforce model-mediated execution, add explicit competing hypothesis tracking, strengthen reusable planner skills, and evaluate on private validation games to test true game-generalization.
Code availability The project repository is available at https://github.com/astroseger/arc-3-agents-baseline1. The source code will be released publicly after acceptance.
References [1] ARC Prize Foundation. ARC-AGI-3: A new challenge for frontier agentic intelligence, 2026. [2] Nicola Dainese, Matteo Merler, Minttu Alakuijala, and Pekka Marttinen. Generating code world models with large language models guided by monte carlo tree search, 2024. [3] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning, 2020. [4] Gabriel Grand, Lionel Wong, Maddy Bowers, Theo X. Olausson, Muxin Liu, Joshua B. Tenenbaum, and Jacob Andreas. LILO: Learning interpretable libraries by compressing and documenting code, 2023. [5] Peter Grünwald. A tutorial introduction to the minimum description length principle, 2004. [6] David Ha and Jürgen Schmidhuber. World models, 2018. [7] Ziga Kovacic, Justin T. Chiu, Celine Lee, Wenting Zhao, and Kevin Ellis. Refactoring codebases through library design, 2025. 7
[8] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097, 2022. doi: 10.1126/science.abq1158. [9] OpenAI. Codex CLI. https://developers.openai.com/codex/cli, 2026. Accessed 202604-30. [10] Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 625:468–475, 2024. doi: 10.1038/s41586-023-06924-6. [11] Hao Tang, Darren Key, and Kevin Ellis. WorldCoder, a model-based LLM agent: Building world models by writing code and interacting with the environment, 2024. [12] Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476–482, 2024. doi: 10.1038/s41586-023-06747-5.
8