Beyond State Consistency: Behavior Consistency in Text-Based World Models Youling Huang1,4,∗ Guanqiao Chen1 Junchi Yao2 Lu Wang4,† Fangkai Yang4 Chao Du4 ChenZhuo Zhao3,4,∗ Pu Zhao4 Qingwei Lin4 Saravan Rajmohan4 Dongmei Zhang4 1 Dalian University of Technology 2 MBZUAI 3 Peking University 4 Microsoft
Abstract
Instruction：Buy a sweater
...
arXiv:2604.13824v1 [cs.LG] 15 Apr 2026
World models have been emerging as critical components for assessing the consequences of actions generated by interactive agents in online planning and offline evaluation. In textbased environments, world models are typically evaluated and trained with single-step metrics such as Exact Match, aiming to improve the similarity between predicted and real-world states, but such metrics have been shown to be insufficient for capturing actual agent behavior. To address this issue, we introduce a new behavior-aligned training paradigm aimed at improving the functional consistency between the world model and the real environment. This paradigm focuses on optimizing a tractable step-level metric named Behavior Consistency Reward (BehR), which measures how much the likelihood of a logged next action changes between the real state and the world-model-predicted state under a frozen Reference Agent. Experiments on WebShop and TextWorld show that BehR-based training improves long-term alignment in several settings, with the clearest gains in WebShop and less movement in near-ceiling regimes, while preserving or improving single-step prediction quality in three of four settings. World models trained with BehR also achieve lower false positives in offline surrogate evaluation and show modest but encouraging gains in inferencetime lookahead planning. Anonymous code release: https://anonymous.4open.science/ r/behr-wm-787B.
1
Input: State&Action
†
... Real Environment
World Model output 2
output 1
Real State Drop the target
B09GBF B09GFQ Jeans 15.5$ Jacket 29.9$
B09GBF B09GFQ B09JI5 Sweater 26$ Jeans 15.5$ Jacket 29.9$
B09GBF B09GFQ B09JI5 Sweater 26$ Jeans 15.5$ Jacket 29.9$
Output: State
Drop the irrelevant B09FRT Hat 9.9
B09FFH Dress 25.5
B09Y69 Scarf 26$
Bert Score: 0.974 BehR (Ours): 0.100
Agent Behavior
B09FRT Hat 9.9
B09FRT Hat 9.9
B09FFH Dress 25.5
B09Y69 Scarf 26$
Bert Score: 0.859 BehR (Ours): 0.763
Thought: Where is the sweater? Action: click [Jacket]
TARGET MISSING → Action Fails
Thought: I find the target ! Action: click [Sweater]
Ground Maching → Truth Action
Figure 1: Metric inversion in a WebShop interaction. The world model receives the same interaction prefix and produces two candidate next-page states. In Drop Target (left), the predicted page omits the decisioncritical target product, so the agent’s correct action is no longer available; despite this catastrophic functional error, BERTScore and other state similarity metrics remain high because most page tokens are unchanged. In Drop Irrelevant (right), several non-critical products disappear, which lowers state similarity much more strongly but leaves the correct action intact. Our proposed metric, Behavior Consistency Reward (BehR) distinguishes these two cases because it scores whether the predicted state preserves the agent’s next action, not just whether the text looks similar.
world models in text-based settings, including web navigation (Chae et al., 2025) and interactive environments such as WebShop and text adventures (Li et al., 2025). In these settings, the LLM serves as an environment simulator, generating the next state conditioned on a state-action pair expressed in text. Most existing training approaches for LLMbased world models focus on state consistency: high textual similarity between states predicted by WMs and states observed in real environment. This training paradigm relies on supervised nextstate prediction and reinforcement learning methods to optimize token-level likelihood of the gener-
Interactive agents deployed in real-world environments such as web navigation, text-based adventure games, and tool-use should be evaluated for their reliability in planning and executing multi-step actions. As testing in real environments is often slow and expensive, world models (WMs) are widely used as surrogate environments for both online evaluation and offline benchmarking. Recent work has explored using large language models (LLMs) as Microsoft intern.
click [search]
| Enter
Environment
Introduction
∗
Webshop Search...Sweater
Corresponding author.
1
2. BehR as a promising training signal. A decision-critical perturbation test shows that BehR correctly captures behavioral severity that surface-level metrics miss, and an optimizationtarget comparison provides initial evidence that replacing BehR with surface or structured rewards does not recover the same task-level gains in the tested setting (§3.2, §5.4).
ated states. Still, existing studies have shown that even state-of-the-art SFT-trained WMs with high single-step textual similarity can exhibit substantial task-level inconsistency when the predicted trajectories are replayed in the real environment (Li et al., 2025). As illustrated in Figure 1, a world model prediction that reproduces most elements on a WebShop page but omits the targeted product can cause the downstream agent to fail. In contrast, a prediction that captures the critical product while missing several irrelevant products may score lower on textual similarity but would still allow the agent to target the right product. As such, metrics measuring state-level consistency may fail at distinguishing good performance from bad performance at the task stage, a phenomenon we call metric inversion. The appropriate criterion for an LLM-based world model is not “does the text look similar?” but rather “does the agent make the same decision as in the real environment?” To address this issue, we propose a new training paradigm for LLM-based world models: Behavior Consistency Training. We argue that an ideal world model should strive to achieve functional consistency: the predicted state induces the same action distribution as the state from the real environment. Because the full action distribution is typically inaccessible, we formulate Behavior Consistency Reward (BehR) as a tractable step-level proxy rather than an exact surrogate for this ideal objective. Using a frozen Reference Agent, BehR compares the likelihood of the logged next action under the predicted and real states and directly rewards decision preservation. Combined with a reinforcement learning method such as Group Relative Policy Optimization (GRPO; Shao et al. 2024), the resulting world models BehR-WMs are optimized to generate predictions with high Pairwise Consistency Ratio (CRpw ), the fraction of individually successful tasks that remain successful when the action sequence generated in the WMs is replayed in the real environment. Our work makes three main contributions, plus a preliminary downstream finding.
3. Improved long-horizon predictive fidelity in selected settings. In our primary setting, training with BehR substantially improves the Pairwise Consistency Ratio (CRpw ) while singlestep prediction quality remains stable or improves; gains transfer to other settings but are more heterogeneous, especially in near-ceiling regimes (§5.2, §5.3). Additionally, BehR-WMs reduce the calibration gap that inflates weak agent performance during offline evaluation and show promising though still preliminary results as lookahead planning simulators (§6.1, §6.2).
2
Related Work
Text-Based World Models. Traditional world models learn environment dynamics for planning and control in continuous domains and video games (Hafner et al., 2019; Schrittwieser et al., 2020). Recent work uses large language models (LLMs) as text-based simulators for web and game environments (Yao et al., 2022; Sodhi et al., 2023; Zhou et al., 2023). More broadly, pre-LLM work had already explored PLM-based world modeling for text games, so the main novelty in the current line is not merely using language models as simulators, but how they are trained and evaluated. Our closest baseline is Word2World (Li et al., 2025), which formalizes text-based world models as next-state generators under a standard interaction protocol and trains them with Supervised Fine-Tuning (SFT). Its objective, however, remains token-level likelihood rather than downstream behavioral preservation. Behavioral Alignment of World Models. Existing text-based world models mainly optimize token-level state consistency (Li et al., 2025; Chae et al., 2025). Recent RL-based variants add reinforcement learning with text-based rewards, but they still target surface reconstruction and remain vulnerable to metric inversion. We instead align world models with downstream decisions: BehR measures whether the predicted state preserves the
1. A behavior-consistent training paradigm for text-based WMs. We reframe world model learning as a process of improving functional consistency, propose BehR as a tractable steplevel proxy, and formulate CRpw as the tasklevel evaluation criterion (§4, §5.1). 2
logged next action, and with GRPO (Shao et al., 2024) shifts training from surface imitation to behavioral preservation. Evaluation of Simulators. Text simulators are still often judged by surface metrics such as EM or ROUGE, even though these do not guarantee behavioral equivalence. This mirrors the mismatch between reconstruction loss and perceptual quality in image generation (Zhang et al., 2018; Heusel et al., 2017). Building on Word2World’s Consistency Ratio (CR), we introduce Pairwise Consistency Ratio (CRpw ) as our primary task-level criterion for behavior preservation.
3
Agent–Environment Interaction
Agent. The agent is defined as a policy π that selects an action based on the interaction history at each step t:
BehR (Ours) ACS GPT-4o Cont. Token F1 BERTScore EM
Behavioral Behavioral LLM as Judge Text Semantic Surface
0.763 0.600 0.466 0.803 0.859 0%
0.100 0.370 0.824 0.961 0.974 0%
Rank? Dense? ✓ ✓ × × × ×
✓ × × × × ×
Method
We propose a new behavior-consistent training paradigm for text-based world models as substituation to the conventional state consistency paradim. In contrast to the state consistency paradigm that strives to reconstruct the next observation text accurately, we argue that a world model should satisfy functional consistency: the predicted state should induce the same action distribution as the real state. However, directly optimizing this objective is impractical because the full action distribution of an agent is typically inaccessible. We therefore instantiate behavior consistency as a practical training objective, using a tractable step-level proxy, i.e., Behavior Consistency Reward (BehR), based on logged actions rather than the full action distribution. We then optimize the world model using reinforcement learning, with BehR as the training signal. In this section, §4.1 formalizes the ideal objective, §4.2 introduces Behavior Consistency Reward (BehR) as a tractable step-level proxy, and the final subsection describes the GRPO optimization procedure built on BehR.
(1)
World Model. As the surrogate of the real environment E, the world model Ê maps the history of textual observations and actions to the next observation and a task-completion signal: (2)
where ŝt is the predicted next observation and r̂t ∈ {0, 1} is the task-completion signal. In practice, Ê is implemented as a single LLM trained on trajectory data from E. 3.2
DT
4
where si is the textual observation and ai the action at step i. We write the trajectory prefix as ht = (s0 , a1 , s1 , . . . , at ).
Ê : {s0 , (ai , ŝi )t−1 i=1 , at } → (ŝt , r̂t ),
DI
target object and breaks the next action. A behaviorally meaningful metric should then rank DI above DT. Table 1 confirms the metric-inversion pattern shown in Figure 1. Text-based metrics and contentonly LLM judges all rate the decision-critical DT perturbation as less severe because most tokens remain unchanged. BehR is the only metric that ranks the contrast correctly while serving as a usable dense training signal. The full seven-perturbation study and 32B reference-agent replication are deferred to Appendix F and Appendix F.2.
Preliminaries
π : {s0 , (ai , si )t−1 i=1 } → at ,
Type
Table 1: Preliminary empirical study of the Figure 1 contrast: Drop Irrelevant (DI) vs. Drop Target (DT). Correct alignment requires DI > DT.
This section lays out the notation for describing the agent-environment interaction and presents a compact empirical study that quantifies the metric inversion phenomenon illustrated in Figure 1. 3.1
Metric
Preliminary Observation: Decision-Critical Perturbations
Before introducing our method, we provide a simple empirical observation to illustrate the limitation of existing metrics. To quantify the metric inversion phenomenon, we define Drop Irrelevant (DI) as perturbation that removes non-critical content while preserving next action correctly and Drop Target (DT) as perturbation that removes the
4.1
Functional Consistency
We begin with the ideal behavioral objective: functional consistency. 3
(ht , st+1 , a∗t+1 ) consists of the history ht , the real next state st+1 , and the recorded next action a∗t+1 from the offline trajectory. The real next state serves as the behavioral anchor, while the logged next action serves as the probe for whether a predicted state preserves behavior.
Definition 1 (Functional Consistency). A world model Ê is functionally consistent with the real environment E with respect to an agent π if the predicted state ŝt induces the exact same action distribution as the real state st : π(· | ht , ŝt ) = π(· | ht , st ),
∀t.
(3) Reward Definition. Given a WM prediction ŝt+1 , the frozen reference agent scores both the predicted and real states:
Under this definition, two states are only equvivalent when the agent’s behavior is preserved, i.e., agent will make decision with the same distribution given either state. Unfortunately, for a black-box agent, we can not access the full action distribution and thus cannot optimize Eq. 3 directly.
ℓ̄pred = ℓ̄ref (a∗t+1 | ht , ŝt+1 ),
(5)
ℓ̄real = ℓ̄ref (a∗t+1 | ht , st+1 ).
(6)
We define BehR as 4.2
Behavior Consistency Reward (BehR) Rbeh = − ℓ̄pred − ℓ̄real .
We introduce Behavior Consistency Reward (BehR) as a tractable proxy for Eq. 3. Importantly, BehR does not recover the full action distribution, but instead measures consistency with respect to the logged next action. The key idea of BehR is: if a predicted state preserves the agent’s behavior, then a frozen reference agent should assign similar likelihood to the next action under both the predicted and real states.
Rbeh ≤ 0 is maximized at zero when the two states induce identical likelihoods. This proxy focuses on preserving decision-relevant actions observed in data, rather than matching the full decision boundary of the agent. This design makes the objective tractable under black-box agents, focuses the training signal on decision-critical behavior, and provides a stable and dense reward for optimization.
Reference Agent. We use a frozen external LLM πref as a reference agent to estimate the likelihood of an action given a state. Given history ht , next state st+1 , and action at+1 , reference agent outputs the mean per-token log-probability: ℓ̄ref ≜
1 |at+1 |
log πref (at+1 | ht , st+1 ),
(7)
4.3
Behavior Consistency Training for World Models
We optimize the world model using reinforcement learning with BehR as the training signal. Specifically, we use Group Relative Policy Optimization (GRPO) (Shao et al., 2024), which eliminates the critic network and stabilizes training by normalizing rewards within each prompt group. For every prompt ht , the WM generates n candidate next states. The BehR for each state is computed and normalized using the group mean and standard deviation. Figure 2 summarizes the full training pipeline. The list of hyperparameters are given in Appendix B.
(4)
where |at+1 | is the token count of at+1 . To ensure the reward stability against any specific evaluated policy, πref is kept fixed throughout training and is isolated distinct from any downstream agent. Although BehR is defined with respect to a fixed reference agent, it does not train the world model to imitate the reference policy. Instead, the reward depends on the relative likelihood difference of a logged action between the predicted and real states. This removes dependence on the absolute action preferences of the reference agent and focuses the learning signal on how state changes affect decision-making. As a result, the learned behavior is not tied to a specific model and generalizes across different downstream agents.
5
Experiments
5.1
Experimental Setup
Environments. We evaluate world models on WebShop (Yao et al., 2022) (e-commerce) and TextWorld (Côté et al., 2018) (text-adventure). Controlled comparison. We use Word2World (W2W) Qwen2.5-7B world model (Li et al., 2025; Yang et al., 2024), a strong SFT-trained baseline, as the base model, and initialize each BehR model
Offline Training Data. Rewards are computed based on offline trajectories without the need of online interaction. Each training tuple 4
State Consistency: MLE/ RLVR-F1
{𝑠! , (𝑎" , 𝑠̂" )%&$ "#$ , 𝑎% }
World Model
∗ (ℎ! , 𝑠!"# , 𝑎!"# )
∗ ℓ*'() 𝑎%*$ ℎ% , 𝑠̂%*$ )
Δ
Reference Agent
𝑠̂!"#
∗ ℓ*'() 𝑎%*$ ℎ% , 𝑠%*$ )
Behavior Consistency Reward : input from offline trajectory
: optimization
: state consistency training pipeline
−|ℓ*,'(- − ℓ*'(./ | : behavior consistency training pipeline
Figure 2: Behavior Consistency Training for world models. The figure contrasts traditional state consistency training (light purple shaded area) with our proposed functional consistency paradigm (light blue shaded area). Baseline methods optimize directly for textual similarity (e.g., MLE or RLVR-F1) between the predicted state ŝt+1 and the real state st+1 . In contrast, our approach relies on behavioral anchoring. Given a historical trajectory prefix ht and a logged offline action a∗t+1 , a frozen Reference Agent evaluates the action log-likelihoods under both the predicted state (ℓ̄pred ) and the real state (ℓ̄real ). The resulting likelihood gap, ∆ = −|ℓ̄pred − ℓ̄real |, defines the Behavior Consistency Reward (BehR), which is fed back to update the world model, explicitly aligning state generation with agent decision-making.
Evaluation protocol. We evaluate each model from two complementary perspectives: single-step EM on held-out transitions, and task-level evaluation under the Real, WM, and W2R pipelines described below. We use deterministic decoding (T =0) throughout the analysis as stochastic sampling may artificially depress measured consistency. All task-level results in the main text and appendix are computed on 200 held-out initial tasks per domain. Because this evaluation set is modest, differences of only a few tasks should be interpreted cautiously. Appendix M reports 95% Wilson score confidence intervals for all CRpw rows in Table 3 together with an aggregate sign test, but we do not yet report run-to-run variance estimates.
from this checkpoint and post-train it with GRPO on a difficulty-filtered subset of the original training corpus (§A.3). This design ensures that the trained BehR-WM and W2W-WM share the same base model and training corpus. We also report results from a W2W LLaMA3.1-8B (Grattafiori et al., 2024) backbone as a cross-architecture stress test (Appendix G). To verify BehR’s contribution, we run matched GRPO ablations with alternative reward functions (§5.4). Evaluation Agents and Reference Agent. We choose four evaluation agents spanning three families: Qwen3-8B, Qwen3-32B, GPT-4o, and GPT-5 (all T =0), and we pick a frozen Qwen3-8B as the Reference Agent. This supports judge-mediated transfer tests in Table 3, but it does not by itself establish judge-family invariance; we return to that limitation in Appendix L and the Limitations section.
Evaluation metrics. Following Li et al. (2025), we report the following pipelines and metrics: • Real: the agent acts in the real environment E. • WM: the agent acts in the world-model environment Ê. • W2R: the action sequence generated in the WM is replayed in the real environment E. • Consistency Ratio (CR): the aggregate ratio CR = SRW2R /SRReal , where SR denotes task success rate. • Pairwise Consistency Ratio (CRpw ): the fraction of individually Real-successful tasks that remain successful under W2R replay.
Training data. In both domains, we construct GRPO training data as step-level tuples of the form (history, action, next state, next expert action), derived from the original W2W corpus. For WebShop, we keep valid hard examples; for TextWorld, we convert offline trajectories into the same tuple format and apply domain-matched subsampling. This yields 4,321 training tuples for WebShop and 6,000 for TextWorld. Full data-construction details are deferred to Appendix A.3.
We treat CRpw as the primary metric and use aggregate CR as a complementary calibration diagnostic. 5
Base Model
World Model
Qwen3-8B under the Qwen backbone), and BehR achieves the highest CRpw in most configurations. Under the LLaMA backbone the same ordering holds for Qwen3-8B and Qwen3-32B: W2W < F1 < BehR in CRpw for TextWorld. However, with the strongest agent (GPT-5), both F1-WM and BehR-WM reach near-ceiling on TextWorld and produce comparable WebShop CRpw (F1: 0.756– 0.769 vs. BehR: 0.720–0.740), suggesting that GPT-5’s strong planning ability partially compensates for the less targeted F1 training signal. TextWorld is harder to interpret because several baseline rows are already near ceiling, especially for GPT-4o and GPT-5, so BehR often preserves high consistency rather than creating large new gains. Under the LLaMA-base WM, TextWorld CRpw improves for all four agents, but WebShop gains are modest and one row (Qwen3-8B) is a pure tie, indicating substantial variation across settings. Across all 16 W2W-vs.-BehR configurations, BehR improves CRpw in 13 rows, ties in the remaining 3, and does not degrade performance in the displayed table. We view this as encouraging rather than definitive: the largest gains appear when the baseline WM is poorly calibrated, while near-ceiling rows leave little headroom and several modest differences still overlap substantially in Appendix M.
Exact Match (%)
WebShop (N=2,126 test samples) Qwen2.5-7B
W2W-WM BehR-WM
79.05% 79.19%
LLaMA3.1-8B
W2W-WM BehR-WM
77.37% 75.97%
TextWorld (N=1,993 test samples) Qwen2.5-7B
W2W-WM BehR-WM
64.72% 73.11%
LLaMA3.1-8B
W2W-WM BehR-WM
69.54% 72.70%
Table 2: Single-step prediction quality (EM). Exactmatch accuracy on held-out test trajectories. BehR posttraining preserves EM on WebShop and improves it on TextWorld relative to the W2W baseline.
5.2
Single-Step Accuracy
Before disussing the main task-level results, we use single-step EM to verify that BehR-based optimization does not trade away local prediction quality. Our results show that BehR post-training preserves or improves held-out EM in three out of four settings (Table 2). It must be noted that high score on single-step EM does not guarantee good tasklevel performance (Li et al., 2025). The task-level evaluation in Table 3 shows the clearest gains in several WebShop settings, while near-ceiling rows remain more mixed. As we discuss in §5.4, the reward ablation provides supportive but still limited evidence that these gains are tied to BehR rather than RL fine-tuning alone. 5.3
5.4
Why the Optimization Target Matters
To isolate the optimization target, we keep GRPO fixed and vary only the reward: F1 Reward (tokenlevel F1, as in RLVR-World (Wu et al., 2025)), FactR (structured factual accuracy: ASIN, price, page-type F1), and BehR (Behavior Consistency Reward, §4.2). All runs use the same base WM, data, and hyperparameters. Table 3 includes F1WM as an additional GRPO baseline alongside W2W and BehR; Table 4 further compares FactR on WebShop with GPT-4o. As shown in the F1-WM rows of Table 3, GRPO with token-level F1 reward yields improvements over the SFT baseline across both domains and both backbones, but BehR outperforms F1-WM in the majority of configurations—particularly with weaker-to-mid-strength evaluation agents (Qwen38B, Qwen3-32B, GPT-4o). With the strongest agent, GPT-5, F1-WM and BehR-WM produce comparable WebShop CRpw . On WebShop (Table 4), F1 reward also causes a 9-point degradation in EM, while FactR preserves EM but slightly reduces CRpw . We interpret this pattern as evidence
Task-Level Functional Consistency
We now turn to the main results: task-level functional consistency. Table 3 shows the clearest gains in the primary Qwen-base WebShop setting. BehR post-training raises CRpw from 0.345 → 0.483 for Qwen3-8B and from 0.455 → 0.485 for Qwen3-32B. For GPT-4o, aggregate CR improves from 0.92 to 1.05 while CRpw rises from 0.76 to 0.84; for GPT-5, the calibration gap closes from CR = 0.91 to 1.00, but the pairwise gain is small (0.73 → 0.74). Taken together, the strongest improvements are concentrated in WebShop and in weak-to-mid-strength evaluation regimes rather than being uniformly large everywhere. Crucially, the F1-WM rows show that GRPO fine-tuning alone does not explain BehR’s advantage. On TextWorld, F1-WM improves over the SFT baseline (e.g., 0.678 → 0.698 CRpw for 6
TextWorld Agent
World Model
WebShop
Real
WM
W2R
CR
CRpw
Real
WM
W2R
CR
CRpw
Qwen2.5-7B Base World Model Qwen3-8B
W2W-WM F1-WM BehR-WM
87.0% 87.0% 87.0%
100.0% 97.5% 97.0%
64.5% 67.5% 67.5%
0.740 0.776 0.780
0.678 0.698 0.730
14.5% 14.5% 14.5%
16.5% 15.2% 17.0%
12.0% 9.5% 13.5%
0.830 0.655 0.930
0.345 0.310 0.483
Qwen3-32B
W2W-WM F1-WM BehR-WM
97.0% 97.0% 97.0%
100.0% 99.5% 99.5%
49.0% 51.0% 52.0%
0.510 0.526 0.540
0.500 0.521 0.536
16.5% 16.5% 16.5%
19.0% 14.4% 15.5%
14.5% 12.0% 15.0%
0.880 0.727 0.910
0.455 0.424 0.485
GPT-4o
W2W-WM F1-WM BehR-WM
99.5% 99.5% 99.5%
100.0% 93.0% 99.0%
99.0% 92.5% 99.0%
0.995 0.930 0.995
0.990 0.925 0.990
19.0% 19.0% 19.0%
19.0% 22.8% 21.0%
17.5% 18.5% 20.0%
0.920 0.974 1.050
0.760 0.763 0.840
GPT-5
W2W-WM F1-WM BehR-WM
100.0% 100.0% 100.0%
100.0% 100.0% 99.5%
100.0% 100.0% 100.0%
1.000 1.000 1.000
1.000 1.000 1.000
39.0% 39.0% 39.0%
39.0% 43.7% 43.5%
35.5% 41.5% 37.5%
0.910 1.064 0.962
0.730 0.756 0.756
LLaMA3.1-8B Base World Model Qwen3-8B
W2W-WM F1-WM BehR-WM
87.0% 87.0% 87.0%
82.5% 89.5% 90.0%
55.5% 59.0% 58.5%
0.640 0.678 0.670
0.563 0.617 0.621
14.5% 14.5% 14.5%
27.5% 14.9% 12.5%
12.0% 11.5% 10.5%
0.830 0.793 0.720
0.345 0.276 0.345
Qwen3-32B
W2W-WM F1-WM BehR-WM
97.0% 97.0% 97.0%
92.0% 98.0% 99.0%
63.0% 66.5% 69.0%
0.650 0.686 0.710
0.634 0.675 0.706
16.5% 16.5% 16.5%
13.5% 17.5% 17.5%
13.0% 12.0% 14.0%
0.790 0.727 0.850
0.485 0.364 0.515
GPT-4o
W2W-WM F1-WM BehR-WM
99.5% 99.5% 99.5%
99.5% 95.5% 99.0%
94.5% 94.5% 99.0%
0.950 0.950 0.995
0.945 0.950 0.990
19.0% 19.0% 19.0%
19.0% 23.5% 21.5%
17.5% 22.0% 21.0%
0.920 1.158 1.110
0.710 0.816 0.890
GPT-5
W2W-WM F1-WM BehR-WM
100.0% 100.0% 100.0%
99.0% 100.0% 100.0%
93.5% 99.5% 99.0%
0.935 0.995 0.990
0.935 0.995 0.990
39.0% 39.0% 39.0%
36.0% 44.0% 41.5%
34.5% 41.0% 36.5%
0.880 1.051 0.940
0.690 0.769 0.720
Table 3: Multi-agent CR evaluation across two domains. Real: success rate in the real environment; WM: success rate in the WM environment; W2R: WM actions replayed in the real environment; CR = W2R/Real; CRpw = pairwise preservation: the fraction of individually Real-successful tasks that remain successful under W2R replay. CR = 1.0 is ideal; CR > 1.0 indicates a “too-easy” simulator; CR < 1.0 indicates behavioral drift. We adopt CRpw as the primary per-task metric (higher is strictly better) and treat aggregate CR as a complementary calibration measure (closer to 1.0 is better). F1-WM: GRPO-trained with token-level F1 reward (Wu et al., 2025), serving as an RL baseline to isolate the effect of the optimization target from RL fine-tuning itself. The Qwen2.5-7B W2W model (Li et al., 2025) is our primary SFT baseline; the LLaMA3.1-8B W2W model is a cross-architecture stress test.
Target
Type
W2W F1 Reward FactR BehR
— Surface Structured Behavioral
EM
CR
CRpw
on its own.
79.05% 70.06% 78.96% 79.19%
0.92 0.97 0.95 1.05
0.76 0.763 0.737 0.84
6
Downstream Applications
Having established that our BehR-based training improves task-level functional consistency (§5.3), we now show that a better-calibrated world model is a more trustworthy offline evaluator (§6.1) and a safer planning simulator (§6.2).
Table 4: Surface, structured, and behavioral training targets (Qwen2.5-7B WebShop, GPT-4o agent). All GRPO runs use identical hyperparameters; only the optimization target differs.
6.1
that the optimization target matters: surface-level F1 provides a partial training signal, but the behavioral objective captures decision-critical information that surface matching cannot, with the largest gains appearing when the evaluation agent is not strong enough to compensate for world-model drift
Application I: Calibrated Surrogate Evaluation
A key promise of world models is to serve as offline surrogates for agent evaluation. A useful surrogate must preserve agent ranking while remaining calibrated at the task level. 7
Agent WM
TP
TN
FP↓
Agree
Agent
Qwen3 W2W-WM 0.6B BehR-WM
0 0
115 181
85 (42.5%) 19 (9.5%)
57.5% 90.5%
Qwen3-8B
Qwen3 W2W-WM 1.7B BehR-WM
6 5
124 159
69 (34.5%) 34 (17.0%)
65.0% 82.0%
Qwen3 W2W-WM 8B BehR-WM
162 161
0 5
38 (19.0%) 33 (16.5%)
81.0% 83.0%
W2W-WM BehR-WM
200 199
0 0
0 (0.0%) 0 (0.0%)
100.0% 99.5%
GPT-5
Planning WM
SR
∆ vs. Base
ReAct (base) W2W-WM BehR-WM
15.5% 24.5% 27.0%
— +9.0pp +11.5pp
ReAct (base) W2W-WM BehR-WM
13.5% 16.0% 18.0%
— +2.5pp +4.5pp
ReAct (base) W2W-WM BehR-WM
19.0% 26.0% 26.5%
— +7.0pp +7.5pp
Qwen3-32B
GPT-4o
Table 5: Task-level agreement on TextWorld (representative rows; full table in Appendix I). SR: aggregate success rate; TP/TN: both WM and real succeed/fail; FP: WM succeeds but real fails; Agree: overall agreement rate; FN omitted (all ≤5). BehR-WM reduces FP by 3×–5× for weak agents, raising agreement from 57–65% to 82–90%.
Table 6: Lookahead planning on WebShop (K = 5). Lookahead improves over the no-planning baseline for all three agents, and BehR-WM attains the highest point estimate in every pair, although some margins over W2W-WM are small. Additional analysis appears in Appendix J.
Agent ranking preservation. Both WMs preserve leaderboard order well when agents are ranked by the aggregate success rate (SR): Spearman correlation with the real environment is ρ = 0.946/0.958 on WebShop and ρ = 0.846/0.898 on TextWorld for W2W/BehR, respectively. The full eight-agent leaderboard appears in Appendix I.
6.2
Application II: Preliminary Evidence for Lookahead Planning
In addition to offline evaluation, a WM with higher functional consistency should achieve better performance as a planning simulator at inference time. To test this, we employ a lookahead strategy in which the agent proposes K=5 candidate actions per step, the WM simulates the resulting next states, and the agent selects the most promising action. Table 6 shows that lookahead improves over the no-planning baseline for all three tested agents. Most of the gain comes from lookahead itself, while the incremental benefit from BehR is smaller and should be interpreted cautiously under a 200task evaluation, especially for GPT-4o (26.0% vs. 26.5%). We therefore view these results as preliminary evidence that better-calibrated WMs can help as simulators, not as a definitive demonstration of a large BehR-specific planning advantage. We omit TextWorld because strong agents are already near ceiling; additional analysis appears in Appendix J. In summary, functionally aligned world models are more trustworthy evaluators and appear to be more effective planning simulators in the tested WebShop setting, though the planning evidence remains modest.
Calibration. Aggregate SR alone can hide false positives, where the same task is marked as successful by the WM but fails in the real environment, making agents look stronger than they are. We therefore also examine agreement at the task level. In Table 5, TP/TN count simultaneous successes/failures in both WM and real environment, FP counts WM-only successes, and Agree represents the overall agreement rate. We show representative TextWorld rows for Qwen3-0.6B, Qwen3-1.7B, Qwen3-8B, and GPT-5. The calibration problem is concentrated in weaker agents. Under W2W, Qwen3-0.6B and Qwen3-1.7B both suffer from high false positive rates at 42.5%/34.5%. BehR-WM reduces these errors significantly and raises agreement from 57.5% to 90.5% and from 65.0% to 82.0%, respectively. For stronger agents, both WMs are already near the performance ceiling with very small gaps. This is expected because the calibration problem is usually concentrated in the weak-to-mid regime. Additional discussions, including episode-length asymmetries and WebShop breakdowns, are deferred to Appendix I.
7
Conclusion
We argue that the usefulness of a world model lies not in its capability of resembling the environment linguistically, but in its capability of preserving the 8
Ethics Statement
downstream agent’s decisions. We therefore advocate a shift in world model research from text reconstruction to behavior preservation. This perspective yields both a task-level evaluation target (CRpw ) and a trainable step-level objective (BehR).
This work uses only publicly available benchmarks and releases all trained models and datasets. Surrogate evaluators carry a risk of miscalibration a too easy WM can systematically overestimate weak agents which we mitigate by measuring the calibration gap (CR) and recommending W2R replay as the definitive validation step. We encourage users to validate WM-based results against real-environment replay, especially when comparing agents from different model families than the frozen judge agent. World models trained with the proposed BehR objective are intended for research purposes and are not substitutes for evaluation in real environments. Functional consistency observed in simulation does not imply behavioral equivalence in realworld settings. In particular, world-model-based surrogate evaluation may overestimate agent capability, and therefore real-environment replay or execution (e.g., W2R) should be treated as the definitive validation step when assessing downstream agent performance. Moreover, BehR preserves the likelihood of logged next actions under a frozen reference agent and does not guarantee preservation of the downstream agent’s full action distribution. These considerations are important when interpreting planning or evaluation results obtained through world-model-based simulation.
Empirically, optimizing BehR with GRPO improves task-level consistency in most of the 16 reported agent–domain–backbone configurations while maintaining single-step prediction quality. The clearest gains appear in WebShop and in weaker-to-mid-strength evaluation regimes; TextWorld contains several near-ceiling rows where BehR mainly preserves already-high consistency, and the LLaMA-based WebShop results are more modest. Our optimization-target comparison further suggests that the choice of reward matters, but this evidence is still narrow rather than decisive. Downstream, BehR-WMs sharply reduce false positives for weak agents (42.5% → 9.5% on TextWorld), making offline evaluation better calibrated, and achieve the highest point estimate in all tested lookahead planning pairs on WebShop. We therefore view functional consistency as a practical criterion for judging whether text-based WMs are ready to serve as simulators or evaluators, and as a promising but still preliminary ingredient for planning.
Limitations References The method-comparison evidence is limited in scope: the optimization-target ablation covers only one domain (WebShop), one backbone (Qwen2.57B), one evaluation agent (GPT-4o), and two alternative rewards.
Hyungjoo Chae, Namyoung Kim, Kai Tzu iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. 2025. Web agents with world models: Learning and leveraging environment dynamics in web navigation. In The Thirteenth International Conference on Learning Representations.
BehR also preserves the likelihood of a single logged action under a frozen Reference Agent rather than the downstream agent’s full action distribution. It is therefore best viewed as a practical proxy for functional consistency, especially in settings where preserving the logged next action is a good approximation to preserving the relevant decision boundary.
Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. Textworld: A learning environment for textbased games. In Workshop on Computer Games, pages 41–75. Springer, Springer. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
BehR is mediated by a single frozen judge model (Qwen3-8B). Cross-family analysis (Appendix L) shows that gains transfer on the LLaMA-base WM, cross-family agents improve more than same family ones, but a definitive ablation with a non-Qwen judge remains future work.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. 2019. Dream to control: Learn-
9
ing behaviors by latent imagination. arXiv preprint arXiv:1912.01603.
WebShop (Yao et al., 2022) — open-ended domain Domain : E-commerce shopping States : Open-ended, partially observable Actions : search[query], click[element] Max steps : 50 Success : Purchase the correct item Challenge : Diverse product pages, open catalog
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS).
TextWorld (Côté et al., 2018) — rule-governed domain
Yixia Li, Hongru Wang, Jiahao Qiu, Zhenfei Yin, Dongdong Zhang, Cheng Qian, Zeping Li, Pony Ma, Guanhua Chen, Heng Ji, and 1 others. 2025. From word to world: Can large language models be implicit text-based world models? arXiv preprint arXiv:2512.18832.
Domain : Text adventure game States : Structured, bounded Actions : NL commands (go, take, . . . ) Max steps : 50 Success : Complete all subgoals Challenge : Multi-room navigation, state tracking
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, and 1 others. 2020. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609.
Table 7: Environment summary. The two domains span complementary difficulty profiles for world-model evaluation.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR).
Paloma Sodhi, SRK Branavan, Yoav Artzi, and Ryan McDonald. 2023. Step: Stacked llm policies for web actions. arXiv preprint arXiv:2310.03720.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Jialong Wu, Shaofeng Yin, Ningya Feng, and Mingsheng Long. 2025. Rlvr-world: Training world models with reinforcement learning. arXiv preprint arXiv:2505.13934.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. 2024. Agentgym: Evolving large language model-based agents across diverse environments. Preprint, arXiv:2406.04151.
A
Environment and Dataset Details
We evaluate on two representative text-based interactive environments. Task-level evaluation uses 200 held-out initial tasks per domain, drawn from the AGENT E VAL benchmark suite of Xi et al. (2024), which provides standardized evaluation splits across diverse agent environments. This count follows the AgentGym evaluation protocol: it refers to initial tasks rather than single-step samples. Each initial task yields a multi-turn interaction trajectory, and in our setting the agent and world model typically interact for around ten steps on average before termination. Table 7 summarizes their key characteristics.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable realworld web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
A.1
WebShop
WebShop (Yao et al., 2022) is a simulated ecommerce platform where an agent must find and
10
Step 1: Trajectory Collection. Expert agent (GPT-4o) interacts with real environments to collect multi-turn trajectories containing alternating actions and environment responses.
purchase a product matching a natural-language instruction (e.g., “Find me slim fit men’s henleys with short sleeve, color: blue, size: medium, price lower than $50”). The environment provides search results pages listing multiple products (with ASINs, titles, and prices), item detail pages (with product descriptions, options, and reviews), and an action space consisting of search[keywords] and click[element] commands. The agent must navigate through search, browse product pages, select correct options (color, size), and execute click[buy now] to complete the purchase.
Step 2: Step-Level Decomposition. Each trajectory is decomposed into individual transition tuples (ht , at , st+1 , a∗t+1 ), where ht is the dialogue history, at is the current action, st+1 is the real next state, and a∗t+1 is the logged next expert action. Step 3: Difficulty-Aware Filtering. For WebShop, we select hard samples (SFT baseline Token F1 < 0.35) and remove those with invalid expert actions, yielding 4,321 training samples. For TextWorld, √ we allocate per-actiontype budgets proportional to pool × hardness across 632k step-level samples, yielding 6,000 training samples. Step 4: VeRL-Compatible Formatting. Each sample is formatted as a VeRL-compatible Parquet record with fields: prompt (WM message history ending at at ), reward_model.ground_truth (st+1 ), and extra_info.expert_action (a∗t+1 ).
World model challenges. The WM must generate realistic search result pages containing plausible product listings, maintain consistency of product attributes across navigation, and faithfully reproduce the purchase-completion signal. In practice, WebShop presents the hardest WM challenge due to its open-ended state space: each search query can return different product combinations, and the WM must hallucinate coherent product catalogs that are internally consistent. A.2
Figure 3: GRPO training data construction pipeline. The key contribution is the difficulty-aware filtering and step-level restructuring applied on top of Word2World’s supervised corpora.
TextWorld
TextWorld (Côté et al., 2018) is a procedurally generated text-based interactive fiction framework. Each game instance defines a set of interconnected rooms, objects with interactive affordances, and a sequence of subgoals (e.g., “open the chest drawer, take the old key, unlock the wooden door, go east, take the milk from the refrigerator, place the milk on the stove”). The agent receives natural-language observations describing the current room and available actions, and must issue single text commands per turn.
WebShop
TextWorld
Source trajectories Train samples (step-level) Test samples (step-level) Avg. prompt length (tokens) Avg. response length (tokens) Action types
70,790 4,321 2,201 ∼2,500 ∼350 2
58,805 6,000 1,820 ∼1,800 ∼80 ∼8
Table 8: GRPO training data statistics. Both datasets are derived from Word2World SFT corpora; our contribution is the difficulty-aware filtering and step-level restructuring.
Data statistics. Table 8 provides detailed statistics of the resulting GRPO training datasets.
World model challenges. The WM must track object locations across multiple rooms, maintain inventory state, and correctly implement game-logic constraints (e.g., a locked door requires a specific key). TextWorld’s structured dynamics make it more amenable to world modeling than WebShop, but the multi-step dependency chains demand accurate state tracking over long horizons. A.3
Statistic
B
Implementation Details
B.1
BehR-GRPO Training Setup
We train world models with the VeRL framework, invoking GRPO via its PPO trainer entry point. Unless otherwise noted, the training runs discussed in the main text use the exponential-form BehR reward described in Eq. 7. Representative main runs were conducted on 8×A100 (80 GB) GPUs and typically completed in approximately two days. Table 9 lists the core hyperparameters. We use FSDP (Fully Sharded Data Parallelism) for model parallelism with bfloat16 precision, and vLLM as the rollout backend with tensor parallelism size 2. The KL divergence penalty (β=0.001) prevents the WM from drifting too far from the SFT initializa-
GRPO Training Data Construction
Our BehR-GRPO training data is derived from the original Word2World supervised world-model corpora through a step-level restructuring pipeline. Figure 3 illustrates the overall process.
11
tion. We set the rollout temperature to 1.3, which is higher than typical LLM sampling. The reason is that the base WM has already been trained via SFT on several hundred thousand trajectory samples, causing its output entropy to be extremely low under greedy or standard-temperature sampling. At T =1.0, GRPO rollouts show near-zero entropy, meaning all n=5 candidates per prompt are nearly identical and provide no contrastive signal. Raising the temperature to 1.3 restores entropy to a reasonable range (∼0.1–1.0 nats/token), enabling GRPO to generate meaningfully diverse candidate states for reward comparison. Item
Value
Framework Trainer RL algorithm Learning rate Train batch size PPO mini-batch size Micro-batch / GPU Max prompt length Max response length KL loss Gradient clipping Rollout backend Tensor parallel size Rollouts per prompt (n) Rollout temperature Top-p Model dtype FSDP optimizer offload Reward Total epochs Model save freq Hardware
VeRL (verl) main_ppo GRPO 5 × 10−6 32 32 2 (actor), 1 (rollout) 14,336 1,024 on, coefficient 0.001 1.0 vLLM 2 5
notably improving from 64.7% to 73.1% (+8.4pp), suggesting that BehR-based training also regularizes toward more accurate predictions in structured domains. B.3
Table 10 lists every model used in this paper together with its source. The Qwen2.5-7B and LLaMA3.1-8B backbones each have a W2W variant (from Word2World) and a BehR variant (our contribution); an anonymized artifact containing all four fine-tuned WM variants is prepared for release and summarized in Table 11. All Qwen3 (Yang et al., 2025) evaluation agents are used as-is without fine-tuning: Qwen3-8B and Qwen3-32B appear in the main CR comparison (Table 3), while the full 0.6B–32B scale is used in surrogate evaluation (Table 5 and Appendix I.1). Qwen3-8B additionally serves as the frozen BehR Reference Agent during training; Qwen3-32B is also the planner in lookahead experiments (§6.2). GPT-4o and GPT-5 serve as both evaluation agents and the replay engines for the W2R protocol. Data and model availability. To keep the main text focused on the scientific comparison, we summarize release scope here rather than in the method section. All processed datasets used by our GRPO pipeline are derived from the original Word2World supervised world-model corpora; both contain steplevel samples split into a training set for GRPO and a held-out test set for EM evaluation. Because the Word2World checkpoints are public prior-work artifacts (Li et al., 2025), we list their released identifiers explicitly below. For double-blind review, only our BehR releases remain anonymized; the anonymized release package corresponds exactly to the BehR WM variants and processed datasets used in the main comparisons.
1.3 1.0 bfloat16 on BehR, exponential 5 every 20 steps 8×A100 80 GB
Table 9: Core training hyperparameters. Configuration covers both WebShop and TextWorld runs; domainspecific differences are noted in the text.
B.2
Model and Data Availability
Training Dynamics
During BehR-GRPO training, we observe the following dynamics:
C
BehR Reward Function Details
C.1
Reward Computation Pipeline
The BehR reward is computed for each candidate world-model state ŝt+1 as follows:
• WebShop: Mean BehR reward increases steadily from ∼0.45 (step 0) to ∼0.65 (step 280), with the EM accuracy remaining stable (∼79%) throughout. This confirms that BehR optimization does not sacrifice single-step prediction quality.
1. Reference-Agent prompt construction: Build an agent-perspective prompt from the dialogue history ht and candidate state ŝt+1 , ending with the logged expert action a∗t+1 (see §D.3).
• TextWorld: Mean BehR reward increases from ∼0.50 to ∼0.72, with EM accuracy
2. Log-probability computation: Query the frozen Reference Agent (Qwen3-8B) via the 12
Model
Role(s)
Source
World-Model Backbones (open-weight) Qwen2.5-7B LLaMA3.1-8B
Primary WM (SFT + BehR) Cross-architecture WM (SFT + BehR)
Qwen/Qwen2.5-7B meta-llama/Llama-3.1-8B
Evaluation Agents (open-weight, Qwen3 series) Qwen3-0.6B Qwen3-1.7B Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B
Surrogate eval agent Surrogate eval agent Surrogate eval agent Main eval agent + frozen BehR Reference Agent Surrogate eval agent Main eval agent + lookahead planner
Qwen/Qwen3-0.6B Qwen/Qwen3-1.7B Qwen/Qwen3-4B Qwen/Qwen3-8B Qwen/Qwen3-14B Qwen/Qwen3-32B
API Models (proprietary) GPT-4o GPT-5
Main eval agent + W2R Main eval agent + W2R
OpenAI API (gpt-4o-2024-11-20) OpenAI API (gpt-5-2025-08-07)
Table 10: All models used in this paper. Main eval: appears in Table 3; Surrogate eval: task-level agreement analysis (Table 5 and Appendix I.1). All Qwen3 agents span both roles where applicable. An anonymized summary of the fine-tuned WM releases is given separately in Table 11. Domain
Base WM
W2W release
BehR release
WebShop WebShop TextWorld TextWorld
Qwen2.5-7B LLaMA3.1-8B Qwen2.5-7B LLaMA3.1-8B
X1AOX1A/WorldModel-Webshop-Qwen2.5-7B X1AOX1A/WorldModel-Webshop-Llama3.1-8B X1AOX1A/WorldModel-Textworld-Qwen2.5-7B X1AOX1A/WorldModel-Textworld-Llama3.1-8B
anonymized artifact (omitted) anonymized artifact (omitted) anonymized artifact (omitted) anonymized artifact (omitted)
Table 11: Public Word2World releases and anonymized BehR releases for the main comparison models. Prior-work W2W checkpoint identifiers are shown explicitly because they are already public (Li et al., 2025); our BehR release identifiers remain omitted for double-blind review and will be restored in the camera-ready version.
Domain Samples
Dataset release
WebShop 4,321 / 2,201 TextWorld 6,000 / 1,820
anonymized artifact (omitted) anonymized artifact (omitted)
rollouts per prompt, all 5 candidates share the same real state st+1 and thus the same ℓ̄real . We cache ℓ̄real per prompt, reducing API calls from 32 × 5 × 2 = 320 to 32 × (5 + 1) = 192 per batch—an approximately 40% reduction. In practice, the real-state deduplication achieves up to 80% reduction in API calls to the Reference Agent.
Table 12: Anonymized release summary for the GRPO datasets (train / test splits). Exact hosting identifiers are omitted for double-blind review; both datasets are derived from Word2World supervised corpora with difficulty-aware filtering applied.
C.2
We support multiple reward mapping functions from the behavioral difference ∆ = ℓ̄pred − ℓ̄real (Table 13). The exponential form is used in all main experiments; the Cauchy form is recommended for future work due to its stronger gradient signal at large |∆|.
vLLM HTTP API to obtain per-token logprobabilities for a∗t+1 : let ct = (ht , ŝt+1 ) denote the agent context, and define ℓ̄ref (a∗t+1 | ct ) =
1 log πref (a∗t+1 | ct ) |a∗t+1 |
3. Reward mapping: Compute the ReferenceAgent likelihood difference ∆ = ℓ̄pred − ℓ̄real and apply the exponential form: Rbeh = exp(−coef × |∆|),
Reward Modes
D
Prompt Templates
D.1
World Model Prompt Format
The world model operates in a reversed role convention compared to the agent: the user role contains agent actions, while the assistant role contains environment observations. This design allows the
coef = 1.0
Real-state log-probability caching. A key efficiency optimization: under GRPO with n=5 13
Mode
Formula
Range
Exponential
(0, 1]
Linear
e−c|∆| 1 1 + c|∆| max(0, 1 − c|∆|)
Negative-L1
−|∆|
(−∞, 0]
Negative-L2
−∆2
(−∞, 0]
Cauchy
pairs at each turn. The domain-specific system prompts below are loaded from init_contexts/ at evaluation time. The BehR Reference Agent uses the same system prompt (per domain) when constructing its agent-perspective prompt for logprobability computation (§D.3); it prepends the Action:\n prefix so that log-probabilities are computed only over the action tokens, skipping the Thought portion.
(0, 1] [0, 1]
Table 13: BehR reward modes. c denotes behavior_scale_coef (default 1.0). The Cauchy mode provides polynomial-decay gradient (∼ 1/|∆|2 ) that avoids saturation at large errors.
WebShop Agent Prompt You are web shopping. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation and a list of available actions, you have to respond an action based on the state and instruction. You can use search action if search is available. You can click one of the buttons in clickables. An action should be of the following structure: search[keywords] click[value] If the action is not valid, perform nothing. Keywords in search are up to you, but the value in click must be a value in the list of available actions. Remember that your keywords in search should be carefully designed. Your response should use the following format:
WM to generate the next environment state conditioned on the action. WebShop World Model Prompt [System] WebShop [SEP] Instruction: [SEP] Find me <product description> with color: <color>, size: <size>, price lower than <price> [SEP] Search Your admissible actions: [...] [User] search[<query>] [Assistant] Instruction: [SEP] <task> [SEP] Back to Search [SEP] Page 1 (Total results: 50) [SEP] Next > [SEP] B09BNCFGLY [SEP] <product title 1> [SEP] $<price1> [SEP] ... Your admissible actions: [...] [User] click[b09bncfgly] [Assistant] <item detail page...>
Thought: I think ...
TextWorld World Model Prompt
Action: click[something]
[System] <TextWorld ASCII banner> Welcome to TextWorld! <task description> -= Bedroom =<room description> AVAILABLE ACTIONS: <action list>
TextWorld Agent Prompt You are playing a text-based interactive fiction game (TextWorld). You will receive observations describing the current state. When available, a list of admissible actions may be provided. Always output strictly in the following format:
[User] open chest drawer [Assistant] You open the chest drawer, revealing an old key. -= Bedroom =- 1/3 AVAILABLE ACTIONS: take old key, ...
“Thought: <your reasoning>
[User] take old key from chest drawer
Action: <the single action to take>”
[Assistant] You pick up the old key. ...
D.2
Guidelines: - Prefer actions from admissible commands when provided. - If no list is provided, issue a valid single command (e.g., “look”, “inventory”, “open door”, “go north”, “take key”). - Avoid invalid or multiple actions in one
Agent System Prompts
All agents follow the standard ReAct paradigm (Yao et al., 2023) as implemented in AgentGym (Xi et al., 2024), producing Thought: / Action: 14
Lookahead Candidate Proposal
step.
D.3
You are currently in this state: {observation}
Reference-Agent Prompt Construction
All admissible actions: {numbered list}
The Reference Agent reuses the same agent system prompt (§D.2) and the same ReAct format used during agent evaluation. The WM-generated candidate state ŝt+1 is placed as the final user turn, and an Action:\n prefix is appended as the assistant turn so that log-probabilities are computed only over the expert action tokens a∗t+1 , skipping the Thought portion. We disable Qwen3’s thinking mode (enable_thinking=False) to prevent <think> tokens from interfering with logprobability computation. D.4
Your task is described in the instruction above. From the admissible actions, select the K actions that are MOST LIKELY to help you complete the task successfully. Output EXACTLY K actions, one per line, in the format: 1. action_here 2. action_here ... Only output the numbered list, nothing else.
Stage 2: Best-action selection. Lookahead Best-Action Selection
Lookahead Planning Prompts
The local lookahead planner (§6.2) uses a twostage prompt protocol at each step: candidate proposal selects K promising actions from the admissible set, and best-action selection chooses the final action after observing WM-predicted futures. Both prompts are domain-generic; the only domainspecific element is the agent system prompt inherited from the interaction context.
[System] You are a decision-making assistant. You will be given a current state and multiple action options with their predicted outcomes. Select the BEST option. [User] CURRENT STATE: {observation} AVAILABLE OPTIONS (with predicted outcomes from a world model): Option 1: {action} Predicted next state: {preview} ...
Step-by-step procedure. At each decision step t, the planner executes: 1. Candidate proposal. The planner LLM receives the current observation and the full list of admissible actions, and outputs the top-K actions ranked by estimated promise (Stage 1 prompt below). This costs one LLM call. 2. WM rollout. For each of the K candidate actions, the world model generates a predicted (k) next state ŝt+1 , k = 1, . . . , K. This costs K WM calls (batched). 3. Best-action selection. The planner LLM receives all K (action, predicted state) pairs and selects the action whose predicted outcome best advances the task goal (Stage 2 prompt below). This costs one LLM call. 4. Execution. The selected action is sent to the real environment (or WM); the returned observation becomes the context for step t+1. Total inference cost per step: K+2 LLM calls (K WM + 2 planner). With K=5, this is 7× the cost of standard ReAct, which motivates keeping K small.
Select the option that best advances the task goal. Reply with ONLY the option number (e.g., 1 or 3).
For TextWorld, the selector outputs the exact action text instead of a number, since TextWorld actions are free-form commands rather than clickable elements.
E
Case Study: Complete Trajectory Comparison
Figure 4 presents a complete TextWorld trajectory (textworld_169) comparing agent behavior under three conditions: the real environment, BehR-WM, and SFT-WM. The agent is Qwen3-8B; the WM backbone is LLaMA3.1-8B (cross-architecture setting, cf. Appendix G).
F
Controlled Perturbation Experiments
This section provides the full controlled perturbation tables summarized in §3.2.
Stage 1: Candidate proposal.
15
Task: open trunk → take key → unlock door → open door → go east → open fridge → take lettuce → put lettuce on stove (3 subgoals, 8 optimal steps) Steps 1–7: Shared prefix — all three conditions produce identical agent actions and functionally equivalent WM responses. #
Agent Action
WM / Env Response (abbreviated)
1 2 3 4 5 6 7
open antique trunk take old key unlock wooden door open wooden door go east open refrigerator take lettuce
“You open the antique trunk, revealing an old key.” “You take the old key from the antique trunk.” “(with the old key) You unlock wooden door.” “You open wooden door.” −= Kitchen =− (room description, refrigerator, stove, exits) “You open the refrigerator, revealing . . . and a lettuce.” “You take the lettuce from the refrigerator.” Score +1
Score +1
Minor surface variation: BehR-WM fridge contains “a bell pepper and a lettuce” vs. real “a half bag of chips and a lettuce”—no behavioral impact since the agent only needs the lettuce. Step 8+: Divergence — agent is in Kitchen with lettuce, score 2/3. Real Environment ✓ Success (8 steps) Agent → put lettuce on stove → “You put the lettuce on the stove. *** The End *** Score 3/3” BehR-WM (LLaMA-8B) ✓ Success (9 steps) Agent → rest lettuce on stove → WM: “That’s not a verb I recognise.” (correct error—same as real env) Agent → put lettuce on stove → “You put the lettuce on the stove. *** The End *** Score 3/3” SFT-WM (LLaMA-8B) × Fail (timeout at 50 steps) Agent → go south → WM: “−= Living Room =−” (navigates away from stove) Agent → go north → WM: “You can’t go that way.” (broken room connectivity) Steps 10–50: Kitchen ↔ Living Room navigation loop; “look” in Living Room returns Bedroom description.
Figure 4: Complete trajectory comparison on TextWorld (textworld_169, Qwen3-8B agent, LLaMA-8B WM). Steps 1–7 are functionally identical across conditions. At the critical decision point (step 8), the BehR-WM preserves correct game-state semantics—including error handling for invalid verbs—enabling task completion. The SFT-WM exhibits broken room connectivity that traps the agent in a navigation loop.
F.1
8B Reference-Agent Results
Perturbation
Severity
BehR (Ours)
ACS
Token F1
BERTScore
ROUGE-L
EM
Oracle Shuffle Drop Irrelevant Add Irrelevant Random Noise Drop Target Random Cross
None Mild Mild Moderate Moderate Severe Severe
0.996 0.671 0.763 0.845 0.174 0.100 0.091
0.740 0.660 0.600 0.740 0.090 0.370 0.160
1.000 1.000 0.803 0.891 0.576 0.961 0.598
1.000 0.953 0.859 0.892 0.666 0.974 0.763
1.000 0.726 0.803 0.891 0.297 0.961 0.511
100% 0% 0% 0% 0% 0% 0%
pact DI-vs.-DT comparison, while the full sevenperturbation table above provides the complete controlled validation. F.2
32B Reference-Agent Replication
Table 15 replicates the controlled perturbation experiment (Table 14) using a 32B Reference Agent instead of 8B. The same inverted ranking pattern is observed: surface-similarity metrics (BERTScore, ROUGE-L) rate Drop Target higher than Drop Irrelevant, while BehR correctly identifies Drop Target as more damaging. This confirms that the BehR metric’s advantage is robust to the scale of the Reference Agent.
Table 14: Controlled perturbation experiment (Reference Agent: Qwen3-8B, N =100 shared samples). Drop Target removes only the target product—a small text change with catastrophic behavioral impact. BehR correctly assigns 0.100; all text-based metrics (Token F1: 0.961, BERTScore: 0.974, ROUGE-L: 0.961) produce inverted rankings, rating Drop Target as less severe than Drop Irrelevant. EM collapses to 0% for all nonoracle conditions, providing no gradient signal.
Content-only LLM judges fail for the same reason as BERTScore: they compare semantic resemblance rather than action consequences. Actionconditioned judging can partially recover the right ordering, but it either depends on unavailable oracle information or yields subjective discrete ratings. The main text therefore focuses on a com-
16
Perturbation
Severity
BehR (Ours)
ACS
Token F1
BERTScore
ROUGE-L
EM
Oracle Shuffle Drop Irrelevant Add Irrelevant Random Noise Drop Target Random Cross
None Mild Mild Moderate Moderate Severe Severe
0.978 0.876 0.863 0.827 0.599 0.652 0.481
0.584 0.608 0.531 0.553 0.237 0.383 0.078
1.000 1.000 0.780 0.848 0.506 0.975 0.448
1.000 0.963 0.927 0.976 0.807 0.988 0.847
1.000 0.769 0.780 0.848 0.310 0.975 0.409
100% 0% 0% 0% 0% 0% 0%
Metric SR (K=5, SFT-WM) Avg Reward Buy Now count Avg Buy Step Click Option count Max steps reached
Table 15: Controlled perturbation with a 32B Reference Agent (N =100 shared samples). Same inverted ranking pattern as with the 8B Reference Agent (Table 14): BERTScore rates Drop Target (0.988) higher than Drop Irrelevant (0.927).
G
Both succeed Only 8B succeeds Only 32B succeeds Both fail
WM SR W2R Real SR
Word2World-LLaMA3.1-8B SFT Baseline 13.0% 14.5% 19.0% 36.0%
10.0% 12.5% 17.5% 34.5%
15.50% 13.50% 19.00% 39.00%
Qwen3-8B Qwen3-32B TextWorld GPT-4o GPT-5
81.5% 92.5% — —
58.5% 70.5% 94.5% 93.5%
83.50% 0.70 91.00% 0.77 99.50% 0.950 100.0% 0.935
0.65 0.93 0.92 0.88
Table 16: LLaMA3.1-8B SFT baseline WM CR across WebShop and TextWorld. LLaMA-base WMs show generally lower CR than Qwen-base WMs (Table 3), confirming that base WM architecture matters.
H
16.0% 0.394 114 5.0 1502 43.0%
19 29 13 139
The “overconfident buyer” pattern arises because the 32B model assigns higher expected rewards to early purchase actions, triggering buy now before selecting all required product options. This results in higher average reward (0.394 vs. 0.302) but lower strict success rate (16.0% vs. 24.0%). The 32B agent issues 114 buy-now actions compared to 75 for the 8B agent, and buys at an earlier average step (5.0 vs. 6.3), confirming the premature commitment behavior. In head-to-head comparison, the 8B agent uniquely succeeds on 29 tasks where the 32B agent fails, suggesting that the 8B agent’s more cautious exploration strategy is better suited to WebShop’s all-or-nothing evaluation protocol.
CR
Qwen3-8B Qwen3-32B WebShop GPT-4o GPT-5
24.0% 0.302 75 6.3 2093 63.0%
Table 17: 8B vs. 32B Lookahead on WebShop (K=5, SFT-Qwen WM). 32B achieves higher partial reward but lower SR due to premature buy now without selecting all required options—an “overconfident buyer” pattern specific to WebShop’s evaluation protocol.
Table 16 reports the raw Word2World LLaMA3.18B SFT baseline that underlies the LLaMA rows in Table 3. We include it to make the crossarchitecture starting point explicit before the paired W2W-vs.-BehR comparison in the main results. Agent
Qwen3-32B
Head-to-head on the same WebShop evaluation set
Cross-Architecture: LLaMA Results
Dataset
Qwen3-8B
Lookahead Analysis: 32B Overconfident Buyer
I
Trajectory-Level Analysis
We present detailed trajectory-level analyses that support the calibrated-surrogate findings in §6.1.
Table 17 compares 8B and 32B planners under lookahead on WebShop.
17
Agent
Real
W2W
I.1
BehR
WebShop Qwen3-0.6B Qwen3-8B Qwen3-32B GPT-4o GPT-5
6.5 14.5 16.5 19.0 39.0
4.5 16.5 19.0 19.0 39.0
4.0 17.0 15.5 21.0 43.0
0.0 3.5 87.0 97.0 100.0
42.5 37.5 100.0 100.0 100.0
9.5 19.5 97.0 99.5 99.5
Failure modes. WebShop failures look qualitatively different in Real and in WM-based evaluation. In the real environment, weaker agents most often buy the wrong item (72.2% for 0.6B; 38.9% for 1.7B), while stronger agents more often stop before purchasing (74.3% for 4B and 74.6% for 14B). In WM-based evaluation, failures are overwhelmingly timeouts: 98.9–100% for W2W-WM and 99.0–100% for BehR-WM across the same agents. This structural difference explains why WM-internal SR is not directly comparable to real SR and motivates W2R replay as the definitive evaluation protocol.
TextWorld Qwen3-0.6B Qwen3-1.7B Qwen3-8B Qwen3-32B GPT-5
Table 18: Calibrated surrogate evaluation across agents. Success rates (%) in the real environment vs. W2W-WM and BehR-WM. Both WMs broadly preserve leaderboard order, while BehR remains better calibrated for weak TextWorld agents.
Task-level agreement. BehR-WM improves WebShop agreement for every Qwen evaluation agent we tested, from 91.0% to 92.5% (0.6B), 81.2% to 84.4% (1.7B), 92.4% to 92.9% (4B), 77.8% to 81.0% (8B), 90.3% to 91.3% (14B), and 81.4% to 83.9% (32B). Unlike TextWorld, where the dominant gap between W2W-WM and BehRWM is concentrated in false positives for weak agents, WebShop shows a more mixed FP/FN pattern; still, false positives fall or stay flat in every configuration, including 7 to 5 (0.6B), 17 to 12 (8B), and 21 to 15 (32B). The consistency across scales complements the TextWorld result in Table 5 and supports the claim that BehR improves calibration rather than merely shifting one operating point.
Episode Length and Timeout. Table 19 reports mean episode length and max-step timeout rates across conditions. In real WebShop, episodes average 8–17 steps, with zero timeouts. In WMbased evaluation, 82–97% of episodes reach the 50step limit: agents enter exploratory loops because the WM cannot faithfully reproduce WebShop’s purchase-completion signal. TextWorld shows no such effect—strong agents (14B) maintain nearly identical episode lengths across all conditions (∼12 steps), confirming that the TextWorld WM preserves the task’s temporal structure. WebShop Agent Cond.
Episode Length Calibration on TextWorld. On TextWorld—where the timeout pathology does not confound episode structure—BehR-WM produces mean episode lengths consistently closer to the real environment than W2W-WM. Across all four agent–backbone configurations, the mean step gap |T̄WM − T̄Real | is smaller for BehR-WM: Qwen38B with Qwen-WM: 1.6 → 0.8 steps; Qwen3-8B with LLaMA-WM: 5.1 → 2.3 steps; Qwen3-32B with Qwen-WM: 0.6 → 0.4 steps; Qwen3-32B with LLaMA-WM: 4.7 → 1.5 steps. The effect is largest on the LLaMA-base WM, where W2WWM inflates mean episode length by 37–42% relative to Real; BehR-WM reduces this to 14–16%. We do not report per-task step analysis on WebShop, where 82–97% of WM episodes reach the 50-step limit regardless of WM variant (Table 19), rendering step-level comparisons uninformative.
TextWorld
Mean
T/O%
Mean
T/O%
0.6B
Real W2W-WM BehR-WM
8.5 47.8 48.0
0% 94.5% 94.0%
49.2 36.5 46.2
100% 57.5% 90.5%
1.7B
Real W2W-WM BehR-WM
12.6 43.8 42.4
0% 85.6% 82.2%
48.6 37.1 42.9
96.5% 62.5% 80.5%
4B
Real W2W-WM BehR-WM
16.6 48.2 48.9
0% 96.0% 97.0%
21.5 14.5 17.0
25.5% 8.5% 15.5%
14B
Real W2W-WM BehR-WM
17.3 46.7 46.5
0% 91.5% 91.3%
11.9 11.0 11.8
1.5% 0% 2.5%
WebShop Calibration Details
Table 19: Episode length and timeout rates. Mean steps per episode and percentage hitting 50-step limit. WebShop WM episodes show >80% timeout rates vs. 0% in Real. TextWorld strong agents (14B) maintain consistent episode structure.
18
J
Lookahead Planning Results
Agent
Table 6 in the main text reports the completed lookahead planning results on WebShop (K=5). This appendix section provides supporting behavioral analysis referenced from §6.2. The dominant effect is the value of world-modelbased planning over no planning; the BehR versus W2W comparison should therefore be interpreted as a secondary effect on top of the planning benefit itself.
K
From Functional Consistency to Practical Proxy
BehR / 95% CI
0.345 0.455 0.763 0.731 0.678 0.500 0.990 1.000
[0.199, 0.527] [0.298, 0.620] [0.608, 0.870] [0.623, 0.817] [0.606, 0.743] [0.430, 0.570] [0.964, 0.997] [0.981, 1.000]
0.483 / [0.314, 0.656] 0.485 / [0.325, 0.648] 0.842 / [0.696, 0.926] 0.744 / [0.637, 0.827] 0.730 / [0.659, 0.790] 0.536 / [0.466, 0.605] 0.990 / [0.964, 0.997] 1.000 / [0.981, 1.000]
LLaMA3.1-8B base WM Qwen3-8B 29 WS Qwen3-32B 33 WS GPT-4o 38 WS GPT-5 78 WS Qwen3-8B 174 TW Qwen3-32B 194 TW GPT-4o 199 TW GPT-5 200 TW
0.345 0.485 0.711 0.692 0.563 0.634 0.945 0.935
[0.199, 0.527] [0.325, 0.648] [0.553, 0.832] [0.582, 0.784] [0.489, 0.635] [0.565, 0.698] [0.906, 0.968] [0.893, 0.961]
0.345 / [0.199, 0.527] 0.515 / [0.352, 0.675] 0.895 / [0.759, 0.960] 0.718 / [0.609, 0.806] 0.621 / [0.547, 0.690] 0.706 / [0.639, 0.766] 0.990 / [0.964, 0.997] 0.990 / [0.964, 0.997]
Table 20: CRpw with 95% Wilson score confidence intervals. Results are grouped by base WM. WS = WebShop; TW = TextWorld. Bold BehR values exceed the upper bound of the corresponding W2W interval. Across the 13 non-tied rows in the full comparison, BehR is always directionally better; a binomial sign test gives p=0.000244.
Functional consistency (Eq. 3) requires preserving the agent’s full action distribution π(· | ŝt ) ≈ π(· | st ), whereas BehR (Eq. 7) monitors only the likelihood assigned by a frozen Reference Agent πref to the next action a∗t+1 recorded in the offline trajectory, rather than the downstream agent π itself. The gap is twofold: what is preserved (one action vs. the full distribution) and who measures it (Reference Agent vs. downstream agent).
gap shrinks to +0.064 vs. +0.045 after excluding TextWorld rows where GPT agents already hit the CRpw ≥ 0.990 ceiling. The LLaMA-base WM provides the cleanest test: Reference Agent (Qwen38B), WM backbone (LLaMA3.1-8B), and evaluated agents (GPT-4o/5) are three distinct families. Cross-family agents average ∆CRpw = +0.078, exceeding same-family +0.040—directly contradicting the confounding hypothesis.
Why this relaxation works. In our environments, the critical behavioral question at each step is often close to binary: does the agent still select the correct action? Preserving the logged expert action’s likelihood can therefore approximate the decision boundary between the correct action and the most relevant alternatives. The controlled perturbation study (Table 14) supports this interpretation: BehR assigns 0.100 to Drop Target (optimal action destroyed) vs. 0.763 to Drop Irrelevant (optimal action preserved), suggesting that the single-action proxy captures a task-relevant part of the decision boundary.
M
Statistical Analysis: Confidence Intervals
We report 95% Wilson score confidence intervals for all CRpw values in Table 3. Since CRpw is a binomial proportion, its effective sample size is the number of Real-successful tasks nreal rather than the full 200-task evaluation set.
Limitations. The proxy is weakest when multiple plausible actions exist and behavior depends on their relative ranking rather than just the top choice. It is also mediated by a specific frozen Reference Agent (Qwen3-8B); the cross-agent results in Table 3 provide partial reassurance, but transfer is heterogeneous.
L
nreal Domain W2W W2W 95% CI
Qwen2.5-7B base WM Qwen3-8B 29 WS Qwen3-32B 33 WS GPT-4o 38 WS GPT-5 78 WS Qwen3-8B 174 TW Qwen3-32B 194 TW GPT-4o 199 TW GPT-5 200 TW
Interpretation. The intervals clarify two complementary facts. First, several rows with small nreal — especially WebShop rows with nreal in the 29–38 range—have wide uncertainty bands, so modest row-level gains should not be overstated. This is consistent with the cautious interpretation in the main text. Second, the overall direction is still favorable to BehR: all 13 non-tied comparisons move upward, and several rows show stronger separation, including GPT-4o on LLaMA-WebShop, Qwen332B on LLaMA-TextWorld, and GPT-4o/GPT-5 on LLaMA-TextWorld. These statistics should still be read as partial un-
Reference-Agent Family Overlap
A natural concern is that BehR gains are inflated when the evaluated agent belongs to the same family as the Qwen3-8B Reference Agent. For the Qwen-base WM, same-family agents gain +0.064 on average versus +0.023 for cross-family, but the
19
certainty quantification rather than a complete stability analysis. Wilson intervals and the aggregate sign test help characterize sampling uncertainty on the fixed 200-task evaluation set, but they do not replace repeated-run variance estimates under different seeds, decoding perturbations, or API nondeterminism.
20