ReWorld: An Interactive World Model with Long-Horizon Memory

ReWorld: An Interactive World Model with Long-Horizon Memory Zhifei Chen1,2,∗ , Luozhou Wang1,∗ , Guibao Shen1,2 , Dongyu Yan1 , Shuai Yang1 , Tianshuo Xu1 , Yihua Du1 , Wei Wang2 , Tianyi Gui2 , Lianghua Huang2 , Yingcong Chen1,†

arXiv:2608.23565v1 [cs.AI] 24 Aug 2026

1

∗

HKUST(GZ)

2

ATH, Alibaba

Equal contribution, † Corresponding author

Figure 1 ReWorld follows camera intents (top), regenerates a revisited view after an out-and-back excursion (middle), and streams long rollouts (bottom).

Abstract An interactive world model must do three things at once: follow the user’s actions, remember the places it has already shown, and keep streaming in real time. The tension is structural—control wants a short horizon, memory wants an unbounded one. ReWorld resolves it by separating the two during training and bounding them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources—Unreal-rendered fly-throughs, game roaming, and real-world footage—on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704 × 1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video

1

quality, against six recent interactive world models it attains the best control fidelity (11.95◦ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts (64 s, 384 latents), its fixed 12-chunk cache still regenerates the starting view—at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory. Website: https://zhifeichen097.github.io/ReWorld/

1

Introduction

A world model simulates an environment an agent can act in [10]. Recent interactive systems turn a stream of user actions into video of a coherent, explorable environment [2, 27, 29, 30, 35]. Expectations have converged: the model should react, reflecting a key press in the very next frames; remember, so that a revisited place looks the way it did before; and stream, generating at interactive rates over unbounded rollouts. Current systems pursue these goals along two axes. For control, one route folds camera pose into attention, so attention logits depend on relative pose [7, 39]; the other directly injects an explicit action signal [30, 35]. For memory, camera pose is the retrieval key of choice, in three forms: selecting which past frames re-enter the context [44], indexing an external memory bank [36, 38], or entering the attention index itself [39]. Learning a world model comes down to learning two abilities: control, so that the next frames follow the current action, and memory, so that a revisited place looks the way it did before. When the two are trained together, control learns well but memory does not: adding direct action injection improves every control metric while revisit fidelity drops (Sec. 4.5, Table 7). The two abilities also ask for different attention windows. Control should not depend on the window size: at inference the model must respond to the current action correctly whatever window it is given, and what the response requires—the current scene and the current command—sits inside a short window. Memory is the opposite: retrieval can only be learned under a long window, because a model that cannot see the far past has nothing to retrieve from. The learning of the two abilities can therefore be split by window: control learned under short windows, memory under long ones. This paper presents ReWorld, an action-controllable streaming world model designed in two steps: split the training of control and memory by window, then consolidate memory under a fixed KV budget at inference. We keep both conditioning channels: pose-indexed attention (MRoPE) finds cached content by pose, and direct action injection states the command. What we separate is the window each attention head trains under. Mixed attention windows give the model both window lengths in every iteration: most heads are local and attend only to a short recent window, while a small set of global heads attends over the entire causal past. Random head routing then switches which heads are global at every step, cycling through a fixed pool of random partitions. The routing is what makes the split workable. The split itself cannot be kept at inference: the full history is gone, and all that remains is one bounded cache that every head reads, so there is no way to give some heads a long window and others a short one. A fixed partition would train heads to specialize in windows that deployment cannot provide. Routing removes this dependence: which heads are global changes at every step, every head trains under both windows, and neither ability binds to particular heads—at inference any head can read the shared bounded cache. The recipe adds no parameters, losses, or modules, and it makes a testable prediction: swapping cache-compression policies at inference should leave control error unchanged (Sec. 4.5). With control insulated, the second step strengthens memory at inference: the rollout is unbounded but the cache is not. ReWorld therefore consolidates, keeping few chunks but keeping them intact: a chunk aging out of the recent window is stored in a bounded landmark bank only if the camera has travelled far enough since the last stored landmark—one full-resolution snapshot per stretch of camera travel—and once the bank is full, each admission evicts the member most spatially redundant with the rest. Retrieval then fills the fixed cache with a sink chunk [37], a recent window, and the landmarks closest to the current camera pose. Inference thus reads a sparse, non-contiguous history, while standard training sees only a complete, contiguous prefix—a The project was done during an internship at Alibaba.

2

train–test mismatch [14]. Chunk-drop training closes the gap: a random subset of past chunks is masked at every step, teaching the model to reconstruct scene state from incomplete memory and making the spliced caches of deployment in-distribution (Secs. 2.3 and 2.3). Two components complete the system. A metric-scale-aligned pipeline places synthetic, real, and game footage on a single physical action scale, so the same key press moves the camera the same physical distance in every source, with palindrome trajectories supplying the revisit evidence memory training needs (Sec. 3). Distribution-matching distillation with self-forcing rollouts [14, 42, 43] compresses sampling into a few denoising steps inside a LoRA adapter [11], so one backbone serves both a high-fidelity multi-step operating point and a real-time interactive one (Sec. 2.4). In summary, our contributions are: • Window-split training of control and memory. Mixed per-head attention windows train control under short windows and memory under long ones, and random head routing keeps control in every head complete within the short window, so cache compression at deployment does not disturb action following (Secs. 2.2 and 4.5). • Chunk-drop training with consolidation-based inference. A training augmentation that makes sparse, non-contiguous KV caches in-distribution, paired with an inference mechanism that consolidates aged chunks into a landmark bank kept bounded by redundancy-based eviction and retrieves them by pose proximity, so spatial memory persists over unbounded rollouts at a fixed KV budget (Secs. 2.3 and 2.3). • A metric-aligned multi-source data pipeline. Synthetic, real, and game footage aligned to one physical action scale, with palindrome augmentation providing revisit supervision (Sec. 3). • Real-time deployment and evaluation. LoRA-confined few-step distillation that gives a single backbone both a high-fidelity and a real-time operating point, together with an evaluation protocol covering action following, long-horizon recall, and general video quality (Secs. 2.4 and 4).

2

Method

2.1

Overview

ReWorld is an action-controllable streaming world model: given a text prompt, an optional reference image, and a stream of 6-DoF camera actions, a causal flow-matching diffusion transformer [8, 22] generates video one latent chunk at a time, each chunk driven by a per-chunk camera action. The generated stream must both follow the commanded trajectory and remain spatially consistent with everything already generated—for instance, when the camera revisits a location seen long ago. The two requirements are learned under different conditions: action following needs only what a short window holds, while spatial memory can be learned only when distant content is visible in the attention window. Deployment adds a hard constraint: the KV cache holds a limited number of chunks, so the model cannot simply attend to its full history. ReWorld meets both requirements with a small set of co-designed components. The camera-control section builds up from pose-indexed attention (MRoPE) to direct action injection, then trains control under short windows and memory under long ones with mixed per-head attention windows and random head routing (Sec. 2.2); memory consolidation then strengthens what the decoupling protects—at inference a pose-retrieved landmark cache fills the fixed budget with the most relevant old chunks (Sec. 2.3), and chunk-drop training makes attention robust to the sparse caches this policy produces (Sec. 2.3); four-step LoRA distillation makes the pipeline real time (Sec. 2.4)—the overall development route, AR-training the base model then plugging in a step-distilled LoRA, follows LongLive-2.0 [5]; and a metric-aligned, eight-source data mixture lets a single action space transfer across synthetic and real footage (Sec. 3). Figure 2 summarizes the design.

2.2

Camera-Controllable Streaming Generation

Backbone and chunked causal generation. ReWorld builds on Wan2.2-TI2V-5B [33], a video diffusion transformer [28] operating in the latent space of a causal VAE. We turn this bidirectional backbone into a streaming generator by imposing chunkwise causality: video is generated as a sequence of latent chunks (L=12 chunks

3

Figure 2 Overview of ReWorld. Left: a metric-aligned data pipeline places UE-rendered, real-world, and game footage on a single physical action scale, and palindrome routes—the camera retracing its own path—supply revisit supervision (Sec. 3). Middle: teacher-forcing training turns the bidirectional backbone into a streaming world model, with a DMD LoRA trained alongside for few-step real-time inference (Secs. 2.2 and 2.4); control (action injection, local short-window heads) and memory (MRoPE pose-indexed attention, global long-window heads) are decoupled, and the partition into global and local heads is switched every step through a fixed pool of random partitions so that neither capability binds to particular heads (Sec. 2.2); chunk-drop training masks random parts of the history, matching the sparse cache the model will read at inference (Sec. 2.3). Right: at inference, a fixed cache budget holds a sink chunk, pose-retrieved landmarks, and a recent window next to the chunk being generated; chunks aging out of the recent window are consolidated into a bounded landmark bank and retrieved by pose proximity upon revisits, so spatial memory persists over unbounded rollouts at constant cost (Sec. 2.3).

of four latent frames per window), with full attention within a chunk and causal attention across chunks, so at inference the model emits one chunk per denoising pass and appends its keys and values to the cache C. Pose-indexed attention as implicit memory. Standard RoPE indexes attention by time and space, so a revisited location is, positionally, just a distant timestamp—retrieval must be inferred from content alone. Our first component therefore builds spatial memory into the attention index itself. We adopt Memory-RoPE (MRoPE), following the camera-phase RoPE design of PM-RoPE [39]: each latent frame f carries a relative camera-to-world pose Pf ∈ SE(3) (anchored to the first frame) with descriptor cf = [vec(Rf ); tf ] ∈ R12 , which a zero-initialized MLP maps to a phase offset δf applied on top of RoPE to queries and keys within the same attention pass, q̃ = RoPE(q) e i δf (q) ,

k̃ = RoPE(k) e i δf (k) ,

⟨q̃, k̃⟩ ∝ e i (δf (q) −δf (k) ) ,

(1)

so attention depends on pose difference rather than temporal distance: similar viewpoints are pulled together no matter how far apart in time. The cache thus acts as an implicit spatial memory—each chunk is stored with the pose it was seen from, and a revisit retrieves it by pose proximity. Zero-initialized SE(3) residuals on the values and outputs (ṽ = v + Wv (Pf−1 ◦ v), out = Wo y + Wp (Pf ◦ y)) complete the conditioning. Unlike the two-pass designs in Table 1, which add a separate camera-aware attention pass so that pose can steer generation, MRoPE uses pose as a retrieval index in the existing pass: the cost is one small MLP plus two linears, and the attention kernel and per-head windows stay untouched. MRoPE alone gives strong revisit memory but imprecise control (Sec. 4.5, Table 7). Action injection. For control authority, we therefore inject the camera command directly. The commanded pose of each latent frame is expanded into a Plücker ray map: every spatial position receives the 6-D Plücker 4

Table 1 Pose conditioning in attention: design comparison. Two-pass designs such as HY-World 1.5 add a camera-aware attention pass next to the temporal-RoPE pass and fuse the outputs; E-PRoPE trims the second pass to spatially reduced tokens; MRoPE folds pose into the same pass, leaving the attention kernel, mask layout, and per-head windows unchanged. Design

Attention index

Where pose enters

Temporal RoPE HY-World 1.5 (two-pass) [30] E-PRoPE [7] MRoPE (ours)

time & space time + projective pose time + projective pose time, space & pose

— separate camera-aware pass, fused separate reduced-token camera pass, added back Q/K phase + zero-init V/O residuals

Attn. passes 1 2 2 1

Extra cost — ≈2× attention FLOPs one reduced pass (∼4.5× fewer tokens) 1 small MLP + 2 linears

Figure 3 Mixed per-head attention windows, routed from a pattern pool. Left: local heads attend to the last w=12 frames (reactive control); global heads attend to the full causal history (long-range memory). Right: each optimizer step switches to the next of |P|=12 random head partitions, so neither ability binds to particular heads.

coordinates [d, o × d] of its viewing ray under that pose (direction d, camera center o, expressed in the same first-frame coordinate system as the MRoPE poses). An MLP projects the map to the model width, and the result is added token-wise to the patch embeddings at the transformer input (Figure 2). The two channels now have clearly separated jobs: the ray map tells each token of the current chunk where its camera should look, while MRoPE poses tell attention where every cached token was seen. They still encode the same trajectory, however, and when every head receives both, the action signal crowds out pose-keyed retrieval: control improves, but long-horizon memory degrades (Sec. 4.5). Mixed per-head attention windows. The two abilities need different windows to be learned. Action following does not depend on the window size: the next chunk must answer the current command however much history is visible, and the current scene plus the command fit in a short window. Retrieval, by contrast, can only be learned when distant content is visible. We therefore train under both windows at once (Fig. 3, left): we split the H=24 attention heads of every block into 18 local heads, which attend only to the last w=12 latent frames (three chunks), and a set G of |G|=6 global heads, which attend to the entire causal history: ( full causal history, h ∈ G (6 global heads), context(h) = (2) last w=12 frames, otherwise (18 local heads). Every iteration thus trains both abilities at once: local heads learn control under the short window, and global heads are the only heads that can see distant content, so they are where retrieval is learned (a fixed 1:3 global-to-local budget). Random head routing. A fixed partition would not survive deployment. At inference the per-head split is hard to realize: the full history is gone, and all heads read the same bounded cache (Sec. 2.3), so no head may be trained to depend on a particular window. We therefore draw a pool P of |P|=12 random six-head global sets, and every optimizer step switches to the next set in the pool (Fig. 3, right); the 1:3 global-to-local ratio is unchanged. Every head thus spends steps in both roles: neither ability binds to particular heads, and since a head is local on most steps, the control it learns must be complete within the short window; retrieval keeps living in whichever heads currently see far. Changing the inference cache policy should then leave control error unchanged—a prediction Sec. 4.5 tests. 5

Figure 4 Memory consolidation under a fixed KV budget. Left: at inference the cache holds one sink, six poseretrieved landmarks, and five recent chunks (B=12); a chunk aging out of the recent window is consolidated into the bounded landmark bank—kept sparse and diverse—and re-enters the cache when the camera returns nearby (a full bank evicts its most redundant member). Right: chunk-drop training randomly masks past chunks at every step, so such sparse caches are in-distribution.

This is the control-side design of ReWorld: pose-indexed attention supplies the memory index, action injection supplies the command, and mixed-window training with random head routing lets each ability learn under the window it needs. Its memory-side counterpart, chunk-drop training and landmark retrieval, follows in the next sections.

2.3

Memory Consolidation under a Bounded KV Budget

ReWorld generates video autoregressively over latent chunks zk , so its KV cache C grows linearly with the rollout horizon: at 720p, an unbounded cache exhausts device memory well before the horizons at which spatial memory is actually tested. A deployable interactive world model must therefore read from a cache of constant size while still recalling scene content observed arbitrarily far in the past [30, 35, 36, 44]. ReWorld addresses this by separating the write side of memory from the read side. On the write side, consolidation keeps few chunks and keeps them intact and diverse: aged chunks are admitted selectively into a bounded landmark bank at full resolution—one snapshot per stretch of camera travel—and the bank is held at its capacity by evicting the member most spatially redundant with the rest—rather than keeping the whole history at degraded fidelity. On the read side, retrieval spends a fixed budget of B=12 chunks on a pose-indexed working set. A training-time augmentation, chunk drop, makes the backbone robust to the sparse caches this policy produces. The pose-indexed positional encoding of Sec. 2.2 is what makes retrieval effective: because attention keys carry camera-pose structure, a retrieved chunk remains addressable by pose regardless of how distant it is in time. Landmark consolidation and retrieval. At deployment (Fig. 4, left), ReWorld maintains a bounded cache whose active set at chunk k is Ck = { 1 } ∪ { k−5, . . . , k−1 } ∪ |{z} | {z } sink

recent window

Tk |{z}

,

|Tk | = 6,

|Ck | = B = 12,

(3)

retrieved landmarks

where the sink chunk anchors the global scene layout in the spirit of attention sinks [37], and every retained chunk is stored at full resolution—no pooling, merging, or token-level pruning is applied. The retrieved landmarks are drawn from a bounded bank M (|M| ≤ K, K=30) held in a tiered store: a pinned host-memory master copy of all K members, with only the retrieved working set staged on device. Each chunk carries a single camera pose Pk ∈ SE(3) (first-frame rotation with block-mean translation), and the bank is managed by pose alone. Admission keeps the bank diverse with an odometer rule: a chunk aging out of the recent window is stored only if the camera has travelled at least a stride δ (measured in median step lengths) since the last stored landmark; the chunks in between are the densest, most redundant samples along a path, and are dropped. Crucially, the rule measures distance travelled—never whether a place is already banked—so a revisit pass is stored just like the first visit, whereas gating against the whole bank would discard exactly 6

the chunk a later revisit needs (only ∼64% far-revisit coverage in simulation). Once the bank is full, each admission evicts the member most redundant in pose space (the one whose nearest neighbour is closest), and the first two members are never evicted, since the earliest region is the one revisited across the longest gaps. After every generated chunk, the six landmarks nearest to the current pose re-enter the cache as Tk , where they are read through pose-indexed attention. Device memory thus stays at O(B) chunks regardless of rollout length, with the full bank in host memory; landmark transfers are prefetched on a separate stream and overlap with denoising. Chunk-drop training. The cache above hands the model a sparse, non-contiguous subset of its history at inference, whereas standard teacher forcing conditions every denoising query on a contiguous, complete causal prefix—a train–test gap that manifests as blur and drift as soon as chunks are evicted. We close it by randomly dropping cached KV chunks during training (Fig. 4, right): for a training clip of L=12 chunks (Sec. 2.1), the sink chunk is always kept and five further survivors are drawn at random at every optimization step, so each step exposes the model to a random half of its history (6 of 12 chunks). The attention mask is restricted accordingly: the query for chunk zk attends only to the clean keys of surviving earlier chunks and to its own chunk, and queries are never dropped, meaning that every chunk still receives its denoising loss. The keep-set is resampled independently at every step and broadcast so that all data- and sequence-parallel ranks apply an identical mask. We compare this deployment policy against four alternatives under the identical budget B=12: an unbounded full-KV cache (undeployable upper bound), a sliding-window cache with sink (forgetting baseline), mean-pooled compression of aged chunks, and a retrieval-free variant that replaces pose retrieval with a static bank of six pose-deduplicated landmarks attended in full; results are reported in Sec. 4.5 (Tab. 6).

2.4

Real-Time Distillation with a Lightweight Adapter

To run in real time, ReWorld generates with only four denoising steps, obtained by step distillation confined to a plug-in LoRA in the AR-train-then-distill route of the LongLive series [5, 40] (Fig. 5): the multi-step model is the teacher, and the student is the same backbone, frozen, plus a rank-128 LoRA on the attention and feed-forward layers— the LoRA is the only thing trained (the DMD critic is also a LoRA over the frozen weights). Training follows distributionmatching distillation with self-forcing rollouts [14, 42, 43]: the student streams chunks from its own KV cache, just as at deployment, so distillation needs only text prompts—no video. The prompts are generic long captions from another domain (248K, extended from VidProM), and the action and pose injection path carries no LoRA, so the model’s control is untouched by distillation. Figure 5 Real-time distillation. A captionOne backbone thus has two modes: adapter off is the origdomain DMD LoRA plugs into the frozen backinal multi-step model; adapter on streams in real time at bone; the control path carries no LoRA. four steps. The 2.6 GB adapter also transfers across base checkpoints, re-distilling in 2k–4k steps when needed. Two practical rules: distill at the deployment resolution, with prompts that resemble deployment prompts; the production student uses block size 16 with a single rollout and folds classifier-free guidance into the adapter.

3

Data Pipeline

ReWorld is trained on an eight-source joint corpus of 220,724 pose-annotated clips (Fig. 6). Two of the sources are rendered by our own Unreal Engine (UE) pipeline and play a double role: they anchor the metric scale to which all other sources are aligned, and their trajectories are generated specifically for camera control.

7

Figure 6 The four-stage data pipeline. Eight UE, real-world, and game sources; an automated UE service renders 337 environments with NavMesh-based trajectory planning (the control pattern anchors each path at two points of interest and fills the segment between them with direction-balanced random motion); a curation funnel filters each pool and evens out on-screen speed by dropping too-fast and too-slow clips; unification then rescales the pose numbers to one shared metric scale — the same action moves the camera the same distance — before clip-level sampling. Table 2 The eight-source joint training corpus. #Clips are counts after per-source filtering (Sec. 3.4); Duration is the per-clip length; Pose gives the camera-trajectory provenance; Caption gives the text-annotation provenance.

Source

#Clips

Duration

Pose

Caption

UE control (ours) UE expansion (ours) DL3DV RealEstate10K Sekai real-walking-hq Game roaming (79 games) OmniWorld-Game Sekai game-walking

95,895 42,536 29,864 12,065 14,730 18,387 5,629 1,618

15 s 30–60 s 15 s ≥ 8s 60 s 15 s ∼12 s 60 s

engine GT (c2w, cm) engine GT (c2w, cm) VIPE VIPE (re-estimated) MegaSaM (c2w) VIPE (metric) engine (w2c→c2w) engine GT (UE5, c2w)

pipeline-generated scene summary + camera Qwen-VL pre-computed official CSV Qwen-VL per-clip JSON official CSV

Total

220,724

3.1

Sources

Table 2 summarizes the eight sources; each clip is stored as an RGB video, a per-frame camera trajectory, and a text caption. The sources fall into three groups — UE-rendered (a control set and an expansion set, produced by the pipeline of Sec. 3.2), real-world footage (DL3DV [21], RealEstate10K [47], and Sekai real-walking-hq [19]), and game footage (game roaming, OmniWorld-Game [16], and Sekai game-walking) — with per-source construction details in Appendix A.

3.2

UE Rendering Pipeline

An automated service inside the UE editor drives scene loading, geometry survey, navigation-mesh construction, point-of-interest discovery, trajectory planning, and Movie Render Queue rendering, and a batch orchestrator sweeps this chain over a library of 337 environments (Fig. 6). Each clip is exported with RGB frames, exact engine ground-truth camera poses (pinhole intrinsics and per-frame camera-to-world matrices), and scene and trajectory metadata; control clips additionally carry per-frame action labels. Trajectories are planned for control, not only for scene coverage: they are drawn from six planned modes, each walking at a constant, mode-specific speed. The control mode anchors each trajectory at two points of interest and fills the path between them with randomized motion segments, so that backward, strafing, and diagonal motion are sampled as evenly as forward motion (Fig. 6). The action-following accuracy reported in Sec. 4 8

owes as much to these direction-balanced trajectories as to the model design, since forward-biased footage gives little supervision for the rarer directions.

3.3

Metric-Scale Alignment

The action ak ∈ R6 that conditions each latent chunk is a frame-to-frame 6-DoF camera increment, so the same commanded action must correspond to the same motion in every source — both on screen and in the pose numbers (Fig. 6). The curation stage already settles the on-screen half: its speed filter drops clips that move too fast or too slow, leaving the survivors at a similar pixel speed (Sec. 3.4). What still disagrees is the numbers attached to that motion — estimated poses (VIPE, MegaSaM) are defined only up to scale, and engine poses differ in units. We therefore rescale the translations of each source s by a single divisor — the ratio of pooled per-latent-step translation-increment medians against the UE anchor, measured by replaying the exact training-time windowing with augmentation disabled: σs =

meds [∥∆t∥] , medUE [∥∆t∥]

medUE [∥∆t∥] = 0.3667,

t ← t/σs .

(4)

A single global scalar per source suffices because the per-clip scale spread within each source is bounded; all sources are further unified to camera-to-world matrices in a common camera basis, and per-source divisors, spread statistics, alignment audits, and convention verification are given in Appendix A.

3.4

Filtering and Sampling

Filtering. The UE control renders pass the funnel of Fig. 6 — splitting, a speed filter that drops clips moving too fast or too slow, captioning, and a darkness/wall filter — reducing 110,204 clips to 95,895, while the expansion set is drawn from the raw render pool by scene round-robin and darkness-filtered from 47,726 to 42,536 clips. The game-roaming source is curated from 494.7 hours of gameplay across 168 games down to 18,387 free-roaming clips from 79 games. Filters for the remaining sources are simple duration and completeness checks; all thresholds and per-source details are given in Appendix A. Sampling. The eight sources are concatenated and sampled uniformly at the clip level, so the training mixture equals the clip-count proportions of Table 2 — 63% UE, 26% real, and 11% game — independent of clip duration. Each dataset access draws a random 189-frame window resampled to 24 fps (≈ 8 s, one training window; Sec. 2.1); with probability 0.2 on the two UE sources the window is replaced by a palindrome — a random half of the window concatenated with its temporal reverse — injecting the explicit revisit evidence that long-horizon spatial memory requires. From each window the loader emits the RGB frames, the per-chunk 6-DoF actions ak , and the relative SE(3) trajectory Pk consumed by MRoPE (Appendix A).

4

Experiments

4.1

Setup

Implementation. ReWorld is trained in two resolution stages on top of the Wan2.2-TI2V-5B backbone [33]: a 480p (384 × 640) pre-training stage, followed by a 720p (704 × 1280) warm-start with interpolated spatial RoPE positions; optimizer, guidance, and parallelism settings follow the training details of Sec. 2.2. Each training window covers L=12 latent chunks zk , conditioned on per-chunk actions ak and poses Pk as defined in Sec. 2.1. Of the H=24 attention heads, the global set G (|G|=6) attends over the full causal history while the remaining heads use a local window of w=12 frames; during training, G cycles through a fixed pool of 12 random six-head partitions, switching every optimizer step (random head routing, Sec. 2.2), and KV chunks are randomly dropped down to 6 kept chunks plus one sink chunk (chunk drop, Sec. 2.3). Real-time inference. Unless otherwise stated, all deployed and timed results are produced with a 4-step DMD-distilled [42, 43] LoRA (rank 128, block size 16, single-rollout student) applied to the EMA weights of the multi-step model, following the self-forcing style distillation recipe [14]. Inference runs at 720p with 4 denoising steps, CFG scale 1, and 16 latent frames per block; per-head attention windows are a training-time construct and are not applied at evaluation—all heads attend over the bounded cache. Bounded-memory 9

Table 3 Camera controllability on the 40-image × 6-trajectory benchmark (240 clips per method). Overall RotErr, TransErr, and CamMC (↓) averaged over all trajectories, followed by per-trajectory RotErr (◦ , ↓) after Sim(3) and global orientation alignment. Best per column in bold. Overall ◦

RotErr◦ ↓ per trajectory

RotErr ↓

TransErr ↓

CamMC ↓

dolly

strafe

arc_yaw

arc_pitch

s_curve

palindrome

SANA-WM [48] DreamX [7] HY-WorldPlay [30] Matrix-Game 3.0 [35] LingBot-World [29] Yume-1.5 [26]

13.02 13.10 14.66 15.45 12.59 14.24

0.123 0.114 0.114 0.075 0.107 0.131

0.388 0.381 0.427 0.394 0.354 0.428

4.95 2.50 0.26 8.19 3.10 0.84

1.51 1.21 0.24 1.00 6.01 2.01

32.01 33.00 43.53 39.93 26.21 29.05

21.33 20.71 22.00 20.59 26.83 24.57

17.54 18.06 21.73 17.75 12.23 25.60

0.80 3.13 0.20 5.24 1.16 3.38

ReWorld (ours)

11.95

0.102

0.332

2.69

1.07

27.63

23.61

16.06

0.64

Method

arms operate under a KV-cache chunk budget of B=12 on the cache C; the landmark bank M is capped at |M| ≤ K with K=30 (Sec. 2.3). Evaluation suite. We evaluate four axes: (i) camera controllability against six interactive world-model and camera-controlled video generation baselines on a shared trajectory benchmark (Sec. 4.2); (ii) long-horizon memory with a needle-in-a-haystack (NIAH) protocol built from palindromic revisit trajectories of up to 384 latents (Sec. 4.3); (iii) video quality on seven video-intrinsic VBench [15] dimensions (Sec. 4.4); and (iv) ablations isolating the training recipe (chunk drop, random head routing), the inference-time cache policy, and the action/pose fusion design (Sec. 4.5).

4.2

Camera Controllability

Protocol. We construct a controllability benchmark of 40 start images (drawn from the six baselines’ official repositories at 704 × 1280, so that no single method is favored) × 6 canonical trajectories, i.e., 240 clips per method. The trajectories are dolly (sustained forward), strafe (lateral left–right), arc_yaw (forward with yaw), arc_pitch (forward with pitch), s_curve (forward with alternating yaw), and palindrome (forward then exact return); every trajectory contains forward translation, since pure-rotation ground truth has zero displacement and degenerates the similarity alignment. Each method receives the same camera intent translated into its native control interface (action keys, pose sequences, or text, as appropriate). Generated clips are re-tracked with ViPE [12], and the estimated trajectory is registered to the intended trajectory by a Sim(3) position alignment together with a global orientation alignment (restricted to det = +1), which absorbs per-method conventions in step magnitude, handedness, and coordinate frame so that the comparison measures whether the trajectory shape follows the intent. We report per-trajectory rotation error RotErr (geodesic, degrees) and aggregate translation error TransErr and camera motion consistency CamMC. We compare against SANA-WM [48], DreamX [7], HY-WorldPlay [30] (its lightweight Wan-based variant on this benchmark), Matrix-Game 3.0 [35], LingBot-World [29], and Yume-1.5 [26]. Table 3 reports per-trajectory rotation error together with aggregate metrics for all seven methods. Because all methods are aligned with the same Sim(3) and orientation registration against a method-agnostic ground-truth intent, the comparison isolates trajectory-following fidelity from per-method magnitude conventions. ReWorld attains the best overall RotErr and CamMC, sits in the first tier on the translation-dominant trajectories (dolly, strafe, palindrome), and remains competitive on the rotation-heavy arcs, which are the hardest regime for every method.

4.3

Long-Horizon Memory

Protocol. We probe spatial memory with a needle-in-a-haystack protocol built on palindrome trajectories: the camera moves out and then retraces its path, so views generated early in the clip must be reproduced after a long temporal gap—these early views are the needles. The baseline benchmark pairs 12 start images with 3 canonical palindromes at two rollout lengths k ∈ {48, 96} latents (≈8 s and ≈16 s), giving 36 clips per method and length. The palindromes are strafe, dolly, and yaw (Fig. 7). Every method receives the same camera intent through its native control interface, with per-image matched prompts and a fixed seed. Scored 10

(a) Strafe

(b) Dolly

(c) Yaw

translate left, then retrace right

forward, then straight back

rotate away, then back (in place)

out

turnaround

back

needle / revisit

rollout k = 48 / 96 latents ( ≈ 8 / 16 s)

t = 0 (needle)

scored mirror pairs (t, k−1−t)

turnaround

t = k−1 (revisit)

Figure 7 Constructed palindrome trajectories. The camera moves out (blue) and retraces its path (orange), so the needle view at t=0 must be regenerated at t=k−1; scored mirror pairs (t, k−1−t) span at least half the rollout.

revisit pairs are the mirror pairs of the palindrome, anchored in the earliest fifth of the clip and kept only when their gap spans at least half the rollout, up to five pairs per clip. For our own arms we additionally run a longer version of the memory test: grouped-action explorations completed into palindromes, at nested lengths k ∈ {96, 192, 288, 384} latents; each length is a prefix of the same trajectory, so scores are comparable across k. This longer test feeds the ablations of Sec. 4.5. Memory metrics. For each revisit pair we compare the predicted frame at the needle with the predicted frame generated when the camera returns to that pose, and score their similarity four ways. SSIM measures pixel-level structure. LPIPS is a perceptual distance between deep features (lower is better). DINO is the cosine similarity of DINO ViT features and asks whether the two frames show the same scene layout. ORB is the fraction of matched ORB keypoints and asks whether specific landmarks reappear in place. Each score is averaged over revisit pairs, then over clips, and reported as an absolute value. A rollout that barely moves scores high on any similarity metric, so the table also reports each method’s executed path length: the accumulated median optical flow of the outbound half, i.e., how much visual ground the rollout actually covers before turning back. Comparison to baselines. Table 4 reports all four scores at both lengths for the same six methods as Sec. 4.2. For HY-WorldPlay we evaluate its flagship autoregressive model, as in Sec. 4.4. ReWorld runs its default deployment configuration (consolidation with the landmark bank, B=12). The two lengths play different roles. At k=48 the revisit gap reaches 47 latents, and several baselines still hold the start view in context—LingBotWorld keeps a 52-latent window, while SANA-WM (linear attention) and Yume-1.5 are unbounded—so k=48 serves as a sanity check. At k=96 the gap reaches 95 latents: the start view has left every bounded window and must be recalled from memory, making this the discriminative length. HY-WorldPlay posts the strongest baseline scores, but it also moves the least: at k=96 its median outbound path length is 210 px, versus 332–834 px for the other baselines (Table 4), and a shorter path is easier to reproduce from memory. Figure 8 shows this regime qualitatively on a k=96 strafe-and-return rollout: the final chunks must reproduce the input view after a long round trip, and ReWorld returns to the start with the landmark layout intact, whereas the baselines either drift off the commanded path or regenerate the revisited region with altered geometry. The effect of the KV budget and cache policy on our model, including its per-length scaling, is isolated on the long-rollout test in Sec. 4.5 (Table 6).

4.4

Generation Quality

We assess perceptual quality on the seven video-intrinsic VBench [15] dimensions that require no text prompt: Imaging Quality, Aesthetic Quality, Subject Consistency, Background Consistency, Temporal Flickering, Dynamic Degree, and Motion Smoothness, computed on the same clip set as the controllability benchmark. Native outputs differ across methods in length, resolution, and frame rate, so all clips are normalized to a common specification before scoring—32 uniformly sampled frames spanning the full clip, resized to 1280 × 704, re-encoded at 16 fps; the comparison is therefore internally controlled, but not directly comparable to numbers 11

Table 4 Long-horizon memory against the six interactive world-model baselines on the palindromic revisit bench: absolute revisit-similarity scores (SSIM, LPIPS, DINO, ORB) at rollout lengths k=48 and k=96 latents, alongside the executed path length (accumulated median optical flow of the outbound half, px). Every method receives the same action commands, but how far it actually travels differs, and a shorter path generally yields higher revisit scores. k=48 is a sanity length that several baselines can solve from context alone; k=96 forces recall from beyond every bounded window. Best per column in bold, second best underlined; path length is descriptive, not ranked. k=48 (≈8 s) Method

k=96 (≈16 s)

Path Length SSIM↑ LPIPS↓ DINO↑ ORB↑ Path Length SSIM↑ LPIPS↓ DINO↑ ORB↑

SANA-WM [48] DreamX [7] HY-WorldPlay [30] Matrix-Game 3.0 [35] LingBot-World [29] Yume-1.5 [26]

296 279 113† 312 352 165

0.318 0.214 0.448 0.318 0.274 0.268

0.433 0.553 0.163 0.369 0.557 0.533

0.810 0.740 0.969 0.891 0.663 0.747

0.174 0.180 0.464 0.292 0.177 0.179

499 454 210† 724 834 332

0.313 0.194 0.427 0.275 0.251 0.269

0.528 0.627 0.247 0.478 0.635 0.586

0.700 0.603 0.942 0.850 0.509 0.661

0.161 0.159 0.460 0.247 0.162 0.152

ReWorld (ours)

286

0.349

0.282

0.913

0.325

615

0.384

0.332

0.932

0.379

† HY-WorldPlay executes by far the shortest paths: 113/210 px vs. 279–352/454–834 px for all other methods.

camera strafes LEFT

strafes back RIGHT

latent 0

latent 8

latent 28

latent 48

latent 72

latent 94

input frame

strafing left

further left

furthest left

strafing back right

back at the start view

Ours

PROMPT “Low-poly stylized 3D desert scene with red rock cliffs, green cacti and palm trees, wooden ladders, and sandy ground under warm daylight, flat cartoon CG rendering.”

SANA-WM

DreamX

HY-WorldPlay Look for — whether column 6 returns to the input frame in column

Matrix-Game 3.0

1: the same landmarks, in the same places. Ours — bounded KV, 12 chunks,

LingBot-World

with the landmark bank; four-step distilled. Camera — a lateral slide: sideways

Yume-1.5

left, then back along the same line. It never turns.

Figure 8 Qualitative memory comparison on a strafe-and-return rollout. All methods start from the same input frame and execute the same lateral palindrome—the camera strafes left, then retraces the same line back to the start, without ever turning—so the last column should reproduce the first: the same landmarks, in the same places. Columns are latents 0/8/28/48/72/94 of a k=96 rollout; each baseline is driven through its native control interface. ReWorld (top row; bounded cache B=12 with the landmark bank, four-step distilled) returns to the input view with the layout intact, whereas the baselines either drift off the commanded path or regenerate the revisited region with altered geometry.

reported on native-length outputs. For HY-WorldPlay, this benchmark evaluates its flagship autoregressive model. Table 5 summarizes the seven quality dimensions. ReWorld attains the best mean: no single dimension is dominant, but it is in the first tier on the temporal axes (Motion Smoothness, Temporal Flickering) while sustaining a high Dynamic Degree, whereas the per-dimension leader HY-WorldPlay pays for its consistency scores with markedly lower motion. We note that Dynamic Degree should be read jointly with the revisit 12

Table 5 VBench quality on the seven video-intrinsic dimensions (higher is better), computed on the shared benchmark clips normalized to 32 frames / 1280 × 704 / 16 fps. Best per column in bold. Method

Mean ↑

Imaging

Aesthetic

Subject Cons.

Background Cons.

Temporal Flick.

Dynamic Degree

Motion Smooth.

SANA-WM [48] DreamX [7] HY-WorldPlay [30] Matrix-Game 3.0 [35] LingBot-World [29] Yume-1.5 [26]

0.835 0.828 0.842 0.836 0.841 0.844

0.668 0.670 0.670 0.713 0.692 0.669

0.537 0.542 0.601 0.461 0.577 0.565

0.908 0.880 0.976 0.894 0.905 0.878

0.919 0.904 0.957 0.919 0.915 0.911

0.942 0.909 0.973 0.921 0.918 0.934

0.900 0.946 0.733 0.983 0.933 0.988

0.969 0.946 0.987 0.962 0.947 0.964

ReWorld (ours)

0.850

0.665

0.579

0.929

0.929

0.952

0.912

0.979

Figure 9 Inference cache policies, qualitatively. On out-and-back trajectories, sliding window and KV compression drift or reinvent the revisited scene; the landmark cache (ours) returns to the start view.

metrics of Sec. 4.3, as consistency and revisit metrics alike favor low-motion rollouts.

4.5

Ablations

Training recipe and cache policy. Table 6 ablates two axes on the long-rollout memory test of Sec. 4.3, reporting revisit SSIM at rollout lengths k ∈ {96, 192, 288, 384} latents. The upper block varies the training recipe—chunk drop only (CD), random head routing only (RH), and both combined (ours). All variants are trained from scratch under identical settings (720p, 20k steps) and evaluated with the same consolidation arm, so only the recipe differs. The lower block fixes the final recipe and varies only the inference cache policy under the same budget B=12. Every bounded arm keeps the same sink chunk and differs only in how the rest of the budget is filled: window uses a sliding window of recent chunks; naive-merge pools aged chunks instead of evicting them; consolidation w/o bank adds six static landmarks (no retrieval); consolidation w/ bank retrieves the six 13

landmarks nearest the current pose from the bounded bank (Sec. 2.3); full-KV keeps everything and is the undeployable upper bound (OOM beyond k=192). The far-gap lengths k ∈ {288, 384}, where a sliding window has necessarily evicted the needle, are the discriminative setting; Fig. 9 shows the contrast qualitatively. Control–memory decoupling. Table 7 tests whether the routed head structure decouples action-following from long-range recall, comparing three fusion variants trained under the same protocol. MRoPE only drops the action embedding and sets all H=24 heads global, so control must be inferred solely from pose-indexed positional encodings; Action + MRoPE keeps the additive action embedding with all heads global, i.e., fusion without routing; Routing is our full configuration with the mixed global/local head structure and per-step head routing. We report the control metrics of Sec. 4.2 on the long-rollout memory test. Adding action injection improves every control metric over MRoPE only (RotErr 17.66◦ → 13.21◦ , TransErr 0.204 → 0.107, CamMC 0.394 → 0.371) but drops revisit SSIM from 0.3898 to 0.3376—the interference that motivates the window split. Routing matches the unrouted variant on control (12.94◦ , 0.093, 0.346) while restoring revisit SSIM to 0.3752: the structural constraint costs no control authority and preserves long-gap recall.

5

Related Works

Streaming video generation. Large video diffusion models denoise a clip as a single bidirectional block [18, 33, 41], an interface at odds with interaction: no frame can appear before the whole clip is finished. Autoregressive reformulations restore a temporal arrow. Per-frame noise schedules [3], chunked causal attention with KV caching [32], context packing [45], and streaming pipelines [17] realize it, while few-step distillation [25, 34, 42, 43], trainingfree attention reuse [46], self-rollout training that closes the exposure gap of AR [14, 23], and ARtrain-then-distill systems scaled to long video [5] make the streams real-time. ReWorld follows this recipe—chunked causal generation over a bidirectional backbone, DMD distillation with self-forcing rollouts—but confines distillation to a LoRA [11], so one backbone retains a high-fidelity multi-step mode beside the real-time one.

Table 6 Revisit SSIM↑ on the long-rollout memory test (Sec. 4.3) at four rollout lengths k. Upper block: the training recipe is varied with the inference arm fixed to our consolidation. Lower block: the inference cache policy is varied on the final recipe; full-KV is the unbounded upper bound (OOM beyond k=192). Revisit SSIM↑ Method

k=96 k=192 k=288

Training recipe (consolidation arm) base 0.4019 0.3747 + CD 0.4043 0.3941 + RH 0.4319 0.3787 + CD + RH (ours) 0.4358 0.3929 Inference cache policy full-KV window naive-merge consolidation w/o bank consolidation w/ bank

0.4463 0.4129 0.3741 0.4376 0.4358

0.3698 0.3846 0.3836 0.3943

k=384 0.3387 0.3463 0.3565 0.3752

0.4231 (OOM) (OOM) 0.3687 0.3505 0.3476 0.3371 0.3467 0.3541 0.3758 0.3743 0.3629 0.3929 0.3943 0.3752

Interactive world models. From recurrent latent Table 7 Control–memory decoupling study: control metrics simulators [10] through latent-action pretraining [2] on the long-rollout memory test (Sec. 4.3) for the three and its frontier successors [27], interactive generafusion variants. tion now spans action-conditioned diffusion trained on game and UE footage [13, 29, 35, 48], explorable Method RotErr◦ ↓ TransErr↓ CamMC↓ Revisit SSIM↑ scene generation [6, 24, 26, 31], and low-latency MRoPE only 17.66 0.204 0.394 0.3898 streamed deployments [4, 7]. The two prevailing Action + MRoPE 13.21 0.107 0.371 0.3376 control signals differ in kind: relative pose folded Routing (ours) 12.94 0.093 0.346 0.3752 into attention localizes past content [7, 39], whereas an injected action supplies the motion command directly [35], and some recent systems carry both [30]. ReWorld likewise keeps both channels, but treats their interference as a training problem, splitting the window each attention head trains under so that control is learned within short windows and memory under long ones. Memory in world models. Long rollouts stay consistent only if views that have left the context remain recallable [9]. Existing designs differ chiefly in what they store and what that costs: archives of past frames that grow with the rollout, from which pose-relevant views are pulled back as conditioning [38, 44]; an explicit 3D reconstruction that keeps memory outside the generator’s own representation [36]; or pose-dependent 14

attention over the retained history [39]. Sinks and sliding windows [37] bound the cache instead, at the price of forgetting everything beyond the window. ReWorld takes the bounded route without the forgetting: chunks aging out of the recent window consolidate into a fixed-capacity landmark bank inside the model’s own KV space, redundancy-based eviction keeps the bank diverse, pose-proximity retrieval fills the cache, and chunk-drop training teaches the model to read the spliced result.

6

Conclusion

We presented ReWorld, an interactive streaming world model built in two steps: split the training of control and memory by window—mixed per-head attention windows with random head routing—then consolidate memory at inference, where a bounded cache backed by a pose-indexed landmark bank holds the entire past under a fixed KV budget and chunk-drop training makes its sparse caches in-distribution. Trained on metrically aligned multi-source data and distilled to a few-step LoRA student, ReWorld streams high-resolution video in real time and leads recent interactive world models on control fidelity and visual quality, while revisit fidelity persists at rollout lengths where a sliding window has long evicted the evidence. Memory is still keyed on camera pose alone; extending consolidation to dynamic scenes and richer, non-navigational interaction is the natural next step.

15

References [1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. [2] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In International Conference on Machine Learning (ICML), 2024. arXiv:2402.15391. [3] Boyuan Chen, Diego Marti Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. In Advances in Neural Information Processing Systems, 2024. [4] Ming Chen, Liyuan Cui, Wenyuan Zhang, Haoxian Zhang, Yan Zhou, Xiaohan Li, Songlin Tang, Jiwen Liu, Borui Liao, Hejia Chen, et al. Midas: Multimodal interactive digital-human synthesis via real-time autoregressive video generation. arXiv preprint arXiv:2508.19320, 2025. [5] Yukang Chen, Luozhou Wang, Wei Huang, Shuai Yang, Bohan Zhang, Yicheng Xiao, Ruihang Chu, Weian Mao, Qixin Hu, Shaoteng Liu, Yuyang Zhao, Huizi Mao, Ying-Cong Chen, Enze Xie, Xiaojuan Qi, and Song Han. Longlive-2.0: An nvfp4 parallel infrastructure for long video generation. arXiv preprint arXiv:2605.18739, 2026. [6] Xinhua Cheng, Haiyang Zhou, Wangbo Yu, Tanghui Jia, Bin Lin, Yunyang Ge, Weiqi Li, and Li Yuan. 360explorer: Exploring 4d controllable world in panoramic videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 3300–3308, 2026. [7] DreamX Team. Dreamx-world 1.0: A general-purpose interactive world model. arXiv preprint arXiv:2606.16993, 2026. [8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024. [9] Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025. [10] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. [11] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. arXiv:2106.09685. [12] Jiahui Huang, Qunjie Zhou, Hesam Rabeti, Aleksandr Korovko, Huan Ling, Xuanchi Ren, Tianchang Shen, Jun Gao, Dmitry Slepichev, Chen-Hsuan Lin, et al. ViPE: Video pose engine for 3D geometric perception. arXiv preprint arXiv:2508.10934, 2025. [13] Siqiao Huang, Jialong Wu, Qixing Zhou, Shangchen Miao, and Mingsheng Long. Vid2world: Crafting video diffusion models to interactive world models. arXiv preprint arXiv: 2505.14357, 2025. [14] Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009, 2025. [15] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, 2024. [16] InternRobotics Team. OmniWorld: A multi-domain and multi-modal dataset for 4D world modeling. arXiv preprint arXiv:2509.12201, 2025. [17] Akio Kodaira, Tingbo Hou, Ji Hou, Markos Georgopoulos, Felix Juefei-Xu, Masayoshi Tomizuka, and Yue Zhao. Streamdit: Real-time streaming text-to-video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 29200–29210, 2026. [18] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. [19] Zhen Li, Chuanhao Chen, Haiyang Yang, Yang Han, Bo Zhu, Zhuangzi Zhou, et al. Sekai: A video dataset towards world exploration. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2506.15675.

16

[20] Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. MegaSaM: Accurate, fast, and robust structure and motion from casual dynamic videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. [21] Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. DL3DV-10K: A large-scale scene dataset for deep learning-based 3D vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [22] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023. [23] Kunhao Liu, Wenbo Hu, Jiale Xu, Ying Shan, and Shijian Lu. Rolling forcing: Autoregressive long video diffusion in real time. arXiv preprint arXiv:2509.25161, 2025. [24] Wei Liu, Ziyu Chen, Zizhang Li, Yue Wang, Hong-Xing Yu, and Jiajun Wu. Realwonder: Real-time physical action-conditioned video generation. arXiv preprint arXiv:2603.05449, 2026. [25] Zhengyao Lv, Chenyang Si, Tianlin Pan, Zhaoxi Chen, Kwan-Yee K. Wong, Yu Qiao, and Ziwei Liu. Dual-expert consistency model for efficient and high-quality video generation. https://arxiv.org/abs/2506.03123, 2025. [26] Xiaofeng Mao, Zhen Li, Chuanhao Li, Xiaojie Xu, Kaining Ying, Tong He, Jiangmiao Pang, Yu Qiao, and Kaipeng Zhang. Yume-1.5: A text-controlled interactive world generation model. arXiv preprint arXiv:2512.22096, 2025. [27] Jack Parker-Holder and Shlomi Fruchter. Genie 3: A new frontier for world models. Google DeepMind Blog, https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/, 2025. Announced August 5, 2025. [28] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195–4205, 2023. [29] Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, et al. Advancing open-source world models. arXiv preprint arXiv:2601.20540, 2026. [30] Wenqiang Sun, Haiyu Zhang, Haoyuan Wang, Junta Wu, Zehan Wang, Zhenwei Wang, Yunhong Wang, Jun Zhang, Tengfei Wang, and Chunchao Guo. Worldplay: Towards long-term geometric consistency for real-time interactive world modeling. arXiv preprint arXiv:2512.14614, 2025. [31] HunyuanWorld Team, Zhenwei Wang, Yuhao Liu, Junta Wu, Zixiao Gu, Haoyuan Wang, Xuhui Zuo, Tianyu Huang, Wenhuan Li, Sheng Zhang, et al. Hunyuanworld 1.0: Generating immersive, explorable, and interactive 3d worlds from words or pixels. arXiv preprint arXiv:2507.21809, 2025. [32] Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tianning Zhang, WQ Zhang, Weifeng Luo, et al. Magi-1: Autoregressive video generation at scale. arXiv preprint arXiv:2505.13211, 2025. [33] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. [34] Fu-Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency model. arXiv preprint arXiv:2405.18407, 2024. [35] Zile Wang, Zexiang Liu, Jiaxing Li, Kaichen Huang, Baixin Xu, Fei Kang, Mengyin An, et al. Matrix-game 3.0: Real-time and streaming interactive world model with long-horizon memory. arXiv preprint arXiv:2604.08995, 2026. [36] Tong Wu, Shuai Yang, Ryan Po, Yinghao Xu, Ziwei Liu, Dahua Lin, and Gordon Wetzstein. Video world models with long-term spatial memory. arXiv preprint arXiv:2506.05284, 2025. [37] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations (ICLR), 2024. [38] Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xingang Pan. Worldmem: Long-term consistent world simulation with memory. arXiv preprint arXiv:2504.12369, 2025. [39] Tianshuo Xu, Yichen Xie, Depu Meng, Chensheng Peng, Quentin Herau, Bo Jiang, Yihan Hu, and Wei Zhan. Teaching video generators to remember: Eliciting dynamic memory for out-of-sight state evolution. arXiv preprint arXiv:2605.25333, 2026. [40] Shuai Yang, Wei Huang, Ruihang Chu, Yicheng Xiao, Yuyang Zhao, Xianbang Wang, Muyang Li, Enze Xie, Ying-Cong Chen, Yao Lu, Song Han, and Yukang Chen. Longlive: Real-time interactive long video generation. arXiv preprint arXiv:2509.22622, 2025.

17

[41] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In ICLR, 2025. [42] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T. Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405.14867, 2024. [43] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T. Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828, 2024. [44] Jiwen Yu, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Context as memory: Scene-consistent interactive long video generation with memory retrieval. ICCV, 2025. [45] Lvmin Zhang and Maneesh Agrawala. Packing input frame context in next-frame prediction models for video generation. arXiv preprint arXiv:2504.12626, 2025. [46] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. In ICLR, 2025. [47] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Transactions on Graphics (SIGGRAPH), 37(4), 2018. [48] Haoyi Zhu, Haozhe Liu, Yuyang Zhao, Tian Ye, Junsong Chen, Jincheng Yu, Tong He, Song Han, and Enze Xie. SANA-WM: Efficient minute-scale world modeling with hybrid linear diffusion transformer. arXiv preprint arXiv:2605.15178, 2026.

18

Appendix A

Data Pipeline Details

This appendix expands Sec. 3 with per-source construction details, the per-source scale divisors and their audits, and the loader-level sampling and conditioning specifics.

A.1

Per-Source Construction

UE-rendered fly-throughs (metric anchor). Our Unreal Engine rendering pipeline produces camera fly-throughs with exact ground-truth trajectories: each clip directory contains the rendered video, a camera.json with per-frame absolute 4 × 4 camera-to-world matrices (right-handed, Y-up, centimeters) together with intrinsics, and a summary.json recording the scene and trajectory metadata (plus an overhead trajectory visualization). The full rendering run yields 130,986 raw clips over 337 purchased and in-house environments (indoor, urban, and landscape scenes), drawn from six trajectory modes planned at generation time — local_explore (∼29%), reveal_pan (∼26%), aerial (∼15.5%), control (∼15%), wandering (∼8%), and boomerang (∼6%) — each walking at a constant, mode-specific speed (180 cm/s for control, 120 cm/s for the exploratory modes). Because raw mode and scene frequencies are skewed, we curate a balanced subset by scene round-robin: for each mode we target ∼8,000 clips while cycling over environment and level identifiers, which yields 47,726 clips spanning 248–324 environments per mode; darkness filtering (Sec. 3.4) reduces this to the 42,536-clip expansion set of Table 2, whose captions combine the pipeline scene summary with a camera-motion sentence. The 95,895-clip control set of Table 2 comes from an earlier, control-heavy rendering run (27,551 renders of 60 s, split into 110,204 clips) and is obtained from 109,132 captioned clips by the same darkness filter (removing black-frame and wall-facing clips); its trajectories are generated from the explicit discrete action space of the control mode, with per-frame action labels kept consistent with the realized motion, and its captions are generated by the rendering pipeline itself. Because UE poses are exact and metric, these sources serve as the scale anchor in Sec. 3.3. UE luminance and motion filtering. UE renders fail in a characteristic way: when the camera clips into geometry or faces an unlit surface, frames are near-black, so both UE sets pass through a luminance filter. Each clip is probed with 16 uniformly sampled grayscale frames and dropped if its mean luminance is below 35 (on 0–255) or if more than half of the sampled frames are dark; a lightweight motion probe during captioning separately removes frozen clips whose frames barely change. This filter reduces the control set from 109,132 captioned clips to 95,895 (−12.1%) and the round-robin-selected expansion set from 47,726 clips to 42,536 (−10.9%). DL3DV. From DL3DV-10K real-scene walkthroughs [21] we cut frame-accurate 15 s slices whose pose subsegments are aligned to the video slice with zero drift, yielding 29,864 clips with no missing modality. Poses are estimated with VIPE [12] (near-metric). Each clip is captioned with Qwen-VL [1] (average 149 words per caption, no duplicates or empties). RealEstate10K. We use RealEstate10K [47] indoor/real-estate footage (watermark-inpainted video), retaining clips of at least 8 s, which leaves 12,065 of the original 22,275. Rather than the released SLAM trajectories, poses are re-estimated with VIPE so that pose provenance matches DL3DV; captions come from a pre-computed caption store. Game roaming. The game-roaming source is curated from a 494.7-hour collection of gameplay recordings spanning 168 games. A roaming whitelist keeps 88 titles suited to free exploration and walking (pure shooters are excluded), which are sliced into 24,801 clips of 15 s at 1920 × 1080, 30 fps. Each clip receives VIPE metric camera-to-world poses (OpenCV convention, first frame anchored to the identity) and a Qwen-VL caption; successive cleaning passes remove pose-quality outliers, dark clips, and full-screen menu or loading content, leaving 18,387 clips across 79 games (nine whitelisted titles yield no surviving clips). Because the VIPE poses of this source are already metric, it enters Sec. 3.3 with σ = 1. OmniWorld-Game. From the simulator/game domain of OmniWorld [16] (479 scenes) we normalize the released quaternion extrinsics, invert the world-to-camera matrices to camera-to-world at build time, and merge only 19

frame-contiguous reconstruction splits into runs (splits across index gaps are independent reconstructions — largely teleports — and are never merged). Runs are cut into 12 s windows with 9 s stride (25% overlap), discarding runs shorter than 8.5 s, and each clip is re-anchored so that its first pose is the identity. This yields 5,629 clips (median 12.0 s) with per-clip JSON captions. Sekai real-walking-hq and game-walking. From the Sekai corpus [19] of first-person walking videos we intersect available 720p/60 s videos with released pose files, obtaining 14,730 real-walking clips; clips are kept as uncut 60 s segments. Poses are per-frame MegaSaM [20] camera-to-world estimates (the c2w convention was verified empirically by forward-direction alignment, and pose–video synchronization was verified by cross-correlating optical-flow speed with pose angular speed, with median lag 0). The companion game-walking split contributes 1,618 clips rendered in UE5 with engine ground-truth c2w trajectories, downscaled to 720p without re-cutting. Both splits use the official CSV captions (caption plus location, weather, crowd, and time-of-day tags).

A.2

Alignment Details and Audits

Per-source divisors. The divisor of Eq. (4) is measured by replaying the exact training-time windowing (a 189-frame window resampled to 24 fps, i.e. 0.1640625 s per latent step, with all trajectory augmentation disabled), converting poses to per-latent translation increments ∆t, and pooling the increment magnitudes over sampled clips. The resulting divisors are: UE σ = 100 (exact cm→m conversion; anchor), DL3DV 0.0709/0.3667 = 0.1923, RealEstate10K 0.0825/0.3667 = 0.2273, OmniWorld-Game 0.641/0.3667 = 1.75 (pooled over the locomotion subset, since the UE anchor is locomotion-dominated and including the 29% vehicle clips would inflate the divisor), Sekai real-walking 0.0032/0.3667 = 0.0086, and Sekai game-walking 0.0039/0.3667 = 0.0107; the game-roaming source is already metric under VIPE and uses σ = 1. The divisor is applied to the translation column of every pose in the loader. The UE expansion set moves genuinely more slowly than the control set (its raw per-latent-step translation median is ≈ 20 cm versus ≈ 37 cm); we deliberately do not renormalize it separately, since the speed difference is real rather than a scale error. Spread and post-hoc audits. A single global scalar per source suffices because the per-clip scale spread is bounded: the p90 /p10 ratio of per-clip median increments is 3.65 (DL3DV), 2.66 (RealEstate10K), and 2.86/2.67 (Sekai real/game) — under one order of magnitude and attributable to genuine capture-speed variation — so neither per-scene normalization nor metric-depth rescaling is needed. A post-hoc audit of the model-facing increments after rescaling confirms alignment: the per-source median ∥∆t∥ relative to UE is 1.00 (UE, by construction), 0.92 (DL3DV), 1.00 (RealEstate10K), 1.21 (OmniWorld-Game, a vehicle-tail effect), 0.96 (Sekai real), and 0.94 (Sekai game). Coordinate-convention normalization. Coordinate conventions are normalized jointly with scale. All sources are brought to camera-to-world matrices in the OpenCV camera basis (forward +z, down +y): the UE camera basis (forward +x, up +y) is conjugated by a non-trivial 90◦ rotation about the vertical axis (a naive diag(1, −1, −1) sign flip is incorrect and was ruled out by an empirical derivation against engine trajectories); OmniWorld world-to-camera extrinsics are inverted at clip-build time; Sekai extrinsics are already c2w and are ingested without inversion; the legacy camera-referenced (w2c) RealEstate10K mode is not used in the joint configuration. The audit further verifies that forward motion maps to a dominant +tz in every source (DL3DV is tx -dominated in aggregate, but this reflects its orbital/strafing capture style rather than a convention error) and that yaw maps to the same Euler index with consistent sign. Rotations are already dimensionless (radians) and are never rescaled; only translation passes through Eq. (4).

A.3

Sampling and Conditioning Details

Window extraction. Within a long clip the 189-frame window — the native training-window length of 48 latent frames, i.e. L=12 chunks of 4 — starts at a uniformly random frame, and frames are resampled to 24 fps by uniform striding (clips shorter than the required span are stretched by linear index interpolation). A 60 s Sekai clip therefore contributes a different random ≈ 8 s excerpt (7.88 s) on every epoch, which converts long uncut footage into diverse window-level supervision without offline re-slicing. Palindrome augmentation. The palindrome augmentation of Sec. 3.4 forces the camera to retrace its own path within a single training window and additionally provides reverse-motion coverage. Under this augmentation, 20

pose downsampling is SE(3)-aware (rotation slerp), since arithmetic averaging across the palindrome mirror point produces singular matrices. A whole-window reversal probability exists in the loader but is disabled in the joint configuration. Emitted conditioning. From each window the loader emits the RGB frames, the per-chunk actions, and the pose context consumed by MRoPE. The 189 frame poses are downsampled to the 48 latent steps, re-anchored so the first camera is the identity, and differenced into per-step 6-DoF increments (three translations plus three Euler angles), which are grouped per latent chunk into the actions ak ∈ R6 . The MRoPE context is the relative camera-to-world trajectory Pk ∈ SE(3) itself, with translations additionally normalized by the window’s maximum radius (a pure rescaling that preserves the identity anchor), so that pose-indexed attention operates on a bounded, source-agnostic coordinate frame.

21

Related documents

• IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training#819629
• IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training#809633
• CAER: Causal Action Effect Reweighting for World Model Training#818935
• CAER: Causal Action Effect Reweighting for World Model Training#808943
• World Observer: Joint Actor-Observer Generation for Persistent World Modeling#1157510