arXiv CS open access

WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models ∗

∗

Bohai Gu1, , Yueyang Yuan2, , Taiyi Wu3 , Dazhao Du1 , Jian Liu1 , Xiaoyi Pang1 , Jie Zhang1 , † Xiaocheng Lu1 , Haobin Zhong3 , Xiaotong Zhao3 , Alan Zhao3 , Song Guo1, 1

The Hong Kong University of Science and Technology 2 Wuhan University 3 AI Technology Center, Tencent Video, Tencent

arXiv:2608.04964v1 [cs.AI] 5 Aug 2026

Abstract Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Post-training methods such as reinforcement learning (RL) can improve these models, but they hit a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift. Our key insight is that reversible action cycles make this verification possible: a sequence composed with its inverse must analytically return to the initial state, yielding annotation-free supervision on long-horizon correctness. Building on this, we introduce WorldCycle, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions. These rewards force the model to learn actions as consistent state operators rather than memorized temporal patterns, and extend naturally to out-of-distribution composite action cycles that the base model handles poorly. We further release CycleBench, a diagnostic benchmark for state-returning ability under complex action structures. WorldCycle reduces state returning drift by up to 44% and lifts composite-action accuracy nearly 4× over the base model, providing a vital foundation for physically grounded world models.

Introduction Interactive video world models (IWMs) generate future visual observations conditioned on an initial observation and a stream of user or agent actions, turning generative video models into interactive simulators (Yang et al. 2024; Wu et al. 2024; Zhu et al. 2025; Xiang et al. 2024). Such models support embodied interaction, robot learning, visual planning, and game-like simulation. Related action-conditioned video world models have also been developed for autonomous driving (Hu et al. 2023; Wang et al. 2024; Gao et al. 2024; Zhang et al. 2025a). These models commonly employ an autoregressive generation paradigm: each newly generated observation becomes the context for predicting the next one. Consequently, small single-step errors can propagate and compound as the rollout extends, gradually eroding the physical and geometric consistency required by downstream longhorizon tasks (Bengio et al. 2015; Janner et al. 2019; Huang et al. 2026; Su et al. 2026). ∗ †

These authors contributed equally. Corresponding author

Post-training methods based on reinforcement learning (RL) have recently emerged as a promising approach for improving video world models (Wu et al. 2025a; Ye et al. 2025). Yet existing approaches predominantly optimize short-horizon visual quality or per-step action alignment. The fundamental obstacle to long-horizon improvement is the absence of effective supervision. For an arbitrary action sequence, no ground-truth future state is available to measure accumulated error. The most recent work, WorldCompass (Wang et al. 2026a), attempts to address this by rewarding clip-level action following, but such objectives primarily improve local action accuracy and remain insufficient for correcting fine-grained spatial drift over long horizons. We identify a new angle on this problem by focusing on a specific and ubiquitous class of dynamics: reversible trajectories such as camera ego-motion. For a reversible action cycle, physical law dictates that the final state must be identical to the initial state, providing an absolutely precise, annotationfree reference. Yet as shown in Figure 1, both WorldPlay and WorldCompass (Sun et al. 2026; Wang et al. 2026a) fail this minimal check: an inverse action sequence fails to recover the starting view (spatial closure failure), and the same action produces different displacements at different rollout positions (temporal consistency failure). These failures compound with horizon length and are invisible to any short-horizon reward. Crucially, the same physical prior that exposes the failure also resolves the supervision bottleneck: the cycle-return constraint provides an exact, label-free target for the long-horizon trajectory. Starting from this insight, we identify two remaining challenges: (i) reward sparsity, where an endpoint-only cycle signal provides too little gradient to localise intermediate errors; and (ii) temporal drift, where AR accumulation makes the same action behave differently at different rollout depths. The harder second level concerns composite actions such as moving forward while turning, which must also return after their inverse programs. Pre-trained baselines such as WorldPlay exhibit a 5× accuracy collapse on composite-action versus simple-action settings of identical rollout length (Table 1), indicating that composite motions are out-of-domain: pretraining data rarely provides GT video demonstrations of composite trajectories. We propose WorldCycle, a Self-Verifiable Reinforcement Learning framework specifically designed to tackle statereturning failure in long-horizon video world models under

253 frames

WorldPlay

WorldCompass

Ours

Cycle 1: Start

Cycle 1: Mid

Cycle 2: Start

Cycle 2: Mid

Figure 1: State-returning failure in interactive video world models. Even state-of-the-art models fail a minimal sanity check: an inverse action sequence such as forward-then-backward does not recover the starting view (spatial closure failure), and executing the same action from what should be the same state at different rollout steps produces inconsistent displacements (temporal consistency failure). Both failures accumulate as autoregressive rollouts extend, yet neither is captured by shorthorizon rewards. WorldCycle turns closed action cycles into annotation-free supervision on long-horizon correctness. reversible action trajectories. The guiding principle is to conditions. Our contributions are threefold: turn closed action programs into dense supervision. We ❶ A new long-horizon supervision perspective. We reexploit the mirror structure of an inverse action program: veal state-returning failures in long-horizon video IWMs every partial forward trajectory has a corresponding reverse and introduce reversible action cycles as a self-verifiable state that should coincide with it. We construct mirrored supervision signal, requiring no ground-truth trajectories. frame pairs at every intermediate depth of a cycle and com❷ WorldCycle: a self-verifiable RL framework for repare them with dense visual evidence, yielding a spatial cloversible dynamics. We introduce two complementary sure reward that converts each intermediate chunk into an trajectory-level rewards, spatial cycle closure and temindependently verifiable closure check, matching the chunkporal state consistency, that overcome autoregressive wise structure of AR post-training and removing the need drift on in-domain actions and generalize to out-offor a single sparse endpoint signal. We further repeat cycles distribution composite action cycles. and compare co-indexed frames across cycles, producing a ❸ CycleBench and empirical gains. We establish the first temporal state consistency reward that penalises drift of benchmark for state-returning consistency across reidentical actions over time, directly targeting the temporal versible, repeated, and composite actions. WorldCycle drift obstacle. These two rewards force the model to learn reduces state returning drift by up to 44% and improves actions as consistent state operators rather than memorized composite-action accuracy nearly 4× over the baseline. temporal patterns. The same cycle-based objective also applies to composite actions. Although the base model may Related Work not reliably execute such action combinations, WorldCycle can directly optimize them through cycle closure rewards, Video World Models. Classical world models learn predicwithout requiring ground-truth video supervision. tive environment dynamics for control and planning, typiWe further introduce CycleBench, a benchmark suite covercally through compact internal or latent representations (Ha ing inverse, repeated, and composite cycles, enabling video and Schmidhuber 2018; Hafner et al. 2019; Schrittwieser IWMs to be evaluated as simulators rather than only as shortet al. 2020; Hafner et al. 2025). Recent video world models horizon generators. Because no existing benchmark diaginstead represent predicted environment evolution directly in noses state-returning consistency, existing evaluations ask pixel space, building on advances in video generation and viwhether the model follows actions correctly, not whether sual world modeling (Ho et al. 2022; Kondratyuk et al. 2024; its induced state-transition system is cycle-consistent (Ying Brooks et al. 2024; Alonso et al. 2024). Interactive systems et al. 2026; Ye et al. 2026). CycleBench fills this gap with such as Genie, GameNGen, Oasis, WorldPlay, and HY-World four task types and four scenario settings spanning short-, condition generation on user actions for longer-horizon or mid-, and long-horizon lengths as well as composite-action real-time interaction (Bruce et al. 2024; Valevski et al. 2024;

Decart and Etched 2024; Sun et al. 2026; HY-World Team 2026; Google DeepMind 2024, 2025; Zhang et al. 2025b; Gao et al. 2026). These models pursue fidelity along a single forward direction: each step should look right and follow the current action. Long-range coherence, when addressed, is treated as an appearance- or geometry-level property (Chen et al. 2025; Wu et al. 2025b; Yan et al. 2023; Po et al. 2025). None of these objectives can ask whether the rollout has reached the state the actions demand, because along an arbitrary open trajectory no reference state exists. Reversible and closed action trajectories are the exception: their composed transformation must be the identity, fixing the correct outcome analytically and without annotation. WorldCycle turns this structure into a trajectory-level training signal. Reinforcement Learning for World Model Post-training. RL-based post-training has produced strong gains in language models, from RLHF to GRPO (Ouyang et al. 2022; Shao et al. 2024; Guo et al. 2025), and the paradigm has been extended to diffusion models via DDPO, AlignProp, DiffusionDPO, Flow-GRPO, DanceGRPO, and DiffusionNFT (Black et al. 2024; Prabhudesai et al. 2023; Wallace et al. 2024; Liu et al. 2025; Xue et al. 2025; Zheng et al. 2025). For world models specifically, RLVR-World and RLIR apply RL post-training with verifiable or inversedynamics rewards (Wu et al. 2025a; Ye et al. 2025), and WorldCompass scores each generated clip against an inversepose action-following reward and a visual-quality preference model (Wang et al. 2026a). All existing methods supervise individual clips or short segments against an external scorer, placing no constraint on the trajectory as a whole. The correct long-horizon state is unknown for arbitrary action sequences, so trajectory-level error cannot be penalized directly. WorldCycle resolves this by constructing rewards whose target is fixed by the algebraic identity of reversible cycles, yielding a label-free objective that requires no external annotation.

Method Preliminaries Interactive Video World Models. We formulate an interactive video world model as a learned state-transition system Mθ = (S, A, fθ ), where fθ : S × A → S maps a state and an action to the next state, st+1 = fθ (st , at ). For a reversible action a, let Ta : S → S denote its true state transformation and T̂a the transformation realized by the learned model. Given a trajectory γ = (a1 , . . . , aT ), we define Tγ = TaT ◦ · · · ◦ Ta1 and define T̂γ analogously. Even a well-trained model realizes each action only up to a residual transformation, T̂a = Ta ◦ Ea with Ea ≈ I. Consequently, T̂γ = T̂aT ◦ · · · ◦ T̂a1 (1) = (TaT ◦ EaT ) ◦ · · · ◦ (Ta1 ◦ Ea1 ) ̸= Tγ . Because autoregressive predictions become the context for subsequent generation, these residual transformations generally do not cancel and instead compound with the rollout horizon. Directly measuring the resulting deviation would require access to the correct transformation Tγ , which is unknown for an arbitrary action sequence at training time. We refer to this as the verification bottleneck: long-horizon transition error exists, but no reference state is available against

which it can be directly scored, forcing existing post-training objectives to rely on locally verifiable proxies. Action-induced Transformation Group. The verification bottleneck admits a structural exception. We consider the subset of reversible interactions whose induced transformations form a group G = (T , ◦). Every action a admits an inverse a−1 , while the inverse trajectory γ −1 cancels the complete transformation induced by γ: Ta ◦ Ta−1 = I,

Tγ −1 ◦γ = I.

(2)

The identity target is determined entirely by the algebraic structure of the actions and is therefore independent of scene content, the initial state, and intermediate observations. Closed reversible trajectories consequently turn an otherwise unknown long-horizon target into an analytically known one, providing a content-independent and annotation-free verifier for accumulated transition error.

Self-Verifiable Reinforcement Learning WorldCycle converts this analytical identity constraint into a trajectory-level training signal. For a closed trajectory γ, recurrently executing the learned transition function, denoted by fθ (s, γ), should recover the state from which the trajectory started. We define state returning through the condition Tγ = I

=⇒

fθ (s, γ) = s,

∀s ∈ S.

(3)

This condition is stronger than local action following. A model may respond correctly to every individual action while its composed transformation T̂γ still deviates from identity because of accumulated residuals. We therefore construct two complementary objectives. The spatial objective densely verifies mirrored states within a reversible cycle, whereas the temporal objective verifies whether corresponding cycle phases remain stable across repeated executions. State Discrepancy Measurement. Because the underlying physical state s is inaccessible, we leverage dense point tracking within the image space as an observable proxy to measure state consistency. Given two frames oi and oj , we track N valid correspondence points and denote their image coordinates by pin , pjn . The frame-pair discrepancy is the size-normalized mean displacement N X 1 d(oi , oj ) = pi − pjn 2 , (4) N (H + W ) n=1 n where H, W are the image dimensions. Invalid or lowconfidence matches are excluded. Spatial Closure Reward. We construct a symmetric cycle consisting of a forward action sequence followed by its −1 exact inverse, γc = (a1 , . . . , am , a−1 m , . . . , a1 ), for which Tγc = I. The state reached after the first i forward actions should match the state reached at the mirrored phase 2m − i of the reverse trajectory. Rather than supervising only the final endpoint, we use every mirrored pair to provide dense evidence throughout the rollout: Rspatial =

m 1 X exp[−αd(oi , o2m−i )] . m + 1 i=0

(5)

Spatial Cycle

Input A bright low-poly fantasy garden plaza in a stylized 3D game environment……

…

Text Prompt

First Frame

Noise

… S …… …S

Cycle 1

Cycle k

DiffusionNFT Gaussian Noise 𝜺~𝑵(𝟎, 𝑰)

Forward

Calculate the Spatial Reward with CoTracker

Cycle Action Input

𝑹𝒔𝒑𝒂𝒕𝒊𝒂𝒍 New Policy 𝝅𝜽

Auto-Regressive Video Diffusion 𝝅𝒐𝒍𝒅

Backward

Old Policy 𝝅𝒐𝒍𝒅

Temporal Cycle

1st Spatial Cycle

𝑹𝒕𝒆𝒎𝒑𝒐𝒓𝒂𝒍

Calculate the Temporal Reward with CoTracker

||𝒗− − 𝒗 ||𝟐𝟐

||𝒗+ − 𝒗||𝟐𝟐

𝒓

𝟏−𝒓

… Combined Reward W

kth Spatial Cycle

𝒎𝒊𝒏𝜽

Figure 2: Overview of WorldCycle. From ordinary action sequences we construct closed reversible cycles and their repeated executions. The spatial closure reward compares every mirrored forward-reverse frame pair within a single cycle, providing dense supervision on where the trajectory begins to drift. The temporal consistency reward compares phase-aligned frames of the same action across repeated cycles, penalizing drift of identical actions over time. Both rewards are trajectory-level, annotation-free, and jointly optimized with a DiffusionNFT-style objective. Here oi denotes the generated observation at phase i, and has not been grounded. We therefore begin with a spatial-only d(·, ·) measures the discrepancy between two observations warm-up (t < tw ) before activating the temporal reward. that should correspond to the same underlying state. This (i) (i) (i) mirrored supervision localizes where a reversible trajectory Rcycle (t) = λs Rspatial + 1[t ≥ tw ] λt Rtemporal . (7) begins to drift, avoiding the sparse credit assignment produced by an endpoint-only reward. After tw , both rewards remain jointly active. A sequential Temporal State Consistency Reward. Spatial closure veristrategy that switches to temporal-only risks catastrophic fies state agreement within a single cycle, but does not guarforgetting of spatial closure (Kirkpatrick et al. 2017; Kaantee that the recovered state behaves consistently when the planis, Shanahan, and Clopath 2019; Yu et al. 2020). An cycle is executed again. A valid state-transition system should alternating strategy avoids forgetting but introduces gradient satisfy st = st′ ⇒ fθ (st , a) = fθ (st′ , a). Identical states conflict: each objective switch partially reverses the previous under identical actions must induce identical transitions. To optimization direction. Joint optimization resolves both isenforce this, we repeat the cyclic action program for K cysues simultaneously: spatial closure provides the geometric cles. Let oik denote the observation at step i of cycle k (where grounding on which temporal consistency depends, and the each cycle has length L). We anchor all subsequent cycles to two gradients reinforce rather than compete with each other. the first cycle and define the reward as: Multi-Scale Cycle Sampling. A fixed cycle length may inK L troduce a temporal-position shortcut, where the model memXX 1 orizes when to reverse rather than learning the inverse relaRtemporal = exp −αd(oik , oi1 ) . (K − 1)(L + 1) tion between actions. We therefore enforce cycle closure at k=2 i=0 multiple temporal scales. Given a rollout horizon H = n·2Q , (6) where n is the residual factor and 2Q denotes the largest Using the first cycle as a shared anchor avoids allowing the power-of-two factor of H, we construct hierarchical cycles reference state to drift through adjacent-cycle comparisons. by partitioning the rollout into 2k (k ∈ {0, . . . , Q − 1}) cyThe resulting reward verifies not only whether each cycle cles of length H/2k , each consisting of a forward segment closes, but also whether repeated returns produce consistent followed by its inverse. We randomly sample one hierarchy subsequent evolution over an extended horizon. level k during training, thus preventing reward hacking. Cycle-Consistent RL Training Reward Composition. The cycle reward is the trajectorylevel backbone of our objective, but on its own it underTrajectory-level cycle rewards require long rollouts, which constrains two ingredients that a video world model must still are computationally expensive and challenging for standard get right: local action fidelity and per-frame visual quality. RL algorithms to optimize. To make training practical, we To ensure the cycle reward is not satisfied through visually introduce a structured curriculum to prevent learning shortdegenerate shortcuts, we compose the final training reward cuts, alongside an efficient diffusion RL objective. from three complementary signals: Warm-up-and-Combine Schedule. Temporal consistency is unreliable if individual cycles do not close: phase-aligned (i) (i) (i) comparisons across cycles are uninterpretable when the cycle R(i) (t) = Rcycle (t) + λa Ract + λv RHPS , (8)

Metrics Action Accuracy Vision Quality Reverse-path Symmetry Repeated-Cycle Stability

Endpoint State Closure

Scenario Settings

Four Benchmark Tasks T1: Reversible Cycle

T2: Closed Cycle

T3: Repeated Cycle

T4: Cascaded Cycle

A D

Long Term

Mid Term

W A

D S

W A

Short Term

W + ↑

Composite Action

Figure 3: Overview of CycleBench. Four complementary tasks probe different aspects of accumulated transition error: T1 measures where error appears along a reversible path; T2 measures the residual after a non-symmetric closed trajectory; T3 tracks error growth across repeated cycles; T4 measures error propagation across structurally different cycles. All tasks are further evaluated under three horizon settings (short/mid/long) and one out-of-domain composite-action setting. where Ract is an action-following score (Wang et al. 2026a) rather than memorizing specific trajectories. Consequently, and RHPS is a visual-quality score (Ma et al. 2025). the learned dynamics generalize to unseen combinations of translations, rotations, and their compositions. Trajectory-Level Optimization with Efficient Rollouts. Cycle-consistency is a global property: rewards are comCycleBench puted over the complete rollout before optimization. The reward design is optimizer-agnostic and can be integrated CycleBench measures how transition errors accumulate into any diffusion RL framework supporting autoregresover extended autoregressive rollouts. For an arbitrary sive rollout. However, policy-gradient methods such as open trajectory, the correct future state is unavailable, so DanceGRPO (Xue et al. 2025) require timestep-wise loglong-horizon error cannot be scored directly. CycleBench probability tracking and often reduce rollout diversity under sidesteps this by constructing reversible and closed action long-horizon optimization (Wang et al. 2026a). We therefore trajectories whose composed transformation is analytically adopt a DiffusionNFT-style negative-aware objective (Zheng the identity: any discrepancy between rollout states that et al. 2025), which performs supervised regression toward should coincide is an observable estimate of accumulated reward-weighted positive and negative velocity directions: error. The benchmark consists of 47 action trajectories evaluated from 380 initial frames, organized as four tasks × (i) LRL = r(i) ∥vθ+ − v (i) ∥22 + (1 − r(i) )∥vθ− − v (i) ∥22 , (9) four settings, and provides correspondence-based metrics that localize where errors emerge and how they propagate. (i) (i) where r = Norm(R ) is the normalized cycle reward. Benchmark Tasks. Each task uses analytically known frame Following existing work (Wang et al. 2026a), we backpropacorrespondences within closed trajectories to probe a disgate only through the trailing chunk while keeping previous tinct aspect of error accumulation. T1 (Reversible-Cycle) chunks as frozen context to reduce memory cost for long rollexecutes a forward action sequence followed by its exact inouts, preserving trajectory-level supervision with a per-step verse and evaluates whether mirrored forward-reverse frames cost comparable to single-clip training. coincide, providing a dense path-wise probe. T2 (ClosedAnalysis on Composite Action Generalization. ComposCycle) executes a non-retracing closed path (e.g., a rectite actions lie outside the base model’s pretraining distriangle) whose net transformation is still identity, testing bution. WorldCycle addresses this directly: a composite acendpoint closure without symmetric replay. T3 (Repeatedtion sequence composed with its inverse still forms a closed Cycle) repeats the same closed cycle K times and tracks both cycle, so the cycle-closure reward provides annotation-free endpoint-error growth and phase-aligned drift across repetisupervision on composite trajectories without requiring any tions. T4 (Cascaded-Cycle) composes two structurally difground-truth video. The mechanism that makes this effective ferent closed cycles C (1) ◦ C (2) and asks whether residuals is transition residual reduction. Consider a composite trajecfrom the first cycle contaminate the second. tory γcomp = (c1 , . . . , cT ), where each ct may be a primitive Scenario Settings. Every task is evaluated under four setaction or a combination of multiple motion components. Its tings along two axes—rollout horizon and action complexlearned rollout can be written as ity: short-term (125 frames, basic in-domain case), midT̂γcomp = Tγcomp ◦ Eγcomp , (10) term (253 frames, tests error amplification under longer generation), long-term (381 frames, primary stress-test for where Eγcomp denotes the accumulated transition residual compounding error), and composite-action (125 frames along the trajectory. For a nominally closed composite trawith multi-component controls, out-of-domain w.r.t. prejectory, Tγcomp = I, and the final drift is therefore determined training). Matched trajectory structure across the three horiby the residual accumulation. By optimizing cycle closure, zons attributes differences to autoregressive accumulation WorldCycle reduces local transition errors and their proprather than task semantics. agation across action sequences, thereby encouraging the model to recover the underlying action composition algebra Metrics. To decouple evaluation from the CoTracker-based

Table 1: CycleBench results (weighted average over four benchmark tasks). Model size and venue are indicated after each baseline. WorldCompass and WorldCycle are both post-trained from WorldPlay; reported gains are relative to WorldCompass. Short-term (125 frames) Method

ESC↓

RPS↓

RCS↓

Acc↑

Mid-term (253 frames) Qual↑

Lingbot World v2 (14B) 0.198 0.143 0.085 0.464 12.11 WorldPlay (8B, ICML’26) 0.076 0.057 0.066 0.635 9.66 WorldCompass (ICML’26) 0.038 0.034 0.026 0.829 11.06 WorldCycle 0.026 ↓32% 0.019 ↓44% 0.020 ↓23% 0.833 +0.5% 11.24 +2%

ESC↓

RPS↓

RCS↓

Acc↑

Experiments Implementation Details Baselines. We compare WorldCycle against three publicly available interactive video world models. Lingbot World v2 (Gao et al. 2026) is a 14B-parameter open-source foundation model with the largest capacity among the compared systems. WorldPlay (8B, ICML 2026) (Sun et al. 2026) and WorldCompass (ICML 2026) (Wang et al. 2026a) are the strongest smaller-scale baselines for action-conditioned generation; WorldCompass is the closest prior work, applying reinforcement-learning post-training to WorldPlay using perclip inverse-pose and visual-quality rewards. Both WorldCompass and our WorldCycle are initialized from WorldPlay, isolating the effect of the post-training objective. Evaluation. All methods are evaluated on CycleBench (380 videos) under four task types (T1–T4) and four scenario settings: short-term (125 frames), mid-term (253 frames), longterm (381 frames), and composite-action (125 frames). Primary metrics are ESC↓, RPS↓, RCS↓, and Action Accuracy↑; HPSv3↑ quality is reported to verify that cycle training does not degrade visual fidelity. Training Data and Setup. We post-train the 8B WorldPlay AR checkpoint on approximately 4,000 image–caption pairs from real-world scenes. Each training instance samples a base or combined action sequence and constructs closed reversible cycles and repeated executions on the fly, so no cycle-specific supervision is annotated. Rollouts are chunked into 16-frame video clips as in the base model. We use rollout group size G = 16, up to K = 8 repeated cycles per hierarchical level,

Acc↑

Qual↑

0.480 0.695 0.868 0.878 +1%

10.58 9.00 10.67 10.81 +1%

Long-term (381 frames) Qual↑

Lingbot World v2 (14B) 0.204 0.143 0.102 0.414 10.81 WorldPlay (8B, ICML’26) 0.119 0.088 0.057 0.136 8.98 WorldCompass (ICML’26) 0.073 0.059 0.040 0.499 10.12 WorldCycle 0.057 ↓22% 0.042 ↓29% 0.026 ↓35% 0.553 +11% 10.42 +3%

reward used during post-training (Karaev et al. 2024), CycleBench adopts RoMa (Edstedt et al. 2024) as an independent dense correspondence estimator. For any two frames that should coincide, the mean 2D displacement of valid RoMa pixel matches, measured in pixels, defines the framepair distance dRoMa (Iˆi , Iˆj ); pairs with too few matches are assigned the image diagonal. Three complementary metrics are reported: ESC↓ (Endpoint State Closure), the distance between the generated endpoint and the initial frame; RPS↓ (Reverse-Path Symmetry), the average distance between mirrored forward-reverse pairs; and RCS↓ (Repeated-Cycle Stability), the average phase-aligned distance across cycles.

RCS↓

0.227 0.233 0.062 0.092 0.065 0.053 0.048 0.039 0.025 0.038 ↓21% 0.028 ↓28% 0.018 ↓28%

Composite-action (125 frames) Method

RPS↓

ESC↓

RPS↓

RCS↓

Acc↑

Qual↑

0.221 0.186 0.123 0.033 11.73 0.302 0.202 0.173 0.073 4.71 0.187 0.104 0.074 0.084 9.81 0.163 ↓13% 0.087 ↓16% 0.049 ↓34% 0.095 +13% 10.31 +5%

and sample one hierarchy level per training step to keep the per-step rollout cost fixed. To optimize the model for longhorizon generation, we set the maximum generation length to 16 clips. The spatial warm-up runs for tw = 300 steps, after which the temporal reward is activated. The reward weights are set to λs = 1.0, λt = 0.5, λv = 1.0 and λa = 2.0. Optimization uses AdamW with learning rate 1×10−5 . The training process runs 3 days on 8 H200 GPUs.

Qualitative and Quantitative Results Table 1 reports results across all CycleBench tasks and settings. We break down the key findings into three distinct aspects of model performance: Scaling Alone Does Not Solve State Consistency: Lingbot World v2 (14B) achieves the highest HPSv3 quality due to its larger capacity, but lags every RL post-trained baseline on cycle-consistency metrics (ESC, RPS, RCS are 1.4–8.3× worse than WorldCycle), confirming that state-returning consistency is not solved by scaling alone. Superiority Across Extended Horizons: WorldCycle consistently outperforms both WorldPlay and WorldCompass. On short-term simple actions it reduces ESC by 32% and RPS by 44% over WorldCompass while maintaining comparable action accuracy (0.833 vs. 0.829), confirming that cycle-consistency gains do not come at the cost of local interaction following. The gains persist as the horizon extends: at 253 frames ESC drops by 21% and RPS by 28%; at 381 frames RCS falls by 34%, the largest relative gain across horizon settings, reflecting the direct contribution of the temporal consistency reward to long-range stability. Strong Generalization to Out-of-Domain Composite Actions: On composite actions, where the base WorldPlay’s accuracy collapses to 0.136, WorldCycle achieves an accuracy of 0.553 (a 4× improvement over the base and +11% over WorldCompass), confirming that cycle supervision generalizes effectively to unseen action compositions. Figure 4 provides qualitative confirmation. While baselines generate locally plausible frames, they accumulate visible drift upon returning to the initial state: colors shift, geometric layouts distort, and structural details diverge. This degradation is particularly severe in the out-of-domain compositeaction setting (left). WorldCycle recovers states substantially

253 frames

253 frames WorldPlay

W →

S ←

LingBot World v2

W →

S ←

WorldCompass

W →

S ←

Ours

W →

S ←

First Frame

WorldPlay

LingBot World v2

WorldCompass

Ours

WorldPlay

First Frame

LingBot World v2

WorldCompass

Ours

Post-Trained Model

Base Model

Figure 4: Qualitative comparison on CycleBench (253 frames). Left: composite-action setting (forward+turn then inverse). Right: simple-action setting (forward then backward). Each method row contains eight sampled frames in total, with four keyframes from each setting: two from the outward phase and two from the return phase. The bottom strip shows a cropped zoom of the return frame alongside the first-frame reference for each method. Table 2: Ablation on CycleBench. Left: basic actions across three horizons (short / mid / long). Right: OOD composite actions reported with the full metric set. Reward isolates each cycle signal; Schedule isolates the warm-up-and-combine training order. Short-term

Mid-term

Long-term

Composite-action (OOD)

Variant

ESC↓

Acc↑

ESC↓

Acc↑

ESC↓

Acc↑

ESC↓

RPS↓

RCS↓

Acc↑

Qual↑

Base model

0.076

0.635

0.092

0.695

0.302

0.073

0.119

0.088

0.057

0.136

8.98

Reward ablation Spatial only Temporal only

0.033 0.059

0.712 0.651

0.050 0.078

0.744 0.712

0.212 0.268

0.081 0.076

0.071 0.098

0.054 0.076

0.038 0.048

0.473 0.378

9.77 9.20

Schedule ablation Sequential (S→T) Joint from scratch

0.028 0.030

0.797 0.769

0.041 0.044

0.845 0.821

0.176 0.180

0.088 0.085

0.061 0.065

0.045 0.052

0.030 0.036

0.529 0.510

10.21 10.04

Full (Ours)

0.026

0.833

0.038

0.878

0.163

0.095

0.057

0.042

0.026

0.553

10.42

closer to the starting view in both settings, as the bottom crop strips make this visually unambiguous.

Ablation study We analyze component contributions in Table 2. Each reward addresses a distinct failure mode. Removing the temporal signal primarily degrades long-range stability and composite-action generalization (e.g., long-term ESC increases from 0.163 to 0.212). Conversely, removing the spatial signal causes broader failure across all settings, because temporal supervision relies on individual cycles approximately closing; without the spatial anchor, phase-aligned comparisons become meaningless. Warm-up-and-combine is essential. The schedule ablation confirms that the ordering of the two rewards is non-trivial. Joint-from-scratch training consistently underperforms our warm-up schedule: the temporal reward has no meaningful reference before individual cycles are grounded, so its gradient adds noise rather than signal. Sequential training (spatialonly then temporal-only) recovers some performance but ultimately drifts because it discards the spatial constraint during the temporal phase. Our warm-up-and-combine strategy

retains the spatial anchor throughout optimization, yielding the best results across all settings. Training does not harm visual quality. Table 2 shows that the full WorldCycle model achieves the highest HPSv3 score (10.42) among all variants, a 16% gain over the base model (8.98), ruling out the hypothesis that cycle-consistency improvements are purchased at the cost of visual fidelity.

Conclusion We presented WorldCycle, a self-verifiable RL framework that turns reversible action cycles into dense, annotation-free supervision for long-horizon video world models. Spatial closure and temporal state consistency rewards jointly drive the model to learn actions as consistent state operators, overcoming reward sparsity and temporal drift while generalizing to composite action cycles outside the base model’s pretraining distribution. We further introduced CycleBench, the first benchmark evaluating video world models as state-transition simulators. Extending the cycle-return principle to approximate reversibility and other action modalities such as robotic manipulation are natural directions for future work.

References Alonso, E.; Jelley, A.; Micheli, V.; Kanervisto, A.; Storkey, A.; Pearce, T.; and Fleuret, F. 2024. Diffusion for World Modeling: Visual Details Matter in Atari. In Advances in Neural Information Processing Systems. Bengio, S.; Vinyals, O.; Jaitly, N.; and Shazeer, N. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In Advances in Neural Information Processing Systems, volume 28. Black, K.; Janner, M.; Du, Y.; Kostrikov, I.; and Levine, S. 2024. Training Diffusion Models with Reinforcement Learning. In International Conference on Learning Representations. Brooks, T.; Peebles, B.; Holmes, C.; DePue, W.; Guo, Y.; Jing, L.; Schnurr, D.; Taylor, J.; Luhman, T.; Luhman, E.; Ng, C.; Wang, R.; and Ramesh, A. 2024. Video Generation Models as World Simulators. OpenAI technical report. Bruce, J.; Dennis, M.; Edwards, A.; Parker-Holder, J.; Shi, Y.; Hughes, E.; Lai, M.; Mavalankar, A.; Steigerwald, R.; Apps, C.; Aytar, Y.; Bechtle, S.; Behbahani, F.; Chan, S. C. Y.; Heess, N.; Gonzalez, L.; Osindero, S.; Ozair, S.; Reed, S.; Zhang, J.; Zolna, K.; Clune, J.; de Freitas, N.; Singh, S.; and Rocktäschel, T. 2024. Genie: Generative Interactive Environments. In Proceedings of the 41st International Conference on Machine Learning. Chen, T.; Hu, X.; Ding, Z.; and Jin, C. 2025. Learning World Models for Interactive Video Generation. arXiv preprint arXiv:2505.21996. Decart; and Etched. 2024. Oasis: A Universe in a Transformer. Project website. Edstedt, J.; Sun, Q.; Bökman, G.; Wadenbäck, M.; and Felsberg, M. 2024. RoMa: Robust Dense Feature Matching. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19790–19800. Gao, S.; Yang, J.; Chen, L.; Chitta, K.; Qiu, Y.; Geiger, A.; Zhang, J.; and Li, H. 2024. Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability. arXiv preprint arXiv:2405.17398. Gao, Z.; Wang, Q.; Zhu, J.; Chen, J.; Liu, Z.; Bai, Q.; Wang, J.; Yuan, Y.; Wang, H.; Lu, Y.; Cheng, K. L.; Zhang, H.; Gao, J.; Feng, T.; Liu, Y.; Yao, Y.; Xu, Y.; Zhu, X.; Shen, Y.; and Ouyang, H. 2026. Infinite Worlds with Versatile Interactions. arXiv preprint arXiv:2607.07534. Google DeepMind. 2024. Genie 2: A Large-Scale Foundation World Model. Google DeepMind Blog. Google DeepMind. 2025. Genie 3: A New Frontier for World Models. Google DeepMind Blog. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948. Ha, D.; and Schmidhuber, J. 2018. World models. arXiv preprint arXiv:1803.10122. Hafner, D.; Lillicrap, T.; Fischer, I.; Villegas, R.; Ha, D.; Lee, H.; and Davidson, J. 2019. Learning Latent Dynamics for Planning from Pixels. In Proceedings of the 36th

International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 2555–2565. PMLR. Hafner, D.; Pasukonis, J.; Ba, J.; and Lillicrap, T. 2025. Mastering diverse control tasks through world models. Nature, 640(8059): 647–653. Ho, J.; Salimans, T.; Gritsenko, A. A.; Chan, W.; Norouzi, M.; and Fleet, D. J. 2022. Video Diffusion Models. In Advances in Neural Information Processing Systems, volume 35. Hu, A.; Russell, L.; Yeo, H.; Murez, Z.; Fedoseev, G.; Kendall, A.; Shotton, J.; and Corrado, G. 2023. GAIA-1: A Generative World Model for Autonomous Driving. arXiv preprint arXiv:2309.17080. Huang, J.; Ye, Z.; Hu, X.; He, T.; Zhang, G.; Shi, S.; Bian, J.; and Jiang, L. 2026. LIVE: Long-Horizon Interactive Video World Modeling. In Proceedings of the 43rd International Conference on Machine Learning. ArXiv preprint arXiv:2602.03747. Huang, Z.; He, Y.; Yu, J.; Zhang, F.; Si, C.; Jiang, Y.; Zhang, Y.; Wu, T.; Jin, Q.; Chanpaisit, N.; et al. 2024. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 21807–21818. HY-World Team. 2026. HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds. arXiv preprint arXiv:2604.14268. Janner, M.; Fu, J.; Zhang, M.; and Levine, S. 2019. When to Trust Your Model: Model-Based Policy Optimization. In Advances in Neural Information Processing Systems, volume 32. Kaplanis, C.; Shanahan, M.; and Clopath, C. 2019. Policy Consolidation for Continual Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 3242–3251. PMLR. Karaev, N.; Rocco, I.; Graham, B.; Neverova, N.; Vedaldi, A.; and Rupprecht, C. 2024. CoTracker: It Is Better to Track Together. In European Conference on Computer Vision, volume 15120 of Lecture Notes in Computer Science, 18–35. Springer. Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R. 2017. Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences, 114(13): 3521–3526. Kondratyuk, D.; Yu, L.; Gu, X.; Lezama, J.; Huang, J.; Schindler, G.; Hornung, R.; Birodkar, V.; Yan, J.; Chiu, M.C.; Somandepalli, K.; Akbari, H.; Alon, Y.; Cheng, Y.; Dillon, J.; Gupta, A.; Hahn, M.; Hauth, A.; Hendon, D.; Martinez, A.; Minnen, D.; Sirotenko, M.; Sohn, K.; Yang, X.; Adam, H.; Yang, M.-H.; Essa, I.; Wang, H.; Ross, D. A.; Seybold, B.; and Jiang, L. 2024. VideoPoet: A Large Language Model for Zero-Shot Video Generation. In Proceedings of the 41st International Conference on Machine Learning.

Liu, J.; Liu, G.; Liang, J.; Li, Y.; Liu, J.; Wang, X.; Wan, P.; Zhang, D.; and Ouyang, W. 2025. Flow-GRPO: Training Flow Matching Models via Online RL. arXiv preprint arXiv:2505.05470. Ma, Y.; Shui, Y.; Wu, X.; Sun, K.; and Li, H. 2025. HPSv3: Towards Wide-Spectrum Human Preference Score. arXiv:2508.03789. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems. Po, R.; Nitzan, Y.; Zhang, R.; Chen, B.; Dao, T.; Shechtman, E.; Wetzstein, G.; and Huang, X. 2025. Long-Context StateSpace Video World Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 8733–8744. Prabhudesai, M.; Goyal, A.; Pathak, D.; and Fragkiadaki, K. 2023. Aligning Text-to-Image Diffusion Models with Reward Backpropagation. arXiv preprint arXiv:2310.03739. Schrittwieser, J.; Antonoglou, I.; Hubert, T.; Simonyan, K.; Sifre, L.; Schmitt, S.; Guez, A.; Lockhart, E.; Hassabis, D.; Graepel, T.; Lillicrap, T.; and Silver, D. 2020. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature, 588: 604–609. Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300. Su, Z.; Hu, T.; Zhang, J.; Wang, R.; Yi, R.; Ma, L.; and Tao, D. 2026. Cycle-World: Mitigating Error Accumulation in Long-Term Video World Models via Reverse-Prediction Cycle Consistency. arXiv preprint arXiv:2607.11836. Sun, W.; Zhang, H.; Wang, H.; Wu, J.; Wang, Z.; Wang, Z.; Wang, Y.; Zhang, J.; Wang, T.; and Guo, C. 2026. WorldPlay: Towards Long-Term Geometric Consistency for RealTime Interactive World Modeling. In Proceedings of the 43rd International Conference on Machine Learning. ArXiv preprint arXiv:2512.14614. Valevski, D.; Leviathan, Y.; Arar, M.; and Fruchter, S. 2024. Diffusion Models Are Real-Time Game Engines. arXiv preprint arXiv:2408.14837. Wallace, B.; Dang, M.; Rafailov, R.; Zhou, L.; Lou, A.; Purushwalkam, S.; Ermon, S.; Xiong, C.; Joty, S.; and Naik, N. 2024. Diffusion Model Alignment Using Direct Preference Optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. Wang, Y.; He, J.; Fan, L.; Li, H.; Chen, Y.; and Zhang, Z. 2024. Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14749–14759. Wang, Z.; Wang, T.; Zhang, H.; Zuo, X.; Wu, J.; Wang, H.; Sun, W.; Wang, Z.; Cao, C.; Zhao, H.; Guo, C.; and

Zhao, Z. 2026a. WorldCompass: Reinforcement Learning for Long-Horizon World Models. In Proceedings of the 43rd International Conference on Machine Learning. Wang, Z.; Zhang, W.; Zhang, W.; Zhang, F.; Tan, X.; Qin, Y.; and Li, G. 2026b. World Models as Group Actions. arXiv preprint arXiv:2605.24578. Wu, J.; Yin, S.; Feng, N.; He, X.; Li, D.; Hao, J.; and Long, M. 2024. iVideoGPT: Interactive VideoGPTs Are Scalable World Models. In Advances in Neural Information Processing Systems, volume 37. Wu, J.; Yin, S.; Feng, N.; and Long, M. 2025a. RLVR-World: Training World Models with Reinforcement Learning. In Advances in Neural Information Processing Systems. Wu, T.; Yang, S.; Po, R.; Xu, Y.; Liu, Z.; Lin, D.; and Wetzstein, G. 2025b. Video World Models with Long-Term Spatial Memory. arXiv preprint arXiv:2506.05284. Xiang, J.; Liu, G.; Gu, Y.; Gao, Q.; Ning, Y.; Zha, Y.; Feng, Z.; Tao, T.; Hao, S.; Shi, Y.; Liu, Z.; Xing, E. P.; and Hu, Z. 2024. Pandora: Towards General World Model with Natural Language Actions and Video States. arXiv preprint arXiv:2406.09455. Xue, Z.; Wu, J.; Gao, Y.; Kong, F.; Zhu, L.; Chen, M.; Liu, Z.; Liu, W.; Guo, Q.; Huang, W.; et al. 2025. DanceGRPO: Unleashing GRPO on Visual Generation. arXiv preprint arXiv:2505.07818. Yan, W.; Hafner, D.; James, S.; and Abbeel, P. 2023. Temporally Consistent Transformers for Video Generation. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 39062–39098. PMLR. Yang, S.; Du, Y.; Ghasemipour, K.; Tompson, J.; Kaelbling, L.; Schuurmans, D.; and Abbeel, P. 2024. Learning Interactive Real-World Simulators. In International Conference on Learning Representations. Ye, Y.; He, T.; Yang, S.; and Bian, J. 2025. Reinforcement Learning with Inverse Rewards for World Model PostTraining. arXiv preprint arXiv:2509.23958. Ye, Y.; Lu, X.; Jiang, Y.; Gu, Y.; Zhao, R.; Liang, Q.; Pan, J.; Zhang, F.; Wu, W.; and Wang, A. J. 2026. MIND: Benchmarking Memory Consistency and Action Control in World Models. arXiv:2602.08025. Ying, K.; Hu, H.; Ren, S.; Li, J.; Chen, F.; Wang, Z.; Cao, X.; Cai, X.; and Ding, H. 2026. WBench: A Comprehensive Multi-Turn Benchmark for Interactive Video World Model Evaluation. arXiv preprint arXiv:2605.25874. Yu, T.; Kumar, S.; Gupta, A.; Levine, S.; Hausman, K.; and Finn, C. 2020. Gradient Surgery for Multi-Task Learning. In Advances in Neural Information Processing Systems, volume 33, 21326–21337. Curran Associates, Inc. Zhang, K.; Tang, Z.; Hu, X.; Pan, X.; Guo, X.; Liu, Y.; Huang, J.; Yuan, L.; Zhang, Q.; Long, X.-X.; Cao, X.; and Yin, W. 2025a. Epona: Autoregressive Diffusion World Model for Autonomous Driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 27220–27230. Zhang, Y.; Peng, C.; Wang, B.; Wang, P.; Zhu, Q.; Kang, F.; Jiang, B.; Gao, Z.; Li, E.; Liu, Y.; and Zhou, Y. 2025b.

Matrix-Game: Interactive World Foundation Model. arXiv preprint arXiv:2506.18701. Zheng, K.; Chen, H.; Ye, H.; Wang, H.; Zhang, Q.; Jiang, K.; Su, H.; Ermon, S.; Zhu, J.; and Liu, M.-Y. 2025. DiffusionNFT: Online Diffusion Reinforcement with Forward Process. arXiv preprint arXiv:2509.16117. Zhu, F.; Wu, H.; Guo, S.; Liu, Y.; Cheang, C.; and Kong, T. 2025. IRASim: A Fine-Grained World Model for Robot Manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 9834–9844. Zhu, Z.; Huang, K.-R.; Xu, Z.; Li, R.; Wu, B.; Bai, R.; Wu, M.; Paul, S.; and Tu, Z. 2026. 4KLSDB: A LargeScale Dataset for 4K Image Restoration and Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 3052–3061.

Benchmark Details Benchmark Overview CycleBench measures how transition errors accumulate over extended autoregressive rollouts. For an arbitrary open action trajectory, the correct future state is unavailable, so long-horizon error cannot be scored directly. CycleBench sidesteps this verification bottleneck by constructing reversible and closed action trajectories whose composed transformation is analytically the identity. Any discrepancy between rollout states that should coincide therefore provides an observable estimate of accumulated transition error. The benchmark consists of 47 action trajectories evaluated from 380 initial frames, organized as four benchmark tasks and four scenario settings. The four tasks probe where transition error emerges, how much residual remains at closure, how it grows across repeated executions, and whether it propagates from one closed cycle to the next. The four settings vary rollout horizon and action complexity, covering shortterm, mid-term, long-term, and composite-action conditions. Initial-Frame Collection. CycleBench contains 380 initial frames from two complementary sources. We select 236 images from 4KLSDB (Zhu et al. 2026), which predominantly cover realistic, photographic scenes. To increase the diversity of scene content and visual styles, we additionally synthesize 144 images using GPT Image 2. Together, the collected and generated images provide a balanced set of realistic and synthetic environments for evaluating world-model rollouts.

Benchmark Tasks Each task uses analytically known frame correspondences within closed trajectories to probe a distinct aspect of accumulated transition error. T1: Reversible-Cycle. T1 executes a forward action sequence H = (a1 , . . . , aK ), followed by its exact inverse −1 H −1 = (a−1 K , . . . , a1 ). Under ideal dynamics, the complete forward–inverse trajectory induces the identity transformation. Moreover, each state reached during the forward segment should coincide with its mirrored state during the inverse segment. T1 therefore evaluates whether mirrored forward–reverse frames coincide, providing a dense pathwise probe that localizes where transition error first appears and how it develops along the rollout. T2: Closed-Cycle. T2 executes a non-retracing closed action sequence C, such as a rectangular or polygonal path. Although the trajectory does not symmetrically replay its earlier path, its ideal net transformation is still the identity, so the generated endpoint should coincide with the initial observation. T2 therefore tests endpoint closure after a structurally closed but non-symmetric trajectory, preventing success through simple reverse-path replay. T3: Repeated-Cycle. T3 repeats the same closed cycle C for K executions. If one cycle has length L, the endpoints at times L, 2L, . . . , KL should all coincide with the initial observation, while phase-aligned states across executions should remain consistent. T3 tracks both endpoint-error growth and phase-aligned drift, directly measuring whether small residuals are amplified as the rollout horizon increases.

T4: Cascaded-Cycle. T4 sequentially executes two structurally different closed cycles, C (1) and C (2) , where the second begins from the generated endpoint of the first: (1)

(2)

C C I0 −−−→ IˆT1 −−−→ IˆT1 +T2 .

Because both cycles ideally induce identity transformations, the intermediate and final endpoints should each coincide with I0 . T4 asks whether the residual produced by the first cycle contaminates the second, thereby probing cross-cycle error propagation under different action compositions.

Scenario Settings Every task is evaluated under four settings along two complementary axes: rollout horizon and action complexity. The short-, mid-, and long-term settings preserve matched trajectory structures while progressively extending the rollout, so differences can be attributed primarily to autoregressive error accumulation rather than to different task semantics. The composite-action setting instead tests more complex, multi-component controls. Short-term. The short-term setting contains 125-frame rollouts and serves as the basic in-domain case for measuring where transition error first emerges and how much residual remains after a relatively short reversible or closed trajectory. Mid-term. The mid-term setting extends the rollout to 253 frames while preserving the corresponding trajectory structure. It tests whether residuals observed at short horizons are amplified under longer autoregressive generation. Long-term. The long-term setting further extends the rollout to 381 frames and serves as the primary stress test for compounding error, including endpoint residuals, reversepath discrepancies, and repeated-cycle drift. Composite-action. The composite-action setting uses 125-frame rollouts with controls that combine multiple motion components, such as simultaneous translation and rotation. These controls are out of domain with respect to the base model’s pretraining distribution and test whether cycle consistency generalizes to unseen action compositions rather than only to primitive controls.

RoMa Correspondence Distance To decouple evaluation from the CoTracker-based correspondence reward used during post-training, CycleBench adopts RoMa as an independent dense correspondence estimator. Given two generated frames Iˆi and Iˆj , RoMa produces corNij (i) (j) (i) (j) respondences {(pk , pk , ck )}k=1 , where pk , pk ∈ R2 are matched pixel coordinates and ck is the associated confidence. We retain the valid matches Vij = {k | ck ≥ τmatch } and define the frame-pair distance as 1 X (i) (j) dRoMa (Iˆi , Iˆj ) = pk − pk . (11) |Vij | 2 k∈Vij

For frame pairs that should correspond to the same underlying state, this distance provides an observable estimate of the

Algorithm 1: WorldCycle Training Process Input: World model πθ , EMA reference πθold , training data D, level weights ω, group size G, repeat count K, warm-up threshold tw , weights λs , λt , λa , λv Output: Optimized world model πθ for training step t = 1, 2, . . . do /* Multi-scale cycle sampling: one hierarchy level per step */ k∼ω for (o0 , a1:T ) ∈ D do /* Construct closed cycle at hierarchy level k */ γc ← (Bk ∥ Bk−1 ); ΓK ← (γc , . . . , γc ) ;

// K repetitions

/* Group rollout from EMA reference (G independent samples) */ {τ (i) }G i=1 ∼ πθold (o0 , ΓK ) /* Compute per-rollout rewards */ for i = 1, . . . , G do (i) Rs ← SpatialClosure(τ (i) ) ; (i) Rt ← 1[t ≥ tw ] · TemporalStationarity(τ (i) ) ; (i) (i) (i) (i) (i) R ← λs Rs + λt Rt + λa Ract + λv RHPS end

// Dense mirrored-pair verification // Activated after spatial warm-up

/* Group-normalize rewards and optimize (DiffusionNFT) */ r(i) ← Norm {R(i) }G // (R(i) − µ)/σ within group i=1 ; G i 1 Xh (i) + 2 2 r vθ −v (i) 2 + (1−r(i) ) vθ− −v (i) 2 LRL ← G i=1 θ ← AdamW(θ, ∇θ LRL );

θold ← EMA(θold , θ) ;

// Update model and reference

end end

accumulated transition residual. A smaller value indicates better alignment between the matched visual structures. Because all evaluated videos share the same resolution, CycleBench reports the displacement directly in pixels. To prevent failed matching from being interpreted as low error, frame pairs with fewer than Nmin valid correspondences are √ assigned the maximum image-space distance Dimg = H 2 + W 2 , where H and W denote the frame height and width.

Evaluation Metrics CycleBench reports three complementary correspondencebased metrics, ordered as in the main results table: ESC measures the final closure residual, RPS measures discrepancies along mirrored forward–reverse paths, and RCS measures phase-aligned drift across cycle executions. Lower values indicate better state-returning consistency.

denote the set of non-trivial mirrored pairs. We define X 1 (ξ) LRP S = sym dRoMa (Iˆi , Iˆj ). (13) |Pξ | sym (i,j)∈Pξ

Unlike an endpoint-only measure, RPS provides a dense path-wise diagnostic of where residual error emerges during reversible generation. Repeated-Cycle Stability (RCS). RCS is the average distance between phase-aligned states across cycle executions. For T3, we anchor all later executions to the first cycle. If each cycle has length L, the correspondence set is Pξcyc = {(ℓ, (r − 1)L + ℓ) | r = 2, . . . , K, ℓ = 0, . . . , L}. The metric is (ξ)

LRCS =

ˆ ˆ

dRoMa (Ii , Ij ). |Pξcyc | (i,j)∈P cyc

(14)

Endpoint State Closure (ESC). ESC is the distance between the generated endpoint and the initial frame: (ξ) LESC = dRoMa (I0 , IˆT ).

(12)

It summarizes the total residual remaining after the complete closed trajectory as a single global closure error. Reverse-Path Symmetry (RPS). RPS is the average distance between mirrored forward–reverse frame pairs. For a standard T1 trajectory ξ of length T = 2K, let Pξsym = {(t, T − t) | 1 ≤ t < K}

For T4, the same principle is applied to analytically coincident closure states at the cycle handoff and after the second cycle. A low RCS indicates that repeated or cascaded cycle execution does not progressively amplify transition residuals. Auxiliary Metrics. In addition to the three stateconsistency metrics, CycleBench reports Action Accuracy, using the action-following evaluator adopted by WorldCompass, and HPSv3 visual quality. These auxiliary metrics verify that improvements in cycle consistency do not come at the cost of local action following or visual fidelity.

Table 3: VBench video quality comparison across four settings. All six dimensions are higher-is-better (↑). Results are weighted averages over T1–T4 tasks within each setting (380 videos total). WorldCycle consistently matches or exceeds WorldCompass and improves over WorldPlay across all settings. Short-term (125 frames)

Mid-term (253 frames)

Method

Aesth.↑ S.Cons↑ B.Cons↑ Img↑ T.Flick↑ Mot↑

WorldPlay (8B, ICML’26) WorldCompass (ICML’26) WorldCycle

0.653 0.676 0.676

0.950 0.963 0.968

0.948 0.953 0.955

0.744 0.770 0.773

0.956 0.946 0.945

0.985 0.983 0.985

0.631 0.665 0.665

0.930 0.954 0.961

0.939 0.948 0.948

0.723 0.772 0.771

0.955 0.944 0.943

0.983 0.983 0.984

Composite-action (125 frames)

Long-term (381 frames)

Method

Aesth.↑ S.Cons↑ B.Cons↑ Img↑ T.Flick↑ Mot↑

WorldPlay (8B, ICML’26) WorldCompass (ICML’26) WorldCycle

0.637 0.666 0.672

+ WorldCycle

0.948 0.960 0.966

0.943 0.948 0.948

0.733 0.762 0.768

Pre-train

+ WorldCycle

Figure 5: Evolution of cycle consistency and visual quality metrics during the RL training of WorldCycle. These metrics are evaluated on a fixed CycleBench validation subset at every training step.

0.960 0.943 0.944

0.985 0.981 0.984

0.579 0.665 0.697

0.818 0.871 0.907

0.891 0.921 0.925

0.572 0.707 0.729

0.980 0.958 0.961

0.989 0.984 0.988

while, HPSv3 increases and remains above the WorldPlay initialization after convergence. These trajectories indicate that the combined objective provides stable optimization of long-horizon consistency without sacrificing visual quality. Long-Horizon Qualitative Stability. Figure 6 provides a qualitative analysis under the 381-frame long-term setting, in which the model executes five repeated cycles. The baselines exhibit distinct forms of accumulated visual error. LingBot World v2 undergoes pronounced color drift, with the sky becoming progressively oversaturated and unnaturally blue. WorldPlay and WorldCompass instead develop pronounced visual artifacts in the sky region, including whitening and unnatural linear patterns in the final return frame. In contrast, WorldCycle maintains stable colors, textures, and scene content throughout the rollout, producing a final return frame that remains closely aligned with the initial observation. These results demonstrate that the cycle-consistency rewards effectively suppress the accumulation of both color drift and visual artifacts over repeated long-horizon transitions.

WorldCycle Training Algorithm

VBench Video Quality Evaluation

Algorithm 1 summarizes the complete training procedure of WorldCycle. At each training step, we sample a hierarchy level and construct a closed action cycle by concatenating an action block with its inverse, which is then repeated to form a long-horizon trajectory. The EMA reference model generates a group of independent rollouts, from which we compute the spatial closure, temporal consistency, actionfollowing, and visual-quality rewards. The temporal reward is activated only after the spatial warm-up stage, while both cycle rewards remain jointly active thereafter. The groupnormalized rewards are finally used in a DiffusionNFT-style objective to update the world model, followed by an EMA update of the reference model.

To verify that WorldCycle’s training does not degrade perceptual video quality, we evaluate all methods on six VBench dimensions (Huang et al. 2024): Aesthetic Quality, Subject Consistency, Background Consistency, Imaging Quality, Temporal Flickering, and Motion Smoothness. These dimensions are computed independently of the CycleBench reward, providing a complementary quality signal. Lingbot-World-2 VBench scores are omitted here due to limited sample size (12 videos); overall VBench scores for Lingbot-World-2 are: Aesthetic 0.743, Subj. Cons. 0.882, Bg. Cons. 0.923, Imaging 0.746, Temp. Flick. 0.961, Motion 0.979. WorldCycle matches or exceeds WorldCompass on nearly all dimensions across all four settings, and consistently outperforms the base WorldPlay model. Subject Consistency and Imaging Quality show the largest gains over WorldPlay across all settings, consistent with the observation that reducing transition residuals also stabilizes frame-level appearance. The improvements are sustained in the compositeaction setting, confirming that cycle training on out-of-

Additional Analysis Details Training Dynamics. Figure 5 further examines the optimization process when WorldCycle is initialized from WorldPlay. All three cycle-consistency metrics decrease steadily during post-training and converge below their initial values. Mean-

381 frames (5 cycles) WorldPlay

LingBot World v2

WorldCompass

Ours

First Frame

WorldPlay

LingBot World v2

Base Model

WorldCompass

Ours

Post-Trained Model

Figure 6: Long-horizon qualitative comparison over five repeated cycles (381 frames). Each row shows sampled frames from the same long-term rollout. The bottom row enlarges the red-boxed region in the final return frame and compares it with the corresponding region in the first frame. LingBot World v2 exhibits severe color drift, most visibly through the increasingly saturated blue sky, whereas WorldPlay and WorldCompass develop pronounced visual artifacts, including whitening in the sky and unnatural linear patterns around the rooftop structure. In contrast, WorldCycle preserves stable colors, textures, and scene content, yielding a final return frame that remains close to the initial observation after all five cycles. distribution action sequences does not introduce quality degradation.

Discussion and Limitations WorldCycle does not treat reversible action cycles merely as a specialized state-returning task. Instead, it uses them as a controlled mechanism for exposing and correcting long-horizon transition errors that would otherwise be difficult to observe. Because the composed transformation of a closed trajectory is analytically known, mirrored and repeated states provide dense verification without requiring ground-truth video supervision. The consistent improvements across different rollout horizons and action structures indicate that reducing local transition residuals improves the overall trajectory-level dynamics. Moreover, the gains on outof-distribution composite actions suggest that the learned dynamics increasingly capture the underlying compositional structure of actions rather than simply memorizing specific action programs. Closed cycles should therefore be understood as a self-verifiable supervision mechanism, rather than a restriction on the model’s applicable trajectory space. The main limitation of WorldCycle is that its exact supervision signals rely on action structures with explicit inverse operations or analytically known closure relations. For irre-

versible or partially irreversible processes, such as object deformation, contact-rich interaction, and persistent state changes, an action sequence cannot generally recover its initial state by simply applying inverse controls. The current cycle construction therefore cannot be directly applied to these dynamics. Extending self-verifiable supervision to irreversible processes through conservation laws, equivalent terminal states, or other analytically verifiable constraints represents an important direction for future work.

Relation to Concurrent Cycle-Based Methods Two concurrent works explore cyclic constraints for improving video world models from complementary perspectives. Cycle-World (Su et al. 2026) introduces reverse-prediction cycle consistency for long-horizon video generation: an auxiliary reverse model reconstructs the preceding latent chunk from the current generated chunk and is further reused for inference-time latent correction. World Models as Group Actions (Wang et al. 2026b) instead formalizes actionconditioned dynamics through group actions and enforces identity, inverse, and composition consistency using synthesized latent-space supervision. Despite these high-level similarities, the central distinction lies in the role of cyclic structure in the optimization

pipeline. Both concurrent approaches formulate cycle consistency as a differentiable loss for model training or fine-tuning. In contrast, WorldCycle focuses specifically on reinforcement-learning post-training of a pretrained interactive world model. Our action cycles are not introduced merely as another structural regularizer. Their analytically known closure relations instead serve as self-verifiable reward functions, converting sampled long-horizon rollouts into annotation-free scalar feedback. This directly addresses the verification bottleneck in world-model post-training, where ground-truth future videos are generally unavailable. The three methods also differ in how their cycles are instantiated. Cycle-World forms a temporal prediction loop between adjacent latent chunks, whereas World Models as Group Actions regularizes the latent dynamics to respect predefined algebraic relations. WorldCycle constructs closed action programs in the environment and evaluates their complete generated trajectories using spatial closure and temporal consistency rewards. Thus, the cyclic structure serves three distinct purposes: reverse-prediction regularization, group-action regularization, and self-verifiable reward construction for RL post-training, respectively.

WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

Analyze & explore related research in Conceptio

Related documents