Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation Jiaqi Peng∗ 1,2 Xiqian Yu∗ 2 Delin Feng∗ 2 Yuqiang Yang2 Jing Xiong2,3 Ganlin Yang2,4 Jinliang Zheng1,2 Jiafei Cao2 Jiangmiao Pang2 Yuan Shen† 1 Tai Wang† 2 1
Wenzhe Cai2 Xueyuan Wei2
Tsinghua University 2 Shanghai AI Laboratory 3 Peking University https://steinate.github.io/cortex.github.io ∗ Equal contribution. † Corresponding author.
4
USTC
arXiv:2607.05377v1 [cs.RO] 6 Jul 2026
Abstract: While recent Vision-Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature—relying solely on current observations. Hierarchical dual-system methods address this but suffer from a gap between high-level planning semantics and low-level execution kinematics. We introduce Cortex, a bidirectionally aligned embodied agent framework with a customized planning interface that conveys executable and tractable subtask plans from high-level VLM to low-level VLA. Specifically, we standardize manipulation subtasks into 32 canonical skill primitives and inject tractability principles, such as representative object attributes and improved trajectory reachability, into the data generation pipeline. This enables automatic annotation of over 4k hours of open-source video data and generation of 30 hours of simulation data. We further devise an event-balanced sampling strategy to construct training data for fine-tuning the framework to better handle planning ambiguity during subtask transitions, enhanced by carefully designed harness engineering from task contexts to skill constraints during inference. Both open-loop VLM and closed-loop system evaluations demonstrate Cortex’s efficacy, e.g., it outperforms monolithic baselines by 3.1% on Liberolong and 4.1% on RoboTwin. Notably, Cortex’s generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks, such as multi-stage chemistry experiments, by simply combining with a fine-tuned VLA—a capability infeasible through VLA fine-tuning alone. Keywords: Long-horizon Manipulation, Vision-Language-Action Model Global Instruction: Wash the beaker
◼
Pick up the beaker ◼
◼
Place the bottle on the table
Place the beaker on the platform ◼ Pick up the beaker
◼
Pick up the bottle ◼ ◼
Unscrew the cap ◼
Pour the water into the kettle ◼
Same Observation, Ambiguous Subtask
RGB Obs.
Instruction
Unscrew the cap√
Place back
Pick up the bottle
Pick up the bottle
Place the bottle× RGB Obs.
Semantic Ambiguity Instruction
VLM
Referring &Timing?
Fetch the cup? Bottle? Beaker?
fail
Temporal Ambiguity Unscrew the cap
Pour the …
Subtask
Trigger
Instruction
RGB Obs.
2Hz Unscrew the cap Event-balanced Sampling
Dual System RGB Obs.
memory
VLM
Action
I have picked up the beaker and place it on the platform, …
Pick up
VLA
Pour the water into the beaker
Place the beaker on the table
Cortex framework
Monolithic VLA Which stage?
Memory
Subtask
VLA
10Hz
Unscrew the cap
Unscrew the cap
Bidirectionally Aligned Embodied Agent Framework
Figure 1: Compared to previous works, Cortex is an embodied agent framework aligning the highlevel VLM and low-level VLA on the subtask executability and tractability, enabling synergistic planning and execution for long-horizon manipulation.
1
Introduction
Vision-Language-Action (VLA) models have fundamentally reshaped embodied AI by directly mapping multi-modal inputs to continuous motor control, achieving remarkable zero-shot generalization in short-horizon tasks [1, 2, 3, 4]. However, as tasks scale in temporal complexity, monolithic VLAs face a critical bottleneck: Markovian short-sightedness. Operating purely reactively without continuous progress verification or spatial-temporal memory, these models struggle to differentiate actual task progress from instantaneous visual observations [5, 6]. Consequently, when executing monolithic long-horizon instructions, they frequently lose track of intermediate states and blindly repeat actions, inevitably leading to compounding execution errors. Recent efforts to mitigate this via visual frame buffering [7, 8] are constrained by limited context windows and struggle to form the semantic memory required for logical planning. Alternatively, hierarchical dual-system paradigms decouple cognitive planning from reactive execution. However, they fundamentally suffer from a lack of bidirectional alignment, i.e., the high-level VLM should consider the capabilities of low-level VLA, while the VLA should be robust and adaptive to the VLM’s outputs. Early dual-system planners [9, 10] act as disembodied observers: The output plans lack explicit embodied constraints, and thus serve as kinematically ungrounded instructions for downstream executors, leading to a severe semantic-kinematic domain gap. Recent works [5, 11, 12] equip the VLAs with the memorization capability but still fall short in the planning alignment aspect. To tackle these challenges, we propose Cortex, a cognitive orchestrator aligned with execution. It is achieved by a bidirectionally aligned embodied agent framework featuring a customized planning interface that conveys executable and tractable information from the high-level VLM to the lowlevel VLA. Specifically, we first define an interface subtask formulation and construct the metadata accordingly. For executability, we standardize manipulation subtasks into 32 canonical skill primitives, enabling the automatic annotation of over 4k hours of open-source video data. For tractability, we inject physical principles—such as representative object attributes and improved trajectory reachability—directly into the data generation pipeline, yielding over 30 hours of high-quality simulation data. Built upon these metadata, we further devise an event-balanced sampling strategy to construct subtask execution and transition samples for fine-tuning the framework to better handle the planning ambiguity during different phases. During inference, we augment the agent framework with carefully designed harness engineering, involving task contexts and skill constraints to ensure the alignment between planners and executors. Both open-loop VLM and closed-loop dual system evaluations validate the efficacy of Cortex. In the open-loop VLM evaluation, our Cortex’s VLM outperforms state-of-the-art generalist models including GPT-5 [13] and Gemini [14]. In closed-loop simulation benchmarks, Cortex achieves state-of-the-art performance, reaching a 95.5% success rate (+3.1%) on Libero-long and 86.8% (+4.1%) on RoboTwin, decisively outperforming its underlying monolithic VLA baseline (π0.5 ). More importantly, Cortex’s generalist VLM enables the zero-shot completion of unseen real-world long-horizon tasks, such as complex multi-stage chemistry experiments, by simply combining with a fine-tuned VLA—a capability fundamentally infeasible through VLA fine-tuning alone. Compared to the zero success rate of end-to-end methods, Cortex achieves up to 65% success rate on such tests, which is even comparable to the performance of humans with VLA policies.
2
Related Work
Vision-Language-Action Models. Vision-Language-Action (VLA) models reframe continuous control as multimodal sequence modeling. Early milestones like RT-1 [15] and PaLM-E [16] integrated sensory streams with language, while RT-2 [15] co-fine-tuned web-scale VLMs directly on robotic data for semantic-to-motor translation. Recently, diverse open-weight frameworks (Octo [17], OpenVLA [4], InternVLA-M1 [18]) and foundation models (π0 [19], GR00T [3]) have achieved robust zero-shot and cross-embodiment generalization. However, despite ex2
Wash the beaker!
I have picked up the beaker and … unscrewed the cap.
Instruction
Observation
Memory
I have picked up the beaker and … poured the water.
Observation
Memory …
VLM
(Cognitive Orchestrator) … Pour the water into the beaker
I have picked up the beaker and … unscrewed the cap.
Place the bottle on the table
I have picked up the beaker and … poured the water.
Place the beaker on the table
Task completed
VLA
(Reactive Execution)
Figure 2: Overview of the Cortex framework. The cognitive orchestrator dynamically updates semantic memory to continuously stream executable subtask to ground the reactive System-1 executor. celling at short-horizon primitives, these monolithic end-to-end architectures remain fundamentally constrained by Markovian assumptions, severely limiting their performance on long-horizon tasks. Hierarchical Planning and Dual-System. To address monolithic limitations, “System-1/System2” paradigms decouple cognitive planning from reactive execution. Early methods like SayCan [9] and Code as Policies [20] used explicit linguistic or code interfaces to orchestrate predefined skills. To bypass rigid textual boundaries, recent frameworks couple slow orchestrators with fast executors via continuous latent spaces for manipulation [21, 22] and navigation [23]. However, latent-coupled systems sacrifice interpretability and explicit progress tracking, while unconstrained language-coupled systems often fail to physically ground their plans. Crucially, both paradigms largely operate open-loop, lacking continuous visual success verification. Embodied Agents and Temporal Memory. Long-horizon autonomy demands persistent state representations. While classical Task and Motion Planning (TAMP) [24, 25] enables compositional reasoning, it struggles in partially observable, open-world environments. To maintain temporal consistency, recent embodied agents [26, 27, 28, 29] augment large models with memory mechanisms via textual histories, episodic retrieval, or visual buffering. Yet, existing approaches [30] primarily treat memory as passive observation storage. They lack an action-oriented, physically grounded state abstraction necessary to resolve ambiguities during dynamic, multi-stage interactions.
3
Approach
This section delineates the Cortex framework by first establishing a bidirectionally aligned subtask interface via scalable metadata construction, then introducing an event-balanced episode sampling strategy for ambiguity-resolving training, and finally deploying an asynchronous inference pipeline with harness engineering for closed-loop execution. 3.1
Framework Overview
As Fig. 5 shows, the Cortex framework consists of a VLM and VLA. The VLM takes the instruction and image observations as input and performs subtask planning with textual memory records. The VLA takes the subtask as input for reactive execution. The key problem here is to define a customized subtask interface to align VLM and VLA, which should meet the requirements of executability and tractability. Next, we first introduce the formulation and then present how to construct data for training in the subsequent subsections. Interface Formulation. To achieve generalist embodied autonomy, a hierarchical agent must reliably translate open-ended human intentions (the global instruction I) into continuous, high3
Public Real-world Data
Self-collected Data
Public Simulation Data
Procedural Data
Automatic Annotation Pipeline
Interface Property pick
grasp
fetch
Executability
[Pick]
Tractability
Figure 3: Long-horizon metadata construction and interface standardization. Our data generation pipeline leverages automated annotation of over 4,000 hours of open-source video data and synthesizes high-quality procedural simulation data. frequency motor commands (the action sequence at ). However, directly mapping unconstrained high-level cognitive reasoning to low-level reactive control inevitably induces a severe semantickinematic gap. We resolve this by instantiating the subtask (st ) as a bidirectionally aligned interface. For example, given a complex global instruction I such as “Wash the beaker”, the interface grounds the plan into executable skill primitives, such as s1 : “[Pick] the beaker from the table”. While the interface st guarantees immediate kinematic grounding, a system executing isolated subtasks easily trapps in execution loops when visual states appear ambiguous. Therefore, an actively updated semantic memory, M (t) , is introduced as the temporal bridge connecting discrete subtasks. At any given timestep t during the k-th subtask phase, this temporal prior aggregates all previously Lk−1 achieved milestones: M (t) = M (0) ⊕ i=1 Φ(si ) where ⊕ denotes semantic concatenation and Φ(si ) encodes the completed state of the i-th subtask. Starting from an initial memory M (0) (e.g., “This is the first subtask, and no subtasks have been completed yet”), M (t) actively aggregates prior milestones into a consolidated history. To illustrate, before generating the 4th subtask (s4 : “[Unscrew] the cap of the water bottle”), the memory explicitly logs: “The robot has picked up the beaker and placed it on the platform, then picked up the water bottle”. 3.2
Metadata Annotation & Generation
Given the interface definition, this section details the corresponding scalable pipeline used to construct the structural metadata for subsequent training. Open-source Data Annotation. We aggregate over 4,000 hours of long-horizon episodes (with an average of more than 7 subtasks) from established benchmarks (e.g., AgibotWorld [31, 32], Galaxea [33], BEHAVIOR-1K [34], RoboCerebra [35]) and newly collected real-world teleoperation data. To ensure executability, we standardize 32 canonical skill primitives. The full definitions of these skill primitives are elaborated in Appendix A.3. Each primitive is assigned a skill category and a strict language template (e.g., [Unscrew] [object], [Stack] [object1] on [object2]) to establish a uniform syntax. For existing open-source datasets, we utilize Qwen3-VL-235B [36] to re-annotate subtasks according to our templates and explicitly merge static frames. For real-world demonstrations, we deploy an automated boundary inference pipeline that converts raw trajectories into structured subtask annotations. First, coarse boundary priors are generated for reference episodes by prompting Qwen-VL on uniformly sampled frames. For remaining episodes, we frame segmentation as a multimodal sequence partitioning problem. At each frame t, we fuse state-action (ot , at ) and visual features It into a joint representation xt = [ϕs (ot , at ), ϕv (It )]. Dynamic programming is then applied to compute the optimal monotone boundaries b1 < · · · < bK−1 4
Pick up the cucumber from the shelf.
Place the cucumber …
... 𝑡0
𝑡0
𝜖1
𝑡1
𝑡2
Boundary trasition
𝑡3
Final tail
𝑡1
𝑡2
𝜖2
𝑡4
𝑡5
𝑡6
Sample frame
Figure 4: Event-balanced Sampling. Training trajectories are strategically divided into boundary transition phases and intra-task execution phases. The sampling mechanism employs asymmetric temporal margins around the subtask boundaries, explicitly balancing the learning of continuous state maintenance with discrete logical memory updates. by minimizing the frame-to-subtask compatibility cost, regularized by duration priors and lowmotion boundary penalties. Comprehensive implementation details are provided in Appendix A.1. Simulation Procedural Data Generation. We utilize simulators, including RoboTwin [37] and RMBench [12], to procedurally synthesize novel, optimal sequences from high-level tasks. We generate subtask annotations online during expert demonstration collection. Subtask boundaries are programmatically defined during expert execution as described in Append A.4: a scope initiates upon entering a semantic stage and terminates when constituent low-level primitives complete. To enforce tractability, we extract detailed simulation assets to determine object categories and derive colors directly from their material properties (e.g., ”blue toy car”). When multiple identical assets exist within a scene, we access object pose to disambiguate them using relative spatial identifiers (e.g., ”the right white stapler”). Interaction counts are annotated using rule-based methods. Crucially, the data generation pipeline explicitly incorporates descriptions of the robot embodiment to evaluate reachability, ensuring the synthesis of optimal subtask routing and kinematically constrained execution paths. 3.3
Training
Event-balanced Sampling. Based on the curated metadata, fine-tuning the dual-system framework requires addressing the planning ambiguity challenge during subtask transitions. Standard uniform frame sampling heavily biases the model toward ongoing execution, obscuring critical task boundaries. Therefore, we devise an event-balanced sampling strategy to construct the training data. First, we define three distinct temporal phases within a trajectory based on their proximity to groundtruth subtask boundaries tk . To capture state changes, we introduce asymmetric temporal margins, ϵ1 (pre-boundary) and ϵ2 (post-boundary). The boundary transition phase is defined as t ∈ [tk − ϵ1 , tk + ϵ2 ]. Because the visual completion landmarks typically emerge slightly after the exact subtask boundary, we empirically set ϵ2 > ϵ1 , maintaining the total duration of this transition phase at approximately 1 second. Here, the model must visually verify physical completion to resolve transition ambiguity, thereby triggering discrete state advancement by updating the memory and planning the next subtask sk+1 . The intra-task execution phase spans the remaining intervals, defined as t ∈ (tk−1 + ϵ2 , tk − ϵ1 ). During this phase, observations represent steady actions, and the model must learn “semantic patience” by maintaining the current active subtask sk without modifying the memory state. The final tail phase, t ∈ [tK − ϵ1 , tK ], concludes the trajectory. 5
Help me wash the beaker! Pick up the beaker and place it on the platform, then pick up the bottle to pour water.
Get it
You are a robot program for high-level manipulation. System skill
Pick, Place, Screw, Navigate, Pull, Push, Press, Sweep, Handover, Tie
I have picked up the beaker and … unscrewed the cap.
1. Pick up the beaker 2. Place beaker on platform 3. Pick up the water bottle 4. Pour water into beaker
Pour the water into the beaker
Working
1. Pick up the beaker 2. Place beaker on platform 3. Pick up the water bottle 4. Pour water into beaker Pour the water into the beaker
[0.26, -0.50, 0.29, -0.19, …]
Figure 5: Harness Engineering. The deployment harness acts as a lightweight arbitration layer that maps diverse, open-ended global instructions into a unified instruction interface. The model processes the final memory state and current visual observations to explicitly output a termination token, signifying that the global instruction has been fully realized. To balance the learning of continuous state maintenance and discrete logical advancement, we target an near equal sampling ratio between these phases. Given the scarcity of transition and final frames, we employ a denser sampling stride within these boundary phases, dynamically adjusting these strides per dataset to accommodate varying subtask lengths. Specific data ratios are detailed in Appendix A.2. We first finetuning on the System-2 VLM [36] using this event-balanced dataset, optimizing it to actively predict the current subtask and update memory from visual streams. Subsequently, we finetune the System-1 VLA [19] on the corresponding motor commands, strictly conditioned on the executable subtasks provided by the VLM interface. 3.4
Inference
Asynchronous Inference. To maximize reactivity without stalling the high-level orchestrator, the dual systems operate continuously and asynchronously. System-2 runs at a lower frequency, processing streaming visual observations alongside the recurrent memory log to monitor progress. Harness Engineering. To operate as a versatile generalist, an agent must dynamically bridge the gap between open-ended human intentions and the kinematically aligned skills. Cortex achieves this by seamlessly mapping a diverse global instruction I-spanning coarse abstract goals, detailed procedural descriptions, and explicit subtask lists—into a unified instruction interface. The harness constructs rigorous prompts that provide the VLM with rich task contexts, primarily combining the instruction I and the actively updated memory log M (t) . To ensure executability, we enforce strict skill constraints on the VLM’s output space. During the post-processing stage, the harness utilizes sequence matching to accurately map the generated raw subtask description to the closest matching canonical primitive within the predefined VLA skill library as shown in Appendix A.5.3.
4
Experiments
Our experimental evaluation is rigorously designed to answer three primary questions, explicitly mapped to the core architectural components of the Cortex framework: (Q1) [Interface & Metadata]: How effectively do the curated structural metadata (ensuring executability and tractability) and multi-granular instruction priors ground the cognitive orchestrator against kinematic hallucinations? (Q2) [Resolving Ambiguities]: How successfully does Cortex resolve semantic ambiguity (via Spatial/Numerical Grounding) and temporal ambiguity (via Event-balanced Sampling) to surpass monolithic VLAs in long-horizon simulations? (Q3) [Asynchronous Bidirectional Loop]: Can the joint asynchronous inference framework reliably transfer to complex real-world environments to achieve dynamic, closed-loop physical error recovery? 4.1
Open-loop VLM Evaluation
To address (Q1) and validate the interface design, we evaluate System-2’s ability to generate physically grounded plans using an LLM-as-a-Judge (Qwen-3.5-9B [38]) across three axes: Spatial Information, Long-Horizon Task consistency, and Counting Task accuracy (see Appendix A.3 for detailed task configurations). We employ two rigorous paradigms: (1) Step-Level (Teacher-Forced) isolates single-step generation accuracy utilizing ground-truth memory, while (2) Episode-Level 6
Table 1: Performance evaluation on Step-level and Episode-level across different spatial and temporal tasks. Metric scores include Subtask accuracy, Memory (Mem.) accuracy, and Total (Tot.) score. Level
Method
Step-level
Episode-level
Spatial Information
Avg. Tot.
Long-horizon Task
Counting Task
Subtask
Mem.
Tot.
Subtask
Mem.
Tot.
Subtask
Mem.
Tot.
Qwen3-VL-8B-Instruct [36] GPT-5 [13] Gemini [14]
6.739 6.268 6.925
2.162 2.755 2.521
4.263 3.668 4.176
6.424 6.422 6.697
2.611 2.481 2.705
4.165 3.682 4.215
6.775 6.163 6.920
2.948 2.790 3.135
4.070 3.429 4.025
7.018 6.220 7.159
Cortex (baseline) Cortex (w/o harness)
7.051 7.213
2.761 2.994
3.744 3.933
6.505 6.927
2.940 3.112
4.033 3.952
6.973 7.064
3.503 3.435
4.173 4.213
7.676 7.648
Cortex (harness on skills) Cortex (full harness)
7.392 8.318
2.885 3.800
4.150 4.254
7.035 8.053
3.144 3.909
4.121 4.250
7.265 8.160
3.527 4.237
4.348 4.504
7.875 8.741
Qwen3-VL-8B-Instruct[36] GPT-5[13] Gemini [14]
6.292 7.231 6.860
2.548 2.920 2.896
3.920 4.400 4.033
6.468 7.321 6.929
2.274 2.711 2.774
3.746 4.285 3.870
6.021 6.996 6.644
2.614 3.320 3.060
3.775 4.056 3.945
6.388 7.376 7.006
Cortex (baseline) Cortex (w/o harness)
6.978 7.155
2.549 3.033
3.969 3.771
6.518 6.805
2.238 3.004
4.362 3.922
6.600 6.925
3.525 3.741
4.290 3.995
7.815 7.736
Cortex (harness on skills) Cortex (full harness)
7.314 7.810
3.017 3.543
3.958 4.044
6.985 7.587
2.743 3.576
4.123 3.804
6.866 7.380
3.950 4.203
4.142 4.262
8.091 8.464
(Self-Forced) demands autoregressive memory input to measure resilience against compounding semantic drift. To evaluate the individual contribution of each component in Cortex, we compare four strategic variants (results detailed in Table 1): (1) Cortex (baseline), a stripped-down variant relying solely on event-balanced sampling; (2) Cortex (w/o harness), which complements the baseline with interface information; (3) Cortex (harness on skills), which restricts the harness engineering to skills; and (4) Cortex (full harness), the complete proposed framework. As shown in Table 1, step-by-step introduction of each component yields consistent performance gains, with the full harness configuration achieving the optimal average total scores at both step-level (8.318) and episode-level (7.810). 4.2
Closed-loop Simulation Evaluation
Table 2: Zero-shot comparison on LIBERO-Long
To address (Q2), we evaluate the complete dual-system in challenging, multi-stage simulation environments. Crucially, since groundtruth subtasks are inaccessible during closedloop rollouts, all evaluations are strictly conditioned on raw, high-level global instructions.
Paradigm
Method
Success (%)
π0 [39] π0.5 [19] End-to-End MemoryVLA [7] OpenVLA-OFT [40]
85.2 92.4 93.4 94.5
Agentic
RoboBrain [41] Qwen3-VL-8B [36] GPT-5.4 [42] Gemini-3.1-Pro [14] Cortex (Ours)
57.0 68.0 72.0 91.0 95.5 (+3.1)
LIBERO-Long Benchmark. As detailed in Table 2, we rigorously isolate reasoning capabilities by coupling all agentic methods (including Cortex) with π0.5 as their shared System-1 executor. Crucially, Cortex achieves a state-of-the-art zero-shot success rate of 95.5%. While end-to-end models like π0.5 (92.4%) perform adequately, they operate purely reactively. Leading generalist agents like Gemini-3.1-Pro (91.0%) act as disembodied observers; lacking strict physical grounding, they occasionally hallucinate subtasks that violate immediate kinematic constraints.
Success Rate (%)
RoboTwin 2.0 RoboTwin Evaluation. As illustrated in Fig88.0 86.0 86.8 ure 6, monolithic VLAs suffer performance Short Horizon 82.6 83.0 82.7 Long Horizon 77.1 80 72.5 72.8 degradation as the task horizon scales, sucOverall 65.9 66.3 62.7 61.5 cumbing to premature task completion due to 60 53.5 55.2 temporal ambiguity. Conversely, Cortex utiliz42.1 40 33.2 35.732.834.5 32.838.3 ing π0.5 and maintains an exceptionally high 29.7 24.5 success rate (88.00%) on long-horizon splits. 20 We attribute this robustness directly to our pro0 posed ambiguity resolution mechanisms: (1) 0 0.5 ortex ACT RDT-1BpenVLA-T DP3 X-VLA C urs) O OF (O Resolving Semantic Ambiguity: Fine-grained Figure 6: Success rates on RoboTwin benchmark. attribute and spatial grounding yield significant
7
improvements in visually dense tasks like place object basket (80% → 85%), conditioning the underlying VLA to predict kinematically favorable chunking. Cortex actively accounts for reachability constraints, dynamically inserting intermediate “handover” subtasks in dump bin bigbin (92% → 98%). (2) Resolving Temporal Ambiguity: Through event-balanced sampling, System-2 actively dictates the temporal flow rather than passively sleeping, dynamically injecting intermediate subtasks (e.g., press button task in Table 9) to ensure continuous, uninterrupted physical progression. 4.3
Closed-loop Real-world Evaluation
Figure 7: Zero-shot Real-world Deployment in Multi-stage Chemistry Tasks. A 14-step continuous experiment demonstrating Cortex’s exceptional capability in fine-grained spatial-numerical grounding, robust long-horizon progress tracking, and closed-loop physical execution. To answer (Q3), we deploy the Cortex architecture zero-shot on an ARX ACONE physical platform. We design complex, multi-stage chemistry experiments and kitchen assembly workflows that strictly prohibit purely reactive execution. The remarkable zero-shot success of Cortex (Table 3) stems from the seamless integration of our asynchronous bidirectional loop. The reactive executor is instantiated Table 3: Method comparison across two long-horizon task as a MEM-style System-1 policy [5], suites. Progress and success rate are average over 20 trials. sub πmem , with a several-second memChemical Washing Paradigm Method ory window. We fine-tune it on Prog. ↑ SR ↑ Prog. ↑ SR ↑ about 10 hours of automatically segπ 2.5/14 0 3.7/14 0 mented subtask-to-action data. As End-to-end π0.5 4.1/14 0 6.5/14 0 mem end-to-end comparisons, we evaluate Cortex 11.0/14 65 10.5/14 55 π0.5 without explicit memory and a Agentic sub Human + πmem 12.2/14 75 11.6/14 70 task-level πmem with the same shortmemory design. As shown in Table 3, both remain brittle because they must infer global progress and subtask boundaries from the raw instruction stream, causing repeated primitives and local errors to compound over long horizons. Detailed visual comparisons and failure analyses are provided in Appendix A.5.7 and Appendix A.5.8, sub with representative cases in Figs. 16 and 18. Encapsulating πmem within Cortex lets System-2 continuously ground execution through optimally routed, unambiguous subtasks. 8
Crucially, the Cortex architecture natively supports Closed-Loop Physical Error Recovery, while a mature agent harness improves both fault tolerance and execution fluency across the asynchronous dual-system interface. Real-world deployment inevitably involves perception-action desynchronization: minor visual occlusions or delayed state transitions can cause reactive policies to stall. Rather than failing passively, Cortex triggers a timeout-driven kinematic reset that introduces a controlled physical perturbation to refresh visual evidence and resolve perceptual deadlocks. The continuously streaming System-2 then immediately re-processes the updated observation, re-validates task progress online, and dispatches the corrected subtask. As a representative real-world case, Figure 7 shows that Cortex autonomously completes a 14-subtask long-horizon chemistry procedure under this mechanism. More implementation and evaluation details are provided in Appendix A.5.
5
Conclusion
This paper introduces Cortex, a bidirectionally aligned dual-system framework designed to overcome the Markovian short-sightedness of monolithic Vision-Language-Action (VLA) models in long-horizon manipulation. To bridge the semantic-kinematic gap, Cortex deeply couples a cognitive orchestrator with a reactive executor via a structured subtask interface, constrained by 32 canonical skill primitives and kinematically optimal routing. This bidirectional alignment synchronizes the dual systems by explicitly resolving both semantic and temporal ambiguities. Consequently, Cortex decisively outperforms purely end-to-end monolithic VLAs in long-horizon simulations, while demonstrating robust zero-shot physical error recovery in complex, real-world chemistry tasks. By formalizing this continuous reasoning-execution loop, Cortex establishes a highly scalable foundation for long-horizon robot autonomy.
6
Limitations
Despite its robust planning capabilities, Cortex faces two primary limitations. Memory Representation: Text-based memory discards spatial coordinates and visual nuances, disrupting objectinstance correspondence during large-scale mobile manipulation. Future work will integrate visual memory retrieval and pixel-level grounding to extend Cortex into a unified dual-mode framework. High-Frequency States: Relying on standard vision encoders renders Cortex insensitive to highfrequency micro-state changes. Although we explored tokenizing historical proprioceptive states (analogous to π0.5 ) to inject continuous kinematic priors, seamlessly fusing these with vision for rapidly changing, highly dynamic environments remains a formidable challenge.
9
References [1] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. [2] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024. [3] J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025. [4] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. [5] M. Torne, K. Pertsch, H. Walke, K. Vedder, S. Nair, B. Ichter, A. Z. Ren, H. Wang, J. Tang, K. Stachowicz, et al. Mem: Multi-scale embodied memory for vision language action models. arXiv preprint arXiv:2603.03596, 2026. [6] I. Liu, A.-C. Cheng, R. Yan, G. Chen, R.-Z. Qiu, X. Zou, S. Yi, H. Yin, X. Wang, and S. Liu. Long-horizon manipulation via trace-conditioned vla planning. arXiv preprint arXiv:2604.21924, 2026. [7] H. Shi, B. Xie, Y. Liu, L. Sun, F. Liu, T. Wang, E. Zhou, H. Fan, X. Zhang, and G. Huang. Memoryvla: Perceptual-cognitive memory in vision-language-action models for robotic manipulation. arXiv preprint arXiv:2508.19236, 2025. [8] H. Li, S. Yang, Y. Chen, Y. Tian, X. Yang, X. Chen, H. Wang, T. Wang, F. Zhao, D. Lin, et al. Cronusvla: Transferring latent motion across time for multi-frame prediction in manipulation. arXiv e-prints, pages arXiv–2506, 2025. [9] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. [10] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. [11] A. Sridhar, J. Pan, S. Sharma, and C. Finn. Memer: Scaling up memory for robot control via experience retrieval. arXiv preprint arXiv:2510.20328, 2025. [12] T. Chen, Y. Wang, M. Li, Y. Qin, H. Shi, Z. Li, Y. Hu, Y. Zhang, K. Wang, Y. Chen, et al. Rmbench: Memory-dependent robotic manipulation benchmark with insights into policy design. arXiv preprint arXiv:2603.01229, 2026. [13] OpenAI. Gpt-5 system card, 2025. URL https://arxiv.org/abs/2601.03267. Released: August 7, 2025; arXiv published: August 13, 2025. [14] G. DeepMind. Gemini 3.1 pro (preview) model card, 2 2026. URL https://storage. googleapis.com/deepmind-media/Model-Cards/Gemini-3-1-Pro-Model-Card. pdf. Preview release, accessed: 2026-05-26. [15] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 10
[16] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. [17] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024. [18] X. Chen, Y. Chen, Y. Fu, N. Gao, J. Jia, W. Jin, H. Li, Y. Mu, J. Pang, Y. Qiao, et al. Internvlam1: A spatially guided vision-language-action framework for generalist robot policy. arXiv preprint arXiv:2510.13778, 2025. [19] D. Driess, K. Black, N. Brown, J. Darpinian, K. Dhabalia, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. π0.5 : A vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054. [20] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International conference on robotics and automation (ICRA), pages 9493–9500. IEEE, 2023. [21] Q. Bu, H. Li, L. Chen, J. Cai, J. Zeng, H. Cui, M. Yao, and Y. Qiao. Towards synergistic, generalized, and efficient dual-system for robotic manipulation. arXiv preprint arXiv:2410.08001, 2024. [22] J. Zhang, Y. Guo, X. Chen, Y.-J. Wang, Y. Hu, C. Shi, and J. Chen. Hirt: Enhancing robotic control with hierarchical robot transformers. arXiv preprint arXiv:2410.05273, 2024. [23] M. Wei, C. Wan, J. Peng, X. Yu, Y. Yang, D. Feng, W. Cai, C. Zhu, T. Wang, J. Pang, et al. Ground slow, move fast: A dual-system foundation model for generalizable visionand-language navigation. arXiv preprint arXiv:2512.08186, 2025. [24] L. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE international conference on robotics and automation, pages 1470–1477. IEEE, 2011. [25] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021. [26] R. Li, Y. Zhou, Y. Zhu, K. Chen, J. Wang, S. Wang, K. Hu, M. Yu, B. Jiang, Z. Su, et al. Roboclaw: An agentic framework for scalable long-horizon robotic tasks. arXiv preprint arXiv:2603.11558, 2026. [27] P. Xu, J. Zheng, and Y. Mu. Roboagent: Chaining basic capabilities for embodied task planning. arXiv preprint arXiv:2604.07774, 2026. [28] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible planner-independent interface layer. In 2014 IEEE international conference on robotics and automation (ICRA), pages 639–646. IEEE, 2014. [29] K. Fang, A. Toshev, L. Fei-Fei, and S. Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 538–547, 2019. [30] Y. Zhang, J. Shu, Y. Ma, X. Lin, S. Wu, and J. Sang. Memory as action: Autonomous context curation for long-horizon agentic tasks. arXiv preprint arXiv:2510.12635, 2025. [31] Q. Bu, J. Cai, L. Chen, X. Cui, Y. Ding, S. Feng, S. Gao, X. He, X. Hu, X. Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025. 11
[32] A. Team. Agibot world 2026, 2026. URL https://huggingface.co/datasets/ agibot-world/AgiBotWorld2026. Open-source real-world embodied AI dataset. [33] T. Jiang, T. Yuan, Y. Liu, C. Lu, J. Cui, X. Liu, S. Cheng, J. Gao, H. Xu, and H. Zhao. Galaxea open-world dataset and g0 dual-system vla model. arXiv preprint arXiv:2509.00576, 2025. [34] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martı́n-Martı́n, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80–93. PMLR, 2023. [35] S. Han, B. Qiu, Y. Liao, S. Huang, C. Gao, S. Yan, and S. Liu. Robocerebra: A large-scale benchmark for long-horizon robotic manipulation evaluation. Advances in Neural Information Processing Systems, 38, 2026. [36] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [37] T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, et al. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025. [38] Qwen Team. Qwen 3.5: Open multimodal models, February 2026. URL https://qwen.ai/ blog?id=qwen3.5. Qwen 3.5-9B, accessed: 2026-05-26. [39] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0 : A vision-language-action flow model for general robot control, 2024. URL https://arxiv.org/abs/2410.24164. [40] M. J. Kim, C. Finn, and P. Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025. [41] H. Tan, E. Zhou, Z. Li, Y. Xu, Y. Ji, X. Chen, C. Chi, P. Wang, H. Jia, Y. Ao, et al. Robobrain 2.5: Depth in sight, time in mind. arXiv preprint arXiv:2601.14352, 2026. [42] OpenAI. Gpt-5.4 thinking system card, 3 2026. URL https://deploymentsafety. openai.com/gpt-5-4-thinking/gpt-5-4-thinking.pdf. Official safety and model documentation. [43] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. [44] S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. In International Conference on Learning Representations, volume 2025, pages 29982–30009, 2025. [45] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.03954, 2024. [46] J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng, et al. Xvla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model. arXiv preprint arXiv:2510.10274, 2025. 12
[47] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025.
13
A
Appendix
This appendix provides supplementary implementation details, evaluation protocols, and qualitative analyses for Cortex. The additional material is organized to clarify how the proposed bidirectional alignment is instantiated in data construction, model training, simulation evaluation, and real-world deployment. A.1
Automatic Annotation Pipeline
To scale long-horizon supervision without dense human trajectory annotation, we adopt an annotation-free subtask boundary inference pipeline that converts raw robot demonstrations into structured pseudo-subtask annotations. Consider a raw robot trajectory τ = {(ot , at , It )}Tt=1 ,
(1)
where ot denotes the robot state, at denotes the action, and It denotes the multi-view visual observation at frame t. Each task is associated with an ordered task-level subtask schema S = (s1 , s2 , . . . , sK ).
(2)
The objective is to infer a monotone boundary sequence 0 = b0 < b1 < · · · < bK = T,
(3)
so that each segment [bk−1 , bk ) corresponds to the k-th subtask sk . In this context, annotation-free denotes the absence of manual frame-level or trajectory-level boundary labels; only the ordered task-level subtask names are assumed to be available, for example from task instructions, program structure, or automatically parsed language descriptions.
Figure 8: Annotation-free subtask boundary inference on Wash the beaker. Each thumbnail is sampled at the inferred start boundary of a subtask. The bottom bar shows the monotone temporal segmentation over the full trajectory. Monotone subtask segmentation. Subtask segmentation is formulated as a multimodal sequence partitioning problem. For each frame, we extract state-action and visual features and fuse them as hp i p xt = λs ϕs (ot , at ), λv ϕv (It ) , (4) 14
Figure 9: Annotation-free boundary inference for a representative Wash the beaker trajectory. Thumbnails indicate inferred subtask onsets, and the lower bar shows the resulting monotone segmentation. where ϕs encodes robot state, action, and local temporal differences, while ϕv captures downsampled appearance statistics and short-term visual change. The coefficients λs and λv balance state and visual modalities. For each subtask sk , we estimate a prototype distribution p(xt | sk ) = N (µk , Σk ),
(5)
and define the frame-to-subtask compatibility cost as ck (t) = (xt − µk )⊤ Σ−1 k (xt − µk ) + log |Σk |.
(6)
For a candidate segment [i, j) assigned to subtask sk , the average observation cost is Ek (i, j) =
j−1 1 X ck (t). j − i t=i
(7)
To avoid degenerate segmentations, we introduce a duration prior. Let ρk denote the expected relative duration of subtask sk ; then 2 j−i Pk (i, j) = λd − ρk . (8) T Subtask boundaries also tend to occur near low-motion regions or action-switch points. Let mj denote the normalized motion magnitude around frame j; the boundary penalty is defined as M (j) = λm mj .
(9)
Boundary inference is then performed by dynamic programming over candidate boundary sets Bk , constructed from sampled frames near expected transition locations together with local low-motion minima. The first segment is initialized as D1 (j) = E1 (0, j) + P1 (0, j) + M (j),
j ∈ B1 ,
(10)
and the recurrence for the k-th subtask is Dk (j) =
min
i∈Bk−1 , i<j
[Dk−1 (i) + Ek (i, j) + Pk (i, j) + M (j)] ,
(11)
for k = 2, . . . , K − 1, with the terminal segment fixed at T : DK (T ) = min [DK−1 (i) + EK (i, T ) + PK (i, T )] .
(12)
i∈BK−1
Backtracking yields the optimal monotone boundary sequence b∗1:K−1 = arg
min
b1 <···<bK−1
K X
[Ek (bk−1 , bk ) + Pk (bk−1 , bk )] +
k=1
K−1 X
M (bk ).
(13)
k=1
This monotone constraint preserves subtask order and prevents temporal overlap between adjacent segments. 15
Pseudo-annotation generation. Once the optimal boundaries are obtained, each trajectory is converted into a structured subtask annotation containing segment spans, executable action text, and canonical skill labels. A typical record stores, for each subtask, its start frame, end frame, normalized action text, and mapped primitive skill. These pseudo labels are produced entirely by multimodal feature matching and dynamic programming rather than by manual trajectory annotation. During VLA training, the dataloader can therefore replace a sampled frame with its corresponding subtask instruction, such as pick up the beaker, unscrew the cap, or pour into the kettle, thereby providing subtask-conditioned supervision without dense human labeling. Role in bidirectional alignment. This inference pipeline is central to the Cortex Suite because it bridges task-level semantic decomposition and frame-level executable supervision. On the top-down side, it distills heterogeneous free-form demonstrations into standardized subtasks and canonical skills. On the bottom-up side, it provides temporally localized transition labels that are subsequently used to construct ongoing and transition samples for System-2 memory training. Consequently, the same inferred boundary structure supports both kinematic grounding and temporal alignment within a unified data engine. A.2
System-2 Training Protocol
This section details the concrete training configurations, structural data formatting, and empirical validations for the System-2 cognitive planner. Infrastructure and Hyperparameters. We fine-tune the Qwen3-VL-8B-Instruct architecture utilizing a distributed cluster of 32 NVIDIA A800 (80GB) GPUs. To maximize representation learning capacity, all model parameters—including the vision encoder, multimodal projector, and language model backbone—are fully unfrozen during the optimization process. The entire training trajectory spans approximately 14.2M effective multi-modal samples. For reproducibility, a comprehensive breakdown of the optimization hyperparameters and distributed infrastructure is dynamically tabulated in Table 4. Table 4: System-2 training configuration and infrastructure parameters. Hyperparameter / Item
Configuration
Model Backbone Hardware Infrastructure Distributed Engine Numerical Precision Optimizer Peak Learning Rate Learning Rate Schedule Warmup Ratio Weight Decay Global Batch Size Max Sequence Length Visual Resolution Budget
Qwen3-VL-8B-Instruct 32 × NVIDIA A800 GPUs (80GB VRAM) DeepSpeed ZeRO-3 bfloat16 AdamW 3 × 10−6 Cosine decay 0.03 0.0 512 (32 × 16 micro-batch per node) 8192 tokens Min pixels: 3,136; Max pixels: 307,200
Unified Instruction Interface. Rather than employing dataset-specific formats, we map heterogeneous supervision sources into a standardized structured prediction problem via a unified prompt template. Concretely, given the Global Task Goal, Input Language Memory, a fixed 32-way atomic skill vocabulary, and the current observations, the model is trained to generate a structured JSON payload containing current skill, current subtask, and active language memory. For visual inputs, RoboCerebra features a single head-view image, while other datasets supply three synchronized views (head, left wrist, and right wrist). The finalized training mixture reflects a balanced distribution of instruction paradigms, consisting of 42.5% detailed procedural descriptions, 20.0% explicit subtask lists, and 37.5% coarse abstract goals. 16
Table 5: Ablation study of temporal sample composition on the Galaxea dataset under a leave-episode-0-out validation protocol. Evaluated on the held-out episode 0 split across all tasks, the event-balanced mixture confirms superior sample efficiency and task performance. Sample counts are reported in millions (M). Training Sample Composition
Sampling Mixture
Evaluation Score
Intra-task (M) Boundary (M) Total (M) Ratio (Intra:Bound) Subtask Memory Avg. Total Intra-task-dominant Event-balanced (Ours)
2.33 1.78
0.62 0.80
3.10 2.72
3.77:1 2.23:1
3.40 3.85
4.17 4.33
7.58 8.18
Event-Balanced Sampling Implementation and Ablation. To ground the event-balanced sampling paradigm, the temporal margin parameter ϵ is adaptively customized across heterogeneous datasets to accommodate variations in execution pace and annotation granularity. Specifically, ϵ is decoupled into dataset-specific configurations based on the execution pacing: for rapid, highvelocity tasks (e.g., RoboCerebra), we enforce a tighter margin of ϵ = 0.5 s; conversely, for slower, more protracted execution sequences (e.g., Galaxea), the window expands up to ϵ = 1.5 s to accommodate transitional variance. Through this adaptive stratification, the finalized training corpus exhibits a highly balanced composition between the intra-task execution phase and the boundary transition phase, shifting from a heavily long-tailed raw distribution to an empirical mixture of approximately 76% intra-subtask phase samples and 24% boundary transition phase frames. This deliberate distribution densifies high-fidelity supervisory signals precisely at cognitive pivoting points while ensuring sufficient representation of nominal execution. To further verify that the observed empirical benefits are inherently attributable to boundaryfocused supervision rather than a mere artifact of increased data volume, we conduct a controlled ablation study on the Galaxea [33] dataset utilizing a rigorous leave-one-episode-out (leave-episode-0-out) cross-validation protocol. As quantified in Table 5, our event-balanced sampling mixture consistently outpaces the intra-task-dominant baseline across all evaluation metrics, despite utilizing fewer total optimization samples (2.72M vs. 3.10M). Specifically, by optimizing the sampling ratio between the intra-task execution phase and the boundary transition phase from 3.77:1 down to a more balanced 2.23:1, the average Subtask accuracy, Memory retention, and Total scores improve from 3.40, 4.17, and 7.58 to 3.85, 4.33, and 8.18, respectively. This substantial gain under a reduced data regime strongly validates our core hypothesis: indiscriminately accumulating redundant steady-state observations within the intra-task execution phase is counterproductive for high-level decision-making. Instead, scaling high-fidelity supervision proximal to the boundary transition phase is paramount, as these critical junctions are precisely where the agent must resolve temporal ambiguity, update its semantic memory, and orchestrate discrete logical advancements. A.3
LLM-as-a-Judge Protocol & Benchmark Details
To complement Sec. 4.1, we instantiate three evaluation buckets aligned with the three axes in Table 1: Spatial Grounding, Long-Horizon Logical Consistency, and Object Counting Accuracy. Scenario Buckets and Task Allocation. To probe complementary facets of System-2 planning, we organize the evaluation suite into three scenario buckets corresponding to spatial grounding, long-horizon sequential reasoning, and counting-sensitive manipulation. Each bucket contains five tasks drawn from AgibotWorld [31], Galaxea [33] and BEHAVIOR-1K [34], yielding 15 rollout tasks in total. Representative scenes for the three buckets are shown in Fig. 10, which may be used to visualize the characteristic observation complexity and planning challenges associated with each evaluation axis. For episode-level evaluation, we sample 10 trajectories for each task, yielding 50 trajectories per scenario bucket and 150 trajectories in total. For step-level evaluation, we construct a balanced benchmark from the AgibotWorld [31], Galaxea [33] and BEHAVIOR-1K [34] validation splits, yielding approximately 1,000 judged samples for each bucket. 17
Figure 10: Representative raw head-view frames extracted from the 15 tasks in the evaluation suite. The three rows correspond to Spatial Information, Long-horizon Task, and Counting Task, respectively, while the five columns in each row follow the task ordering in Table 6. Table 6: Scenario buckets used by the System-2 step-level and episode-level evaluations. Bucket
Evaluation Focus and Source Tasks
Spatial Grounding
Primary challenge: Reference-target disambiguation under distractors, relative position reasoning, and local state verification after manipulation. Source tasks: AgibotWorld: task 598 Galaxea: Arrange The Sofa Cushions 20250722 008, Organize Potato Chip Shelves 20250701 003, Enter Exit Door Turn On Off lights 20250620 001 BEHAVIOR-1K: task-0000. Primary challenge: Multi-stage causal dependencies, strict execution order, and completion verification before advancing to the next step. Source tasks: AgibotWorld: task 357, task 527 Galaxea: Organize The Beverage Cooler 20250630 003, Pour Water20250708 006 BEHAVIOR-1K: task-0011. Primary challenge: Repeated-object tracking, quantity-sensitive termination conditions, and memory updates driven by count changes. Source tasks: AgibotWorld: task 422, task 487 Galaxea: Arrange The Fruits 20250716 006, Organize Toys 20250628 002 BEHAVIOR-1K: task-0001.
Long-Horizon Logical Consistency
Object Counting Accuracy
Atomic Skill Vocabulary. System-2 does not generate unconstrained free-form actions. Instead, both training and evaluation use a fixed candidate vocabulary of 32 robot atomic skills. SYSTEM SKILLS: Pick, PickAndPlace, Place, Remove, Press, Push, Pull, Navigate, Fold, Wipe, Close, Open, Pour, Cut, Rotate, Handover, Sweep, Stack, Unstack, Screw, Unscrew, Scan, Aim, Clamp, Rinse, Spread, Release, Retreat, AdjustPosture, Tie, Strike, Stir
Given the current observation, task instruction, and active language memory, System-2 must first select exactly one skill from this vocabulary when the task is in progress, and then generate the corresponding subtask and updated memory. This constrained skill space reduces open-vocabulary 18
drift and makes the generated subtasks more stable for downstream command normalization and System-1 execution. Evaluating Prompt Modes. Following the Omni-Format Instruction Interface, we evaluate System 2 under three complementary instruction modalities: coarse abstract goals, detailed procedural descriptions, and explicit subtask lists. These modalities differ in the amount and structure of taskorder supervision provided to the planner, while sharing a common prediction objective: conditioned on the current observation and accumulated language memory, the model must identify the currently executable subtask together with the corresponding actively compressed memory state. The unified mode-aware system prompt used across these settings is given below. SYSTEM PROMPT: You are a robot program for high-level manipulation. Given the global task goal, the input language memory, an optional task-order cue, a list of candidate atomic skills, and the camera observations, first choose exactly one current atomic skill from the candidate skill list when the task is in progress, then predict the subtask the robot should currently be in and the language memory that should be active now. The optional task-order cue can be either a Detailed Global Task Instruction, a Subtask List, or absent; Detailed Global Task Instruction and Subtask List are mutually exclusive. When a Detailed Global Task Instruction is provided, it is ordered according to the subtask sequence that should be predicted. When a Subtask List is provided, use its item order as the subtask sequence and prefer the corresponding item text for current subtask. When no task-order cue is provided, infer the current progress from the global task goal, input language memory, and observations. Use any provided task-order cue as a constraint when reasoning about progress, and do not skip ahead to later subtasks unless the observations and input language memory clearly indicate that the earlier subtasks have already been completed. If the observations show that the current subtask is still ongoing, keep the same subtask and keep the active language memory unchanged. If the observations show that the previous subtask has just been completed or the robot has already entered the next subtask, predict the next subtask and switch to the new active language memory that reflects the completed progress. If the task has already been completed, set current subtask to task completed and set current skill to null. Return a JSON object only with keys "current skill", "current subtask", and "active language memory". "current skill" must be exactly one skill copied from the candidate atomic skill list when the task is in progress, or null when the task is already completed. "active language memory" should be a concise semantic summary containing only task-relevant completed progress, without low-level visual details or speculation about future subtasks.
The templates below show the three user-prompt instantiations used consistently in both step-level and episode-level evaluation. They share the same skill-aware prediction format and differ only in how task-order information is supplied to the planner. Fig. ?? shows the inference details. Mode 1: Coarse abstract goals User Input = { Global Task Goal: task text, Input Language Memory: input language memory, Candidate Atomic Skills: [skill 1, ..., skill 32], Current Observation Images (in order): Image 1 (view 1): [image 1] ... Image N (view N): [image N] Choose exactly one atomic skill from the candidate list when the task is in progress; if the task is already completed, set current skill to null and current subtask to task completed. Predict the subtask and active language memory that should be active now. Return JSON only with keys "current skill", "current subtask", and
19
"active language memory". }
Mode 2: Detailed procedural description User Input = { Global Task Goal: task text, Input Language Memory: input language memory, Detailed Global Task Instruction: detailed task text, Candidate Atomic Skills: [skill 1, ..., skill 32], Current Observation Images (in order): Image 1 (view 1): [image 1] ... Image N (view N): [image N] Choose exactly one atomic skill from the candidate list when the task is in progress; if the task is already completed, set current skill to null and current subtask to task completed. Predict the subtask and active language memory that should be active now. Return JSON only with keys "current skill", "current subtask", and "active language memory". }
Mode 3: Explicit subtask list User Input = { Global Task Goal: task text, Input Language Memory: input language memory, Subtask List: 1. subtask 1 2. subtask 2 ... K. subtask K Candidate Atomic Skills: [skill 1, ..., skill 32], Current Observation Images (in order): Image 1 (view 1): [image 1] ... Image N (view N): [image N] Choose exactly one atomic skill from the candidate list when the task is in progress; if the task is already completed, set current skill to null and current subtask to task completed. Predict the subtask and active language memory that should be active now. Return JSON only with keys "current skill", "current subtask", and "active language memory". }
Qwen-3.5-9B Judge Prompt. We use Qwen-3.5-9B [38] as the LLM judge for scoring. For each sample, the judge assigns discrete normalized ratings in {0, 0.4, 0.9, 1.0} to subtask correctness and memory correctness, which are subsequently mapped to the reporting scale used in Table 1, yielding per-axis scores out of 5 and aggregate scores out of 10. To prevent exact lexical matches from being under-scored by the judge, predictions whose subtask or memory strings are character-identical to the reference are deterministically assigned full credit. The judge prompts are listed below:
JUDGE SYSTEM PROMPT: You are a strict JSON-only judge. Never output thinking process, analysis, or markdown. Return exactly one JSON object and nothing else.
20
JUDGE USER PROMPT: You are a strict evaluator. Do NOT output any thinking process, analysis, explanation, markdown, or extra text. Output exactly one JSON object only. Allowed JSON keys: subtask score, memory score, total score, verdict, reason. Scoring rules: - First normalize wording before scoring: * Pronouns are equivalent for the acting agent (e.g., I/me/my/you/your) and can be ignored when meaning is unchanged. * If the subject is the robot, pronouns and explicit robot mentions are equivalent (this can still be exact match), and optional robot descriptors are non-essential. * Treat hand, arm, and gripper as equivalent references to the manipulator. - Use only these discrete scores: 0, 0.4, 0.9, 1. - subtask score rubric: * 1: exact match with reference after applying the normalization above. Exact match MUST be 1.0 (never 0.9). * 0.9: semantically equivalent overall but not exact after normalization (e.g., synonyms, different word order, minor non-critical attribute omission, or non-contradictory object-attribute addition/omission). * 0.4: describes essentially the same high-level thing but is incomplete or substantially rephrased without explicit contradiction. For multi-part subtasks, predicting only the earlier/front part while missing later parts should be 0.4. * 0: any logical or factual error exists (wrong action/object/state/relation /attribute, missing required action/object, contradiction, or unsupported extra claim). For multi-part subtasks, predicting only a later part while missing the earlier/front part is a logical error ≥ 0. - memory score rubric: * 1 and 0.9: same criteria as subtask score. * 0.4: memory is broadly related but substantially rephrased/coarse, with no contradiction and no omitted required subtask. * 0: any logical/factual error, contradiction, unsupported extra claim, or omission of any required subtask. Omission ≥ 0 (not 0.4). - total score = subtask score + memory score. - verdict: correct if total score=2, partial if 0 < total score < 2, wrong if total score=0. Keep reason concise (≤ 40 words). [Question] Task Goal: task text Input Language Memory: input language memory [Reference Answer] "current subtask": "gt current subtask", "active language memory": "gt active language memory" [Assistant Answer] pred text Now output JSON only.
A.4
Simulation Evaluation Protocol
Implementation Details. For all simulation benchmarks, we conduct full-parameter finetuning for π0.5 model as the System-1 executor. The detailed fine-tuning hyperparameters are summarized in the following table 7. LIBERO-Long. For LIBERO-Long, all agentic baselines are coupled with the same System-1 executor when feasible, isolating the contribution of high-level planning. A rollout is counted as successful only if the full long-horizon instruction is completed without violating the benchmark success predicate. Intermediate subtask correctness is not directly rewarded unless it contributes to final task success. 21
Table 7: Simulation π0.5 executor training configuration. Item
Configuration
Model Model size Backbone Action expert Distributed training Precision Optimizer Learning rate LR schedule Warmup steps Weight decay Global batch size Training step Max text length
Pi05 3.62B parameters PaliGemma, width 2048, depth 18 width 1024, depth 18 DDP bfloat16 AdamW, β = (0.9, 0.95), ϵ = 10−8 5 × 10−5 Cosine decay with min LR 5 × 10−6 2,000 1 × 10−10 8 × 8 = 64 30, 000 200 tokens
RoboTwin. For procdural data generation, subtasks come from the expert script itself. Each task file defines an expert procedure in its play once() function. We manually wrap consecutive expert commands with record subtask. Each wrapped block corresponds to one semantic part of the task. For example, in a bottle-to-dustbin task, the expert script contains blocks of the following form:
with self.record_subtask("pick up the cola"): self.move(self.grasp_actor(bottle, arm_tag=arm_tag, pre_grasp_dis =0.1)) self.move(self.move_by_displacement(arm_tag, z=0.1)) with self.record_subtask("place the cola into dustbin"): self.move((ArmTag("left"), [left_end_action])) self.move(self.open_gripper("left"))
The text passed to record subtask becomes the raw subtask description. The commands inside the block define the time span of that subtask. During data generation, the switching signal is the control flow of the expert script. More specifically, a subtask starts when the program enters a record subtask block, and it ends when that block exits. The implementation is a Python context manager. When the block is entered, the current simulator frame index is saved as start frame. When all commands inside the block finish and the program leaves the block, the current frame index is saved as end frame. The next subtask begins when the program enters the next record subtask block. RoboTwin evaluation follows the data-scaling protocol described in the main paper. We report success rates on short-horizon tasks, long-horizon tasks, and the overall benchmark. In RoboTwin, we evaluate the proposed dual-system architecture using an online hierarchical control protocol that decouples semantic task progress estimation from low-level action generation. For each evaluation episode, System-1 operates as a vision-language-action policy server that produces short-horizon action chunks conditioned on the currently committed subtask instruction, while System-2 runs as an independent semantic scheduler that periodically infers the active subtask and updates a language memory from multi-view observations and recent robot state history. In the no-subtask-list setting, System-2 is not provided with the ground-truth ordered subtask sequence; instead, it receives only the global task goal, the previously committed language memory, historical visual observations, and proprioceptive state traces, and outputs a free-form prediction of the current subtask together with an updated memory. To prevent unstable frame-wise semantic predictions from directly perturbing 22
the controller, all System-2 outputs are first processed by a local scheduler, which matches the predicted subtask against an episode-specific subtask plan maintained only on the evaluator side and accepts a transition only when the match confidence, maximum allowed subtask advance, and minimum dwell-time constraints are satisfied. System-1 therefore never consumes raw System-2 predictions directly; it is re-invoked only after the local scheduler commits a new subtask context. This asynchronous evaluation design allows System-1 to execute continuous action chunks at a high control frequency, while System-2 performs lower-frequency semantic monitoring and subtask switching, providing a stable interface between long-horizon task reasoning and closed-loop robotic control. Table 8: Success rates (%) on the RoboTwin 2.0 benchmark under the data-scaling setting (50 clean + 500 randomized demonstrations). Method
short horizon
long horizon
RoboTwin (Overall)
ACT [43] RDT-1B [44] OpenVLA-OFT [40] DP3 [45] π0 [39] X-VLA [46] π0.5 [19]
33.20 35.73 42.10 62.67 61.50 77.13 82.60
24.50 32.75 32.80 53.45 72.55 66.30 82.95
29.70 34.50 38.30 55.24 65.92 72.80 82.74
Cortex (Ours)
86.00
88.00
86.80(+)
RMBench. RMBench emphasizes task memory complexity. We therefore evaluate not only physical success but also whether the generated subtask sequence preserves task-relevant memory variables such as object order, object count, and previously modified states. This setting is particularly useful for diagnosing failures caused by semantic drift rather than low-level motor errors. Table 9: RMBench benchmark results. We report success rates across seven manipulation tasks for six policies, each trained with 50 synthesized demonstrations and evaluated over 100 rollouts. Method
Observe and Pick Up
Rearrange Blocks
Put Back Block
Swap Blocks
Swap T
Battery Try
Press Button
DP [47] ACT [43] π0.5 [19] X-VLA [46] Mem-0 [12]
1% 1% 9% 9% 4%
0% 29% 13% 13% 89%
0% 0% 11% 18% 90%
11% 2% 24% 16% 67%
20% 2% 15% 3% 14%
10% 19% 16% 26% 28%
0% 0% 0% 0% 0%
Cortex (Ours)
14%
100%
100%
99%
63%
37%
20%
State history as text. Some tasks require high-frequency progress estimation that is not reliably recoverable from sparse image observations alone. For example, the robot may need to infer how many times a button has been pressed or whether a handover has already occurred. We therefore inject a causal history of robot states as text. For each sample, we read the most recent H = 30 state slots with stride 1 from observation.state. We truncate the raw state to at most Dmax = 32 dimensions; lower-dimensional states keep their native dimensionality. Each retained scalar is then converted to a bounded integer token: s̄i,d = tanh(si,d ), B (s̄i,d + 1) , 0, B − 1 , bi,d = clip 2 23
(14) B = 256.
(15)
normalized value
1
P state motion, d |st,d − st−1,d | completed press count
0.5
0
0
2
4
6
8
10
12
14
16
18
time in episode 0 of press button (s)
Figure 11: State history exposes high-frequency progress in a representative press button episode. The blue curve plots the normalized per-frame motion magnitude of the raw robot state, and the red staircase indicates the number of completed button presses from the subtask annotation. The model does not receive this hand-drawn curve; it receives the 30-step quantized state history and delta summary as text. The plot visualizes why this signal is useful when the same subtask phrase is repeated many times. Missing history rows are represented by the token value −1. The prompt contains the quantized sequence from oldest to current, together with summary features including per-step L1 changes, the number of changed dimensions, the oldest-to-current delta, and whether the most recent transition changed. This textual state representation is intentionally model-agnostic: it can be parsed by the language backbone without adding a new numeric encoder. Figure 11 illustrates why this state channel is important. In the RMBench press button task, the same action phrase (“Press the middle button one time”) appears repeatedly, and the correct next subtask depends on how many presses have already happened. Sparse visual queries may only show the arm near a button, while the robot state contains high-frequency joint and gripper motion at every control frame. We therefore do not require System-2 to infer the count solely from isolated images; the quantized state-history text exposes the recent actuation pattern. Prompt variants and supervision target. Each training example is formatted as a chat-style instruction to Qwen-VL. The input contains the global task goal, the input language memory, optional additional guidance, the robot state history, and the multi-view observation history. The assistant target is a JSON object: zt = "current subtask" : ay(t) , "active language memory" : my(t) .
(16)
The target memory is not a chain-of-thought trace; it is a concise semantic state that can be fed back to System-2 at the next query. Figure 12 shows the exact structure of the training prompt. The example is taken from press button, where the task requires pressing the left button twice, the middle button seven times, and the confirm button once. For space, the state history is abbreviated; in training, all 30 state rows and all selected state dimensions are emitted. SYSTEM: You are a robot program for high-level manipulation. Given the global task goal, an optional detailed global task instruction, an optional ordered subtask list, the input language memory, and the history and current camera observations, predict the subtask the robot should currently be in and the language memory that should be active now. Return JSON only with keys "current subtask" and "active language memory". USER: Global Task Goal: "Observe the two numbers on the table. Press the left button the number of times corresponding to the number on the left, and press the middle button the number of times corresponding to the number on the right. Then press the right button once to confirm."
24
Input Language Memory: I pressed the left button twice and the middle button six times. Robot State History (causal, oldest to current, 30 state slots, stride=1 state step, 14-D, values are 256-bin integers after tanh-squashing raw state to [-1,1]; -1 means padded/missing state dimension): t-29: 128 249 230 37 126 173 128 128 ... t-28: 128 249 230 37 126 173 128 128 ... ... t-2 : 128 249 230 37 126 173 128 128 ... t-1 : 128 248 229 37 126 171 128 128 ... t : 128 247 228 37 126 169 128 128 ... Robot State Delta Summary: per step l1 x1000: 0 0 0 4 19 48 86 74 12 ... changed dims per step: 0 0 0 2 4 5 6 5 2 ... oldest to current delta x1000: 0 97 83 0 0 41 0 0 ... state change run count: 1 current transition changed: 1 ... ASSISTANT: {"current subtask": "Press the confirm button.", "active language memory": "I pressed the left button twice and the middle button seven times."}
Figure 12: Training prompt template for System-2. The prompt contains task-level language, input memory, optional ordered subtask guidance, quantized causal state history, delta summaries, and multi-view image tokens. The target is a minimal JSON object rather than free-form reasoning. A.5 A.5.1
Real-World Robot Experiments Hardware Setup
Real-world experiments are conducted on an ARX ACONE dual-arm platform equipped with RGB cameras for scene observation. The robot is evaluated in tabletop manipulation scenes containing household objects, kitchen tools, and chemistry-style containers. Unless otherwise specified, all experiments are initialized from manually reset scene configurations, while subsequent subtask transitions are determined online by Cortex rather than by human intervention. System-1 runs closed-loop action generation at approximately 10 Hz, while System-2 performs high-level verification and subtask generation at approximately 2 Hz. A.5.2
System-1 Executor
sub The real-world executor is instantiated as a MEM-style VLA policy [5], denoted as πmem , with a short several-second memory window. It is trained on long-horizon demonstrations collected in the target robot setup, but its action prediction is conditioned on the current RGB observation, the short memory window, and the executable subtask accepted by the harness. In contrast to the tasklevel end-to-end baseline πmem , which is conditioned on the original long-horizon task instruction, System-1 receives a short, physically grounded command such as “pick up the glass beaker” or “place the cup into the microwave”. This conditioning suppresses irrelevant future steps and reduces ambiguity when multiple visually similar objects or repeated skill primitives appear in the same episode. Training hyperparameter are shown in the following table 10.
Executor training data. System-1 training data are segmented into subtask-level clips with paired language annotations and continuous action trajectories. For complex real-world tasks, manually calibrated boundaries are used for a subset of demonstrations to establish reliable supervision around 25
Table 10: Real-world πmem executor training configuration. Item
Configuration
Model size Backbone Action expert Distributed training Precision Optimizer Learning rate LR schedule Warmup steps Weight decay Global batch size Training step Max text length Action dimension Action horizon Temporal frames Temporal stride Temporal state tokens Temporal attention
3.62B parameters PaliGemma, width 2048, depth 18 width 1024, depth 18 DDP bfloat16 AdamW, β = (0.9, 0.95), ϵ = 10−8 5 × 10−5 Cosine decay with min LR 5 × 10−6 2,000 1 × 10−2 4 × 8 = 32 30,000 200 tokens 32 100 4 frames 20 Enabled Every 4 layers
contact-rich transitions such as grasping, pouring, placing, and tool release. The remaining trajectories can be aligned by the automated segmentation pipeline described in Section A.1. This hybrid strategy is intended to preserve execution precision while reducing the amount of manual annotation required for long-horizon data collection. A.5.3
Deployment Harness
Long-horizon execution requires close coordination between System-2 planning and System-1 control, while real-world deployment often introduces transient occlusions, delayed state changes, and unexpected perturbations. To stabilize this asynchronous dual-system loop, the deployment harness serves as a lightweight arbitration layer between the low-frequency planner and the high-frequency executor. At each System-2 inference step, the harness receives a candidate subtask and memory update. Instead of forwarding every prediction directly to System-1, it performs command holding and abnormal-transition filtering. A new command is accepted only when it can be mapped to an executable System-1 command and does not exhibit abnormal high-frequency switching across adjacent System-2 predictions. Otherwise, the previous command is preserved, allowing System-1 to continue high-frequency action generation without discontinuous interruptions. Standardized Command Mapping. Because System-2 generates open-ended language while System-1 is trained with standardized command annotations, the harness normalizes each generated subtask into an executor-compatible command. Exact template matching is used when the generated command follows a canonical skill template. For semantically equivalent but surface-level different expressions, such as “grasp the beaker” and “pick up the beaker”, keyword matching is used as a conservative fallback. When multiple executable labels are plausible, the harness selects the output according to the abnormal-switching filter described above. These standardization and conservative selection rules reduce distribution shifts caused by surface-form language variation, encouraging System-1 to condition on the intended subtask command rather than scene-specific biases in the training data. Timeout-driven Action Processing. Real-world perception and execution are not perfectly synchronized. In some cases, System-1 may complete a motion before System-2 visually confirms the transition, causing the robot to pause near a boundary state. To prevent deadlock, the harness applies 26
a timeout mechanism. If System-2 produces no accepted transition within a predefined interval, the robot executes a slow, low-amplitude corrective or reset motion. This motion does not force task progress; instead, it refreshes the visual evidence available to System-2 while minimally perturbing the task state. This mechanism is particularly important in the beaker washing and chemical liquid stirring tasks (Appendix A.5.7 and Appendix A.5.8), where the most informative head-view observation comes from a relatively high camera viewpoint and therefore differs from much of the training distribution. Under this harder zero-shot setting, the timeout-driven escape from local deadlocks substantially improves real-world reliability without requiring additional System-1 augmentation. A.5.4
Zero-Shot Deployment Capabilities
The following real-world experiments analyze two capabilities that are central to Cortex’s zero-shot deployment: prompt-mode flexibility and subtask-level progress feedback. The former allows Cortex to accommodate different prompt specifications while preserving the correctness of the overall plan, whereas the latter stabilizes long-horizon execution by deciding when to preserve the current subtask, update memory, or advance the procedure. Prompt-mode flexibility. Cortex supports coarse goals, detailed procedural instructions, and explicit subtask lists within a unified deployment interface. This flexibility is important in zero-shot settings because real-world tasks differ in how much procedural structure the prompt should provide. We use the trash disposal experiment as the primary case study for prompt-mode comparison (Fig. 14) because it is compact enough for controlled comparison while still requiring object grounding and completion verification. Subtask-level feedback—enabling trial-and-error. Cortex maintains real-time feedback during physical execution. When the observed physical state deviates from the nominal plan, Cortex can identify the mismatch instead of blindly switching to the next subtask according to a fixed order or common procedural logic, or persistently maintaining the current subtask without reassessment. Benefiting from memory compression and temporal alignment during training, Cortex jointly considers the input memory, task instruction, and current observation to determine whether the current subtask has been completed and whether the state recorded in memory should be updated. It then updates both the executable subtask and the memory accordingly. This capability is particularly important under real-world perturbations, partial execution failures, or delayed visual verification. We evaluate this capability through two representative cases: a human-induced state perturbation and a local execution failure during robot manipulation. In the perturbation case, the nominal task requires the robot to pick up a water bottle, open its cap, and pour water. Before execution, we manually open the bottle cap. Cortex observes this changed state and automatically appends the memory update “I opened the cap of the bottle”. After picking up the bottle, it directly proceeds to the pouring subtask instead of redundantly outputting an open-cap command. In the local-failure case from the chemical liquid stirring task shown in Fig. 17, the robot attempts to grasp a stopper from the tube rack. Because the stopper rotates under contact and becomes difficult to pinch, the first attempts fail. Cortex detects that the stopper has not been picked up, preserves the current memory and subtask, and only advances to the next subtask after the third attempt succeeds. These cases directly test whether Cortex performs online success verification rather than blindly following a precomputed subtask list. A.5.5
Experiment 1: Oven Heating
The oven heating experiment represents a common kitchen task with only a few relevant objects and a clear causal order. The task requires the robot to open the oven door, pick up the dish, place it into the oven, and close the door. It mainly evaluates whether Cortex can switch between subtasks at the correct physical boundaries while preserving the logical dependency between consecutive actions. Experiment setting. 27
Task instruction: “Put the dish into the oven to heat it up.”
Figure 13: Multi-view rollout of the oven heating experiment. From top to bottom, the rows show the head, left-wrist, and right-wrist views. Each column shows an action-switching frame, and the displayed output is the current subtask. Qualitative results. For such common household tasks, Cortex can produce a correct task plan from a short input instruction and select appropriate transition times between subtasks. In general, the model switches to the next subtask only after confirming that the previous subtask has been fully completed. As shown in Fig. 13, before switching to the second subtask, the oven door has already fallen completely onto the table surface rather than remaining in the air, indicating that System-2 waits for a stable completion state before issuing the command to pick up the plate. A.5.6
Experiment 2: Trash Disposal
The trash sorting experiment is an unordered manipulation task over multiple objects from similar categories. The core challenge is not procedural complexity, but whether the system can robustly identify each target object and generate expressions with a unique visual referent. We use this experiment to verify the effectiveness of the multi-mode System-2 interface and, based on these observations, further optimize the training of System-1. Experiment setting. Task instruction: “Sort the garbage on the desktop into recyclable and non-recyclable.” Detailed task setting: “Pick up the red plastic wrapper, the yellow can and the orange can with the right arm, place each item into the blue bin.” Qualitative results. Figure 14 shows that both prompt modes produce valid long-horizon executions, but they differ in how explicitly the intermediate subtasks are grounded. In the detailed mode, the generated commands are more object-specific and the action switches align closely with the intended pick-and-place sequence. In the coarse mode, the model must infer both the relevant targets and their execution order from the scene, leading to a more implicit decomposition of the same task. Prompt-mode qualitative comparison. This experiment compares prompt granularity under the same physical task. In detailed mode, the instruction explicitly names the red plastic wrapper, yellow can, orange can, and blue bin, so System-2 produces more object-specific subtasks and follows a cleaner pickup–place sequence. This mode is advantageous when the task requires precise target grounding, stable action ordering, and reliable completion checks across multiple similar objects. In coarse mode, System-2 receives only the high-level sorting goal and must infer the target objects and disposal order from the scene. This mode is more flexible and better reflects open-ended task understanding, because the system can decompose an underspecified goal into executable subtasks online. The comparison therefore shows a tradeoff: detailed mode improves precision and temporal stability, while coarse mode tests semantic generalization and autonomous task decomposition. 28
Figure 14: Prompt-mode comparison for trash disposal. Each frame is an action-switching frame, and the displayed output is the current subtask. The top row shows detailed mode, and the bottom row shows coarse mode. A.5.7
Experiment 3: Beaker Washing
The beaker washing experiment evaluates a longer manipulation chain involving tool or container handling, placement near a washing area, and repeated state changes. Compared with trash disposal, this task places stronger requirements on memory because the beaker may need to be moved through multiple spatial regions before the final washed state is achieved. Experiment setting. Task instruction: “Wash the beaker using the water from the bottle.” Detailed task setting: “First, pick up the beaker, then place the beaker on the platform. Next, pick up the water bottle from the table. After opening the cap of the water bottle, place its cap on the table. Then, pour the water into the beaker and place the held water bottle back to the table. Later, pick up the beaker from the platform, pour water from the beaker into the kettle. Finally, place the beaker back on the platform.”
Figure 15: Detailed subtask prediction and execution process of Beaker Washing task 29
Qualitative results. The beaker washing task stresses both long-horizon state tracking and local execution stability. The robot must first move the beaker to the platform, manipulate the bottle and its cap, pour water into the beaker, return the bottle, and finally transfer the washed beaker state to the kettle-side operation. These stages contain repeated primitives such as picking, placing, and pouring, but the correct action depends on the accumulated procedural state rather than the current image alone. Cortex handles this task by separating procedural tracking from local control. System-2 maintains the active memory over bottle state, cap state, beaker location, and washing progress, and only sub advances when the current subtask is visually verified. The executor πmem therefore receives a localized command at each stage, which reduces ambiguity when the same object appears in multiple visually similar configurations. Comparison with end-to-end VLA. Figure 16 shows representative end-to-end failures in the beaker washing task. The first baseline, π0.5 , has no explicit memory. It can execute short primitives, but it often fails to decide when a reciprocal manipulation phase has ended. For example, after picking the bottle or entering a pouring-related motion, it may repeatedly revisit the same local behavior instead of committing to the next subtask, which prevents the full washing procedure from completing. The second baseline, the task-level πmem , uses the same short several-second memory design as our executor and can therefore finish some short local sequences more reliably. However, conditioning it on the original long-horizon task instruction forces the policy to infer the current subtask stage from visual context alone. In this long-horizon task, several stages differ only subtly in the image, such as holding the bottle before versus after cap opening, or approaching the beaker before versus after water has been poured. As a result, πmem may confuse the cap-opening, bottle-lifting, and pouring phases, causing out-of-order behavior such as attempting to pour before the cap has been opened. sub In contrast, Cortex combines explicit subtask-memory tracking with πmem . System-2 preserves the intended procedure and dispatches only the currently executable subtask, while the low-level policy focuses on executing that localized command. This design keeps the task order stable even when adjacent stages are visually similar.
A.5.8
Experiment 4: Chemical Liquid Stirring
The liquid stirring experiment evaluates a chemistry-style workflow with rare objects, strict action order, and fine-grained success conditions. It is the most suitable task for detailed or subtask-mode prompts because the system must preserve procedural order while maintaining stable memory over multiple object interactions. Experiment setting. Task instruction: “Pour liquid from the graduated cylinder into the round-bottom flask with the aid of a funnel, then place the flask on the stirrer and press the switch to start stirring.” Detailed task setting: “First, pick up the funnel on the tube rack. Then insert it into the roundbottom flask on the platform. Next, pick up the graduated cylinder from the right side of the table and lift the held graduated cylinder close to the funnel. Then, pour the liquid from the graduated cylinder into the flask through the funnel. Place the held cylinder back on the table. Next, approach the top of the flask on the platform and split the upper part funnel. Place the held funnel back on the rack. Then, pick up the round-bottom flask from the table and place it on the black stirrer. Next, pick up the stopper from the tube rack. Insert the stopper into the round-bottom flask. Finally, press the white switch on the right bottom of the stirrer to turn it on.” Qualitative results. The chemical liquid stirring task is more order-sensitive than the household tasks. It involves rare laboratory objects and a fixed procedure: inserting the funnel, lifting the graduated cylinder, pouring through the funnel, removing and returning the funnel, moving the flask 30
Figure 16: Performance comparison across different baselines in the beaker washing task. π0.5 tends to repeat memory-sensitive reciprocal motions, while the task-level πmem can confuse visually similar task phases; Cortex maintains the correct order through explicit subtask routing and transition verification. ✓ indicates successful subtask execution, × indicates failure and ※ indicates wrong subtask order.
31
to the stirrer, inserting the stopper, and pressing the switch. Many stages are contact-rich and visually subtle, so advancing too early can invalidate the remaining procedure. Cortex maintains the current subtask during repeated or contact-rich motions and advances only after the relevant completion evidence is observed. This behavior is especially important for insertion and removal operations, where a small pose change can determine whether the object is actually inserted, lifted, or released. The memory state provides the long-horizon procedural context, while sub πmem executes each verified local command. Retry behavior. When stopper grasping fails locally, Cortex preserves the active subtask and reattempts execution until visual evidence supports completion, rather than prematurely advancing the procedure.
Figure 17: Local execution failure during stopper grasping. Cortex keeps the current subtask and memory until the grasp succeeds. Comparison with end-to-end VLA. Figure 18 summarizes representative chemical-task failures. Because π0.5 does not maintain an explicit memory state, it struggles to terminate insertion and removal-style subtasks. In practice, it can remain stuck around local contact behaviors, such as repeatedly trying to insert or remove the funnel, because the current observation alone does not provide a reliable signal that the procedural stage has been completed. The task-level πmem reduces some of this short-term repetition, but it still conditions only on the the overall task instruction. This makes adjacent chemistry stages easy to entangle: the visual difference between different subtask may be small, while the required action order is strict. A typical failure is executing a pouring motion before the graduated cylinder has been properly lifted toward the funnel. Cortex avoids these failures by using System-2 to maintain the long-horizon procedural state and sub . This keeps the controller grounded to the by dispatching only the verified current subtask to πmem correct stage while preserving the ability to repeat local motions until visual verification confirms completion.
32
Figure 18: End-to-end failure cases in the chemical liquid stirring task. π0.5 fails to terminate memory-sensitive insertion/removal phases, while the task-level πmem can confuse adjacent procedural stages; Cortex preserves task order through explicit memory and subtask-conditioned execution. ✓ indicates successful subtask execution, × indicates failure and ※ indicates wrong subtask order.
33