Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy Junhao Shi1,2,∗ Zezheng Huai2,3,∗ Siyin Wang1,2 Jia Chen2 Yubang Wang2 Zhaoye Fei1 Hechang Chen2,3 Jingjing Gong2 Xipeng Qiu1,2,† Yu-Gang Jiang1,†
arXiv:2606.27251v1 [cs.RO] 25 Jun 2026
[email protected], [email protected] 1 Fudan University 2 Shanghai Innovation Institute 3 Jilin University
Abstract Building persistent embodied agents in unstructured environments demands unified orchestration of heterogeneous tools spanning both cyber (APIs, IoT) and physical (manipulation, navigation) domains, coupled with autonomous recovery from physical failures that inevitably arise over extended operation. Existing systems treat these as separate problems: VLM-based planners lack a unified cyber-physical action space, agent frameworks accumulate unbounded context that degrades temporal coherence, and VLA policies execute open-loop without detecting their own failures. We argue that persistent autonomy requires not a monolithic model but a hierarchical asynchronous architecture with explicit separation of planning, memory, and verification. To this end, we present OmniAct, a framework integrating a multimodal semantic planner for skill routing across unified action spaces, an adaptive hierarchical memory with event-boundary-driven compression for sub-linear context growth, and an asynchronous visual preemption engine that closes the semantic loop during physical execution. Across 40 real-world long-horizon tasks on two robotic platforms coordinating four IoT devices, OmniAct achieves consistent improvements in end-to-end success across all complexity levels, maintains near-flat token consumption over under 100k+ accumulated interaction tokens, and elevates mid-scale open-weight models to proprietary-level performance. Project Page: https://huaizz-shawen.github.io/OmniAct GitHub Code: https://github.com/EmbodiedForge/RAS_interactivate_planner
1
Introduction
The fundamental goal of embodied intelligence is to build persistent physical agents that operate reliably in unstructured real-world environments over extended temporal horizons. This requires the simultaneous integration of multimodal perception spanning speech, vision, and language, unified actuation across heterogeneous tools including IoT devices, Web APIs, robotic manipulators, and mobile platforms, as well as autonomous recovery from unforeseen physical failures. However, current embodied systems, whether end-to-end vision-language-action models or LLM-based planning frameworks, encounter fundamental lim∗ Equal Contribution. † Corresponding Authors.
1
(a) Cyber-Physical Coordination
(b) Resilient Long-Horizon Execution
Figure 1 Two key challenges for real-world long-horizon embodied deployment. (a) A single user command may require joint orchestration of IoT devices, robotic manipulators, mobile platforms, and web APIs. (b) Over extended horizons, unexpected physical failures must be autonomously detected and recovered from without human intervention.
itations when deployed in such demanding real-world scenarios [1–4]. Existing approaches fall short in three critical aspects. First, current systems treat tool invocation (APIs, IoT) and physical manipulation as separate pipelines, yet realistic user interactions through natural speech often implicitly require coordination of multiple heterogeneous tools within a single task flow [5–7]. The absence of a unified cyber-physical action space forces these methods to rely on brittle hand-crafted interfaces between domains, severely limiting their capacity to handle naturally interleaved instructions. Second, agent frameworks that linearly concatenate interaction history accumulate context rapidly during long-term deployment, increasing memory overhead, losing critical historical information, and introducing semantic conflicts between long-term preferences and recent states [8–11]. Third, though current end-to-end VLA policies [12–16] have achieved notable progress, their inherently open-loop nature cannot support autonomous operation: undetected physical failures cascade into irreversible faults from which the system cannot recover [3]. As illustrated in Figure 1, achieving everyday physical autonomy faces two key challenges: (i) coordinating heterogeneous cyber-physical tools within single task flows, and (ii) autonomously recovering from physical failures over extended horizons. To address these challenges jointly, we observe that persistent real-world autonomy requires not a monolithic end-to-end model, but a hierarchical asynchronous architecture that explicitly decouples high-level planning, long-horizon state tracking, and real-time physical verification into specialized, cooperating modules. Consequently, we introduce OmniAct, an embodied framework achieving true “omni-action” by seamlessly bridging omnimodal perception (speech, vision, language) with a unified cyber-physical action space (IoT, Web APIs, robotic control). OmniAct comprises three tightly integrated modules: (1) a multimodal semantic planner for structured skill-routing; (2) an adaptive hierarchical memory utilizing event-boundary-driven compression to maintain sub-linear context growth; and (3) an asynchronous visual preemption engine that periodically verifies physical execution and triggers immediate replanning upon detecting anomalies. We evaluate OmniAct across 40 real-world long-horizon tasks on two structurally distinct platforms (UR5e manipulator and wheeled mobile robot) coordinating four household IoT devices. Experiments show that OmniAct substantially outperforms both open-loop and step-wise reasoning baselines, achieving higher endto-end success and improved perturbation recovery(Sec. 4.2), reduced token consumption with near-flat context growth (Sec. 4.3), and strong cross-model generalizability (Sec. 4.4).
2
Our contributions are threefold: 1. We introduce OmniAct, a multimodal embodied framework that unifies discrete cyber tools (APIs, IoT) and continuous physical control into a single event-driven loop with asynchronous visual preemption, enabling closed-loop persistent deployment across diverse robotic platforms without force sensing. 2. We design an adaptive hierarchical memory that condenses long-horizon interaction history into timestamped semantic cues through event-boundary-driven compression and integrates explicit physical reflection, enabling the agent to maintain bounded context growth while autonomously recovering from failures and adapting to dynamic environmental shifts. 3. We conduct extensive real-world empirical validation across diverse physical environments and robotic platforms. Compared to reactive policies and LLM-based planning baselines, our framework significantly improves end-to-end success rates in failure-prone scenarios while substantially reducing token consumption and inference latency.
2
Related Work
2.1
Embodied Agents with Foundation Models
The integration of foundation models into embodied systems has progressed rapidly. Early work leverages large language models as high-level planners that decompose natural language instructions into executable skill sequences, either through affordance grounding [5], code generation [17, 18], or iterative replanning with textual feedback [19]. Subsequent agent architectures extend these planners with autonomous reasoning capabilities, incorporating chain-of-thought tool invocation [8], verbal self-reflection for trial improvement [9], and persistent skill accumulation for open-ended exploration [10]. More recent efforts further integrate multimodal perception into the agentic loop: PaLM-E [20] grounds planning in visual and sensor observations, WAP [21] advances planning toward closed-loop continuous setting. RoboBrain [22, 23] combines manipulation planning with affordance reasoning, and hierarchical frameworks such as RT-H [24] bridge high-level language to low-level motor execution through structured sub-goal decomposition. Despite this progress, existing approaches operate within either the cyber or physical domain without a unified action space, mostly focus on short interactions without addressing long-horizon context accumulation toward real-world autonomy. OmniAct addresses these gaps by unifying cyber tools and physical control into a single skill-routing space, introducing event-boundary-driven hierarchical memory for bounded context growth, and employing asynchronous visual preemption for semantic-level closed-loop verification.
2.2
Vision-Language-Action Models
Vision-Language-Action (VLA) models directly map visual observations and language instructions to lowlevel control signals. Early architectures explored various fusion strategies including feature modulation [25, 26], cross-attention [12, 27], and token concatenation [28]. Subsequent large-scale pre-training on diverse robotic datasets significantly improved model capabilities [13, 29]. Concurrently, advances in action representation such as autoregressive tokenization [30] and diffusion-based generation [14, 15, 31], together with richer input modalities including 3D geometry [32] and tactile feedback [33, 34], have further enhanced generalization across environments and task horizons. Most recently, RoboOmni [16] extends VLA to accept speech input and enable proactive interaction, moving closer to natural human-robot communication. Nevertheless, current VLA models remain limited in cross-embodiment transfer, multi-task versatility, and zero-shot generalization to novel scenarios. More critically, they lack inherent reasoning capabilities and cannot autonomously detect or correct their own execution failures. Omni does not seek to replace VLA models but instead incorporates them as low-level executors within a unified orchestration framework, compensating for these limitations through asynchronous visual verification for failure detection and hierarchical memory for long-horizon state coherence.
3
Multimodal Perception & Planning Speech
Adaptive Hierarchical Memory Sensory Memory
Multimodal Semantic Planner Task Decomposition
Intent Interpretation
Image
Timestamp
State
Function Return
Recent frames buffer
Skill Routing
Compressed Memory
Success / Preempt Skill + Instructions
Physical Executor
Cyber Executor Skill Dispatch
Chat history Event Boundary Compression
Instruction + Image
Control Policy
Execution
Interrupt (Anomaly)
Execution
Success Failure
Episodic Memory
Periodic image Sampling
Normal
VLM Verification
Reflective Memory Tool Schemas
Failure Reflections
User Preference
Learned Experience
Anomaly Failure Reflect
Figure 2 Overview of the OmniAct framework. The system comprises three modules: (1) Multimodal Semantic Planner fuses speech, image, and state inputs for intent interpretation, task decomposition, and skill routing; (2) Closed-Loop Execution Engine with asynchronous visual preemption that periodically verifies physical progress via a VLM and triggers interruption upon anomaly; and (3) Adaptive Hierarchical Memory (sensory, episodic, reflective) employing eventboundary compression for bounded context growth and storing failure reflections for cumulative adaptation.
3
Methodology: The OmniAcr Framework
3.1
Problem Formulation
A key challenge in building persistent embodied agents is to enable a single planning system to coordinate diverse executors spanning both cyber (APIs, IoT devices) and physical (robotic manipulation, navigation) domains under partial observability. We formulate this as a decision process over a unified cyber-physical action space. At each time step t, the system receives a multimodal observation ot = vt , st , where vt is a visual frame and st denotes discrete state information from sensors or tool APIs. The multimodal context ct integrates ot with the current user intent (e.g., transcribed speech) and system state. Since the system must dispatch to fundamentally different executors ranging from stateless API calls to continuous robotic control, a single low-level action representation is insufficient. We therefore define a skill-routing action space: each action is a tuple at = τt , gt , where τt ∈ T selects the target skill and gt specifies skill-specific conditioning parameters. The historical trajectory Ht , retrieved from hierarchical memory (Sec. 3.4), provides compressed prior experience. Given the planning model π, the high-level planning policy is: at ∼ π· | ct , Ht , T .
3.2
(1)
Multimodal Semantic Planner
The semantic planner instantiates model π and is responsible for task decomposition, skill selection, and sub-goal generation. Given context ct and memory Ht , the planner performs three functions: (1) interpreting potentially ambiguous user intent by grounding speech and visual observations against the current tool state; (2) decomposing long-horizon instructions into executable sub-goals, selecting appropriate skills τt from inventory T based on task requirements and environmental constraints; and (3) leveraging historical experience from Ht to avoid previously observed failures and respect long-term user preferences. The planner outputs a structured JSON specification supporting parallel dispatch of multiple skill calls within a single step. Details is provided in the Appendix.
4
3.3
Closed-Loop Execution with Asynchronous Visual Preemption
A fundamental limitation of VLA policies is their open-loop nature: once committed, the executor cannot perceive whether the physical world has deviated from expectations, and a single undetected failure propagates into cascading faults. OmniAct addresses this through an asynchronous visual preemption mechanism that maintains a low-frequency semantic closed-loop concurrent with physical execution. The execution engine dispatches actions along two parallel streams. Discrete cyber commands (IoT control, API calls) are verified through deterministic return codes. Continuous physical commands are forwarded to the control policy, for which no such verification signal exists. To compensate, an asynchronous visual monitor periodically samples frames and submits them to a VLM for semantic-level assessment of execution progress and physical safety. If anomaly is detected, the system immediately halts the VLA executor, packages the failure context, and returns control to the planner for replanning. Unlike geometric constraints in model-predictive control that verify joint limits or collision distances, this mechanism reasons about high-level task semantics, whether the target object remains grasped, whether the robot is progressing toward the goal—capturing failure modes invisible to low-level controllers. Upon preemption or failures from the cyber executor, the failure episode simultaneously triggers a reflection process that stores causal analysis and corrective strategies in long-term memory (Sec. 3.4) to prevent recurrence.
3.4
Adaptive Hierarchical Memory
A persistent agent must maintain awareness of its full interaction history, yet naively accumulating raw observations causes linear context growth that quickly exceeds model capacity. Our key observation is that physical interaction naturally segments into discrete semantic events, and compression should operate at these event boundaries rather than at fixed intervals or uniform turn-level granularity. We design a threelevel memory hierarchy with increasing abstraction: Sensory Memory. A lightweight FIFO queue maintaining recent visual frames for the control policy and visual monitor, decoupled from the planning context. Episodic Memory. This level manages interaction history through event-boundary-driven compression. Each record is annotated with absolute timestamps for temporal ordering. Upon detecting a semantic boundary (e.g., sub-task completion, state change), the raw observation sequence is compressed into a compact keyframe summary retaining semantic content, state transitions, and action outcomes while discarding procedural redundancy. Context thus grows proportionally to distinct semantic events rather than raw turns, yielding sub-linear scaling over extended deployment. Reflective Memory. This level stores both static priors (tool schemas, skill inventory) and dynamically accumulated experience. When execution failure or visual preemption is triggered, the system generates a structured record of root cause and corrective strategy. During subsequent planning, relevant reflections are retrieved and injected into context, transforming isolated error recovery into cumulative adaptation over the agent’s operational lifetime.
4
Experiments
We structure our evaluation around three axes: cross-domain orchestration across heterogeneous hardware (Sec. 4.2), memory effectiveness and scalability under long-horizon deployment (Sec. 4.3), and generalizability across foundation models of varying scale (Sec. 4.4).
4.1
Experimental Setup
Platforms and Tools. We evaluate OmniAct on two robotic platforms with distinct morphologies: a UR5e 6-DoF manipulator for precision tabletop manipulation and a Keenon wheeled mobile robot for indoor navigation. The system interfaces with 4 household IoT devices (lighting, climate control, appliances) and 12
5
Task Progress (%)
Manipulation (UR5e) 100
92 64 62
62
31
0
56
61
40 25
21
L1 E2E
L2 Sub SayCan
13 12
4
L2 E2E
31 23
50
44
4
Direct Policy
Task Progress (%)
62 50
50
72
80
75
67 67
0
L3 Sub
Code as Policy
L3 E2E
Average
RoboBrain2
OmniAct (Ours)
Navigation (Keenon) 100
83
73 75 58
67
50
83 62 62 42
51 58
50
L1 E2E
L2 Sub SayCan
15
38 25
Direct Policy
51
54
46
38 12
0
74
81 67
32
22 12
L2 E2E
L3 Sub
Code as Policy
L3 E2E RoboBrain2
Average OmniAct (Ours)
Figure 3 End-to-end and sub-task success rates (%) on long-horizon cyber-physical tasks across three complexity levels. Performance is evaluated on two structurally distinct real-world platforms: a UR5e manipulator (top) and a Keenon mobile robot (bottom). While baseline methods exhibit severe performance degradation as cross-modal coordination depth increases from L1 to L3, OmniAct demonstrates a significantly more graceful decline, substantially outperforming prior approaches on both partial progress (Sub) and strict task completion (E2E). Error bars in the rightmost column denote standard deviation. Empty bars indicate that the baseline lacks the architectural capability for that specific complexity level; these are counted as zero when computing the overall average.
virtual tool APIs (web search, checklist, delivery status, etc.), forming a heterogeneous cyber-physical environment representative of realistic domestic deployment scenarios. Task Suite. To reflect the multimodal, cross-domain nature of realistic human-robot interaction, we construct a benchmark of 40 real-world tasks divided into two categories by physical modality: manipulation (20 tasks, deployed on UR5e) and navigation (20 tasks, deployed on Keenon mobile robot) , each grouped into three complexity levels by modality involvement and coordination depth: Level 1 (Physical Only, 20%): Tasks requiring only low-level robotic control (e.g., pick-and-place). Level 2 (Cyber-Physical, 40%): Tasks additionally requiring IoT device coordination or web API querie (e.g., activating an appliance before manipulation). Level 3 (Full Multimodal, 40%): Tasks involving speech interaction, web API queries, IoT control, and physical execution in an integrated flow. This stratification exposes how gracefully each method scales with cross-modal coordination depth while ensuring fair comparison on subsets accessible to all baselines. Each task is repeated 3 times with success adjudicated by human evaluation. The complete task list is provided in the appendix. Baselines. We compare OmniAct against four methods: Direct Policy, our self-trained VLA / navigation policies without planning or memory (L1 difficulty only); SayCan [5], affordance-grounded scoring with LLM planning; Code as Policy [17], one-shot code generation planning for full task plans; and RoboBrain2 [23], a 32B embodied vision-language model that unifies perception, reasoning, and planning, surpassing proprietary models on multiple embodied benchmarks [23, 35]. All methods share identical tool interfaces. Except RoboBrain2, which uses its own backbone, all employ Gemini-3.1-Pro to isolate architectural differences.
6
4.2
Long-Horizon Cyber-Physical Orchestration
To disentangle partial progress from true task completion, we report two complementary metrics: Sub-task Success Rate (Sub), the average fraction of correctly completed sub-tasks, and End-to-End Success Rate (E2E), a strict binary measure crediting only full task completion. The gap between them quantifies how severely single-stage errors cascade through the pipeline, a critical diagnostic for long-horizon autonomy where an unrecovered failure can invalidate all subsequent stages. Figure 3 reveals a consistent trend: as task complexity increases from L1 to L3, all baselines suffer steep performance degradation, while OmniAct maintains comparatively graceful decline. On the most demanding L3 tasks, OmniAct achieves 50.0% and 54.2% E2E success on manipulation and navigation respectively, significantly exceeding the best baseline (SayCan: 43.8% on manipulation; Code as Policy: 37.5% on navigation). Furthermore, OmniAct achieves the highest Sub-task success across all complexity levels, with L3 Sub reaching 80.5% on manipulation and 80.6% on navigation, compared to the best baseline scores of 60.9% and 57.9% respectively. This directly validates the contribution of asynchronous visual preemption and reflective memory: when a VLA execution fails mid-task (e.g., grasp slippage), baselines proceed obliviously and corrupt all downstream stages, whereas OmniAct detects the deviation, halts execution, and replans from the current state, converting catastrophic failures into recoverable delays. An instructive case is RoboBrain2, which despite achieving state-of-the-art on established embodied planning benchmarks, obtains a surprisingly low score. Its failures concentrate on generating well-formed API calls and maintaining state across interleaved cyber-physical sub-tasks, likely compounded by the absence of IoT-domain training data, reinforcing that cross-modal tool coordination is a fundamentally different competency from spatial scene understanding and demands an explicit orchestration layer. Finally, the consistency of gains across both the fixed manipulator and the mobile robot deserves emphasis. Despite their fundamentally different kinematic structures, workspace scales, and task semantics, OmniAct achieves comparable relative improvements on both platforms without any platform-specific architectural modification or additional tuning. This confirms that the framework’s core contributions—unified skill routing, hierarchical memory, and visual preemption—operate at an abstraction above embodiment specifics and transfer directly to new hardware configurations.
4.3
Adaptive Hierarchical Memory
Temporal Conflict Resolution. To evaluate whether OmniAct’s memory preserves long-term coherence under extended deployment, we construct three scenarios (tool use, home assistant, robotic manipulation), each containing 10 samples exceeding 10 hours of interaction with 40k+ accumulated tokens. Temporal preference conflicts are embedded throughout: early turns establish persistent constraints (e.g., no sugar”) while later requests introduce seemingly conflicting needs (e.g., something flavored”), requiring the agent to satisfy both simultaneously. We ablate against four variants: Raw History concatenates all past turns verbatim; Sliding Window retains only the most recent 10 turns; Turn Summary compresses each turn independently and concatenates linearly; Episodic Only applies event-boundary compression without reflective memory. As shown in Table 1, Sliding Window achieves the most aggressive compression (16.3%) but catastrophically discards early-established constraints, collapsing to near-zero satisfaction across all scenarios—demonstrating that recency alone is entirely insufficient for preference-aware long-horizon agents. Raw History retains all information yet performs only moderately (46.67%– 66.67%): the sheer volume of 40k+ to-
Table 1 Constraint Satisfaction Rate (%) and context compression ratio across three long-horizon scenarios. Compression is relative to raw history length.
Method
Tool Use
Home Asst.
Manip.
Compress.
Raw History Sliding Window Turn Summary Episodic Only
66.67% 13.33% 56.67% 56.67%
46.67% 6.67% 50.0% 46.67%
66.67% 6.67% 63.33% 66.67%
100.0% 16.3% 27.4% 21.2%
OmniAct (Full)
86.67%
66.67%
80.0%
24.5%
7
kens introduces positional dilution that makes relevant constraints difficult to locate, and semantic conflicts between temporally distant preferences often mislead the planner. Turn Summary and Episodic Only both achieve meaningful compression while maintaining reasonable accuracy, but neither consistently surpasses Raw History—indicating that compression without explicit long-term knowledge retrieval trades one failure mode (dilution) for another (information loss). OmniAct’s full system adds reflective memory atop episodic compression, boosting satisfaction by 13–20 points over the best ablation variant. The gain stems from the reflective layer explicitly surfacing long-term constraints and failure-derived corrective strategies at planning time, rather than relying on the planner to implicitly extract them from compressed episode sequences.
Per-call estimated input tokens
Long-Horizon Scalability. To stresstest scalability under continuous deManip. - Raw Manip. - OmniAct 160000 Home Asst. - Raw Home Asst. - OmniAct ployment, we extend the interaction 140000 Tool Use - Raw Tool Use - OmniAct traces to 100k+ tokens (approximately 120000 Raw input 100000 270 rounds) and track per-call input to80000 ken consumption over time. As shown 60000 in Figure 4, raw history replay exhibits 40000 strictly linear growth, exceeding 120k 20000 tokens by round 260 and approach0 ing the context window limits of most 10000 OmniAct foundation models—at which point ei7500 ther truncation or degraded genera5000 tion quality becomes inevitable. In 2500 contrast, OmniAct’s event-boundary00 50 100 150 200 250 Interaction round driven compression maintains per-call token usage within a narrow 4k–8k Figure 4 Per-call estimated input tokens over 270 interaction rounds band that plateaus early and fluctuacross three scenarios. Top: raw history concatenation grows linearly, ates only with local episode complexexceeding 160k tokens. Bottom: OmniAct’s hierarchical memory reity rather than accumulated history mains bounded within 2.5k–10k tokens regardless of accumulated hislength. This near-flat scaling profile tory length. fundamentally decouples deployment duration from inference cost, enabling persistent operation over arbitrarily long horizons without context overflow or progressive quality degradation.
4.4
Cross-Model Generalizability
To determine whether OmniAct’s gains stem from architectural design rather than backbone strength, we replace the planner across six models spanning proprietary (Gemini-3.1-Pro, Gemini3-Flash, Qwen3.6-Plus/Flash) and openweight (Qwen3-VL-30B-A3B, Qwen3VL-8B) families, using the same longhorizon setting as Sec. 4.3.
Table 2 Constraint Satisfaction Rate (%) with and without OmniAct across foundation model backbones. Δ represents absolute improvement.
Backbone
Raw
OmniAct
Δ
Gemini-3.1-Pro-Preview Gemini-3-Flash-Preview Qwen3.6-Plus Qwen3.6-Flash Qwen3-VL-30B-A3B Qwen3-VL-8B
76.7 70.0 63.3 56.7 50.0 30.0
80.0 76.7 80.0 73.3 80.0 36.7
+3.3 +6.7 +16.7 +16.6 +30.0 +6.7
As shown in Table 2, OmniAct improves all backbones with sufficient baseline capability. Mid-tier models exhibit the most substantial gains: Qwen3-VL-30B-A3B improves from 50.0% to 80.0% (+30.0), reaching parity with Gemini-3.1-Pro’s vanilla performance and demonstrating that OmniAct’s structured orchestration can close the gap between open-weight and proprietary systems without ad-
8
ditional training. At the upper end, even Gemini-3.1-Pro benefits (+3.3), indicating that hierarchical memory and asynchronous verification address systematic failure modes—such as long-horizon preference drift and undetected execution errors—that persist regardless of model scale. At the lower end, Qwen3-VL-8B shows only marginal improvement (+6.7), as its limited capacity to reliably produce well-formed structured outputs constrains the framework’s ability to orchestrate multi-step tool invocations—establishing an approximate lower bound on backbone capability required for effective integration.
5
Conclusions
We presented OmniAct, a multimodal embodied framework that addresses persistent real-world autonomy by decomposing planning, memory, and verification into three cooperating modules: a unified cyberphysical semantic planner, an event-boundary-driven hierarchical memory for bounded context growth, and an asynchronous visual preemption engine for closed-loop failure recovery. Real-world experiments across diverse robotic platforms and backbone models demonstrate consistent improvements over both monolithic VLA and LLM-based baselines, with near-flat token consumption over extended horizons. Notably, OmniAct elevates mid-scale open-weight models to proprietary-level performance without additional training, underscoring that architectural clarity and structured orchestration complement model capacity rather than competing with it. These results suggest that explicit separation of responsibilities, rather than monolithic model scaling, offers a more viable and resource-efficient path toward persistent embodied deployment in unstructured real-world environments.
9
References [1] Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied AI. CoRR, abs/2405.14093, 2024. [2] Siyin Wang, Junhao Shi, Zhaoyang Fu, Xinzhe He, Feihong Liu, Chenchen Yang, Yikang Zhou, Zhaoye Fei, Jingjing Gong, Jinlan Fu, Mike Zheng Shou, Xuanjing Huang, Xipeng Qiu, and Yu-Gang Jiang. World action models: The next frontier in embodied ai, 2026. URL https://arxiv.org/abs/2605.12090. [3] Kento Kawaharazuka, Jihoon Oh, Jun Yamada, Ingmar Posner, and Yuke Zhu. Vision-language-action models for robotics: A review towards real-world applications. IEEE Access, 13:162467–162504, 2025. doi: 10.1109/ACCESS. 2025.3609980. [4] Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai, 2025. URL https://arxiv.org/abs/2407.06886. [5] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not as i say: Grounding language in robotic affordances, 2022. URL https://arxiv.org/abs/2204.01691. [6] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023. URL https://arxiv.org/abs/2209.07753. [7] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. [8] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview. net/forum?id=WE_vluYUL-X. [9] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 1b44b878bb782e6954cd888628510e90-Abstract-Conference.html. [10] Yangjie Zhou, Wenting Shen, Jingwen Leng, Shuwen Lu, Zihan Liu, Weihao Cui, Zhendong Zhang, Wencong Xiao, Baole Ai, Yong Li, Wei Lin, Deze Zeng, Yun Liang, Quan Chen, Ning Liu, and Minyi Guo. Voyager: Input-adaptive algebraic transformations for high-performance graph neural networks. In Lieven Eeckhout, Georgios Smaragdakis, Katai Liang, Adrian Sampson, Martha A. Kim, and Christopher J. Rossbach, editors, Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025, pages 247–263. ACM, 2025. doi: 10. 1145/3676642.3736121. URL https://doi.org/10.1145/3676642.3736121. [11] Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems. CoRR, abs/2310.08560, 2023. doi: 10.48550/ARXIV.2310.08560. URL https://doi.org/ 10.48550/arXiv.2310.08560. [12] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael S. Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong T.
10
Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: vision-language-action models transfer web knowledge to robotic control. CoRR, abs/2307.15818, 2023. doi: 10.48550/ARXIV.2307.15818. URL https: //doi.org/10.48550/arXiv.2307.15818. [13] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Paul Foster, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard, editors, Conference on Robot Learning, 6-9 November 2024, Munich, Germany, Proceedings of Machine Learning Research, pages 2679–2713. PMLR, 2024. URL https: //proceedings.mlr.press/v270/kim25c.html. [14] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π 0 : A vision-language-action flow model for general robot control. CoRR, abs/2410.24164, 2024. doi: 10.48550/ARXIV.2410.24164. URL https://doi.org/10.48550/arXiv.2410.24164. [15] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π 0.5 : a vision-language-action model with open-world generalization. CoRR, abs/2504.16054, 2025. doi: 10.48550/ARXIV.2504.16054. URL https://doi.org/10.48550/arXiv.2504.16054. [16] Siyin Wang, Jinlan Fu, Feihong Liu, Xinzhe He, Huangxuan Wu, Junhao Shi, Kexin Huang, Zhaoye Fei, Jingjing Gong, Zuxuan Wu, Yu-Gang Jiang, See-Kiong Ng, Tat-Seng Chua, and Xipeng Qiu. Roboomni: Proactive robot manipulation in omni-modal context. CoRR, abs/2510.23763, 2025. doi: 10.48550/ARXIV.2510.23763. URL https: //doi.org/10.48550/arXiv.2510.23763. [17] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023, pages 9493–9500. IEEE, 2023. doi: 10.1109/ICRA48891. 2023.10160591. URL https://doi.org/10.1109/ICRA48891.2023.10160591. [18] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023, pages 11523–11530. IEEE, 2023. doi: 10.1109/ICRA48891.2023.10161317. URL https://doi.org/10.1109/ICRA48891. 2023.10161317. [19] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, Proceedings of Machine Learning Research, pages 1769–1782. PMLR, 2022. URL https://proceedings.mlr.press/v205/huang23c.html. [20] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Proceedings of Machine Learning Research, pages 8469–8488. PMLR, 2023. URL https://proceedings.mlr.press/v202/driess23a.html. [21] Junhao Shi, Zhaoye Fei, Siyin Wang, Qipeng Guo, Jingjing Gong, and Xipeng Qiu. World-aware planning narratives enhance large vision-language model planner. CoRR, abs/2506.21230, 2025. doi: 10.48550/ARXIV.2506.21230. URL https://doi.org/10.48550/arXiv.2506.21230.
11
[22] Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, Xinda Xue, Qinghang Su, Huaihai Lyu, Xiaolong Zheng, Jiaming Liu, Zhongyuan Wang, and Shanghang Zhang. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pages 1724–1734. Computer Vision Foundation / IEEE, 2025. doi: 10.1109/CVPR52734.2025.00168. URL https://openaccess.thecvf.com/content/CVPR2025/html/Ji_RoboBrain_A_Unified_Brain_Model_for_ Robotic_Manipulation_from_Abstract_CVPR_2025_paper.html. [23] BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Xiansheng Chen, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, Yi Han, Yingbo Tang, Xiangqi Xu, Wei Guo, Yaoxu Lyu, Yijie Xu, Jiayu Shi, Mengfei Du, Cheng Chi, Mengdi Zhao, Xiaoshuai Hao, Junkai Zhao, Xiaojie Zhang, Shanyu Rong, Huaihai Lyu, Zhengliang Cai, Yankai Fu, Ning Chen, Bolun Zhang, Lingfeng Zhang, Shuyi Zhang, Dong Liu, Xi Feng, Songjing Wang, Xiaodan Liu, Yance Jiao, Mengsi Lyu, Zhuo Chen, Chenrui He, Yulong Ao, Xue Sun, Zheqi He, Jingshu Zheng, Xi Yang, Donghai Shi, Kunchang Xie, Bochao Zhang, Shaokai Nie, Chunlei Men, Yonghua Lin, Zhongyuan Wang, Tiejun Huang, and Shanghang Zhang. Robobrain 2.0 technical report, 2025. URL https://arxiv.org/abs/2507.02029. [24] Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quan Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. RT-H: action hierarchies using language. In Dana Kulic, Gentiane Venture, Kostas E. Bekris, and Enrique Coronado, editors, Robotics: Science and Systems XX, Delft, The Netherlands, July 15-19, 2024, 2024. doi: 10.15607/RSS.2024.XX.049. URL https://doi.org/10.15607/RSS.2024.XX.049. [25] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-Z: zero-shot task generalization with robotic imitation learning. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK, Proceedings of Machine Learning Research, pages 991–1002. PMLR, 2021. URL https://proceedings.mlr.press/v164/jang22a.html. [26] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT1: robotics transformer for real-world control at scale. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi: 10.15607/RSS.2023.XIX.025. URL https://doi.org/10.15607/RSS.2023.XIX.025. [27] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: general robot manipulation with multimodal prompts. CoRR, abs/2210.03094, 2022. doi: 10.48550/ARXIV.2210.03094. URL https://doi.org/10.48550/arXiv.2210.03094. [28] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022. URL https://arxiv.org/abs/2205.06175. [29] Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong, Ted Xiao, Pannag R. Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Dana Kulic, Gentiane Venture, Kostas E. Bekris, and Enrique Coronado, editors, Robotics: Science and Systems XX, Delft, The Netherlands, July 15-19, 2024, 2024. doi: 10.15607/RSS.2024.XX.090. URL https://doi.org/10.15607/RSS.2024. XX.090. [30] Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: efficient action tokenization for vision-language-action models. CoRR, abs/2501.09747, 2025. doi: 10.48550/ARXIV.2501.09747. URL https://doi.org/10.48550/arXiv.2501.09747. [31] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. RDT-1B: a diffusion foundation model for bimanual manipulation. In The Thirteenth International
12
Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=yAzN4tz7oI. [32] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3dvla: A 3d vision-language-action generative world model. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, Proceedings of Machine Learning Research, pages 61229–61245. PMLR / OpenReview.net, 2024. URL https://proceedings.mlr.press/v235/zhen24a.html. [33] Zongzheng Zhang, Haobo Xu, Zhuo Yang, Chenghao Yue, Zehao Lin, Huan-ang Gao, Ziwei Wang, and Hao Zhao. TA-VLA: elucidating the design space of torque-aware vision-language-action models. CoRR, abs/2509.07962, 2025. doi: 10.48550/ARXIV.2509.07962. URL https://doi.org/10.48550/arXiv.2509.07962. [34] Jialei Huang, Shuo Wang, Fanqi Lin, Yihang Hu, Chuan Wen, and Yang Gao. Tactile-vla: Unlocking vision-languageaction model’s physical knowledge for tactile generalization. CoRR, abs/2507.09160, 2025. doi: 10.48550/ARXIV. 2507.09160. URL https://doi.org/10.48550/arXiv.2507.09160. [35] Lu Qiu, Yi Chen, Yuying Ge, Yixiao Ge, Ying Shan, and Xihui Liu. Egoplan-bench2: A benchmark for multimodal large language model planning in real-world scenarios, 2025. URL https://arxiv.org/abs/2412.04447.
13
Appendix Appendix Contents A Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
B Real-World Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
C Action Interface and Skill Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
D Task Suite Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
E Long-Horizon Context Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
F Visual Monitor: Design and Empirical Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
G Planner Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
H Memory Compression Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
I
21
Visual Differencing Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
A Limitations Despite the demonstrated improvements, OmniAct’s adaptive capacity is fundamentally asymmetric: the high-level planner can leverage reflective memory to replan around failures, but cannot improve the frozen downstream VLA policies themselves—when failures stem from inherent motor limitations, recovery is bounded by the existing skill repertoire. Additionally, the asynchronous visual monitor operates at VLM inference frequency rather than control frequency, leaving a latency window during which erroneous actions may continue unchecked, posing residual safety risks in scenarios involving fragile objects or humanproximate operation.
B Real-World Experimental Setup OmniAct is evaluated on two structurally different real-world platforms: a UR5e 6-DoF arm for tabletop manipulation, and a Keenon wheeled mobile robot for indoor navigation. Both platforms share the same high-level planner interface and differ only in the exposed physical skills. The system also interfaces with four household IoT devices (smart light, air conditioner, ambient audio player, and a smart-home mode controller) and twelve API-style virtual tools (web search, weather query, checklist retrieval, etc.). Figures 5a and 5b show the two physical environments.
(a) UR5e tabletop workspace with dual third-person and one wrist-mounted RGB camera.
(b) Multi-room studio environment with the UR5e station, a WidowX area, and a curtained side zone.
Figure 5 Real-world experimental scenes used in the long-horizon cyber-physical benchmark. (a) The UR5e workspace serves as the manipulation testbed for pick-and-place over everyday objects (bottle, sponge, bowl, etc.). (b) The Keenon mobile robot navigates between semantic locations (entrance, corner area, manipulation station) and coordinates with the UR5e arm, enabling cross-platform tasks that combine navigation, IoT control, and embodied manipulation within a single execution loop.
C
Action Interface and Skill Implementations
The high-level planner emits only structured skill calls; the actual realization of each skill is delegated to backend executors. Manipulation skill pick_and_place is implemented by a self-trained π0 -style VLA policy fine-tuned on in-house teleoperation data covering the supported object and location set. Navigation skill navigate_to is realized through a pre-built 2D semantic map of the indoor environment with classical waypoint following. Cyber skills (control_*, query_*, web_search, set_home_mode) are deterministic JSON-returning HTTP calls. Tables 3 and 4 list the full action interfaces exposed to the planner.
15
Table 3 UR5e manipulation action interface exposed to the semantic planner. Action type
Action name
Parameters
Use
talk
speak
message
Final user-facing confirmation or essential status/error report.
tool
store_memory
content, scope, category
Store durable user preference or operational facts only when explicitly relevant.
tool
control_light
action, device, brightness
Turn lights on/off or set brightness.
tool
play_audio
action, audio_type, track, volume
Play or stop ambient audio.
tool
web_search
url, query
Retrieve external information for multimodal tasks.
tool
set_home_mode
device, option_id, option_text
Select a predefined smart-device mode.
act
pick_and_place
item_name, source, target
Move a supported visible object in the UR5e workspace.
sense
get_observation
{}
Request a fresh observation when current visual evidence is insufficient.
D
Task Suite Construction
The benchmark contains 40 long-horizon real-world tasks, evenly split between UR5e manipulation and Keenon navigation. For each platform we design 4 Level-1 tasks (physical only), 8 Level-2 tasks (cyberphysical), and 8 Level-3 tasks (full multimodal orchestration with speech, web/API, IoT, and physical execution). Tasks are designed to cover diverse object categories, IoT device types, and API endpoints, avoiding repeated templates. Each task is repeated three times under varied initial conditions and adjudicated by a human evaluator using a binary end-to-end success criterion. Table 5 lists representative requests for each level; the complete task list will be released with the code. • Level 1: Physical only. Tasks require only physical robotic execution and visual grounding, such as object movement or visual inspection. • Level 2: Cyber-physical. Tasks require physical execution together with IoT or API-style cyber actions. • Level 3: Full multimodal orchestration. Tasks combine speech interaction, web/API access, IoT control, physical execution, and final natural-language reporting.
E
Long-Horizon Context Construction
In real deployment, task-relevant interactions are diluted by environmental status checks, ambient logs, and unrelated household exchanges. We mimic this regime by embedding a small set of seed episodes carrying task-relevant constraints (e.g., dietary preferences, accessibility needs, prior failure lessons) within a much larger volume of irrelevant events drawn from fixed templates (shelf, audit, inventory, display notes). The agent is later queried with a downstream task whose correct execution requires recalling the seed-episode constraints across hundreds of intervening turns. Algorithm 1 details the construction. Stage 1 assembles a chronologically timestamped, interleaved event sequence; Stage 2 streams it through the memory backend so that event-boundary-driven summarization and long-term consolidation are triggered in the loop, matching real deployment rather than post-hoc com-
16
Table 4 Keenon navigation and home-assistant action interface. Action type
Action name
Required parameters
Use
talk
speak
message
Final report, clarification, or completion notice.
talk
send_agent_message
message, recipient
Remote robot message when explicitly required.
tool
control_AC
action, temperature, fan_speed
Home air-conditioner control.
tool
control_light
device, action, brightness
Lighting control.
tool
set_home_mode
device, option_id, option_text
Smart-home mode selection.
tool
query_weather_api
location, query
Weather or temperature query.
act
navigate_to
target_location, with_item
Navigate to a home location; with_item is set to none.
sense
get_observation
target
Inspect a specified area after navigation.
Table 5 Representative task requests for the three complexity levels. Level
Modalities
Example request
L1
VLA + vision
Pick up the pillbox from the tray and put it into the basket.
L2
VLA + IoT + vision
Set the water heater’s target temperature to 55◦ C.
L3
VLA + IoT + web/API + speech
Check the simple steps for organizing the dishes, turn on the light to 80% brightness, put the bowls into the basket, and report the result.
L1
Navigation + vision
Go to the entrance area and check whether the umbrella is still there.
L2
Navigation + IoT + vision
Turn up the brightness of the living-room light, then go to the living room and check whether the remote control is on the sofa.
L3
Navigation + IoT + API + speech
Check whether it is going to rain today, turn on the entrance light, go to the entrance area and check whether the umbrella is still there, then report the result to me.
pression. All randomness is seeded by SHA256 of the case identifier for reproducibility. In the longest configuration the trace contains more than 270 rounds and over 100k accumulated prompt token length.
F Visual Monitor: Design and Empirical Evaluation Real-world task completion is evaluated using an independent visual monitor. After each executed physical action, a separate VLM client is queried under an empty-context setting and receives only the chronologically ordered sequence of images from the transient memory buffer (before, during, and after execution). It is asked to describe visible changes between successive observations and to determine whether the resulting physical state supports the next planning step. This design intentionally decouples task verification from the planner’s reasoning process: since the monitor has no access to the intended command, task history, or prior chain of thought, its judgment is grounded in visual evidence rather than inferred intent, which reduces hallucination-induced verification errors and mitigates task-context leakage. The resulting state update is returned to the planner and can trigger replanning upon grasp failure, object removal, unchanged sourceobject state, or other physical deviations. The exact prompt is provided in Appendix I. 17
Algorithm 1 Long-Horizon Context Construction with Online Consolidation Require: seed episodes Sc , token budget B, time gap Δt, templates T , base length L0 // Stage 1: Interleaved sequence assembly ( ) 1: E ← Sc ; t ← maxe∈Sc e.τ ; αc ∼ U 0.6, 1.6 ( ) 2: while TOKENS E < B do ( ) 3: T ∼ (T ; ℓ ∼ U( 0.5, ) 1.5 · αc L0 ; ) 4: e ← SAMPLE T, ℓ , t, irrelevant 5: E ← E ∪ {e}; t ← t Δt 6: end while ( ) 7: E ← SORTBYTIME E // Stage 2: Online memory consolidation 8: M ← ∅ 9: for e ∈ E do ( ) 10: M ← MEMORYBACKEND M, e 11: end for 12: return M To evaluate whether the visual monitor can serve as an execution-level supervisor rather than a passive visual captioner, we construct a balanced verification set from real manipulation episodes: ten successful pick-and-place executions and ten failed executions, including both natural failures and externally induced perturbations such as manually removing the object during execution. Each candidate VLM receives the same before/during/after triplet and predicts whether the expected physical state has been achieved. Table 6 reports mean latency and binary verification accuracy across candidate backbones. Closed-source Gemini3.1-Pro and Qwen3.6-Flash deliver competitive F1 but incur non-trivial API latency that bottlenecks real-time preemption. Among locally deployable models, Qwen3-VL-30B-A3B matches Gemini-3.1-Pro’s F1 while running an order of magnitude faster on a dual-H100 server, offering the best accuracy-latency trade-off. We therefore adopt Qwen3-VL-30B-A3B as the default visual monitor backbone in all real-world deployments. Table 6 Latency and task-completion verification accuracy of candidate VLM backbones used as the visual monitor. The 20-trial balanced set comprises 10 successful and 10 failed/perturbed pick-and-place executions. A trial is counted as positive if the monitor predicts that the expected post-action state has been achieved. Closed-source models are accessed via remote API; open-weight models are deployed locally on a dual-H100 server.
G
Model
Deployment
Latency (s)
TP ↑
FN ↓
FP ↓
TN ↑
F1 ↑
Gemini-3.1-Pro
Remote API
11.67
8
2
0
10
0.89
Qwen3.6-Flash
Remote API
7.21
9
1
0
10
0.95
Qwen3-VL-8B
Local (2×H100)
0.69
6
4
0
10
0.75
Qwen3-VL-30B-A3B
Local (2×H100)
0.47
8
2
0
10
0.89
Planner Prompts
The semantic planner is prompted to produce exactly one structured next action at each iteration. The execution history is treated as attempted commands rather than guaranteed state changes, and post-action visual differencing feedback is used to update the world state before choosing the next step. The planner must return next_step: null only when the task is complete. UR5e Manipulation Planner Prompt # UR5e Reading Desk Planner
18
You are the user-facing UR5e arm planner for a reading desk setup experiment. You plan one action at a time from visual evidence and structured runtime feedback. Operating loop: 1. At task start, you may receive one initial scene summary for grounding. 2. On every planning step, you receive the current observation image. 3. After each executed action, you may receive post-action visual delta feedback comparing before/process/after images. 4. Treat execution history as attempted commands, not guaranteed state changes. 5. Update the world state from the current image and visual delta feedback before choosing the next action. 6. Return exactly one next step, or next_step:null when the task is complete. Allowed actions: - speak(message) - store_memory(content, scope, category) - control_light(action, device, brightness) - play_audio(action, audio_type, track, volume) - web_search(url, query) - set_home_mode(device, option_id, option_text) - pick_and_place(item_name, source, target) - get_observation({}) Visual feedback rules: - Use post-action visual delta feedback to infer what actually changed. - If visual feedback contradicts requested action parameters, trust observed visual changes. - If an object remains visible at the source after a manipulation attempt, it is still pending unless visual delta says otherwise. - If the target visibly contains the moved object, treat that object as handled. Return exactly one JSON object: { "current_step_analysis": { "visual_state": "Brief current visual state", "task_progress": "What has been completed and what remains", "next_action_reasoning": "Why the next action follows from the image and feedback" }, "next_step": { "step_number": 1, "agent": "ur5e_arm", "location": "home", "action": "pick_and_place", "action_type": "act", "parameters": {"item_name": "water", "source": "tray", "target": "basket"} }, "needs_human_input": false, "user_question": null }
Keenon Navigation Planner Prompt # Vision-Based Autonomous Humanoid Robot Task Planner You are the VLM planner for a Keenon robot in a home-navigation benchmark. You operate in half-open-loop mode: plan exactly one next action, wait for execution, then plan again from the updated context. Allowed actions: - speak(message) - send_agent_message(message, recipient)
19
- store_memory(content, scope, category) - control_air_conditioner(action, temperature, fan_speed) - control_light(device, action, brightness) - set_home_mode(device, option_id, option_text) - play_audio(audio_type) - web_search(url, query) - query_weather_api(location, query) - navigate_to(target_location, with_item="none") - get_observation(target) Return exactly one JSON object: { "current_step_analysis": { "visual_state": "What is visible, without using it to skip required navigation", "task_progress": "What required actions have already been completed", "next_action_reasoning": "Why this single next action is required" }, "next_step": { "step_number": 1, "agent": "Keenon humanoid_robot", "location": "current tracked location or unknown", "action": "specific_action_name", "action_type": "talk|tool|act|sense", "parameters": {} }, "needs_human_input": false, "human_question": null }
H
Memory Compression Prompts
Episodic Memory Compression Prompt System prompt: You are an episodic interaction log writer for an embodied planner. Summarize what happened in this interaction segment as an operational log, not as long-term memory. Preserve chronology, task context, concrete observations, tools, skills, actions, useful parameters, outcomes, errors, and unresolved questions. Do not infer durable user preferences unless the user explicitly stated one; if present, keep it as evidence rather than as the main focus. Return strict JSON with keys: topic, timestamp, source_turn_range, task_request, interaction_summary, actions_or_skills, outcome, important_observations, open_questions, evidence_snippets. User prompt template: Level: Episodic Device: {profile_name} Trigger: {trigger_reason} Suggested topic: {topic} Summary timestamp: {created_at} Source turn range: {source_turn_range} Write this as an episodic interaction log. Prioritize timestamped task flow, user request, planner actions or skills, execution result, and concrete feedback. Do not compress it into stable preference rules; long-term summaries will handle cross-session preference and experience extraction. Source text: {source_text}
20
Long-Term Memory Compression Prompt System prompt: You are a memory compaction assistant for an embodied planner. Do not summarize chronology. Extract the durable decision rules future planning must obey. Return strict JSON with keys: topic, hard_preferences, soft_preferences, current_needs, time_constraints, forbidden_items, decision_rules, stable_user_preferences, priority_order, evidence_snippets. Guidance: Extract only the information that should affect future decisions. Prefer durable user preferences and hard constraints over one-off event narration. If a current temporary need exists, keep it in current_needs instead of stable_user_preferences. For long-term summaries, read episodic interaction logs as evidence and extract stable user preferences, recurring constraints, and reusable task execution experience. Use decision_rules for reusable planning or skill-selection lessons learned from repeated interactions. User prompt template: Level: long_term Device: {profile_name} Trigger: {trigger_reason} Suggested topic: {topic} {level_specific_guidance} Source text: {source_text}
I
Visual Differencing Prompt Visual Differencing Prompt You are an independent visual differencing model for a robot planner. You are not given the robot's intended command. Do not infer what should have happened. If an object appears unchanged, say it remains unchanged. If the images are ambiguous, say so. Compare the images in temporal order. Focus on visible changes between the images. Try to enumerate all scene changes that could matter to the next robot planning step, including: - object presence, absence, position, orientation, and containment changes - objects that remained in their original place despite nearby motion - gripper or arm pose changes, if visible - occlusions, blur, lighting/camera viewpoint shifts, or other uncertainty sources Do not describe the whole scene from scratch. Do not assume an action succeeded. Do not use prior task context. Prefer concrete visual evidence over short generic summaries. If there is no meaningful visible change, say that explicitly. Return JSON only with this schema: { "observed_changes": ["detailed visible changes, one fact per item"], "objects_moved_or_removed": [ { "object": "name or description", "from": "observed source", "to": "observed destination or unknown", "evidence": "specific visual evidence" }
21
], "objects_remaining": [ "relevant objects still visible, including their observed locations" ], "gripper_or_arm_state": "visible state and pose change if relevant, otherwise unknown", "scene_change_notes": [ "camera, lighting, occlusion, blur, or ambiguity notes" ], "action_context_used": false, "contradicts_requested_action": false, "state_update": "planner-facing summary covering the key visible changes and unchanged relevant objects", "confidence": "low|medium|high" }
22