HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System ∗
∗
Tianshuo Yang1,2 , Guanyu Chen3 , Yutian Chen2,4 , Zhixuan Liang1,2 , Yitian Liu3 , Zanxin Chen2,3 , Chunpu Xu2 , Haotian Liang2 , Jiangmiao Pang2 , Yao Mu3,2† , Ping Luo1† The University of Hong Kong 2 Shanghai AI Laboratory 3 Shanghai Jiao Tong University 4 The Chinese University of Hong Kong
arXiv:2604.14125v1 [cs.CV] 15 Apr 2026
1
Abstract. While end-to-end Vision-Language-Action (VLA) models offer a promising paradigm for robotic manipulation, fine-tuning them on narrow control data often compromises the profound reasoning capabilities inherited from their base Vision-Language Models (VLMs). To resolve this fundamental trade-off, we propose HiVLA, a visual-groundedcentric hierarchical framework that explicitly decouples high-level semantic planning from low-level motor control. In high-level part, a VLM planner first performs task decomposition and visual grounding to generate structured plans, comprising a subtask instruction and a precise target bounding box. Then, to translate this plan into physical actions, we introduce a flow-matching Diffusion Transformer (DiT) action expert in low-level part equipped with a novel cascaded cross-attention mechanism. This design sequentially fuses global context, high-resolution object-centric crops and skill semantics, enabling the DiT to focus purely on robust execution. Our decoupled architecture preserves the VLM’s zero-shot reasoning while allowing independent improvement of both components. Extensive experiments in simulation and the real world demonstrate that HiVLA significantly outperforms state-of-the-art endto-end baselines, particularly excelling in long-horizon skill composition and the fine-grained manipulation of small objects in cluttered scenes. The project website is: https://tianshuoy.github.io/HiVLA-page/ Keywords: Vision-Language-Action Models · VLM Agent Systems
1
Introduction
Achieving human-like capabilities in robots that integrates perception, reasoning and execution, is a central pursuit of embodied AI. Recently, the advent of web-scale, pretrained Vision-Language Models [1, 24, 35] (VLMs) has presented a transformative opportunity for robotic manipulation. Exhibiting remarkable zero-shot generalization and deep semantic understanding, VLMs have catalyzed *
Equal contribution † Corresponding authors: [email protected], [email protected]
2
T. Yang et al. Click Bell
1920x1080
VLM
384x384
DiT Action Agent
Action Sequence
Original Image Prompt
Stack Block
37
98
70.6
78.7
88.3
60
45.6 40
95
76
Place Shoe
60
Move Stapler Pad
(a) Our Visual-Grounded-Centric System
H-RDT 100 Ours w/o Skill Ours 80
Press Stapler
Stamp Seal 384x384
Success Rate (9 Tasks) (%)
𝜋!
94
Padding
Text Prompt: Grab the stamp and stamp onto the red mat.
20 0
𝜋!
(b) Success Rate in RoboTwin
H-RDT
Ours w/o Skill
Ours
Fig. 1: (a) Overview of our proposed HiVLA framework. (b) Success rate comparison on RoboTwin benchmark.
the development of Vision-Language-Action (VLA) models [6, 17, 20]. However, current VLA research predominantly adopts end-to-end architectures, utilizing either single-system [7, 19, 40] or dual-system [5, 8, 9] approaches that tightly couple visual reasoning with low-level action generation. Although these integrated paradigms have shown considerable promise, they face a critical bottleneck [13, 15] that fine-tuning VLMs on relatively scarce and domain-specific manipulation data inevitably degrades their original reasoning capabilities. This degradation, widely recognized as catastrophic forgetting, ultimately limits the ability to leverage the full cognitive power of the most advanced VLMs. Hierarchical systems [2, 23, 32] offer a compelling alternative by explicitly decoupling high-level semantic planning from low-level motor control. In this paradigm, the VLM operates purely as a high-level planner, preserving its reasoning capabilities by avoiding low-level fine-tuning, while a dedicated action expert executes the plans. However, the success of this decoupled design heavily depends on the intermediate representation bridging the two modules. A powerful candidate for this interface is visual grounding. This concept is deeply inspired by the “thinking with images” paradigm in VLM agents [18, 21, 31, 41], a framework in which a model explicitly localizes a relevant target region in a high-resolution image before proceeding with complex reasoning. Despite the conceptual elegance of visual-grounded-centric VLAs, existing designs struggle to effectively translate grounded information into physical actions. Current methods typically force a compromise between spatial context and visual fidelity. For example, extracting local image crops often strips away absolute spatial coordinates [14]. Conversely, applying object masks to down-sampled global images discards the nuanced visual details necessary for fine-grained manipulation [42].These shortcomings expose a critical, unresolved question that how can we design a policy capable of fully exploiting a grounded plan including high-resolution local appearance, precise global spatial awareness, and explicit skill-level subtask directives? To address this challenge, we propose HiVLA, a hierarchical manipulation system centered around a robust framework for visual-grounded plan generation and utilization. As illustrated in Fig. 1 (a), our system employs a VLM as a high-level planner that decomposes complex instructions and visually grounds target objects. This process outputs a structured plan consisting of a semantic
HiVLA
3
subtask label and a precise bounding box. To effectively translate this finegrained guidance into physical motion, we design a low-level action expert based on a Diffusion Transformer (DiT) [28]. Within this expert, our key innovation is a cascaded cross-attention mechanism embedded in each DiT block. Rather than naively fusing inputs, this mechanism sequentially conditions the policy on three distinct signals: (1) global visual context for holistic scene understanding, (2) high-resolution, object-centric features from the grounded patch augmented with absolute positional encodings to preserve spatial awareness, and (3) a language embedding representing the specific subtask skill. This architectural grounding design enables the action expert to maximally leverage the VLM’s cognitive output, providing the system with a clear understanding of what to do, where to look, and how to act. Experiments conducted in two challenging, cluttered simulation environments and the real world demonstrate the superiority of our approach. As shown in Fig. 1 (b), HiVLA achieves an absolute success rate improvement of 17.7% over a strong baseline H-RDT [4] and 42.7% over the state-of-the-art π0 [6] on the RoboTwin 2.0 Benchmark [10]. These results validate that our visual-groundedcentric hierarchy significantly enhances robust perception, precise manipulation, and long-horizon task completion. Our contributions are summarized as follows: – We propose HiVLA, a hierarchical VLA framework bridged by a visualgrounded-centric mechanism. This architecture explicitly decouples VLMbased high-level planning from low-level control, eliminating catastrophic forgetting of multi-task manipulation and allowing seperate improvements of VLM and action expert. – We introduce a novel cascaded cross-attention mechanism within the DiT action expert, capable of sequentially integrating global context, spatiallyaware high-resolution local crops, and subtask skill guidance, unlocking the potential of grounded plans for precise action generation. – We perform extensive evaluations in simulation and the real world, demonstratingHiVLA significantly outperforms state-of-the-art VLA models, and showcases exceptional proficiency in long-horizon skill composition and finegrained manipulation within highly cluttered environments.
2
Related Work
2.1
Vision-Language-Action Models
Vision-Language-Action (VLA) models have revolutionized robotic manipulation by leveraging the profound cognitive abilities of large Vision-Language Models (VLMs) to translate multi-modal inputs into executable actions. Current monolithic VLA architectures broadly fall into single-system and dual-system paradigms [30]. Single-system models, such as RT-2 [7] and OpenVLA [20], employ a unified network that directly decodes action tokens autoregressively from sensory inputs. Alternatively, dual-system models like π0 [6] and GR00T-N1.5 [5]
4
T. Yang et al.
utilize a VLM backbone to implicitly guide an action expert through jointly optimized feature spaces. Although these integrated approaches demonstrate significant promise, fine-tuning VLMs on narrow manipulation data severely degrades their original, web-scale reasoning capabilities [13, 15]. This catastrophic forgetting limits the generalization potential of the underlying foundation models. To circumvent this limitation, hierarchical models explicitly decouple highlevel task planning from low-level policy execution via interpretable intermediate representations. This modularity retains the VLM’s zero-shot reasoning power while allowing the action expert to specialize in precise motor control. These intermediate bridges take various forms, including textual subtasks in HiRobot [32]and MemER [34] or spatial keypoints in HAMSTER [22]. By isolating cognitive processes from high-frequency control, hierarchical systems provide a robust and scalable foundation for advancing embodied intelligence. 2.2
Visual-Grounded-Centric VLA
A critical challenge in manipulation is precise visual grounding, which accurately maps high-level instructions to specific spatial regions within the visual input. Early visual-centric VLAs, such as π0.5 [17] and InternVLA-M1 [11], address this by leveraging strong vision-language alignment for spatial localization. To further enforce visual attention, recent works explore integrated grounding techniques. ReconVLA [33] introduces an implicit paradigm that forces a diffusion transformer to reconstruct target gaze regions from visual outputs. Similarly, approaches like InterleaveVLA [14] and 3D-CAVLA [3] attempt to improve scene awareness by interleaving visual tokens with language or incorporating chainof-thought region detection. However, these integrated methods lack explicit architectural decoupling. By compelling the VLM to jointly process semantic reasoning and specific control trajectories, they remain susceptible to catastrophic forgetting and exhibit limited planner generalization in novel scenarios. Addressing these coupling issues, explicit hierarchical grounding methods utilize spatial representations as intermediate bridges between perception and action. Systems like DexGraspVLA [42] and RoboGround [16] employ visual segmentation masks to isolate target objects and guide downstream policies. While conceptually appealing, generating dense segmentation masks is not a native task for standard VLMs, often requiring external expert models that compromise general visual capabilities. Furthermore, RoboGround [16] relies on a traditional GR-1 [39] transformer policy, which struggles to match the continuous control performance of modern Diffusion Transformer(DiT) [28] architectures. Similarly, DexGraspVLA [42] applies masks to heavily down-sampled global images, diluting the high-fidelity visual details crucial for precise manipulation. These collective shortcomings highlight a critical gap: existing systems fail to effectively bridge the VLM and the action expert using a native, computationally efficient grounded plan. Our proposed HiVLA resolves this by utilizing native VLM bounding boxes to extract high-resolution local crops, which are subsequently fused with global context and explicit skill semantics through a novel cascaded DiT architecture.
HiVLA [Instruction] “Pick up the cup and put it on the coaster. ” [Observation]
Action
Environment
VLM
DiT Action Expert
[Sub-task&Skill Reasoning] “A blue cup is on the table. We need to pick up the cup first.” [Target Object Grounding] “The blue cup on the left.” [Bbox of Target Object] [y! , 𝑥! , y" , 𝑥" ] in 1080P HD Img.
Global Img.
(a) HiVLA Framework
5
Robot State Noisy Action Latents Absolute P.E.
Self-Attn.
Global Img. Features
Global Img. Cross-Atten.
Local Img. Features
Local Img. Cross-Atten.
Language Tokens
Language Cross-Atten. FFN
Local Img. (b) DiT Block
Fig. 2: Pipeline of HiVLA. (a) Our decoupled framework utilizes a VLM to decompose user instructions into explicit structured plans, yielding a skill-level subtask and a bounding box used to extract a high-resolution target crop. (b) To execute this plan, the DiT action expert employs a cascaded cross-attention block. This design sequentially conditions the noisy action latents on global visual context, position-aware local features, and language tokens, bridging high-level reasoning with low-level control.
3
Problem Formulation
We formulate the language-guided robotic manipulation task as a conditional sequence generation problem. The objective is to learn a generalized manipulation system πθ with parameters θ, which maps multi-modal observations and human language instructions to a sequence of executable actions. Formally, at each timestep t, the agent receives a set of multi-modal observations St . These observations consist of multi-view visual inputs Ot = {Itk }K k=1 from K cameras (e.g., wrist and head cameras), providing high-resolution images Itk ∈ RH×W ×3 (at a native 1920 × 1080 resolution), and the robot’s current proprioceptive state st ∈ Rds (encoding joint positions and gripper state). Given the history of observations S0:t = ({Oj }tj=0 , {sj }tj=0 ) and the high-level language instruction L, the policy πθ aims to generate a sequence of future actions At = {at+i }H−1 i=0 over a prediction horizon H. Each action ai ∈ Rda specifies the target control commands (e.g., joint positions) for the robot’s arm and gripper. The core challenge lies in bridging the semantic gap between the abstract, long-horizon instruction L and the sequence of precise motor commands At . This difficulty is twofold. First, language instructions demand sophisticated reasoning and task decomposition to translate complex, history-dependent procedures into actionable steps. Second, the visual observations Ot are inherently challenging, as typical manipulation scenes are cluttered with distractor objects and severe perceptual noise. Ultimately, the system must seamlessly deduce the correct subtask from the instruction, visually ground it onto the target object amidst this background clutter, and translate this grounded intent into robust actions.
6
4
T. Yang et al.
Method
This section introduces HiVLA, a visual-grounded-centric hierarchical manipulation system. We will begin by describing the High-Level VLM Planner (Sec. 4.1), which is responsible for decomposing a given task into actionable subtasks and performing the corresponding visual grounding. Then we will detail the architecture of our DiT-based Action Expert (Sec. 4.2 ), with a focus on its mechanism for effectively conditioning on the high-level plans provided by the VLM. Fig. 2 illustrates the overall inference process.
4.1
VLM Planner Agent
The cognitive core of our HiVLA system is a High-Level Planner Agent, implemented with a state-of-the-art Vision-Language Model (VLM). This module serves as the “brain” of the system, responsible for interpreting the high-level language instruction L in the context of the current visual scene Ot to decide what to do next and where to do it. The design of the planner agent is centered around a structured inference process. At each decision step t, the agent is provided with the overall goal L, the robot’s gripper status from state st , the previous subtask executed, and a visual history comprising the scene before and after the last action. Based on these multi-modal inputs, the VLM reasons about the progress towards L and determines the next logical step. The decomposition strategy is contingent on task complexity; simple instructions may map to a single action, whereas complex, long-horizon tasks (e.g., “stack three blocks”) are broken down into a sequence of subtasks. Each subtask is typically a pairing of a primitive skill (e.g., ‘pick’, ‘place’) with a single target object. The agent’s reasoning culminates in the generation of a structured plan, a JSON object containing the next subtask’s description Lsub,t , the action type, the target object’s name, and a normalized bounding box Bt = [ymin , xmin , ymax , xmax ] ∈4 that localizes the target object in the current scene image. A key technical advantage of this design is the decoupling of high-level semantic planning from low-level motion generation. By leveraging a pre-trained VLM, the system inherits sophisticated reasoning capabilities, enabling it to handle a wide range of instructions without requiring exhaustive training for every conceivable task. Furthermore, we frame the VLM planner as an intelligent agent that uses tools to execute its intent. The generation of the bounding box Bt is not merely an output; it is a directive that invokes an Image Crop tool. This tool uses the normalized coordinates in Bt to extract a high-resolution, objectcentric patch Itlocal from the original camera observation Itk ∈ R1080×1920×3 . The complete, structured plan, including the subtask description Lsub,t and the rich visual information from Itlocal , is then passed as a conditional guidance signal to the DiT Action Expert, which can be conceptualized as the final tool the planner uses to translate its intention into physical action At .
HiVLA
4.2
7
DiT Action Expert
The DiT Action Expert serves as the manipulation-focused “hands" of the HiVLA system, responsible for translating the high-level plans formulated by the VLM Planner into precise, low-level motor commands. At its core, this module is a conditional Diffusion Transformer (DiT) designed to model the complex conditional probability distribution p(At |S0:t , Lsub,t , Bt ). To achieve this, we first detail the continuous-time flow-matching framework, followed by a description of the novel hierarchical transformer architecture that implements it. Conditional Flow Matching for Action Generation. Our objective is to learn a deterministic mapping from a simple noise distribution to the complex data distribution of action sequences, conditioned on the rich context provided by the VLM planner. We employ Conditional Flow Matching (CFM), a powerful generative model that learns to approximate the conditional vector field. Let At = {at+i }H−1 i=0 be the ground truth action sequence over a horizon H, and let the comprehensive conditioning context be denoted by Ct = (S0:t , Lsub,t , Bt ). CFM defines a time-continuous probability path between a sample from a standard Gaussian prior, z ∼ N (0, I), and the target action sequence At . We utilize a simple linear interpolation path, defined for a continuous time variable τ ∈ [0, 1]: \mathbf {x}_\tau = \tau A_t + (1-\tau ) \mathbf {z} \label {eq:cfm_path}
(1)
At τ = 0, the path begins with pure noise (x0 = z), and at τ = 1, it culminates in the target action sequence (x1 = At ). The neural network, vθ , which we instantiate as our DiT architecture, is trained to predict the vector field u = At − z that defines the “flow” from noise to data. The training objective is to minimize the L2 distance between the network’s prediction and this target vector field, formulated as the following loss function: \mathcal {L}_{\text {CFM}}(\theta ) = \mathbb {E}_{\tau , A_t, \mathbf {z}} \left [ \left \| v_\theta (\mathbf {x}_\tau , \tau , \mathcal {C}_t) - (A_t - \mathbf {z}) \right \|^2 \right ] \label {eq:cfm_loss} (2) During inference, we generate the action sequence by solving the ordinary differential equation (ODE) defined by the learned vector field: dx dτ = vθ (xτ , τ, Ct ). Starting from an initial noise sample x0 ∼ N (0, I), we integrate from τ = 0 to τ = 1. This is approximated using a numerical ODE solver, such as the forward Euler method, over a discrete number of steps: \mathbf {x}_{\tau +\Delta \tau } = \mathbf {x}_\tau + \Delta \tau \cdot v_\theta (\mathbf {x}_\tau , \tau , \mathcal {C}_t) \label {eq:inference_step}
(3)
where ∆τ is the step size. This process deterministically transforms the initial noise into a coherent action sequence that is precisely conditioned on context Ct . Hierarchical Transformer Architecture. The neural network vθ , which approximates the conditional vector field, is instantiated as a transformer-based architecture. The architecture of our action expert is based on H-RDT, which employs a LLaMA-style transformer backbone featuring RMSNorm for layer normalization and SwiGLU activation functions for enhanced performance. The
8
T. Yang et al.
input to the transformer is a sequence of tokens representing the current proprioceptive state st and a noisy future action sequence Ãt , each projected into the model’s hidden dimension dmodel by dedicated MLP adapters. The diffusion timestep τ is encoded into a vector embedding and integrated into each transformer block via Adaptive Layer Normalization (AdaLN), which modulates the activations without altering the core feature representations. The primary innovation of our action expert lies in its hierarchical conditioning mechanism, which is meticulously designed to leverage the rich, multifaceted plan provided by the VLM planner. Within each transformer block, the model sequentially integrates three distinct forms of guidance through a cascade of cross-attention layers, ensuring a synergistic fusion of global context, local detail, and task-specific instructions. Global Visual Context. The first layer of conditioning provides the model with a comprehensive understanding of the entire scene. The multi-view visual inputs Ot are processed by a pre-trained vision encoder, a powerful combination of DINOv2 and SigLIP, to produce a set of feature tokens C global ∈ RNglobal ×dmodel . A cross-attention mechanism allows the state-action tokens to attend to these global features. This enables the policy to ground its actions within the broader spatial and semantic context of the environment, performing coarse-grained reasoning about object relationships and the overall workspace layout. Position-Aware Local Features. Following the global context integration, a second, specialized cross-attention layer injects fine-grained, object-centric visual information. This guidance originates from the local image patch Itlocal , which is cropped from the original high-resolution (1920 × 1080) camera frame using the bounding box Bt supplied by the VLM planner. Cropping from the full-resolution image is critical as it preserves high-fidelity details of the target object that would be lost in down-sampled inputs. After passing Itlocal through the same vision encoder to obtain feature tokens C local ∈ RNlocal ×dmodel , we introduce a crucial inductive bias: absolute spatial awareness. For each patch token in C local , which corresponds to a specific region in the cropped image, we compute its central coordinate p ∈ R2 within the original high-resolution camera frame. This coordinate is then converted into a fixed sinusoidal positional embedding P E(p) ∈ Rdmodel , inspired by DETR. The final local conditioning signal is formed by element-wise addition: C^{local-pos} = C^{local} + PE(p)
(4)
This position-aware feature set provides the model with a detailed, magnified view of the target object while explicitly informing it of the object’s precise location in global scene, a critical factor for achieving high-precision manipulation. Subtask Language Guidance. The final conditioning stage aligns the policy with the specific skill required for the current subtask. The subtask description Lsub,t from the VLM’s plan is encoded into a sequence of language embeddings C lang ∈ RNlang ×dmodel . A third cross-attention layer allows the model to attend to these
HiVLA
Stack 3 Blocks
Stamp Seal
Place Shoe
Click 3 Bells
Lift Pot
Click Clock
Click 2 Bells
Click 2 Bells
9
(a) RoboTwin Tasks
Place 3 Blocks
Place 1 Blocks
Place 3 Cups
Place 1 Cups
(b) Real-world Tasks
Fig. 3: Visualization of RoboTwin tasks and real-world tasks.
language features, thereby conditioning the generated motion on the precise semantics of required skill (e.g., distinguishing among ‘pick’, ‘place’, or ‘push’). Finally, after passing through all transformer blocks, the output hidden states corresponding to the action sequence are processed by a final MLP-based Action Decoder. This decoder, also modulated by the timestep embedding, maps the hidden states back to the robot’s native action space, producing the denoised action sequence At . Through this cascaded conditioning strategy, the DiT Action Expert maximally utilizes every component of the VLM’s high-level reasoning, effectively grounding abstract plans into robust and precise physical execution.
5
Experiments
5.1
Experimental Setup
To validate the efficacy of the visual-grounded hierarchical design of HiVLA, we conduct extensive experiments in both the RoboTwin2.0 [10] simulation platform and real-world robotic manipulation settings. We aim to answer the following questions: (1) Does our hierarchical VLA outperform state-of-the-art coupled VLA models? (2) How robust is the control policy to reasoning errors from the high-level planner? (3) How do different visual representations and guidance injection strategies affect the system’s performance? Simulation and Robot Configuration. We employ RoboTwin2.0 [10], a high-fidelity simulation platform specifically designed for robot learning, which facilitates the generation of large-scale datasets and enables reproducible evaluation. To closely emulate the challenges of real-world operation, we utilize the ‘domain randomization’ setting for both data generation and testing. This configuration introduces significant visual diversity and perceptual complexity through randomized backgrounds, cluttered tabletops, variable table heights, and dynamic lighting conditions, thereby posing a rigorous test for visual grounding capabilities. For hardware deployment in both simulation and the real world, we utilize Aloha-Agilex-1.0, a widely-adopted bimanual robot platform featuring a total of 14 Degrees of Freedom (DoF), six per arm plus one for each gripper. Dataset Details. We generate a large-scale dataset within the RoboTwin2.0 platform, termed HiVLA-HD (High-Definition). Generated under the ‘Hard’ mode configuration, it comprises 15 manipulation tasks that demand robust
10
T. Yang et al.
Table 1: Main success rates across 9 tasks in the RoboTwin simulator. HiVLA demonstrates superior performance, particularly in long-horizon and visually demanding tasks. Best and second-best results are bold and underlined. Task
π0 [6]
π0.5 [17]
StarVLA [12]
H-RDT [4]
Ours w/o Skill
Ours
Easy Tasks Click Bell Click Clock Press Stapler Lift Pot Average
45% 53% 60% 59% 54.3%
65% 66% 69% 21% 55.3%
71% 83% 63% 18% 58.8%
88% 93% 89% 92% 90.5%
95% 97% 98% 96% 96.5%
94% 97% 97% 96% 96.0%
Hard Tasks Place Shoe Move Stapler Stamp Seal Stack 3 Blocks Click 3 Bells Average
75% 15% 61% 1% 41% 38.6%
68% 17% 42% 1% 54% 36.4%
61% 15% 25% 16% 66% 36.6%
88% 34% 43% 20% 88% 54.6%
94% 42% 68% 26% 92% 64.4%
95% 60% 76% 37% 98% 73.2%
Total Average
45.6%
44.8%
46.4%
70.6%
78.7%
83.3%
visual perception and complex language reasoning. Observations from the head camera are saved at a high resolution of 1920×1080, while wrist cameras operate at 720p. Leveraging the simulator’s capabilities, we obtain precise ground-truth annotations without manual cost: subtask transitions are logged via action planning scripts, and accurate bounding boxes for target objects are derived directly from unique mask IDs. Following rigorous filtering, HiVLA-HD yields approximately 1,000 episodes per task, forming a standardized, high-resolution dataset with fine-grained semantic labels. Crucially, all evaluated models are finetuned on this dataset to ensure a fair comparative analysis. Baseline Selection. To comprehensively evaluate our approach, we benchmark HiVLA against four state-of-the-art (SOTA) models: π0 [6], its advanced variant π0.5 [17], StarVLA [12], and H-RDT [4]. π0 and π0.5 represent SOTA dualsystem VLAs that accomplish perception and reasoning through joint training and parallel inference. StarVLA provides a comprehensive suite of mainstream VLA architectures built upon the Qwen-VL [1] backbone. Specifically, we evaluate its Qwen-GR00T variant (whose performance officially matches GR00TN1.5 [5]), equipped with the exact same Qwen3-VL backbone as our framework, to ensure a strictly fair comparison of the architectural paradigms. H-RDT serves as a critical baseline and an implicit ablation of our visual-grounding mechanism, as it relies entirely on global image features for policy generation. We note that open-source, visual-grounded hierarchical systems designed for general manipulation remain scarce, which informs our baseline choices. For instance, DexGraspVLA [42] is strictly restricted to multi-object grasping and lacks cross-task generalization, precluding a feasible comparison. Proprietary systems like Gemini Robotics [36] and HiRobot [32] are closed-source. Furthermore, InterleaveVLA [14] focuses primarily on object-centric policies rather than offering a complete hierarchical system.
HiVLA
11
System Latency Analysis. Our decoupled architecture resolves the inherent frequency mismatch between slow vision-language reasoning and high-speed motor control via asynchronous inference. While the unoptimized VLM Planner requires 1.9s per reasoning step, leaving ample room for software acceleration, the DiT Action Policy efficiently infers a 16-step action chunk in merely 0.162s. By executing the semantic planner in parallel with the fast control policy and maintaining temporal consistency through mechanisms like real-time tracking, the system effectively bridges this latency gap to achieve an 8Hz control frequency, demonstrating strong practicality for real-world deployment. 5.2
Evaluation in RoboTwin Platform
Evaluation Tasks and Protocols. For a comprehensive assessment, we benchmark all models across a suite of 9 tasks, categorized into four Easy Tasks and five Hard Tasks. Easy tasks typically require a single skill and evaluate precise visual perception (e.g., grasping small objects like a bell or a stapler). Hard tasks involve sequences of multiple skills or demand advanced spatial and semantic reasoning. For instance, ‘Stack 3 Blocks’ requires the model to sequentially infer the correct colored block to manipulate based on a specific visual order, while ‘Click 3 Bells’ presents three identical bells, forcing the model to rely strictly on spatial language reasoning (‘left’, ‘center’, ‘right’) to disambiguate the target. Visualizations of these selected tasks are provided in Fig. 3, with a comprehensive list of textual instructions and visual states detailed in the Appendix. For each task, we conduct 100 independent trials under unseen environment configurations, reporting the average success rate over the last three saved checkpoints to ensure statistical stability. VLM Planner Training and Validation. To validate the high-level reasoning capabilities of our VLM Planner, we curated a specialized 210K-instance dialogue dataset derived from HiVLA-HD. Fine-tuning the Qwen3-VL [1] 8B model on this domain-specific data yields highly robust semantic planning, achieving a bounding box grounding accuracy (mIoU) of 90.37% and a strict exactmatch sub-task prediction of 98.57% (with comprehensive evaluations detailed in the Appendix). Crucially, our decoupled architecture makes the VLM planner readily replaceable. This flexibility allows the system to either undergo lightweight fine-tuning for specific operational domains, or directly integrate offthe-shelf VLMs for zero-shot deployment. While we adopt fine-tuned 8B model for all subsequent evaluations to maximize task performance, this extensible design easily accommodates future advancements in more powerful VLM agents. Action Policy Training and Main Results. To ensure a fair comparison, all baselines are all fine-tuned on the HiVLA-HD dataset for 150K steps using two H200 GPUs (batch size 64). Notably, HiVLA’s DiT is initialized from HRDT weights pre-trained on the EgoDex dataset, where we directly copy the weights from the global image cross-attention layer to initialize our novel local image cross-attention layer. As presented in Tab. 1, HiVLA achieves an unparalleled total average success rate of 83.3%, significantly outperforming SOTA coupled VLAs (π0 , π0.5 ,
12
T. Yang et al.
Table 2: Robustness evaluation of the Action Expert against guidance perturbations. The policy is highly resilient to Bbox noise but strictly adheres to language instructions. Error Injection Rate
Corrupted Modality Noise on BBox Noise on Task (Lang) Noise on Both
0%
20%
40%
60%
80%
100%
83.3% 83.3% 83.3%
78.5% 69.3% 62.0%
74.0% 49.3% 42.5%
70.8% 36.0% 33.3%
59.8% 24.3% 23.0%
57.0% 12.0% 17.3%
StarVLA) and the purely global-vision-based H-RDT. For Easy Tasks, HiVLA’s use of high-resolution, object-centric crops explicitly preserves critical visual features, securing a 96.0% average success rate and showing distinct advantages on small targets. For Hard Tasks, the performance gap widens dramatically. Coupled VLAs struggle to maintain spatial temporal consistency over long horizons (averaging < 40%). In contrast, HiVLA leverages the high-level planner to comprehend task progression and localize targets, resulting in a commanding 73.2% average success rate—a remarkable 18.6% absolute improvement over H-RDT. Skill Decomposition and Error Correction. Our ablation variant, Ours (w/o Skill), further isolates the impact of granular language instructions. While performance is comparable on Easy tasks (where the global instruction intrinsically matches a single skill), replacing specific sub-task skills with the global instruction on Hard tasks causes an 8.8% performance drop. This confirms that decomposed, “one-to-one" language conditions drastically reduce the cognitive load on the diffusion policy, allowing it to focus purely on local geometry and execution. Furthermore, we observed a compelling emergent error-correction property: if the DiT policy fails a grasp (a “phantom execution"), the VLM Planner acts as an independent semantic supervisor. Recognizing the sub-task as incomplete, it seamlessly re-issues the visual-language command, enabling the system to re-attempt the skill—a resilience unattainable in standard coupled VLAs. Robustness to Planner Errors. A pervasive critique of hierarchical systems is their susceptibility to compounding errors, where a planner’s mistake irreversibly crashes the downstream policy. To address this, we subjected our Action Expert to rigorous perturbation testing (Tab. 2) by injecting calibrated noise into the inferred bounding boxes and language instructions. The results reveal a highly desirable decoupling: the policy demonstrates strong resilience to spatial noise. Even with 100% shifting in the target bounding box, it retains a 57.0% success rate, effectively leveraging the auxiliary global image features to self-correct and localize the true target. Conversely, injecting noise into the language instructions causes a proportional degradation in performance, precisely matching the error injection rate. This confirms the policy’s strict semantic compliance, proving that our architecture successfully balances robust visual adaptability with absolute adherence to language commands.
HiVLA
13
Table 3: Real-world success rates. HiVLA excels in multi-object cluttered scenarios requiring semantic grounding, where baseline models struggle. Method H-RDT Ours
5.3
Pick & Place Cup
Pick & Place Block
1 Bell
Click Bell 2 Bells
1 Cup
3 Cups
1 Block
3 Blocks
8/30 13/30
9/30 17/30
4/30 21/30
0/30 6/30
9/30 20/30
0/30 7/30
Evaluation in the Real World
To further validate real-world applicability and robustness of our visual-grounded hierarchical design, we deployed HiVLA in a physical robotic environment. Real-World Task Setup and Protocol. As visualized in Fig. 3, our experiments span 7 object categories and encompass 16 distinct sub-type scenarios. Rather than evaluating on conventional easy tasks, we specifically designed these scenarios to stress-test strong cross-environmental generalization and precise instruction following. By utilizing complex combinations of objects with varying colors and spatial arrangements, the tasks range from manipulating a single primitive to selecting a specific target (e.g., a red block or a green cup) from dense, multi-object clutter. For model training, we collected a dataset of 360 teleoperated episodes, automatically annotated with precise bounding boxes via GroundingDINO [25] and SAM2 [29]. Both H-RDT and HiVLA were initialized from their simulationtrained checkpoints to leverage structural priors, and subsequently fine-tuned for 80K steps on real-world data. During the evaluation phase, each task was attempted for 30 trials, with object positions randomized to prevent rote memorization and rigorously assess policy robustness. Success Rate Analysis. The real-world success rates are detailed in Tab. 3. It is crucial to note that our evaluation suite intentionally targets hard, stronggeneralization scenarios requiring rigorous semantic reasoning. Consequently, the baseline H-RDT exhibits performance degradation. While it performs adequately in isolated single-object scenarios, its success rate collapses to nearly zero in multi-object clutter (e.g., ‘3 Cups’, ‘3 Blocks’). Relying solely on global visual features, H-RDT lacks the fine-grained grounding required to disambiguate identical shapes using color attributes or spatial commands. In contrast, HiVLA’s hierarchical decoupling relieves the DiT Action Expert of the global reasoning burden, allowing it to efficiently map the sparse, limited real-world data to precise local visual-language conditions. As a result, HiVLA effectively navigates complex, cluttered scenes, executing sub-skills with remarkable accuracy and generalizing robustly across demanding physical tasks. 5.4
Ablation Study
We conduct ablation studies on the DiT Action Expert’s architecture to analyze the efficacy of visual representations and cross-attention guidance strategies.
14
T. Yang et al.
Click 3 Bells
Stack Blocks
Stamp Seal
Move Stapler
Place Shoe
Lift Pot
Press Stapler
Click Clock
Method
Click Bell
Table 4: Ablation study on guidance injection strategies and visual-grounding components. Best results are bold. Avg
(A) Guidance Injection Strategy Local→Text 93% 95% 96% 89% 76% 30% 60% 12% 84% 70.4% Global→Text 88% 93% 89% 92% 88% 34% 43% 20% 88% 70.6% Local→Text→Global Global→Text→Local Local→Global→Text
97% 92% 97% 98% 90% 49% 69% 43% 88% 80.1% 90% 88% 95% 98% 94% 48% 65% 39% 89% 78.3% 89% 97% 99% 98% 92% 41% 59% 17% 79% 74.1%
Global→Local→Text 94% 97% 97% 96% 95% 60% 76% 37% 98% 83.3% (B) Visual-Grounding Components w/o HD Crop 85% 92% 94% 79% 94% 57% 63% 31% 82% 75.2% w/o Abs. PE 90% 92% 95% 93% 94% 52% 62% 33% 80% 76.8% Ours (Full)
94% 97% 97% 96% 95% 60% 76% 37% 98% 83.3%
Guidance Injection Strategy. The order in which conditions are injected into the DiT via cross-attention heavily impacts policy learning. We expanded our ablation to isolate guidance contributions (Tab. 4). Relying solely on Local or Global visual features yields suboptimal results (∼70% average). Combining both captures the broader environment while explicitly grounding the target. More importantly, our analysis of the cross-attention ordering confirms that the “Coarse-to-Fine” injection strategy (Global Context → Local Crop → Language Skill ) allows the DiT to progressively narrow its attention from the entire scene to the specific object, and finally to the semantic action, yielding the optimal 83.3% average success rate. Visual-Grounding Components. We investigate two variants: (1) LowRes Crop: Cropping from a down-sampled 640 × 360 image rather than the 1080p source. (2) w/o Absolute PE: Removing absolute sinusoidal positional encoding for the cropped image tokens. As shown in Tab. 4, low-resolution crops significantly degrade performance on tasks involving fine-grained structures (e.g., grasping the thin handle in ‘Lift Pot’). Furthermore, without absolute spatial PE, the model fails to disambiguate identical objects (e.g., ‘Click 3 Bells’), proving that explicit spatial guidance is indispensable.
6
Conclusion
In this work, we presented HiVLA, a hierarchical visual-grounded-centric manipulation system that effectively resolves the fundamental trade-off in end-toend VLA models between preserving VLM reasoning capabilities and achieving precise low-level control. By decoupling high-level planning from action generation, our framework employs a VLM planner for task decomposition and visual grounding, while a novel DiT action expert leverages this grounded plan through a cascaded cross-attention mechanism that integrates global context,
HiVLA
15
position-aware local features, and subtask guidance. Extensive experiments in both simulation and real-world settings demonstrate that HiVLA significantly outperforms state-of-the-art baselines, achieving an 17.7% improvement over HRDT and 42.7% over π0 in simulation, with particular strength in long-horizon skill composition and fine-grained manipulation of small objects in cluttered environments. Beyond performance gains, HiVLA’s modular architecture enables independent scaling of each component and provides interpretability through explicit intermediate plans, establishing a robust and scalable foundation for complex robotic manipulation systems.
References 1. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025) 1, 10, 11, 19 2. Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y., Dwibedi, D., Sadigh, D.: Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01823 (2024) 2 3. Bhat, V., Lan, Y.H., Krishnamurthy, P., Karri, R., Khorrami, F.: 3d cavla: Leveraging depth and 3d context to generalize vision language action models for unseen tasks. arXiv preprint arXiv:2505.05800 (2025) 4 4. Bi, H., Wu, L., Lin, T., Tan, H., Su, Z., Su, H., Zhu, J.: H-rdt: Human manipulation enhanced bimanual robotic manipulation. arXiv preprint arXiv:2507.23523 (2025) 3, 10 5. Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., et al.: Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734 (2025) 2, 3, 10 6. Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al.: π0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 (2024) 2, 3, 10 7. Brohan, Anthony, e.a.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 (2023) 2, 3 8. Bu, Q., Li, H., Chen, L., Cai, J., Zeng, J., Cui, H., Yao, M., Qiao, Y.: Towards synergistic, generalized, and efficient dual-system for robotic manipulation. arXiv preprint arXiv:2410.08001 (2024) 2 9. Cheang, Catherine, e.a.: Gr-3 technical report. arXiv preprint arXiv:2507.15493 (2025) 2 10. Chen, T., Chen, Z., Chen, B., Cai, Z., Liu, Y., Li, Z., Liang, Q., Lin, X., Ge, Y., Gu, Z., et al.: Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088 (2025) 3, 9 11. Chen, X., Chen, Y., Fu, Y., Gao, N., Jia, J., Jin, W., Li, H., Mu, Y., Pang, J., Qiao, Y., Tian, Y., Wang, B., Wang, B., Wang, F., Wang, H., Wang, T., Wang, Z., Wei, X., Wu, C., Yang, S., Ye, J., Yu, J., Zeng, J., Zhang, J., Zhang, J., Zhang, S., Zheng, F., Zhou, B., Zhu, Y.: Internvla-m1: A spatially guided vision-languageaction framework for generalist robot policy (2025), https://arxiv.org/abs/ 2510.13778 4
16
T. Yang et al.
12. starVLA Contributors: Starvla: A lego-like codebase for vision-language-action model developing. GitHub repository (1 2025). https://doi.org/10.5281/ zenodo.18264214, https://github.com/starVLA/starVLA 10 13. Driess, D., Springenberg, J.T., Ichter, B., Yu, L., Li-Bell, A., Pertsch, K., Ren, A.Z., Walke, H., Vuong, Q., Shi, L.X., Levine, S.: Knowledge insulating visionlanguage-action models: Train fast, run fast, generalize better (2025), https:// arxiv.org/abs/2505.23705 2, 4 14. Fan, C., Jia, X., Sun, Y., Wang, Y., Wei, J., Gong, Z., Zhao, X., Tomizuka, M., Yang, X., Yan, J., et al.: Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions. arXiv preprint arXiv:2505.02152 (2025) 2, 4, 10 15. Hancock, A.J., Wu, X., Zha, L., Russakovsky, O., Majumdar, A.: Actions as language: Fine-tuning vlms into vlas without catastrophic forgetting (2025), https: //arxiv.org/abs/2509.22195 2, 4 16. Huang, H., Chen, X., Chen, Y., Li, H., Han, X., Wang, Z., Wang, T., Pang, J., Zhao, Z.: Roboground: Robotic manipulation with grounded vision-language priors. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 22540–22550 (2025) 4 17. Intelligence, P., Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Galliker, M.Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Ren, A.Z., Shi, L.X., Smith, L., Springenberg, J.T., Stachowicz, K., Tanner, J., Vuong, Q., Walke, H., Walling, A., Wang, H., Yu, L., Zhilinsky, U.: π0.5 : a vision-language-action model with openworld generalization (2025), https://arxiv.org/abs/2504.16054 2, 4, 10 18. Jiang, Y., Gu, J., Xue, T., Cheung, K.C., Molchanov, P., Yin, H., Liu, S.: Tokenefficient vlm: High-resolution image understanding via dynamic region proposal. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 24147–24158 (October 2025) 2 19. Kim, M.J., Finn, C., Liang, P.: Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645 (2025) 2 20. Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E.P., Sanketi, P.R., Vuong, Q., et al.: Openvla: An open-source vision-language-action model. In: Conference on Robot Learning. pp. 2679–2713. PMLR (2025) 2, 3 21. Lai, X., Li, J., Li, W., Liu, T., Li, T., Zhao, H.: Mini-o3: Scaling up reasoning patterns and interaction turns for visual search (2025), https://arxiv.org/abs/ 2509.07969 2 22. Li, Y., Deng, Y., Zhang, J., Jang, J., Memmel, M., Yu, R., Garrett, C.R., Ramos, F., Fox, D., Li, A., Gupta, A., Goyal, A.: Hamster: Hierarchical action models for open-world robot manipulation (2025), https://arxiv.org/abs/2502.05485 4 23. Liang, Z., Mu, Y., Ma, H., Tomizuka, M., Ding, M., Luo, P.: Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution (2024), https://arxiv.org/abs/2312.11598 2 24. Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26296–26306 (2024) 1 25. Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In: European conference on computer vision. pp. 38–55. Springer (2024) 13
HiVLA
17
26. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) 19 27. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023) 19 28. Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4195–4205 (2023) 3, 4 29. Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024) 13 30. Shao, R., Li, W., Zhang, L., Zhang, R., Liu, Z., Chen, R., Nie, L.: Large vlm-based vision-language-action models for robotic manipulation: A survey. arXiv preprint arXiv:2508.13073 (2025) 3 31. Shi, B., Li, B., Cai, H., Lu, Y., Liu, S., Pavone, M., Kautz, J., Han, S., Darrell, T., Molchanov, P., Yin, H.: Scaling vision pre-training to 4k resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9631–9640 (June 2025) 2 32. Shi, L.X., Ichter, B., Equi, M., Ke, L., Pertsch, K., Vuong, Q., Tanner, J., Walling, A., Wang, H., Fusai, N., Li-Bell, A., Driess, D., Groom, L., Levine, S., Finn, C.: Hi robot: Open-ended instruction following with hierarchical vision-language-action models (2025), https://arxiv.org/abs/2502.19417 2, 4, 10 33. Song, W., Zhou, Z., Zhao, H., Chen, J., Ding, P., Yan, H., Huang, Y., Tang, F., Wang, D., Li, H.: Reconvla: Reconstructive vision-language-action model as effective robot perceiver. arXiv preprint arXiv:2508.10333 (2025) 4 34. Sridhar, A., Pan, J., Sharma, S., Finn, C.: Memer: Scaling up memory for robot control via experience retrieval. arXiv preprint arXiv:2510.20328 (2025) 4 35. Steiner, A., Pinto, A.S., Tschannen, M., Keysers, D., Wang, X., Bitton, Y., Gritsenko, A., Minderer, M., Sherbondy, A., Long, S., et al.: Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555 (2024) 1 36. Team, G.R., et al.: Gemini robotics: Bringing ai into the physical world (2025), https://arxiv.org/abs/2503.20020 10 37. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) 19 38. Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786 (2025) 19 39. Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, J., Li, X., Liu, M., Li, H., Kong, T.: Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139 (2023) 4 40. Zhao, Q., Lu, Y., Kim, M.J., Fu, Z., Zhang, Z., Wu, Y., Li, Z., Ma, Q., Han, S., Finn, C., et al.: Cot-vla: Visual chain-of-thought reasoning for vision-languageaction models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 1702–1713 (2025) 2 41. Zheng, Z., Yang, M., Hong, J., Zhao, C., Xu, G., Yang, L., Shen, C., Yu, X.: Deepeyes: Incentivizing "thinking with images" via reinforcement learning (2025), https://arxiv.org/abs/2505.14362 2
18
T. Yang et al.
42. Zhong, Y., Huang, X., Li, R., Zhang, C., Chen, Z., Guan, T., Zeng, F., Lui, K.N., Ye, Y., Liang, Y., Yang, Y., Chen, Y.: Dexgraspvla: A vision-language-action framework towards general dexterous grasping (2025), https://arxiv.org/abs/ 2502.20900 2, 4, 10
HiVLA
19
Supplementary Material 1
DiT Model Details
Implementation Details We implemented our model using the PyTorch framework, leveraging the HuggingFace Accelerate library for distributed training. The model was trained on a cluster equipped with 2 NVIDIA H200 GPUs. We utilized the AdamW [26] optimizer with a weight decay of 1 × 10−2 and a gradient clipping threshold of 1.0 to ensure training stability. The learning rate followed a constant schedule with a linear warmup phase of 500 steps, peaking at 1 × 10−4 . To optimize memory usage and computational throughput without compromising performance, we employed BFloat16 (BF16) mixed-precision training. The global batch size was set to 64 (32 per GPU). The model was trained for 150k steps. Tab. 5 presents the detailed hyperparameters and architecture specifications for the DiT Action Expert. Architecture Specifications Our architecture follows a high-capacity Transformer design. The core backbone consists of 16 layers with a hidden dimension of 2,176. We employed Grouped Query Attention (GQA) to balance computational efficiency and performance, utilizing 16 attention heads and 8 key-value heads. For the feed-forward networks (FFN), we adopted the SwiGLU activation function, following the architectural patterns of LLaMA [37]. Layer Normalization (LayerNorm) with an epsilon of 1 × 10−5 was applied before the attention and FFN blocks (Pre-LN). Input Conditioning The vision backbone (DINOv2 [27] + SigLIP [38]) was kept frozen during training to leverage robust pre-trained representations. We utilized distinct Multi-Layer Perceptron (MLP) projectors to map different modalities into the transformer’s latent space. Specifically, a 2-layer MLP with SiLU activation was used for visual and language embeddings, while a deeper 3-layer MLP was employed for state and action embeddings to capture complex kinematic dynamics.
2
VLM Planner Agent Analysis
In this section, we provide a comprehensive analysis of the High-Level VLM Planner Agent. We detail its experimental setup, evaluate the impact of finetuning across different model scales, and ablate key design choices such as visual history injection. Experimental Setup and Metrics. We employ Qwen3-VL [1] as our core VLM Planner Agent. It offers formidable perception and reasoning capabilities while maintaining deployment flexibility. To independently assess its planning proficiency, we curated a dataset of 210K dialogue instances derived from
20
T. Yang et al.
Table 5: Hyperparameters and architecture specifications. Detailed configuration of the HiVLA DiT Action Expert and training settings.
Hyperparameter
Value
Transformer Backbone Hidden size Layers Attention heads Key-value heads (GQA) Activation function Normalization Action chunk size (horizon)
2176 16 16 8 SwiGLU LayerNorm (\epsilon = 1\mathrm {e}{-5}) 16
Encoders & adapters Vision backbone Vision/text adapter State/action adapter State/action dimension Image resolution Patch size
DINO-SigLIP (frozen) MLP (2-layer, SiLU) MLP (3-layer, SiLU) 14 384 \times 384 14 \times 14
Optimization & training Optimizer Learning rate LR schedule Warmup steps Weight decay Gradient clipping Global batch size Mixed precision Training steps
AdamW 1 \times 10^{-4}
Constant with warmup 500 1 \times 10^{-2}
1.0 64 bfloat16 150,000
HiVLA-HD. This dataset is split into an 80/20 ratio for training and testing. For fine-tuning, we trained the models on two NVIDIA H200 GPUs. We used a batch size of 4 and a learning rate of 1e-5, training for 3 epochs. During evaluation, we measure visual grounding using the mean Intersection over Union (mIoU) of the predicted bounding boxes. For subtask prediction, we employ a strict exact-match criterion. The model must correctly predict both the required skill and the target object name to score a success. Fine-Tuning vs. Zero-Shot Capabilities. We evaluate multiple models under both zero-shot and fine-tuned settings. The comprehensive results are reported in Tab. 6. While baseline VLMs possess competent zero-shot reasoning, their out-of-the-box performance is insufficient for precise, long-horizon manipulation. Scaling up the model parameters (e.g., from 8B to 32B or using MoE
HiVLA
21
Table 6: Comprehensive Evaluation of the VLM Planner Agent. We compare different model scales and architectures under zero-shot and fine-tuned settings. Finetuning drastically improves domain-specific performance, while historical visual context is critical for optimal accuracy. Model
Setting
Grounding (mIoU ↑) Sub-task Acc. (↑) Zero-Shot Evaluation
Qwen3-VL-4B Zero-shot Qwen3-VL-8B Zero-shot Qwen3-VL-32B Zero-shot Qwen3-VL-30B-A3B Zero-shot GPT-4o Zero-shot Qwen3-VL-4B Qwen3-VL-8B Qwen3-VL-8B
28.03 12.68 20.17 32.46 3.45
45.51 35.71 39.85 41.41 42.85
Fine-Tuned Evaluation Fine-tuned 92.21 Fine-tuned (w/o history) 89.63 Fine-tuned (Ours) 90.37
97.92 95.24 98.57
architectures like 30B-A3B) steadily improves zero-shot subtask accuracy and grounding. Even proprietary state-of-the-art models like GPT-4o achieve competitive zero-shot subtask accuracy (42.85%), though their native spatial grounding remains weak (3.45% mIoU). Crucially, lightweight fine-tuning on domain-specific visual-language data triggers a massive performance boost. The fine-tuned Qwen3-VL 8B model achieves an exceptional 90.37% mIoU and 98.57% subtask accuracy. This underscores a key advantage of our hierarchical design. It preserves the generalizable priors of pretrained VLMs, yet allows for specialized, highly effective enhancements through scalable fine-tuning. To best validate our DiT Action Expert, we utilize this fine-tuned 8B model for all main paper evaluations.
The Necessity of Visual History. Manipulation tasks are inherently sequential. A robust planner must understand what has already been accomplished. To validate this, we ablate the visual history input. As shown in Tab. 6, removing historical frames during fine-tuning (“w/o history”) leads to a clear performance drop. Subtask accuracy falls from 98.57% to 95.24%. This confirms that historical observations are essential. They provide the necessary temporal context for accurate task progression and target disambiguation.
Extensibility and Future Scaling. Our decoupled architecture makes the VLM planner readily replaceable. Advanced foundation models can serve as direct, plug-and-play replacements for the planner module. While domain-specific fine-tuning remains the most optimal deployment strategy today, the steady improvement in zero-shot capabilities of larger models (e.g., Qwen3-VL-32B, GPT-4o) highlights the strong future potential of our system. As VLM agents continue to evolve, HiVLA will seamlessly inherit their enhanced cognitive limits.
22
T. Yang et al.
Prompt Design. We provide the detailed prompt structure used for the VLM Planner Agent in Tab. 7. The prompt is designed to enforce strict JSON output formatting while providing the agent with context-aware visual and state inputs.
3
Task Visualization
We present comprehensive visualizations of the experimental tasks conducted in both the RoboTwin simulation environment and real-world scenarios. The specific natural language instructions corresponding to each task are detailed in Tab. 8. Visual demonstrations of the execution sequences in the RoboTwin simulation are illustrated in Fig. 4, while the corresponding real-world execution processes are depicted in Fig. 5.
HiVLA
23
Table 7: System prompt for the VLM planner agent. The agent receives historical and current observations together with state information, and generates a structured subtask plan. Role You are the central control unit for a robotic arm. Your goal is to analyze visual and state information to decide the next action needed to complete a high-level task. Provided Information You are given two images in order: 1. Previous Scene Image: The scene after the last action was executed (corresponding to the first <image>). 2. Current Scene Image: The live scene right now (corresponding to the second <image>). Current State Inputs – Overall Goal: {task_instruction} – Previous Subtask Commanded: {previous_subtask} – Current Gripper State: {gripper_state_str} Your Task Based only on the Current State Inputs and the two provided images, you must generate a JSON object describing the next action to perform. Your response must be a single JSON object with no extra text or explanations. The JSON object must contain exactly the following four keys: { "next_subtask_description": "A clear description of the next subtask you are planning.", "action_type": "pick or place", "target_object": "The specific object involved in the action. For pick, this is the object to grasp. For place, this is the object that the robot should place the grasped object onto.", "bbox": "[ymin, xmin, ymax, xmax], a normalized bounding box with coordinates in [0,1000] for the target_object in the Current Scene Image." }
24
T. Yang et al.
Table 8: List of Task Instructions. The specific natural language instructions corresponding to each task in the RoboTwin simulation and real-world experiments. Task Name
Instruction
RoboTwin Simulation Tasks Click Bell Click Alarm Clock Press Stapler Lift Pot Place Shoe Move Stapler Pad Stamp Seal Stack 3 Blocks
Click 3 Bells
Click the bell on the table. Click the alarm clock’s center of the top side button on the table. Use one arm to press the stapler. Use BOTH arms to lift the pot. Grab the shoe from the table and place it on the red mat. Pick up the stapler and place it on the red mat. Grab the stamp and stamp onto the red mat. There are three blocks on the table, the color of the blocks is yellow, green and blue; move the blocks to the center of the table, and stack the blue block on the green block, and the green block on the yellow block. Click the right/left/middle bell.
Real World Tasks Click 1 Bell Click the bell on the table. Click 2 Bells Click the bell on the left/right. Pick & Place 1/3 Cup Pick up the blue/green/yellow cup and place it on the coaster. Pick & Place 1/3 Block Pick up the blue/green/red block and place it on the plate.
HiVLA
Click Bell
Click Clock
Press Stapler
Lift Pot
Place Shoe
Move Stapler
Stamp Seal
Stack 3 Blocks
Click 3 Bells
Fig. 4: Visualization of RoboTwin tasks.
25
26
T. Yang et al.
Click 1 Bell
Click 2 Bells
Click Bells LR
Pick Block 1
Pick Block 3
Pick Cup 1
Pick Cup 3
Fig. 5: Visualization of real-world tasks.