Jump-Start Reinforcement Learning with Vision-Language-Action Regularization Angelo Moroncelli12 , Roberto Zanetti3 , Marco Maccarini12 , Loris Roveda13 University of Applied Science and Arts of Southern Switzerland, Department of Innovative Technologies, IDSIA-SUPSI, Lugano, Switzerland {angelo.moroncelli, marco.maccarini, loris.roveda}@supsi.ch 2 Università della Svizzera Italiana, Faculty of Informatics, Lugano, Switzerland 3 Politecnico di Milano, Mechanical Department, Milano, Italy {roberto2.zanetti@mail., loris.roveda@}polimi.it Large Scale Pre-training
Suboptimal Credit Assignment in RL
JS Real-World RL Expert Beyond Distillation Object Detection + Language-Vision Guidance
State-based On-Policy RL
Case study 1: Long-Horizon Task
x10 steps
Real-world
Language Case study 2: Imperfect Reward Definition
X
X
Jump-Start Training in Simulation
Simulation (infinite data)
Vision
VLA Model Pool
OpenVLA
Octo
VLA Jump-Start (VLAJS)
Improved Exploration Average All Tasks-v2 1 VLAJS
Success Rate
arXiv:2604.13733v1 [cs.LG] 15 Apr 2026
1
0
Complex Tasks Beyond VLA
PPO VLAJS(RPD)
Training Steps Background Generalization
External Disturbance
Directional Loss from VLA Guidance
Sparse Guidance + Free Exploration
Sim-to-Real Gap Reduction
Fig. 1: Overview of Vision-Language-Action Jump-Starting (VLAJS). The figure illustrates the motivation, method, and outcomes of VLAJS. Left: We highlight suboptimal credit assignment in state-based, on-policy RL, focusing on: long-horizon tasks with extended action sequences and environments with imperfect reward design. Center: VLAJS leverages large-scale VLA pretraining from both real-world and simulation data. A pool of pretrained VLA models (e.g., OpenVLA) provides sparse, low-frequency action suggestions conditioned on language and vision. These suggestions are incorporated during RL training through a directional action-consistency loss, enabling jump-start learning while preserving exploration. Right: The VLAJS-based trained RL agent surpasses distillation-based approaches, achieving improved exploration, robustness to background changes and external disturbances, and reduced sim-to-real gaps.
Abstract—Reinforcement learning (RL) enables highfrequency, closed-loop control for robotic manipulation, but scaling to long-horizon tasks with sparse or imperfect rewards remains difficult due to inefficient exploration and poor credit assignment. Vision-Language-Action (VLA) models leverage large-scale multimodal pretraining to provide generalist, task-level reasoning, but current limitations hinder their direct use in fast and precise manipulation. In this paper, we propose Vision-Language-Action Jump-Starting (VLAJS), a method that bridges sparse VLA guidance with on-policy RL to improve exploration and learning efficiency. VLAJS treats VLAs as transient sources of high-level action suggestions that bias early exploration and improve credit assignment, while preserving the high-frequency, state-based control of RL. Our approach augments Proximal Policy Optimization (PPO) with a directional action-consistency regularization that softly aligns the RL agent’s actions with VLA guidance during early training, without enforcing strict imitation, requiring demonstrations, or relying on continuous teacher queries. VLA guidance is applied sparsely and annealed over time, allowing the agent to adapt online and ultimately surpass the guiding policy. We evaluate VLAJS on six challenging manipulation tasks—lifting, pick-and-place, peg reorientation, peg insertion,
poking, and pushing—in simulation, and validate a subset on a real Franka Panda robot. VLAJS consistently outperforms PPO and distillation-style baselines in sample efficiency, reducing required environment interactions by over 50% in several tasks. Real-world experiments demonstrate zero-shot sim-to-real transfer and robust execution under clutter, object variation, and external perturbations.
I. I NTRODUCTION Reinforcement Learning (RL) provides a powerful framework for learning closed-loop control policies directly from interaction [40]. In robotics, RL enables high-frequency, statebased controllers that exploit rich proprioceptive and geometric feedback, enabling precise motor behaviors and online adaptation. These properties make RL particularly attractive for real-world manipulation, where robustness to disturbances, tight feedback loops, and reliability are critical [24]. Despite these strengths, RL faces well-known challenges. Learning complex manipulation behaviors often requires long training times, careful reward engineering, and large amounts of interaction data [3, 21, 41]. These issues are especially
Guidance Behavioral
Transient
Imitation Signal
None
Vanilla RL
JSRL
Persistent
Auxiliary
DAgger-like
This work (VLAJS)
Policy Distillation / RPD
Fig. 2: Comparison of guidance strategies in RL. Methods are categorized by guidance type (behavioral vs. auxiliary) and imitation persistence (none, transient, and persistent). Vanilla RL uses no guidance, DAgger-like methods apply persistent behavioral imitation, and policy distillation/RPD rely on persistent auxiliary losses. JSRL provides transient behavioral guidance, while VLAJS introduces transient auxiliary guidance via a directional actionconsistency loss, accelerating early learning without limiting asymptotic performance.
pronounced in long-horizon or suboptimally rewarded tasks, where delayed rewards lead to weak credit assignment and slow policy improvement [31]. Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) offer a complementary direction [13, 9, 8, 2]. Vision-Language-Action (VLA) models unify perception, language understanding, and control by mapping multimodal observations directly to robot actions [5, 6]. Large-scale systems such as Octo [29], OpenVLA [17], π0 [14], and RDT-1B [23] demonstrate strong semantic understanding and broad generalization, often exhibiting zero-shot and few-shot capabilities. However, current VLA models are not designed to replace RL-based controllers for precise robotic manipulation. Their reliance on large expert datasets limits scalability and adaptation [30, 46, 16, 27], while transformer-based architectures typically operate at low control frequencies, restricting their ability to handle precision requirements, disturbances, and long-horizon closed-loop behaviors [26]. As a result, despite strong semantic priors, standalone VLA performance in realworld manipulation remains limited. In this work, we view RL and VLA models as complementary components for training robotic manipulation policies. RL provides a deployable, high-frequency control backbone, while VLA models offer sparse, high-level action priors that encode semantic knowledge and task structure. Rather than contrasting these approaches, we ask: Can the semantic knowledge captured by VLA models be used to accelerate RL, while preserving the precision, adaptability, and reliability of RL-based controllers? Fig. 1 illustrates our perspective. We treat VLA models as sources of sparse, low-rate guidance that bias early exploration, improve credit assignment, and reduce the sim-toreal gap during training. The RL agent remains an on-policy, state-based learner operating at high control frequency, capable of exploiting precise feedback and ultimately surpassing the
guiding policy. To realize this synergy, we introduce VLAJS, a method for jump-starting on-policy RL using sparse guidance from VLA models. A high-frequency RL controller receives occasional VLA action suggestions during early training, incorporated through a directional action-consistency regularization within a PPO framework. This guidance is transient: it is queried infrequently, temporally propagated across control steps, and gradually annealed. The result is faster early learning without constraining long-term optimization or incurring excessive VLA inference cost. We evaluate VLAJS on six challenging manipulation tasks in ManiSkill and validate a subset on a real Franka Panda robot. Focusing on long-horizon objectives and suboptimal reward design, our results demonstrate substantial gains in sample efficiency while producing policies that are directly deployable on real robotic systems. Therefore, our contributions are: 1) We propose a method for accelerating high-frequency, state-based RL using sparse, low-rate guidance from VLA models, producing policies that are directly deployable on real robotic systems (C1). 2) We introduce a directional action-consistency regularization that enables flexible, transient guidance without constraining asymptotic performance (C2). 3) We demonstrate improved sample efficiency over PPO and distillation-based methods. 4) We introduce long-horizon and suboptimally rewarded ManiSkill environments for studying RL under difficult credit assignment, which we will publicly release. II. R ELATED W ORK We review prior work along two axes most relevant to our approach: (i) reinforcement learning under suboptimal credit assignment and expert guidance, and (ii) Vision-LanguageAction models for robotic control. A. RL, Credit Assignment, and Expert Guidance Reinforcement learning is a natural fit for complex manipulation because it supports high-frequency, closed-loop control from state feedback, but it can struggle in long-horizon and sparsely rewarded settings due to inefficient exploration and weak credit assignment [31, 40, 24, 21, 37]. A common strategy to mitigate poor credit assignment is to incorporate expert guidance [22, 32, 25, 15]. Fig. 2 summarizes the corresponding formal policy/objective definitions, while Fig. 3 provides an intuitive, trajectory-level view of exploration strategies. Persistent behavioral guidance [11, 36, 35] mixes a learned policy πθ with an expert policy πE , π = (1 − β)πθ + βπE , where β > 0 controls expert intervention. Jump-start RL [42, 19, 43, 12] instead applies transient behavioral guidance by delegating control to a guide policy πg only early in training, ( πg , t ≤ h, πt = πθ , t > h,
where h denotes the number of guide steps. Complementarily, distillation-style approaches [15, 45, 39, 1, 10, 7] guide learning via an auxiliary action-matching loss, LRL + ∥aπ − ateacher ∥2 , which can overly constrain optimization when applied persistently, and can be brittle when teacher supervision is sparse or imperfect. These limitations motivate transient auxiliary guidance: use a teacher primarily early in training to bias exploration and improve credit assignment, then anneal and remove the auxiliary signal once the on-policy learner begins to improve reliably. In addition to online guidance, several approaches incorporate prior data or expert policies to improve sample efficiency. Offline-to-online RL pretrains policies or value functions on logged datasets before fine-tuning, while imitation learning/behavior cloning provides initialization but can suffer from distribution shift [28, 27, 34].
(a) Vanilla RL
(b) Behavioral guidance
(c) Auxiliary guidance
Fig. 3: Guidance mechanisms for exploration in RL. (a) Relies on random exploration. (b) Executes an imitation-learned policy for an initial phase (solid path). (c) Continuously biases learning via a teacher-provided signal (dashed red path) without directly executing actions.
A. Preliminaries: PPO for High-Frequency State Control All methods build on Proximal Policy Optimization (PPO) with Generalized Advantage Estimation [37]. Let πθ (a | s) denote a stochastic policy and Vϕ (s) a value function. PPO minimizes the clipped surrogate objective h i LPPO (θ) = −Et min rt (θ)Ât , clip(rt (θ), 1 − ϵ, 1 + ϵ)Ât ,
B. Vision-Language-Action Models Vision-Language-Action models extend foundation-model paradigms to robotics by directly mapping multimodal observations and language instructions to actions [9, 30, 5, 6]. Recent VLAs demonstrate impressive generalization across manipulation tasks, but their practical deployment is constrained by inference latency, low control frequency, and reliance on demonstrations [46, 17, 18]. Techniques such as action chunking and parallel decoding improve throughput but do not fundamentally address the lack of tight closed-loop feedback [4, 14, 29, 23]. Few works explore combining VLAs with downstream RL fine-tuning [20] or distillation [45]. Refined Policy Distillation (RPD) [15] trains a PPO agent under continuous supervision from a VLA teacher using an action-matching loss. While effective in ideal scenarios, RPD assumes expensive teacher access at every timestep, relies primarily on visual inputs, and produces a standalone policy evaluated only under controlled training conditions, without real-world experiments. In contrast, our approach positions the RL agent as a reusable, state-based control layer that is guided—but not dominated—by a VLA. By using sparse, transient auxiliary guidance rather than behavioral (Fig. 3b) or persistent (Fig. 4b) imitation, VLAJS bridges high-level foundation-model reasoning with efficient, high-frequency control, addressing both credit assignment and real-world execution constraints. III. M ETHODOLOGY In this paper, we propose Vision-Language-Action JumpStarting (VLAJS), an on-policy RL method that leverages a pretrained Vision–Language–Action model [17, 29] as sparse, transient auxiliary guidance. VLAJS targets settings with suboptimal credit assignment—in particular (i) long-horizon tasks and (ii) imperfect reward design—where vanilla onpolicy RL often fails to discover rewarding behaviors within practical interaction budgets.
where rt (θ) = πθπθ (a(at |st |st )t ) and Ât are GAE advantages. PPO old is attractive for robotics because it supports stable learning of high-frequency, closed-loop, state-based control without demonstrations; however, it is notoriously inefficient when rewards are sparse/delayed or horizons are long [31]. B. Sparse VLA Queries and Temporal Discretization We assume access to a pretrained VLA teacher that takes a visual observation and a language instruction and outputs a low-rate delta action aVLA (translation, rotation, gripper). Querying the teacher at every environment step is impractical for long-horizon rollouts and parallel simulation. Therefore, we query the VLA sparsely in time: only a small number of calls per rollout. Each teacher delta is then temporally discretized into a short sequence of incremental deltas applied as guidance targets over the next D control steps (linear interpolation for translation and SLERP-style [38] interpolation for rotation). Outside these windows, the teacher target is treated as absent and masked out of the auxiliary loss. This yields guidance that is sparse in two ways: (i) sparse teacher calls across time and (ii) sparse supervision within each rollout, enabling practical wall-clock training while still providing exploration bias early in learning (Fig. 4 explains this concept). C. Reward-Based Jump-Starting (C1) A key principle of VLAJS is that VLA guidance should be transient: the teacher is most useful before PPO has discovered a productive exploration regime. Once PPO begins to learn reliably, persistent guidance can become unnecessary (and even harmful if the teacher is suboptimal), and it remains computationally expensive. We therefore introduce a rewardtrend–based jump-start mechanism that reduces and ultimately deactivates teacher usage (see Fig. 4c).
Our auxiliary objective is
rollout 1 improvement
Ldir = Et 1[validt ]
ℓdir µcθ (st ), ãct ,
c∈{pos,rot}
(a) Vanilla RL rollout 2 improvement
... rollout N improvement
(b) Persistent auxiliary
X
(c) Transient auxiliary (VLAJS)
Fig. 4: Auxiliary guidance during rollouts. (a) The policy generates actions solely through on-policy exploration at a fixed control frequency, learning both direction and action scale incrementally from reward. (b) A teacher provides continuous action targets throughout the rollout, constraining both direction and magnitude and forcing the policy to match the teacher’s action scale (distillation/RPD style). (c) Guidance is applied sparsely within a rollout and progressively annealed across rollouts, biasing action direction while allowing the policy to learn its own action magnitude and eventually explore freely.
where 1[validt ] masks timesteps without guidance and we skip components with near-zero teacher vectors to avoid unstable normalization. We do not constrain the gripper dimension in the auxiliary loss. Rationale for direction-only: Cosine alignment preserves the direction suggested by the teacher while allowing PPO to choose action magnitudes and fine corrections. This is particularly important when (i) teacher actions are discretized across multiple steps, (ii) teacher scale may not match the student’s control frequency, and (iii) the teacher is imperfect and should not be copied exactly. E. Training Objective
a) Adaptive query rate: At each PPO iteration, we compute a reward-improvement statistic from a rolling history of mean rollout rewards. As improvement increases, we reduce the number of teacher calls per rollout by an exponential schedule: Ncalls ← max(Nmin , ⌊Nmax exp(−κ · ∆r̄)⌋), where ∆r̄ denotes a reward-gain signal computed from recent rollouts (clipped at zero), κ controls decay, and Nmax /Nmin bound the calls per rollout. This retains more guidance when PPO is stuck, and quickly sparsifies guidance once learning accelerates. b) Permanent deactivation: We additionally detect monotonic reward improvement over a short window of recent iterations and permanently deactivate guidance once the mean rollout reward (“improvement” in Fig. 5c) exceeds a small threshold of 3. We chose 3 as the smallest value that reliably marks the onset of meaningful learning: across runs reward often hovers around ≈ 2 when the policy is still stuck, while once it goes above 3 learning proceeds reliably, allowing guidance to be turned off as early as possible (see Fig. 5d). D. Directional Action-Consistency Loss (C2) Using a VLA teacher as intermittent guidance differs from classical distillation [15]: the teacher is queried sparsely and can be suboptimal for precise, high-frequency control. In this setting, directly matching teacher actions (e.g., MSE in Fig. 5a) can be too strong and can inject inconsistent gradients when supervision appears intermittently. Instead, we treat teacher outputs as directional hints (see Fig. 5b). Let µθ (st ) denote the policy mean action and ãVLA the t discretized teacher target at time t (only present during discretization windows). We split actions into translation and rotation components and define a cosine misalignment loss ℓdir (x, y) = 1 −
⟨x, y⟩ . ∥x∥∥y∥ + ε
We augment PPO updates with the auxiliary guidance loss: L(θ) = LPPO (θ) + λt Laux (θ), where Laux = Ldir for VLAJS. The coefficient λt follows the same reward-trend schedule used for guidance sparsification and is set to zero after deactivation. F. Baselines Implemented in Our Code We implement three algorithms: a) PPO: Standard PPO [37] trained from scratch using only environment reward. b) Sparse RPD: For long-horizon experiments, we additionally evaluate a persistent sparse-guidance baseline that queries the teacher sparsely throughout training (no deactivation), reflecting “sparse distillation” as a computationally feasible alternative to full RPD [15] when horizons are long. c) VLAJS (RPD): An ablation that keeps exactly the same sparse query mechanism and jump-start deactivation, but replaces directional guidance with an RPD-style MSE actionmatching loss on guided steps: LMSE = Et 1[validt ] ∥µθ (st ) − ãVLA ∥22 . t This isolates the effect of the directional loss under sparse, transient teacher usage. IV. E XPERIMENTAL E VALUATION We evaluate our approach under two complementary sources of suboptimal credit assignment: a) Use Case 1 - Long-horizon task: We test whether sparse VLA guidance is a practical and effective substitute for dense distillation when episodes become very long (e.g., high-frequency control or extended horizons). Here, the goal is primarily to quantify exploration benefits and computational feasibility. We compare Sparse RPD—the building block of VLAJS (RPD)—to PPO.
2
Action Dimension 2
Action Dimension 2
4
3
2
1
0
2
1 1 0 Action Dimension 1
1
0
2
1
2
2
(a)
1 1 0 Action Dimension 1
2
(b)
(c)
(d)
Fig. 5: Auxiliary losses for VLA-guided RL. (a) Distillation-based methods (e.g., RPD) use an MSE loss that penalizes the full Euclidean distance between policy and teacher actions, constraining both action direction and magnitude. (b) VLAJS instead employs a directional action-consistency loss that penalizes angular misalignment between policy and VLA actions, while remaining invariant to action scale. (c–d) Plots show the superimposition of all training runs across tasks, which consistently follow the same trend: VLA guidance produces a pronounced jump-start, visible as a sharp early bump in rollout reward improvement (the metric used for guidance deactivation), after which the signal remains low as guidance is adaptively turned off. PickCube-Sim
PickPlaceCube-Sim
"Pick up the pot lid"-Real
"Pick up the tomato and place it into the bowl"-Real
LiftPegUpright-Sim
PegInsertionSide-Sim
"Pick up the grape"-Real
"Pick up the pepper and place it into the bowl"-Real
PokeCube-Sim
PushCube-Sim
"Pick up the red cube"-Real
"Lift the box upright"-Real
Fig. 6: Simulation and real-world manipulation tasks used in our evaluation. Left: six ManiSkill simulation tasks (PickCube, PickPlaceCube, LiftPegUpright, PegInsertionSide, PokeCube, PushCube). Right: zero-shot real-world deployment on a Franka Panda robot across diverse language-specified tasks.
b) Use Case 2 - Suboptimal reward design: We test whether jump-starting with transient guidance and a directional loss improves learning when rewards are sparse or imperfectly shaped, reflecting realistic reward design constraints. Here we compare PPO, VLAJS (RPD), and VLAJS.
b) Area Under the Success Curve (AUC): AUC integrates success rate over the full training budget [0, B] (where B is the total environment-step budget of the experiment). AUC captures both learning speed and final performance. We report bootstrap 95% confidence intervals across random seeds where available, and macro-averages across tasks.
A. Primary Metrics We report per-task success-based metrics aligned with the summary tables: a) Success Rate at t∗ (SRt∗ ): SRt∗ measures the fraction of evaluation episodes that successfully complete within a task-specific interaction budget t∗ (reported in environment steps). This captures whether a method learns the task within a practical sample budget.
B. Summary Tables For long-horizon experiments, Tab. III reports a macroaverage summary of SRt∗ and AUC across four extendedhorizon tasks, as a preliminary investigation of Sparse RPD. For suboptimal reward experiments, Tab. I reports the same metrics for PPO, VLAJS (RPD), and VLAJS across tasks and a macro-average, summarizing Fig. 8.
Algorithm 1: Vision-Language-Action Jump-Starting Input: PPO policy πθ , value Vϕ , VLA teacher πVLA , rollout horizon H, discretization length D Initialize reward history buffer; vla_disabled←False; for each PPO iteration k do Compute reward-gain ∆r̄ from history; Set Ncalls and λk via exponential decay; if vla_disabled then set Ncalls = 0, λk = 0; Collect rollout {(st , at , rt )}H t=1 with PPO; Query VLA at Ncalls timesteps; discretize each teacher delta into D targets ãVLA ; t Compute GAE advantages and PPO losses; Update (θ, ϕ) using LPPO + λk Laux ; Update reward history; if recent rewards are monotonically improving and ∆r̄ > 3, set vla_disabled←True;
V. T RAINING S ETUP
Fig. 7: Learning curves for long-horizon tasks. Sparse RPD makes distillation feasible in long-horizon tasks significantly accelerating convergence compared to PPO baselines. Standard RPD is omitted due to prohibitive training time.
A. Simulation Environments, Observations, and Actions All simulations are conducted in ManiSkill manipulation environments [41, 44] (see Fig. 6). The RL policy is a statebased controller: observations include robot proprioception and privileged simulator state (e.g., object poses). Actions are continuous delta end-effector controls (translation and rotation) with a gripper command, executed at a high control frequency. This setting reflects the regime where RL excels at precise closed-loop control but struggles with exploration and long-horizon credit assignment. B. Teacher Models and Sparse Querying We use a pretrained VLA—OpenVLA-best with average s.r. of 40%—as an external teacher that maps RGB observations and a language instruction to a delta action. Due to inference cost, the teacher is queried only a few times per rollout (max. 20% of the total). Each teacher delta is discretized into a short sequence of incremental deltas over D steps, producing sparse guidance targets within the rollout and zero targets elsewhere. Teacher actions are never executed directly in the environment; they are used only in auxiliary losses during training. C. Use Case 1 - Long-Horizon Protocol To isolate the impact of long horizons, we take standard ManiSkill tasks and increase the effective horizon length by 10×. This models realistic scenarios where policies operate at higher frequencies or where tasks require extended action sequences, amplifying the difficulty of exploration and reward propagation. We focus on the feasibility and benefit of persistent sparse guidance and therefore compare: (i) PPO and (ii) Sparse RPD variants (teacher queried sparsely throughout training). We report SRt∗ and AUC under fixed step budgets B (Tab. III).
D. Use Case 2 - Suboptimal Reward Design Protocol To model realistic reward engineering constraints, we modify ManiSkill reward functions into simplified, more intuitive variants that provide weaker shaping (e.g., sparse success signals such as rewarding only object pickup rather than dense shaping). This induces suboptimal credit assignment even for tasks that are otherwise solvable with dense rewards. In this use case, we evaluate whether transient jumpstart guidance and directional regularization improve sample efficiency. We compare: (i) PPO, (ii) VLAJS (RPD), and (iii) VLAJS (ours). Performance is summarized via SRt∗ and AUC with task-specific t∗ and macro-averages (Tab. I). E. Training Details and Reproducibility All methods share the same PPO backbone, network architecture, optimizer settings, and rollout configuration. Guidance-related hyperparameters (maximum calls per rollout, decay rate, discretization length D, and deactivation patience) are fixed across tasks. We evaluate each method with multiple random seeds and report mean performance with bootstrap confidence intervals where applicable. VI. R ESULTS We evaluate our approach under two complementary sources of suboptimal credit assignment: long-horizon tasks and imperfect reward design. These settings isolate different failure modes of on-policy RL and motivate different comparisons. A. Use Case 1 - Long-Horizon Task We first study environments with extended episode horizons, which amplify delayed reward propagation and make exploration particularly challenging for PPO. The objective of
Fig. 8: Learning curves and sample-efficiency comparison for suboptimal reward tasks. VLAJS consistently outperforms PPO and distillation-based baselines— VLAJS (RPD). The initial increase in return is driven by the VLA-based jump-start. Once the VLA guidance is deactivated, the agent transitions to a purely learned policy, which results in a temporary reduction in return while maintaining or improving task success (see the additional material for return plots). Task PickCube-v1 PickCube-v2 PickPlaceCube-v1 PickPlaceCube-v2 LiftPegUpright-v1 LiftPegUpright-v2 LiftPegUpright-v3 PokeCube-v2 PushCube-v2 (OOD) Macro Avg
t∗
PPO (SR t∗ /AUC ↑) (%)
VLAJS (RPD) (SR t∗ /AUC ↑) (%)
VLAJS (SR t∗ /AUC ↑) (%)
2.5M 9.9M 11.4M 37.7M 2.0M 7.3M 17.1M 8.8M 1.9M
88.7 [83.3,94.4] / 72.4 [71.5,73.8] 0.0 / 59.6 [52.8,69.2] 45.4 [0.0,93.3] / 70.3 [67.5,74.0] 0.0 / 0.0 84.8 [63.6,100.0] / 85.9 [84.1,87.3] 16.9 [0.0,36.4] / 76.6 [70.4,80.5] 13.2 [0.0,20.0] / 58.4 [39.8,67.9] 9.9 [0.0,16.7] / 36.9 [25.6,43.3] 49.1 [26.7,61.1] / 96.5 [95.7,96.9]
59.0 [33.3,76.9] / 71.0 [68.7,73.5] 1.1 [0.0,2.9] / 76.7 [56.1,89.3] 86.7 [60.0,100.0] / 72.8 [70.2,76.7] 0.0 / 0.0 0.8 [0.0,1.3] / 81.1 [76.7,85.1] 0.3 [0.0,1.0] / 45.3 [17.8,60.0] 19.2 [0.0,37.5] / 33.6 [0.0,59.7] 75.1 [64.8,92.3] / 54.8 [52.7,57.5] 75.6 [65.2,87.8] / 97.7 [97.5,97.9]
91.7 [90.0,93.3] / 72.9 [72.2,74.0] 95.1 [92.3,100.0] / 88.7 [87.4,90.2] 39.1 [0.0,93.8] / 72.3 [65.1,81.3] 65.9 [0.0,100.0] / 66.0 [50.7,86.2] 80.3 [50.0,100.0] / 87.3 [86.3,88.4] 91.5 [85.7,100.0] / 80.5 [76.6,83.0] 63.3 [0.0,100.0] / 79.1 [66.0,90.3] 81.4 [50.0,100.0] / 60.7 [43.8,70.2] 94.4 [92.7,96.7] / 98.4 [98.2,98.6]
–
34.2 / 61.8
35.3 / 59.3
78.1 / 78.4
TABLE I: Per-task performance reported as Success Rate at t∗ (SR t∗ ) / Area Under the Success Curve (AUC0..B ), both in percent. SR t∗ measures the fraction of evaluation episodes that successfully complete the task within the task-specific time budget t∗ (in environment steps). AUC0..B integrates the success rate over the full episode horizon [0, B], capturing both final performance and learning speed. Bracketed values denote bootstrap 95% confidence intervals over random seeds (n = 6). Bold numbers indicate the best mean performance among the baselines.
this experiment is to assess whether sparse VLA guidance is computationally feasible and beneficial when dense teacher supervision is impractical (RPD [15]). Tab. III reports macro-averaged success rate at a fixed interaction budget (SRt∗ ) and area under the success curve (AUC) across four long-horizon tasks. Across all tasks, Sparse RPD consistently outperforms PPO, often by a large margin in both early success and AUC (see Fig. 7). These results demonstrate that even very sparse auxiliary guidance from a VLA provides strong exploration benefits in long-horizon regimes, while remaining computationally tractable. This establishes sparse guidance as a viable building block, but does not yet address whether the agent can learn beyond the teacher or whether persistent supervision is desirable. B. Use Case 2 - Suboptimal Reward Design We next consider environments with deliberately simplified and sparse reward definitions, designed to reflect realistic reward engineering constraints. In these settings, PPO often fails to learn meaningful behaviors within practical interaction budgets, despite moderate episode horizons. Tab. I compares PPO, VLAJS (RPD), and VLAJS using success rate at t∗ and AUC. Unlike the long-horizon case with sparse persistent guidance, imitation-based loss is no longer sufficient when jump-starting: VLAJS (RPD) provides limited or inconsistent improvements over PPO, and in several tasks fails to meaningfully accelerate learning.
In contrast, VLAJS consistently achieves higher success rates (results in Fig. 8) and larger AUC across all tasks, including an out-of-distribution (OOD) task that the VLA was not trained on (PushCube-v2). By combining sparse guidance with reward-aware deactivation and a directional actionconsistency loss, VLAJS effectively jump-starts learning while preserving the ability to optimize beyond the teacher. Notably, performance often continues to improve after VLA guidance is fully deactivated, indicating that the learned policy is not constrained by the teacher. C. Zero-shot Real-World Deployment Real-world results for randomly sampled objects (see supplementary material for the full list) from a fixed subset of tasks are summarized in Table II, demonstrating zeroshot deployment of the learned policies. Figure 6 shows performance under randomized visual conditions, highlighting robustness enabled by the state-based policy and a visually robust detector. In contrast to a VLA baseline, which fails under strong visual perturbations (e.g., a human hand entering the scene), our policies remain stable and successfully complete the task, as shown in Figure 9. Real-world state estimates are obtained using a pretrained YOLO detector [33]; additional details are provided in the supplementary material. VII. D ISCUSSION Our results show that when and how teacher signals are used matters as much as the teacher itself, revealing complementary
Policy OpenVLA-best VLAJS (zero-shot)
Lift Cube
Pick & Place
Peg Reorientation
47% 70%
40% 80%
– 20%
TABLE II: Real robot deployment success rates (20 trials per task). Algorithm PPO Sparse RPD (OpenVLA) Sparse RPD (OpenVLA-best)
SR t∗ ↑ (%) Wins AUC ↑ (%) Wins 0.0 8.3 40.3
0 1 3
7.2 12.8 37.6
0 0 4
TABLE III: Macro averages across the 4 long-horizon tasks. Sparse RPD (OpenVLA-best) achieves both earlier usability (SR t∗ ) and substantially higher overall sample efficiency (AUC), while PPO fails to achieve meaningful success within the training horizon.
roles of VLA guidance in on-policy RL and enabling zeroshot real-world deployment on a real robot (Tab. II). In long-horizon tasks, where delayed reward propagation hinders exploration, sparse but persistent auxiliary guidance provides a practical and deployable alternative to dense distillation, yielding substantial gains over PPO (Tab. III). We then consider a harder setting with suboptimal reward definitions to directly test whether auxiliary guidance can be made transient, reducing reliance on teacher queries and overall computation. In this regime, we find that while persistent guidance may still be effective, distillation-style action matching–VLAJS (RPD)– is no longer suitable for jump-starting learning, motivating the use of a weak, directional consistency loss that bootstraps task-relevant exploration without over-constraining the policy (Tab. I). As a result, policies trained with VLAJS continue to improve after guidance is fully deactivated (Fig. 8). Finally, our framework is compatible with arbitrary VLA teachers. Due to the high cost of VLA fine-tuning, we evaluate two representative models—OpenVLA and Octo, a diffusion policy VLA. We show that even weak or OOD teachers can accelerate
(a) Comparing different VLA teachers and different levels of their fine-tuning (OpenVLA - 10% s.r., OpenVLA-best - 40% s.r. and Octo - 10% s.r.) when used within VLAJS. Success rates averaged across tasks.
(b) Comparing different VLA camera views for OpenVLA. VLAJS(RPD) CAM and VLAJS CAM indicate the performance of our approach when the VLA camera is in a new position (OOD), versus the baselines.
Fig. 10: Comparisons on VLA teachers.
learning, surprisingly suggesting that VLA performance is not critically important in VLAJS (Fig. 10a). The framework also remains robust to changes in the observation setup (Fig. 10b). VIII. L IMITATIONS While VLAJS improves sample efficiency in difficult creditassignment regimes, it still relies on a VLA teacher that provides at least minimally reliable directional cues. Although VLAJS is relatively insensitive to the teacher’s success rate (see VII), in practice current VLAs often require environmentspecific fine-tuning to be useful, and obtaining such adaptation can be expensive. Using a large VLA during training also introduces nontrivial wall-clock overhead and systems complexity, including GPU memory pressure, inference latency, and engineering effort for external serving. The practical benefit therefore depends on the trade-off between fewer environment interactions and higher per-step compute. Finally, our experiments focus on tabletop manipulation with privileged simulator state for the RL policy and RGB observations for the teacher. Extending VLAJS to fully visionbased RL, force-interactive manipulation, or longer-horizon multi-stage tasks may require additional components (e.g., hierarchy, memory, or state estimation). We also rely on a simple reward-based heuristic to deactivate guidance, which may be brittle in highly stochastic settings; more principled uncertainty- or advantage-based gating could improve robustness. IX. C ONCLUSION
Fig. 9: Policy robustness under external perturbations and clutter. VLAJS maintains performance where VLA-only policies fail.
We presented VLAJS, which improves the sample efficiency of on-policy reinforcement learning for robotic manipulation by leveraging pretrained VLA models as sparse, transient auxiliary guidance. VLAJS combines (i) a reward-based jump-start schedule that reduces and permanently deactivates teacher usage, and (ii) a directional action-consistency loss
that interprets VLA outputs as coarse directional hints rather than strict action targets. Across long-horizon tasks and suboptimal reward designs, our approach accelerates learning and improves final performance relative to PPO and RPD-based sparse distillation baselines, while enabling continued policy improvement after guidance is removed. To the best of our knowledge, we are the first to deploy jump-started policies guided by a VLA on a real robotic system, demonstrating zero-shot transfer to a Franka Panda robot. Future work will focus on reducing teacher overhead by querying VLA guidance only when needed, extending the approach to vision-based RL for more complex manipulation and navigation tasks, and exploring direct real-world finetuning of RL policies with the VLA model. ACKNOWLEDGMENTS The research leading to these results was supported by the Swiss Drone and Robotics Centre of the Department of Defence, Civil Protection and Sport, armasuisse S+T under project n°050-44. This work was also supported by the Swiss AI Initiative and utilized computing resources from the Swiss National Supercomputing Centre (CSCS) on the Alps system under project a144. R EFERENCES [1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in neural information processing systems, 35:28955–28971, 2022. [2] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. [3] Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study. In ICLR 2021Ninth International Conference on Learning Representations, 2021. [4] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, and Niccolo Fusai et al. π0: A vision-language-action flow model for general robot control. Robotics: Science and Systems XXI, 2025. [5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. [6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. RT-2:
Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. [7] Wojciech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszcz, and Max Jaderberg. Distilling policy distillation. In The 22nd international conference on artificial intelligence and statistics, pages 1331–1340. PMLR, 2019. [8] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. [9] Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, and Mac Schwager. Foundation models in robotics: Applications, challenges, and the future. arXiv preprint arXiv:2312.07843, 2023. [10] Sam Green, Craig M Vineyard, and Cetin Kaya Koç. Distillation strategies for proximal policy optimization. arXiv preprint arXiv:1901.08128, 2019. [11] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. [12] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [13] Yafei Hu et al. Toward general-purpose robots via foundation models: A survey & meta-analysis. CoRR, 2023. [14] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. [15] Tobias Jülg, Wolfram Burgard, and Florian Walter. Refined Policy Distillation: From VLA generalists to RL experts. In Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2025. Accepted for publication. [16] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, et al. DROID: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024. [17] Moo Jin Kim, Karl Pertsch, et al. OpenVLA: An opensource vision-language-action model. arXiv preprint arXiv:2406.09246, 2024.
[18] Moo Jin Kim, Chelsea Finn, and Percy Liang. Finetuning vision-language-action models: Optimizing speed and success. Robotics: Science and Systems XXI, 2025. [19] Jens Kober and Jan Peters. Imitation and reinforcement learning. IEEE Robotics & Automation Magazine, 17(2): 55–62, 2010. [20] Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, et al. Simplevla-rl: Scaling vla training via reinforcement learning. arXiv preprint arXiv:2509.09674, 2025. [21] Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, and Dieter Fox. Gpuaccelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning, pages 270–282. PMLR, 2018. [22] Gabriele Libardi, Gianni De Fabritiis, and Sebastian Dittert. Guided exploration with proximal policy optimization using a single demonstration. In International Conference on Machine Learning, pages 6611–6620. PMLR, 2021. [23] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. RDT-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024. [24] Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via humanin-the-loop reinforcement learning. Science Robotics, 10 (105):eads5033, 2025. [25] Yecheng Jason Ma, William Liang, Guanzhi Wang, DeAn Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Humanlevel reward design via coding large language models. In The Twelfth International Conference on Learning Representations, 2024. [26] Angelo Moroncelli, Vishal Soni, Marco Forgione, Dario Piga, Blerina Spahiu, and Loris Roveda. The duality of generative ai and reinforcement learning in robotics: A review. Inf. Fusion, 129:104003, 2024. [27] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303–1315. PMLR, 2022. [28] Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:62244–62269, 2023. [29] Octo Model Team et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024. [30] Open X-Embodiment Collaboration, Abigail O’Neill, Amir Rehman, Agrim Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Ajay Mandlekar, Arhan Jain, et al. Open x-embodiment: Robotic learning datasets and RT-X models. arXiv preprint arXiv:2310.08864, 2023.
[31] Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. Transactions on Machine Learning Research, 2024. [32] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems XIV, 2018. [33] J Redmon. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. [34] Scott Reed et al. A generalist agent. TMLR, 2022. [35] Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014. [36] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings, 2011. [37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [38] Ken Shoemake. Animating rotation with quaternion curves. Proceedings of the 12th annual conference on Computer graphics and interactive techniques, 1985. [39] Giacomo Spigler. Proximal policy distillation. arXiv preprint arXiv:2407.15134, 2024. [40] R.S. Sutton and A.G. Barto. Reinforcement learning: An introduction. IEEE TNN, 9(5):1054–1054, 1998. [41] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-Kai Chan, et al. Maniskill3: Gpu parallelized robot simulation and rendering for generalizable embodied ai. In 7th Robot Learning Workshop: Towards Robots with Human-Level Abilities, 2025. [42] Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pages 34556–34583. PMLR, 2023. [43] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817, 2017. [44] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11097–11107, 2020.
[45] Charles Xu, Qiyang Li, Jianlan Luo, and Sergey Levine. Rldg: Robotic generalist policy distillation via reinforcement learning. Robotics: Science and Systems XXI, 2025. [46] Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. Robotics: Science and Systems XIX, 2023.