arXiv CSopen access

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Shuyi Zhang * 1 2 Yunfan Lou * 2 Hongyang Cheng * 2 Yichen Guo 3 Chuyao Fu 2 Yaoxu Lyu 2 3 Xiaojie Zhang 2 Haoran Li 1 Pengwei Wang 2 Zhongyuan Wang 2 Shanghang ZhangB 2 3

arXiv:2606.26006v1 [cs.RO] 24 Jun 2026

Abstract

(Kim et al., 2024; Zitkovich et al., 2023; Ghosh et al., 2024). These models are typically pre-trained through Imitation Learning (IL), enabling robots to understand complex instructions and map them to physical actions (Black et al., 2024; 2025). However, pure imitation learning (such as Supervised Fine-Tuning, SFT) faces a fundamental bottleneck: the policy’s performance is constrained by the quality of the demonstration data, known as the ”imitation ceiling” (Ross et al., 2011). This is not merely an empirical observation but a fundamental statistical limitation, as imitation learning suffers from error compounding that scales quadratically with task horizon (Rajaraman et al., 2020). Since human demonstration data inevitably contains sub-optimal and even inconsistent actions (Hester et al., 2018; Chen et al., 2021), which standard IL struggles to learn from without additional information like trajectory rankings (Guhur et al., 2023), the performance ceiling of VLA models is firmly capped.

Vision-Language-Action (VLA) models are often constrained by the ’imitation ceiling’ imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing onpolicy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy’s own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.

Reinforcement learning offers a clear path for VLA models to surpass the limitations of demonstration data through online interaction with the environment, thereby breaking the ”imitation ceiling” (Vecerik et al., 2017; Kalashnikov et al., 2018). Indeed, recent works have shown that even a small amount of online fine-tuning can dramatically improve the performance of large-scale, pre-trained IL policies (Li et al., 2025; Hu et al., 2025; Lei et al., 2025). Despite this promise, applying RL directly to the online fine-tuning of VLA models faces severe challenges. The most prominent issue is that RL fine-tuning is notoriously sample inefficient in real-world physical interactions (Haarnoja et al., 2018). This sample inefficiency remains the primary bottleneck, even as recent reinforced fine-tuning (RFT) frameworks have shown impressive gains with just minutes of online interaction (Chen et al., 2025; Team, 2024).

1. Introduction Vision-Language-Action models have demonstrated remarkable potential in executing general-purpose robotic tasks

The root of this inefficiency can be traced to two core difficulties in offline-to-online RL on real robots. (1) Catastrophic ’Initial Unlearning’: When transitioning from offline pre-training (which often uses conservative value estimates) to online fine-tuning, policies suffer from a welldocumented performance collapse (Nakamoto et al., 2023; Nair et al., 2021). This ”initial unlearning” is caused by a Q-value scale mismatch, where the highly underestimated

* Equal contribution B Corresponding author 1 Institute of Automation, Chinese Academy of Sciences, Beijing, China 2 Beijing Academy of Artificial Intelligence, Beijing, China 3 State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China. Correspondence to: Shanghang Zhang <[email protected]>.

Preprint. June 25, 2026.

1

FORCE: Efficient VLA Reinforcement Fine-Tuning Offline Stage

Pre-Calibration Stage

Online Stage

Vision-Language-Action Backbone Q-grad

Q-grad

Q-grad

Cal-ql Update

BC Update

Policy 𝜋(𝑠)

Critic 𝑄(𝑠, 𝑎)

Policy 𝜋(𝑠)

Critic 𝑄(𝑠, 𝑎)

Policy 𝜋(𝑠)

Critic 𝑄(𝑠, 𝑎) Cal-ql Update

Environment

Expert BC Update

TD Update

Distill Update

Offline Demos

Static Rollout Offline Demos

Failure Transition Success Transition

Success Transition

Failure Transition

Equal Sampling Success Transition

Offline Demos

Demo Buffer

Success

Failure

……

Mixed Buffer Expert Buffer

Policy Buffer

Figure 1. Overview of the FORCE framework. Our method employs a three-stage reinforcement fine-tuning pipeline that progressively calibrates value estimation and stabilizes policy improvement: (1) offline Cal-QL pretraining to obtain a conservative and well-grounded critic, (2) mixed-rollout value pre-calibration to bridge offline and online distributions and mitigate O2O drift, and (3) online fine-tuning with balanced expert/policy replay coupled with VGPD, enabling safe and sample-efficient policy refinement beyond the imitation ceiling.

offline Q-function is ”deceived” by new online data, leading to a catastrophic adjustment period and negating the benefits of pre-training. (2) Inefficient Policy Updates: The policy update process is severely hindered by low-quality exploratory data (i.e., the policy generates a large number of useless actions when exploring unfamiliar regions). To address these problems and ensure safe exploration, existing methods (Luo et al., 2025; Chen et al., 2025; Kelly et al., 2019) are often forced to rely on costly and difficult-to-scale Human-in-the-Loop (HiL) interventions (Luo et al., 2025), which treat human attention as a scarce resource.

pipeline for intervention-free offline-to-online RL finetuning starting from pre-trained models (e.g., VLA). 2. Distributional Warm-up: A mechanism to mitigate cold-start covariate shift by aligning Q-function support with policy visitation prior to fine-tuning. 3. Value-Guided Policy Distillation: A theoretically grounded regularized policy improvement operator that filters high-variance exploration to accelerate convergence. 4. SOTA Performance and Efficient Adaptation: Empirical validation on physical robots showing that FORCE achieves near-perfect success rates on contactrich tasks with significantly improved sample efficiency compared to baselines.

To address these challenges, we introduce FORCE, an intervention-free framework that bridges the gap between static datasets and dynamic physical interaction. To mitigate catastrophic unlearning, FORCE incorporates a distributional warm-up phase that calibrates the value function on the policy’s visitation distribution prior to any actor updates. This aligns the critic’s support with the agent’s current behavior, ensuring stable value estimates during the offlineto-online transition. Subsequently, the framework employs a value-guided self-distillation mechanism to govern online fine-tuning. Acting as a dynamic filter, this module projects the policy onto a target distribution derived strictly from high-value transitions, effectively eliminating exploration noise and ensuring monotonic improvement without human supervision.

2. Related Work Vision-Language-Action Models Vision-Language-Action (VLA) Models have recently enabled robots to perceive, reason, and act using multimodal inputs (Brohan et al., 2022; Zitkovich et al., 2023). Early architectures treated manipulation as a discrete token generation problem, like OpenVLA (Kim et al., 2024), often resulting in coarse control resolution. This limitation motivated models to utilize flow matching or diffusion processes to chunk continuous motions into high-dimensional latent flows, such as Octo (Ghosh et al., 2024), Rdt-1b (Liu et al., 2024) and π-series models (Black et al., 2024; 2025; Physical Intelligence, 2025). However, these models predominantly rely on supervised imitation learning from static datasets, which prevents them from correcting execution errors or adapting to dynamic environments. In our work, we focus on fine-tuning high-capacity VLA policies using RL to further enhance their performance and generalization.

We evaluated FORCE on several challenging robotic manipulation tasks both in simulation and the real-world. The experimental results demonstrate that FORCE achieves significant performance gains, reaching nearly 100% success rates on several tasks, all without human intervention. Our main contributions are summarized as follows: 1. FORCE Framework:

A principled three-stage 2

FORCE: Efficient VLA Reinforcement Fine-Tuning

RL Finetuning for VLA Models and Real-world RL Many works recently have explored reinforcement learning algorithm to improve the generalization ability of VLA models. Approaches building on PPO (Schulman et al., 2017) or GRPO (Shao et al., 2024)s with transformer backbones remain largely restricted to simulation (Li et al., 2025; Lu et al., 2025; Liu et al., 2025), as their on-policy nature suffers from poor sample efficiency that prohibits physical deployment. Moreover, these standard formulations are structurally limited to autoregressive backbones, failing to accommodate continuous policies required for dexterous manipulation. Real-world RL training strategies for VLA models generally fall into two paradigms: (1) generalized behavioral cloning (Nair et al., 2021; Peng et al., 2019), exemplified by methods such as PA-RL(Mark et al., 2024), which utilizes value preference feedback to steer the policy towards high-quality behaviors while maintaining the stability of supervised imitation; and (2) direct Q-driven optimization augmented with behavioral cloning regularization (Fujimoto & Gu, 2021), like ConRFT (Chen et al., 2025), is to synergize the stability of behavioral cloning (BC) with the optimality-seeking drive of Q-learning via a joint optimization objective. Our framework, FORCE, adopts this hybrid objective but additionally employs a novel value-guided self-distillation mechanism in the online phase to mitigate instability and maximize sample efficiency.

(VGPD) method. In the following section, we provide a detailed account of the training pipeline and the VGPD module. 3.1. Problem Setup We address the problem of fine-tuning pretrained VLA models for complex manipulation tasks. While SFT on offline expert demonstrations is standard, it suffers from the distributional shift problem: the learned policy π inevitably visits states outside the support of the expert data DE , leading to compounding errors. To mitigate this, we formulate the fine-tuning process as a Markov Decision Process (MDP) M = (S, A, P, R, γ). The objective is to learn a policy π that P∞maximizes the expected discounted return J(π) = Eπ [ t=0 γ t R(st , at )]. We the state-action Pdefine ∞ value function Qπ (s, a) = Eπ [ t=0 γ t R(st , at )|s0 = s, a0 = a]. Our framework addresses two fundamental challenges in VLA fine-tuning: (1) Support Mismatch, where the Qfunction is undefined on the current policy’s visitation distribution dπ during the initial offline-to-online transition; and (2) High-Variance Exploration, where standard online policy gradient methods fail due to the sparsity of rewards and the high dimensionality of the VLA action space. 3.2. Training pipeline

Offline-to-Online Shift Training robotic policies from scratch is notoriously sample-inefficient and often demands extensive, infeasible human-in-the-loop intervention(Luo et al., 2025; Chen et al., 2025). The offline-to-online finetuning paradigm has emerged as a compelling alternative(Yu & Zhang, 2023; Zheng et al., 2023; Song et al., 2022; Kumar et al., 2020a), yet it is plagued by the ”initial unlearning” phenomenon (Kumar et al., 2020b). This critical instability arises when the policy, pre-trained on static data, first encounters the online environment. To bridge this gap, RL100(Lei et al., 2025) interleaves imitation re-learning in the offline setting. Inspired by this, our framework introduces a novel value pre-calibration stage. By strategically augmenting the offline phase with diverse, sub-optimal trajectories, we effectively pre-stabilize the Q-function’s distribution before online fine-tuning begins. This pre-calibration acts as a crucial transition step, significantly mitigating initial unlearning and enabling a much smoother and more stable handoff from offline to online learning.

S1. Offline Reinforcement Fine-Tuning with Expert Demonstrations. To circumvent the prohibitive exploration costs associated with learning from scratch, we initialize the policy πpre via offline RL on expert demonstrations. We adopt Calibrated Q-Learning (Cal-QL) to learn a conservative value function from the static dataset DE . Cal-QL minimizes the temporaldifference (TD) error augmented with a calibration regularizer that constrains value estimates based on the behavior policy µD . This constraint prevents the overestimation of out-of-distribution (OOD) actions, a critical requirement for stable offline-to-online transfer. The objective for the critic parameter θ is: LSQ1 (θ) =E(s,a,s′ )∼DE [(Qθ (s, a) − B π Qθ (s, a))2 ] +α(Es∼DE ,a∼π(·|s) [max(Qθ (s, a), Qµ (s, a))] −Es,a∼DE [Qθ (s, a)]) (1) Here, the first term represents the standard Bellman error, while the second term imposes the calibration constraint controlled by hyperparameter α.

3. Method We articulate a three-stage training paradigm spanning offline reinforcement learning and an offline-to-online bridge phase followed by online fine-tuning in interactive environments. To further bolster sample efficiency during online learning, we introduce a value-guided policy self-distillation

While Cal-QL effectively regularizes the value landscape, optimizing the policy solely against a limited dataset can lead to optimization instability. We therefore employ a behavior-regularized policy optimization objective. This 3

FORCE: Efficient VLA Reinforcement Fine-Tuning

combines a Behavior Cloning (BC) term, which anchors the policy to the demonstrated manifold, with a Q-guided policy gradient term that drives performance maximization. The offline actor objective is defined as: 1 1 Lπ (ϕ)S1 = η LBC,S (ϕ) + λ LQ,S (ϕ) π π

(2)

  1 LBC,S (ϕ) = E(s,a)∼DE − log πϕ (a | s) π

(3)

  1 LQ,S (ϕ) = − Es∼DE , a∼πϕ (·|s) Qθ (s, a) π

(4)

A critical challenge in offline-to-online (O2O) RL is the initial performance collapse caused by Covariate Shift. This phenomenon arises from the distributional mismatch between the offline dataset Doffline , sampled from behavior policy πβ , and the visitation distribution dπϕ of the current policy. Standard offline algorithms constrain πϕ close to πβ , but strictly enforcing this constraint inhibits necessary exploration and correction.

where

In Stage 2, we introduce a Distributional Warm-up. We collect a small batch of on-policy trajectories Dwarm = {(s, a, r, s′ ) ∼ πϕ } and merge them with the offline data: Dmix = Doffline ∪ Dwarm . Crucially, we apply the conservative value constraint (Eq. 1) on this mixed dataset. This explicitly expands the valid support of the calibrated Qfunction from supp(πβ ) to supp(πβ ) ∪ supp(πϕ ). By doing so, we ensure that Q-estimates remain well-defined and lower-bounded on the manifold of states the policy is actually visiting, addressing the support mismatch issue before full online adaptation.

Here, η and λ modulate the trade-off between imitation and maximization. In contemporary VLA models, the action head is often implemented as a multi-step diffusion or flow-matching denoiser, which substantially increases computational cost due to iterative denoising. Distilling a diffusion/flow policy into a one-step policy has been shown to be an effective alternative: it largely preserves the expressiveness of the original policy while avoiding iterative denoising and BPTT problem. In this work, we adopt a consistency policy as the actor network, which maintains the original performance while enabling fast inference and more stable Q-gradient propagation.

Consequently, the critic’s ranking of (s, a) pairs aligns with the on-policy distribution, yielding a robust value function that prevents catastrophic unlearning. The actor is trained with an asymmetric objective:

S2. Mitigating Distributional Shift via On-Policy Warmup.

2 2 LSπ2 (ϕ) = η LBC,S (ϕ) + λ LQ,S (ϕ) π π

(5)

where Offline Demos

  1 E(s,a,y)∼Dmix 1{y=1} · (− log πϕ (a | s)) ρ (6)   2 (7) (ϕ) = − Es∼Dmix , a∼πϕ (·|s) Qθ (s, a) LQ,S π

2 LBC,S (ϕ) = π

Failure Transition Success Transition Success Transition

Policy Buffer

Expert Buffer

𝑠𝑒

Here, ρ is the success rate normalization factor. Sampling from Dmix aligns the actor’s training distribution with the expanded critic support, effectively mitigating covariate shift.

𝑎𝑒

𝑠𝑝

Policy 𝜋(𝑠)

S3. Intervention-Free Online Fine-tuning.

𝑎1

VGPD

In the third stage, we perform online fine-tuning without human intervention. We maintain two replay buffers: an expert buffer DE (offline data + successful online trajectories) and a policy buffer Dπ (all online rollouts). At each update, we sample equally from both to balance retaining competent behaviors and learning from new exploration.

𝑎1

𝑎2 𝑎3

… 𝑎𝑛 𝑎𝑝

𝑎2

𝑄𝑚𝑒𝑎𝑛𝑠

… 𝑎𝑚

The critic minimizes the standard temporal-difference loss over the union of buffers: i LSQ3 (θ) =E(s,a,s′ )∼(DE ∪Dπ ) [(Qθ (s, a) − B π Qθ (s, a))2 (8)

𝑎𝑝

Figure 2. VGPD module in the online phase. VGPD serves as a regularized policy improvement mechanism. We maintain an expert buffer and a policy buffer. For states sampled from the policy buffer, we compute a dynamic value baseline Vref (s) (approximated by Qmean ). The policy is updated via filtered importance sampling, distilling only from actions that show positive advantage over this baseline.

The actor optimization leverages our Value-Guided Policy Self-Distillation (VGPD) mechanism: 3 3 LSπ3 (ϕ) = λ LQ,S (ϕ) + η LDistill,S (ϕ), π π

4

(9)

FORCE: Efficient VLA Reinforcement Fine-Tuning 3 where LQ,S (ϕ) encourages high-value actions as in Eq. 7. π 3 The distillation term LDistill,S (ϕ) is detailed below. π

baseline qmean (s) corresponding to Vref (s) in Eq. 13: K

qmean (s) =

3.3. Value-Guided Policy Self-Distillation (VGPD)

1 X Qθ (s, ak ). K

(13)

k=1

We define an indicator ζ(s) = 1{qmean (s) > Qθ (s, abuf )}. The target distribution is constructed as:

Our proposed VGPD mechanism can be theoretically grounded as an approximate solution to a Regularized Policy Improvement problem with a dynamic data-dependent baseline. Following the formulation in trust-region methods (Peng et al., 2019; Nair et al., 2021), we aim to maximize the expected return subject to a KL-divergence constraint to prevent the policy from deviating too far from the data support. The objective is defined as: h J (π) = Es∼D Ea∼π(·|s) [Qθ (s, a)] i (10) − τ DKL (π(·|s)∥πold (·|s))

µVGPD (· | s) = (1 − ζ(s)) δabuf (·) + ζ(s)

K X

w̃k (s) δak (·),

(14)

k=1

where the weights w̃k (s) implement the exponential energy weighting masked by the advantage filter:
1{Qθ (s, ak ) ≥ qmean (s)} exp Qθ (s, ak )/τ w̃k (s) = PK
. j=1 1{Qθ (s, aj ) ≥ qmean (s)} exp Qθ (s, aj )/τ (15) 3 The final distillation loss is LDistill,S (ϕ) = π E LDistill,D (ϕ) + EµVGPD [− log πϕ ]. π

where πold serves as the reference proposal distribution. In the context of our online self-distillation, πold corresponds to the current policy πϕ . The optimal closedform solution is the energy-based distribution π ∗ (a|s) ∝ πold (a|s) exp (Qθ (s, a)/τ ).

4. Experiments

Since sampling strictly from π ∗ is intractable, we project this optimal policy back into our parameterized policy space Π by minimizing DKL (π ∗ ∥πϕ ), which is equivalent to maximizing the weighted log-likelihood:     Qθ (s, a) max Es∼D,a∼πold exp log πϕ (a|s) (11) ϕ τ Filtered Importance Sampling. VGPD approximates the expectation over πold using a set of K samples {âk }K k=1 ∼ πϕ (·|s). To mitigate variance from the exponential weights, VGPD introduces a Dynamic Advantage Filter. We define a state-dependent baseline as the empirical mean value: PK 1 Vref (s) = K Q k=1 θ (s, âk ). The update rule applies a mask I[Qθ (s, âk ) ≥ Vref (s)], effectively performing a Positive Advantage Truncation where samples with negative advantage are discarded.

Pick Cup

Insert USB

Clean Whiteboard

Stack Cube

Open Drawer

Pick Corn

Figure 3. Real-world Experiment Tasks. We conducted realworld experiments using a single-arm Franka robot equipped with two RealSense cameras that supplied complementary visual feedback: a wrist-view and a side-view.

Proposition 1 (Monotonic Value Improvement). Let πtgt be the empirical target distribution constructed by VGPD. Assuming the critic Qθ is consistent, the expected Q-value under the target distribution is lower-bounded by Vref (s), ensuring that the distillation target always represents a monotonic improvement over the average performance of the current policy rollout.

4.1. Overview of Experiments We designed our empirical evaluation to verify the theoretical claims of the FORCE framework. Specifically, we investigate whether addressing distributional shift and enforcing regularized updates leads to superior fine-tuning performance. Our analysis focuses on three key questions:

Implementation. We implement this theoretical framework as follows. For expert data DE , we treat stored actions as optimal, degenerating to behavior cloning:   E LDistill,D (ϕ) = E(s,a)∼DE − log πϕ (a | s) . (12) π

1. Mitigation of Distributional Shift: Does the Distributional Warm-up stage effectively mitigates the ”coldstart” performance collapse typical of offline-to-online transfer?

For the policy buffer Dπ , given a sample (s, abuf ), we sample K candidates {ak }K k=1 from πϕ (·|s). We compute the 5

FORCE: Efficient VLA Reinforcement Fine-Tuning Success Rate (%) Group

Method

BC

RL

Average(%)

StackCube

PullCube

PushCube

PullCubeTool

PlaceSphere

PickCube

Octo(Ghosh et al., 2024) π0 (Black et al., 2024) π0.5 (Black et al., 2025)

0 60 50

0 87.5 87.5

13 70 100

0 7.5 7.5

0 27.5 15

8.5 2.5 5

3.58 42.5 44.17

Cal-QL(Nakamoto et al., 2023) CQL(Kumar et al., 2020b) PA-RL(Mark et al., 2024) ConRFT (no HIL)(Chen et al., 2025) Ours (Octo) Ours (π 0 )

65.2 0 73.9 82.1 94.2 93.2

84.9 0 81.1 91.7 99 100

96.7 92.2 93.7 95.9 100 100

0 0 0 0 19 36.7

0 0 0 69.8 85.1 97.5

24.2 2.1 52.4 87.2 96.7 94.1

45.2 15.7 50.2 71.1 82.3 86.9

Table 1. Success rates of BC and RL methods on ManiSkill tasks. We report the final converged success rates across six tasks on three random seeds. For all other reinforcement learning (RL) baselines, we consistently utilize the same offline pre-trained Octo and pi0 model as the backbone for post-training fine-tuning.