BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models Zhongxi Chen1* Yifan Han2* Yanming Shao3 Huanming Liu4 Congsheng Xu1 Xiaoyu Chen1 Yao Mu1† Wenzhao Lian1†
arXiv:2605.30226v1 [cs.RO] 28 May 2026
1
Shanghai Jiao Tong University (SJTU) 2 CASIA 3 Shanghai AI Laboratory 4 USTC * Co-first authors
† Corresponding authors
Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for grounding visual-language understanding into real-world robotic manipulation. However, dexterous manipulation remains challenging for VLA policies due to high-dimensional hand control and compounding execution errors, which makes real-world RL post-training essential for bridging the gap between visually grounded action generation and physically reliable dexterous execution. However, high-dimensional dexterous exploration often triggers temporal inconsistency, sample inefficiency and hardware risks in the real world. To address these challenges, we propose BORA, an offline-to-online RL post-training framework designed for real-world dexterous VLA models. In the offline phase, BORA constructs a critic that takes both the VLM’s cognition tokens and action chunks as inputs. This design enables action-conditioned value guidance, allowing the critic to evaluate dexterous hand motions beyond visual context alone. During the subsequent online phase, BORA freezes the VLA base and introduces a lightweight, Human-in-the-Loop (HiL) chunk-wise residual adaptation mechanism to mitigate real-world execution errors and further correct the offline-learned intents within the actual physical environment. By inheriting the offline critic and employing intervention-driven rewards, BORA effectively corrects execution discrepancies and adapts to real-world physical variances while preserving the pretrained policy as a stable prior. Extensive evaluations across five complex real-world dexterous tasks demonstrate that BORA significantly outperforms pure imitation learning and traditional decoupled RL baselines, achieving a 33% absolute increase in average success rate under standard settings and up to a 43% improvement in unseen object generalization.Project page is available at https://chenzhongxi-sjtu.github.io/BORA/. Keywords: Dexterous Manipulation, Post-Training, VLA Models
1
Introduction
While Vision-Language-Action (VLA) models have emerged as a powerful paradigm for generalizable robot control, they face severe challenges in real-world dexterous manipulation due to the high degrees of freedom (DoFs) involved. Furthermore, dexterous tasks exhibit inherent action diversity, where multiple distinct hand poses and configurations can successfully execute the same task. In such highly noisy and multimodal continuous action spaces, Imitation Learning (IL) struggles to extract generalized physical interaction intents. Consequently, an efficient Reinforcement Learn-
Figure 1: BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models. We propose an offline-to-online RL post-training framework for dexterous VLAs, bridging semantic intents and physical dynamics to significantly elevate real-world deployment reliability and task success rates. ing (RL) post-training framework that distills intent comprehension from offline data, coupled with online fine-tuning revision, is critical to pushing the boundaries of real-world dexterous control. Nevertheless, existing VLA post-training schemes [1, 2] encounter two fundamental challenges when applied to dexterous manipulation. The first is credit assignment failure in action generation: mainstream generative action architectures (e.g., Diffusion Models or Flow Matching) typically rely on denoising chains spanning tens or even hundreds of steps. Considering that offline dexterous data often contains redundant or unintentional micro-actions [3], and that the actions are highly diverse, the resulting RL gradients inherently possess significant noise. Backpropagating these gradients through temporal computation graphs spanning thousands of steps leads to severe noise accumulation, rendering the model incapable of effectively extracting high-level intents to guide the underlying action manifold [4]. The second challenge is visual occlusion at the perceptual level. Dexterous manipulation frequently involves severe occlusions; during real-robot RL, traditional decoupled critics are prone to overfitting to background visual artifacts. Rather than evaluating the true physical contact consequences of the actions, these critics provide erroneous guidance to the dexterous VLA models. Beyond these offline training challenges, real-world deployment inevitably introduces execution errors arising from complex friction and contact dynamics, resulting in state deviations and actionoutcome mismatches relative to the offline data distribution. Consequently, the offline-learned intents must be continuously adapted to remain effective under such distribution shifts. While online fine-tuning can mitigate these issues by reinforcing intent guidance, directly updating all parameters of a VLA model is impractical. This is primarily because fragile dexterous hardware limits large-scale data collection, and the resulting online intervention data is often noisy, mixing optimal and suboptimal trajectories. Under traditional offline-to-online RL [5], early-stage distribution shifts and noisy gradients can easily trigger catastrophic feature drift in the pre-trained VLM [6]. Consequently, we need a new framework that prevents catastrophic forgetting during full-parameter updates, regularizes the Q-network within the offline manifold during online estimation, and retains sufficient exploration capability. To address these challenges, we propose BORA (Bridging Offline RL and Online Residual Adaptation), an RL post-training and adaptation framework custom-designed for dexterous VLAs spanning the offline-to-online spectrum. In the offline phase, BORA deploys a Consistency Policy [7, 8] as the action expert to generate continuous action chunks in just 1–3 steps, truncating the computation graph for efficient gradient backpropagation. Concurrently, to mitigate the impact of visual occlusions and prevent the critic from overfitting to background artifacts, we design a critic that explicitly fuses the continuous action chunks with the VLM’s cognition tokens. This ensures that the value estimation is fundamentally grounded in actual physical interactions rather than spurious visual features. During the subsequent online phase, BORA freezes the offline-trained VLA base to
2
prevent catastrophic feature drift, introducing a lightweight, Human-in-the-Loop (HiL) chunk-wise residual adaptation mechanism. By directly inheriting the offline critic to ensure stable value estimation, this mechanism guides the residual actor to safely extract corrective priors from human intervention data, robustly compensating for real-world execution deviations and intention mismatches. In summary, our main contributions are threefold: • Action-Conditioned Critic for Dexterous Manipulation: we design a critic architecture that fuses continuous action chunks with the VLM’s cognition tokens. This enables precise, actionconditioned value guidance evaluated on physical execution consequences rather than visual context alone. • Lightweight Residual Online Adaptation: We introduce an HiL chunk-wise residual RL mechanism for the real-world deployment phase. By freezing the VLA base, inheriting the offline critic, and leveraging intervention-driven rewards, we achieve safe and sample-efficient online adaptation. This design effectively corrects execution errors while preventing catastrophic forgetting of the pre-trained representation. • The BORA Unified Framework: We present a comprehensive offline-to-online RL post-training framework tailored for dexterous VLAs. By utilizing a consistency policy to resolve generative credit assignment and employing progressive optimization to bridge offline intent learning with online physical execution, BORA significantly enhances real-world deployment robustness, achieving a 33% absolute increase in average success rate and up to a 43% improvement in unseen object generalization.
2
Related Work
Vision-Language-Action Models for Dexterous Manipulation Vision-Language-Action (VLA) models have emerged as a powerful paradigm for robot manipulation, typically adapting visuallanguage representations from pretrained VLMs to robot control through action heads. While earlier VLA models use autoregressive tokenized actions [9, 10, 11], recent action heads have evolved to diffusion- or flow-based continuous policies [12, 13], and further toward consistency-style few-step generation for efficient closed-loop control [14, 15]. This progress has also motivated dexterous VLA or vision-language-grasp models, as dexterous hands offer substantially greater capability than parallel grippers for fine-grained and contact-rich manipulation [16, 17, 18]. Recent works such as Being-H0 [19] and VITRA [12] study pretraining from human data, while Being-H0.5 [16] explores cross-embodiment transfer across dexterous hands. However, existing dexterous VLA models still suffer from limited real-world success rates, largely due to the high-dimensional hand-arm action space, complex contact dynamics, and noisy dexterous manipulation data, which make offline posttraining unstable and inefficient. Post-Training and Adaptation of Vision-Language-Action Policies Post-training has become an essential step for adapting pretrained VLA policies to downstream robots and task domains. Unlike VLM post-training, where preference optimization can often be conducted on static data, robotic post-training must optimize closed-loop physical interaction under sparse rewards, distribution shifts, and limited real-world samples. Existing methods can be broadly divided into offline and online paradigms. Offline RL and imitation-based fine-tuning can reuse collected robot data and are thus scalable, but they often bring limited gains due to suboptimal demonstrations, action multimodality, and the mismatch between offline trajectories and deployment dynamics [20]. Online RL directly optimizes task success in the target environment and has shown strong potential for robot manipulation, yet it remains costly and unstable, especially for high-DoFs dexterous hands where exploration is sample-inefficient and potentially unsafe [21, 22, 23]. Recent offline-to-online RL methods attempt to combine the scalability of offline data with the adaptivity of online interaction [24, 25]. However, most existing post-training and residual adaptation methods are developed for robot arms or parallel grippers, where the action space is relatively low-dimensional and contact uncertainty is easier to handle. For dexterous VLA policies, high-dimensional hand-arm actions, severe hand-object occlusions, and noisy correction data substantially destabilize value estimation and
3
policy updates. As a result, directly applying conventional offline or online RL post-training methods often leads to inefficient training and limited success-rate improvement in real-world dexterous manipulation.
Figure 2: Illustration of the BORA framework. BORA bridges offline token-action reinforcement learning and online residual adaptation for real-world dexterous VLA policies. In the offline stage, the VLM encoder and action expert produce shared VLM cognition tokens and action chunks, jointly evaluated by an integrated critic Qϕ with semantic anchoring and IQL-based policy optimization. The right panel details the action-conditioned critic, which predicts per-step Q and V values from VLM tokens, action chunks, and position embeddings, with shifted value bootstrap for credit propagation within each chunk. In the online stage, the offline-trained base VLA is frozen, while a lightweight residual chunk actor πres is trained with inherited critic feedback, sparse task rewards, and human-in-the-loop intervention signals. The final action chunk is obtained by residual composition, Afinal = Abase + λAres [26], enabling low-cost physical adaptation and improved dexterous execution.
3
Method
In this work, we propose BORA, a comprehensive offline-to-online reinforcement learning framework tailored for Vision-Language-Action (VLA) models in dexterous manipulation. The core philosophy of BORA is to establish a two-stage adaptation pipeline: it first extracts foundational manipulation skills via offline RL, and subsequently compensates for real-world execution errors through Human-in-the-Loop (HiL) online residual RL adaptation on the physical robot. 3.1
Offline RL with Action-Conditioned Critic and Consistency Policy
Existing generative action architectures typically rely on iterative denoising procedures, which inadvertently exacerbate credit assignment failures in continuous control [27]. This bottleneck is particularly detrimental in dexterous manipulation: given the exceptionally high DoFs, the action manifold is inherently multimodal and riddled with redundant, task-irrelevant micro-actions. Backpropagating RL signals through hundreds of temporal denoising steps over such a noisy space inevitably leads to severe noise accumulation. Consequently, the foundational VLM fails to receive effective gradients, disrupting the alignment between high-level task representations and low-level action execution. To address this, we parameterize the actor as a consistency policy, enabling high-fidelity action chunk generation within just 1–3 denoising steps [7, 8]. By truncating the computation graph, this formulation ensures that informative gradients flow efficiently back into the VLM, thereby facilitating stable offline optimization. Furthermore, to avoid relying on privileged information for reward
4
design [28], we adopt a minimalist sparse reward formulation consisting of a terminal success reward and a step-wise time penalty. However, evaluating high-dimensional action chunks under such sparse rewards—especially when compounded by severe visual occlusions—renders decoupled critics highly susceptible to overfitting to spurious background artifacts rather than actual physical interactions. Action-Conditioned Critic. To counteract this visual overfitting and handle the high dimensionality of chunk-level evaluation, we condition our critic on VLM semantic tokens zt = ΦVLM (st ) and decompose the chunk-level evaluation into atomic, step-wise scores. Specifically, our critic k outputs k-dimensional value vectors defined as Qϕ (st , At ) = [qϕ (zt , at+i , i)]k−1 i=0 ∈ R and k−1 k Vψ (st ) = [vψ (zt , i)]i=0 ∈ R , where the subscript i denotes the relative position encoded by a learnable positional embedding. To enforce temporal consistency across this vectorized horizon, credit is propagated backward within the chunk via a shifted value bootstrap: r + γ(1 − dt+i ) vψ (zt , i + 1), i<k−1 yt,i = t+i (1) rt+k−1 + γ(1 − dt+k−1 ) vψ (zt+1 , 0), i = k − 1 The critic parameters ϕ and ψ are optimized via a masked Bellman residual together with an IQLstyle expectile value objective, which extracts conservative value targets from sub-optimal offline data (see Appendix A for details). Conservative Policy Improvement. Instead of using isolated per-step residuals, we smooth the optimization signal using an intra-chunk Action-level GAE Recursion: Ât,i = δt,i + γλ(1 − dt+i )Ât,i+1 with Ât,k = 0, where δt,i = qϕ (zt , at+i , i) − vψ (zt , i) represents the instantaneous advantage. The consistency-based policy πθ defines a Gaussian action distribution at each atomic |st ) , we step. Simulating policy updates with the importance sampling ratio rt,i (θ) = πθπθ (a(at+i t+i |st ) old optimize the actor using a validity-masked clipped PPO surrogate objective: " LPPO (θ) = −E(st ,At )∼D
k−1 1 X mt,i min rt,i Ât,i , clip(rt,i , 1 − ϵ, 1 + ϵ)Ât,i |Mt | i=0
# (2)
P where mt,i is a validity mask and |Mt | = i mt,i is the effective chunk length. To ensure policy conservatism and prevent out-of-distribution deviation, the full objective combines PPO with a Behavior Cloning (BC) regularization on the normalized action chunk: Lactor = λppo LPPO +λbc LBC . 3.2
Bridging the Deployment Gap: HiL Residual Chunk Adaptation
While offline RL equips the VLA model with robust behavioral priors, direct physical deployment inevitably exposes the policy to compounding execution drift induced by complex contact dynamics. Simultaneously, full-parameter online fine-tuning is computationally prohibitive and highly susceptible to catastrophic feature drift in pre-trained vision-language representations and value collapse [6]. To achieve rapid, sample-efficient real-world adaptation while preserving the foundation model’s priors, BORA freezes the offline-trained VLA base and introduces a lightweight Residual Chunk Actor πres (parameterized by an MLP). Operating natively in the continuous temporal domain, πres generates compensations at the action chunk level. Given the proprioceptive state sprop , the base action chunk Abase , and the VLM tokens zVLM , the final deployed composite chunk is formulated as Afinal = Abase + λres · πres (sprop , Abase , zVLM ), where λres is a scaling coefficient that restricts the intervention magnitude to guarantee the smoothness of the action manifold. During online adaptation, BORA couples Critic Inheritance with an Intervention-Driven RLPD pipeline to stabilize optimization dynamics. The offline-trained critic Qϕ initializes the online value function, preserving a pre-shaped value landscape. To enforce monotonic policy improvement, the residual actor optimizes a conservative value guidance objective (Appendix B.3) that penalizes residual actions underperforming the frozen VLA base. Concurrently, human corrections are integrated 5
via an RLPD protocol that maintains a 1:1 sampling ratio between the static offline dataset and an online buffer dynamically enriched by human interventions [25]. This joint optimization over mixed data streams regularizes the model against non-stationary online trajectories, enabling precise calibration of the residual action advantage. Integrating with the stabilization method, we then design a Human-in-the-Loop (HiL) intervention system following DexHiL [2]. Specifically, BORA allows human operators to take control to recover the task whenever the model encounters failures (e.g., incomplete closing of a finger or wrist misplacement). To mitigate kinematic discontinuities during the transition from autonomous execution to human takeover, a linear interpolation mechanism for action smoothing is employed, ensuring the temporal consistency of the hand poses. Finally, to supply the core optimization signals for the Intervention-Driven RLPD pipeline, we engineer an asymmetric Intervention-Driven Reward function. If the policy drifts OOD and triggers an intervention, an instant penalty rint is imposed. Conversely, upon completion of a human corrective action, a positive recovery reward rrec is granted. This asymmetric reward mechanism directly guides the RLPD value updates, compelling the residual policy to aggressively penalize high-risk states while efficiently learning from high-quality recovery trajectories, ultimately closing the physical adaptation loop with minimal real-world interactions.
4
Experiments
In this section, we design a series of experiments to investigate our proposed BORA by asking the following 3 questions: RQ1:
RQ2: RQ3: 4.1
Can BORA improve offline RL post-training for dexterous VLA policies by providing action-conditioned value guidance under high-dimensional action generation and visual occlusion? How can the proposed online adaptation bridge the real-world deployment gap in a sampleefficient way? How does the inherited action-conditioned critic support residual policy learning during online fine-tuning?
Experimental Setup
As detailed in Appendix C, We deployed BORA on a real-world dexterous manipulation platform composed of a Franka arm equipped with a 12-DoF dexterous hand. We evaluate the models across the following five tasks: Pick the plush toy, Pick and Place, Open the box, Pull the tissue, Press the button. For each task, we conduct 20 trials under the standard configuration and another 20 trials involving novel, unseen objects to assess BORA’s generalization capabilities. To systematically ablate our framework, we compare BORA against four baseline settings. Built upon the pre-trained VITRA VLM, these baselines all adopt the VLA architecture: • VITRA (Fine-tuned Base) [12]: A baseline fine-tuned on top of the vanilla pre-trained VITRA weights, constructed via a multimodal VLM backbone paired with a Diffusion Action Expert. • CP Base (consistency policy): An Imitation Learning baseline utilizing the consistency policy as an action expert without any reinforcement learning fine-tuning. • Decoupled-Critic Baseline [29]: An offline RL baseline with a separately trained critic that does not jointly condition on the VLM cognition tokens and generated action chunks. • BORA-Offline (Ours): Our offline RL framework with an action-conditioned critic that evaluates VLM cognition tokens together with continuous action chunks. Then, our setting, BORA-Full (Ours), is the complete pipeline, featuring BORA-Offline followed by HiL online residual chunk adaptation through 2 rounds of real-world physical reinforcement learning. The online training process incorporating these components is formally summarized in Algorithm 1.
6
Table 1: Real-World Dexterous Manipulation Success Rates under the Standard Configuration. Pure Imitation Learning (IL)
Task Pick the plush toy Pick and Place Open the box Pull the tissue Press the button
Reinforcement Learning (RL) Fine-tuning
VITRA (Diffusion)
CP Base
CP + Decoupled Critic
BORA-Offline
BORA-Full (Ours)
14/20 (70%) 9/20 (45%) 12/20 (60%) 9/20 (45%) 10/20 (50%)
12/20 (60%) 10/20 (50%) 11/20 (55%) 7/20 (35%) 13/20 (65%)
10/20 (50%) 8/20 (40%) 5/20 (25%) 10/20 (50%) 12/20 (60%)
17/20 (85%) 14/20 (70%) 11/20 (55%) 12/20 (60%) 13/20 (65%)
20/20 (100%) 18/20 (90%) 15/20 (75%) 16/20 (80%) 17/20 (85%)
Table 2: Real-World Dexterous Manipulation Success Rates under the Object-Unseen Setting. Pure Imitation Learning (IL)
Task Pick the plush toy Pick and Place Open the box Pull the tissue Press the button
4.2
Reinforcement Learning (RL) Fine-tuning
VITRA (Diffusion)
CP Base
CP + Decoupled Critic
BORA-Offline
BORA-Full (Ours)
9/20 (45%) 6/20 (30%) 2/20 (10%) 7/20 (35%) 9/20 (45%)
7/20 (35%) 6/20 (30%) 3/20 (15%) 3/20 (15%) 8/20 (40%)
11/20 (55%) 8/20 (40%) 1/20 (5%) 10/20 (50%) 11/20 (55%)
15/20 (75%) 9/20 (45%) 7/20 (35%) 10/20 (50%) 11/20 (55%)
17/20 (85%) 14/20 (70%) 10/20 (50%) 14/20 (70%) 15/20 (75%)
Main Results and Analysis
We report the main quantitative results in Tables 1 and 2, and provide a visual summary in Fig. 3. The tables present exact success counts over 20 real-world trials for each task and setting, while the bar plots summarize the relative trends across baselines. Fig. 5 in the Appendix further presents representative real-robot results across the five evaluation tasks. Offline Intent Alignment with Action-Conditioned Value Guidance (RQ1): As shown in Tables 1 and 2, CP Base reaches 53.0% average success under the standard setting but drops to 27.0% under object-unseen evaluation, indicating limited generalization in high-dimensional and multimodal dexterous action spaces. By routing informative policy gradients back into the foundational VLM within minimal steps, BORA-Offline extracts high-fidelity physical intents, improving the averages to 67.0% and 52.0%, respectively, yielding a 14-point gain in the standard setting and a 25point gain under object-unseen evaluation. This indicates that, although standard diffusion architectures hinder direct offline RL fine-tuning due to severe gradient vanishing across lengthy denoising iterations, our truncated consistency policy formulation effectively addresses this bottleneck. Crucially, this architectural synergy successfully immunizes the model against the visual occlusion. Traditional critic-decoupled architectures (Decoupled-Critic baseline) overfit to raw pixel values and spurious background features, propagating erroneous gradients back to the VLM backbone. In the standard setting, the Decoupled-Critic baseline falls below CP Base on average. Under objectunseen evaluation, although it improves over CP Base on some tasks, it collapses on Open-theBox, indicating unstable value guidance under severe occlusion.Conversely, by conditioning our Integrated Critic directly on the VLM’s semantic tokens (zt ), BORA-Offline binds value updates strictly to the multimodal latent space. This is qualitatively supported by the t-SNE visualization in Appendix Fig. 8, where BORA-Offline stays closer to the SFT/BC manifold after offline RL, suggesting better preservation of the policy’s action-representation structure. Appendix Fig. 9 and Fig. 10 further show that BORA assigns stronger saliency to the dexterous hand and task-relevant contact regions, while the decoupled critic exhibits more scattered responses over the table, background, and other non-contact regions. Sample-Efficient Online Physical Adaptation (RQ2): Despite the intent awareness established offline, real-world deployment introduces covariate shifts, execution errors, and intricate contact friction, causing a performance drop on object-unseen tasks. To address this offline-to-online gap, the online residual chunk adaptation in BORA-Full provides a sample-efficient solution. Specifically, BORA-Full exhibits rapid performance convergence within the initial two online reinforcement learning rounds, beyond which further improvement becomes minimal. In practice, the human
7
Standard
Object-Unseen
VITRA (Diffusion)
CP Base
CP + Decoupled Critic
BORA-Offline
VITRA (Diffusion)
CP Base
CP + Decoupled Critic
BORA-Offline
BORA-Full (Ours)
100
100
90
90
80
80
Success Rate (%)
Success Rate (%)
BORA-Full (Ours)
70 60 50 40 30 20
70 60 50 40 30 20
10
10
0
0
Pick Toy
Pick & Place
Open Box
Pull Tissue
Press Button
Pick Toy
Pick & Place
Open Box
Pull Tissue
Press Button
Figure 3: Visual summary of real-world dexterous manipulation results. Success rates are reported across five tasks under the standard and object-unseen settings. BORA-Offline improves over imitation and decoupled-critic baselines, while BORA-Full further improves performance through online residual adaptation. operator only needs to intervene 1–2 times per task, which demands a minimal time investment of approximately 20% of the online trajectory execution. Benefiting from this residual training and the human-intervention value guidance mechanism, BORA-Full improves the overall average success rate to 86.0% in the Standard setting, and elevates it from 52.0% to 70.0% under the Object-Unseen configuration. This efficient adaptation achieves the best performance among the evaluated baselines on dexterous execution tasks, while preventing representation drift in the frozen pre-trained VLM foundation. Mechanistic Analysis of Critic Inheritance (RQ3): To understand the underlying mechanism behind the rapid online adaptation, we analyze the behaviors of the inherited value function, with additional value-profile visualization provided in Appendix Fig. 11. The empirical value profiles reveal that the offline-trained action-conditioned critic, when combined with the online asymmetric intervention-driven reward, provides stable and responsive value estimation. Throughout autonomous execution, the inherited critic exhibits clear discriminative capability: it maintains high value confidence along successful trajectories, while its Q-value estimation remains consistently low along failed trajectories. Importantly, this discriminative capability is preserved even in visually similar states, such as when the multi-fingered hand grasps an object under severe occlusion. Although the raw camera pixels appear nearly identical to those in the decoupled baseline, the actionconditioned critic inherited from the offline stage allows the critic to evaluate state-action pairs at a structural semantic level. Consequently, the inherited critic mitigates superficial visual ambiguities, penalizes high-risk execution states, and guides the lightweight residual policy πres to incorporate human corrective priors with few physical interactions.
5
Limitations
While BORA demonstrates compelling performance, it presents two main limitations. First, the framework relies on visuo-proprioceptive inputs and lacks dense tactile feedback. Integrating highfidelity tactile arrays into VLM tokens could further tighten the perception-action loop under severe visual occlusion. Second, our physical evaluation is constrained to a single arm-hand topology. Verifying BORA’s cross-embodiment generalization across diverse multi-fingered hands with varying kinematics and degrees of freedom remains an important direction for future scaling.
6
Conclusion
In this paper, we presented BORA, an offline-to-online RL post-training framework customdesigned for real-world dexterous VLA models. By seamlessly bridging offline alignment with efficient online fine-tuning, BORA addresses key challenges in dexterous VLA post-training, including critic overfitting to visual artifacts, real-world execution discrepancies, and catastrophic feature drift. Extensive evaluations across five complex real-world dexterous tasks demonstrate that BORA significantly outperforms imitation learning and decoupled RL baselines, achieving a 33% absolute increase in average success rate and up to 43% improvement in unseen object generaliza8
tion. Moving forward, we aim to extend this framework toward high-precision dexterous skills and investigate its scalability to structurally complex, long-horizon operational tasks.
9
Appendix A
Detailed Critic Formulation
Following the chunk-wise Bellman backup defined in the main text, the Q-function parameters ϕ and value-function parameters ψ are updated in the same offline training loop, but with decoupled objectives. For notation brevity, we write qt,i ≜ qϕ (zt , at+i , i) and vt,i ≜ vψ (zt , i). The critic objectives are " # k−1 2 1 X LQ (ϕ) = E(st ,At )∼D mt,i qt,i − yt,i , (3) |Mt | i=0 " # k−1 1 X mt,i τ − 1[qt,i − vt,i < 0] (qt,i − vt,i )2 , LV (ψ) = E(st ,At )∼D (4) |Mt | i=0 where mt,i ∈ {0, 1} is a validity mask that keeps all atomic steps up to and including the first terminal step within the chunk. The parameter τ ∈ (0.5, 1.0) is the expectile asymmetry coefficient, following the value-learning design of IQL. In implementation, the Q estimate used in LV is treated as a stop-gradient target.
B
Implementation Details and Training Safeguards
To stabilize offline post-training of the VLM-based actor, we use a decoupled actor-critic optimization scheme. The critic is trained with the Bellman and expectile objectives above, while the actor is optimized separately using a PPO-clipped likelihood-ratio objective on demonstration actions together with behavior cloning regularization. In the current implementation, no direct pathwise critic gradient ∇a Qϕ is backpropagated into the actor. B.1
Advantage Gating Mechanism
To reduce harmful policy updates under noisy offline value estimates, we optionally gate the actor update using the mean validity-masked chunk advantage: k−1 1 X mt,i Ât,i . Āt = |Mt | i=0
(5)
In practice, this gating score may be computed from either raw or normalized advantages. If Āt ≤ α, the actor update for the current batch is skipped; the critic update is still executed. B.2
Two-Stage Post-Training Pipeline
The offline post-training procedure is divided into two stages: 1. Stage 1: BC Warm-up. The actor is optimized only with the BC objective, while the critic is trained simultaneously from offline transitions. PPO guidance is disabled during this phase. 2. Stage 2: BC+PPO Guidance. After the warm-up stage, the actor is optimized with the combined objective Lactor = λppo LPPO + λbc LBC , using critic-derived action-level advantages as weighting signals. The critic remains decoupled from the actor in the backward pass. B.3
Online Residual Alignment via Conservative Value Guidance
In the online adaptation phase, to guarantee that the lightweight residual actor πres strictly improves execution quality over the frozen VLA base abase t+i , we introduce the conservative improvement hinge 10
loss:
"
# X 1 Limprove = E P mt,i max 0, Qϕ (zt , abase . t+i ) + δ − Qϕ (zt , ât+i ) i mt,i i
(6)
This formulation can be theoretically justified as a relaxation of a constrained optimization problem. Ideally, we seek a residual policy that maximizes the expected return while satisfying a local policy improvement bound: max E [Qϕ (zt , ât+i )] πres
s.t.
Qϕ (zt , ât+i ) ≥ Qϕ (zt , abase t+i ) + δ
(7)
By constructing the Lagrangian function with local multipliers αt+i ≥ 0, we have: L(πres , α) = Qϕ (zt , ât+i ) − αt+i Qϕ (zt , abase t+i ) + δ − Qϕ (zt , ât+i )
(8)
In practical deep reinforcement learning, to avoid gradient instability and maintain manifold smoothness, we parameterize αt+i as a constant coefficient λimprove and convert the constraint into a onesided hinge penalty. When the composite action ât+i fails to outperform the base action by the margin δ, Limprove provides explicit pathwise repulsive gradients through the learned critic Qϕ , regularizing the residual adaptation within a safe optimization boundary.
11
B.4
Online Residual Adaptation Algorithm
The complete online training process incorporating history-aware base behavior and interventiondriven RLPD is detailed in Algorithm 1. Algorithm 1 BORA Online Residual Adaptation (Multi-Round Iteration) old Require: Frozen offline VLA base πbase , Prior round residual actor πres (None if Round 1), Inherited critic Qϕ and value network Vψ , Trainable residual actor πres , Datasets Doffline , Donline , Normalization parameters (µs , σs , µa , σa ). Ensure: Optimized residual chunk actor πres . 1: for episode = 1 to M do 2: Receive initial environment state s0 , set t ← 0 3: while not terminal do 4: Normalize state: snorm ← (st −µs )/(σs +ϵ) and extract VLM tokens zt ← ΦVLM (snorm ) t t 5: Generate prior-policy actions: AVLA ← πbase (zt ) old 6: if πres is None then ▷ Round 1: Use pure VLA as base VLA 7: Set current base action chunk: Abase ← A norm 8: else ▷ Subsequent Rounds: VLA + prior residual as base VLA old norm 9: Set current base action chunk: Abase + λold πres (st , AVLA , zt ) norm ← A 10: end if , Abase 11: Compute new residual chunk: Ares ← πres (snorm norm , zt ) t 12: Linear schedule factor: αt ← αs + (αe − αs ) min(t/Tα , 1.0) 13: Map composite chunk to physical space: Afinal ← (Abase norm + αt Ares ) · σa + µa 14: if Human Intervention Triggered then 15: Aexec ← (1 − β)Afinal + βAhuman , set rint , is int ← True 16: else 17: Set Aexec ← Afinal , rint ← 0, is int ← False 18: end if 19: Step environment: st+1 , renv , done, info ← step(Aexec ) 20: Compute reward: rt ← renv + rint + (rrec if [is int and info.recovered] else 0) 21: Store transition (st , Aexec , rt , st+1 , done) into Donline and update st ← st+1 , t ← t + 1 22: if |Donline | ≥ Nstart then 23: Sample mixed batch (1 : 1 ratio): B ← Bonline ∼ Donline ∪ Boffline ∼ Doffline 24: Compute bootstrap targets yt+i and update Critic Qϕ via LQ 25: Decay imitation weight λBC (t) and update Actor πres via Lactor 26: end if 27: end while 28: end for
12
C
Experimental Details
C.1
Robot Hardware and Teleoperation Setup
Fig. 4 shows the robot hardware and teleoperation setup used for BORA evaluation. The robotic platform consists of a Franka robotic arm equipped with a DexHand021 dexterous hand, together with Intel RealSense D435 RGB-D cameras for visual observation. For human-in-the-loop online adaptation, we use a wearable teleoperation device to provide corrective demonstrations when the autonomous policy enters failure-prone states. The device captures human hand motions and maps them to dexterous hand commands, allowing the operator to recover the task while preserving the physical interaction context. These corrective trajectories are then incorporated into the online residual adaptation stage as intervention data.
(a) Robot hardware setup.
(b) Wearable teleoperation device.
Figure 4: Robot hardware and teleoperation setup. (a) The real-world platform consists of a Franka robotic arm, a DexHand021 dexterous hand, and Intel RealSense D435 RGB-D cameras for multiview visual observation. Red boxes indicate the main sensing and manipulation components used during evaluation. (b) The wearable teleoperation device captures human hand motions and provides corrective demonstrations for human-in-the-loop online residual adaptation. C.2
Rollout Visualizations
Fig. 5 provides representative real-world rollout sequences across the five dexterous manipulation tasks. Each row shows temporally ordered frames from one successful execution, covering grasping, placing, pulling, opening, and pressing behaviors. These qualitative examples illustrate that the learned policy can produce coherent action chunks for contact-rich dexterous manipulation under severe hand-object occlusions. C.3
Object and Task Configurations
Fig. 6 illustrates the object configurations used in our real-world experiments. The object-seen setting contains the object instances used during offline data collection, whereas the object-unseen setting introduces novel instances at evaluation time. Both settings share the same task semantics and robot platform, allowing us to isolate the effect of object-level distribution shift. Fig. 7 further visualizes the task-level configuration variations used during evaluation. In addition to changing object identities, we also vary physical configurations such as the box opening angle, object orientation, object pose, and object position. These variations are designed to evaluate whether the policy remains robust under realistic layout changes, occlusion patterns, and contact-rich manipulation conditions. 13
Figure 5: Representative real-world rollout visualizations. From top to bottom, the rows show successful executions of Pick-the-Plush-Toy, Pick-and-Place, Pull-the-Tissue, Open-the-Box, and Press-the-Button. Each row contains temporally ordered frames from a single rollout, demonstrating coherent dexterous execution across diverse contact-rich manipulation tasks.
(a) Object-seen offline data.
(b) Evaluation with seen and unseen objects.
Figure 6: Seen and unseen object configurations. The object-seen setting corresponds to the offline data distribution, while the object-unseen setting includes novel object instances for evaluating generalization under cluttered and occluded real-world dexterous manipulation.
14
A. Pick the Plush Toy
B. Pull the Tissue
A1
A2
A4
A5
pose shift
A3
B1
A6
B2
pose shift orientation shift
A7
A8
B3
A9
C. Open the Box
opening angle shift C1
C2
C3
C4
C6
C7
C8
D3
D4
orientation shift C5
D. Pick and Place pose shift
position shift D1
D5
D2
D6
D7
Standard: A1 / B1 / C1 / D1 Unseen Object: A4–A9 / C4–C8 / D3–D7
Figure 7: Task-level evaluation variations. We visualize representative standard and object-unseen configurations, together with pose, orientation, position, and box opening-angle shifts used to evaluate real-world robustness.
15
D
Supplementary Visualizations
D.1
Additional Representation Visualization
Fig. 8 provides the full t-SNE visualization of projected action representations used in the main analysis. The left two panels correspond to the External Critic variant on the Pick-the-Plush-Toy task, while the right two panels correspond to BORA-Offline on the Pick-and-Place task. For each setting, we visualize the relation between the SFT/BC manifold anchor and the representation obtained after the same number of offline RL training epochs.
Figure 8: t-SNE visualization of representation drift under offline RL. The left two panels show External Critic on Pick-the-Plush-Toy, and the right two panels show BORA-Offline on Pick-andPlace. The orange star denotes the SFT/BC manifold anchor, the black cross denotes the RL-updated representation, and the green box marks the local BC support. Red density regions indicate criticpreferred high-value areas, while gray regions denote lower-value regions.
16
D.2
V-Critic Saliency Visualization
Fig. 9 and Fig. 10 visualize gradient-based saliency maps of the value critic on the Open-the-Box and Pull-the-Tissue tasks, respectively. The saliency is computed with respect to the visual patch features used by each critic, indicating which visual regions contribute most to the current value estimate. Across both tasks, BORA’s integrated token-action critic places comparatively stronger emphasis on the dexterous hand and task-relevant interaction regions, while the decoupled critic exhibits more scattered responses on object-irrelevant regions. This supports our claim that tokenaction conditioning encourages value estimation to rely more on task-relevant interaction cues under severe occlusion.
(a) Decoupled critic.
(b) BORA’s integrated token-action critic.
Figure 9: V-critic saliency maps on the Open-the-Box task. Each subfigure shows RGB observations from the left, front, and right camera views in the top row, with the corresponding gradientbased saliency maps in the bottom row. Warmer colors indicate larger influence on the current value estimate.
17
(a) Decoupled critic.
(b) BORA’s integrated token-action critic.
Figure 10: V-critic saliency maps on the Pull-the-Tissue task. Each subfigure shows RGB observations from the left, front, and right camera views in the top row, with the corresponding gradientbased saliency maps in the bottom row. Warmer colors indicate larger influence on the current value estimate.
18
D.3
Value-Function Visualization
Fig. 11 visualizes the inherited critic’s value profiles during online execution. The successful episode receives consistently high value estimates, whereas the failure episode remains low and further drops near the failure state. This indicates that the inherited critic provides discriminative value guidance for online residual adaptation.
Figure 11: Value-function visualization of the inherited critic. The left panel shows a successful episode, where the critic maintains high value estimates throughout execution. The right panel shows a failure episode, where the value estimates remain lower and further drop near the failure state. Dashed lines connect representative frames to their corresponding value estimates.
19
E
Hyperparameters and Implementation Details
The complete hyperparameter configurations for both the Phase 1 (Offline VLM pre-training) and Phase 2 (Online residual adaptation) are summarized in Table 3. Table 3: Hyperparameter Configurations for Offline and Online Phases. Category
Hyperparameter
Value
Hardware Setup
Phase 1 (Offline) Cluster Phase 2 (Online) Workstation
8× NVIDIA H100 GPUs 1× NVIDIA RTX 4090 GPU
Phase 1: Offline VLM BC & PPO
Training Mode / Precision Base Optimizer / Weight Decay Total Offline Updates Batch Size per GPU Action Chunk Size (k) Actor VLM Learning Rate Actor Consistency Learning Rate PPO Optimization Start Step Behavior Cloning Coefficient (λBC ) PPO Guidance Coefficient Advantage Normalization Clip OPE Gate / Margin (δOPE )
Full Fine-tuning / BF16 AdamW / 1 × 10−4 70, 000 8 32 1 × 10−5 5 × 10−5 40, 000 1.0 0.01 2.0 Enabled / 0.0
Phase 2: Online Residual RLPD
Base Optimizer / Weight Decay Max Online Updates Batch Size Action Chunk Size (k) Discount Factor (γ) Actor / Critic Learning Rate Offline-to-Online Batch Ratio Target Network Soft Update (τtarget ) Initial / End BC Lambda Critic Regularization (λQ ) Residual Blend Factor (λt ) Actor Anchor Lambda Conservative Improvement Margin (δ) Conservative Improvement Lambda Actor Residual L2 Regularization
AdamW / 1 × 10−4 15, 000 64 − 128 32 0.99 1 × 10−4 / 1 × 10−4 1:1 0.005 2.0 / 1.0 (Decay over 10, 000 steps) 0.25 0.2 → 0.75 (Warmup over 4, 000 steps) 0.03 0.0 0.35 0.0002
20
F
Dataset Statistics and Task Metrics
Table 4 summarizes the task definitions, success criteria, and data statistics used in our real-world experiments. For each task, we report the number of offline trajectories used for offline token-action RL, together with the number of online intervention trajectories collected per adaptation round for BORA-Full. The success criteria are defined at the task level and are used consistently for both the standard and object-unseen evaluations. Table 4: Task Definition, Success Criteria, and Dataset Statistics for Offline and Online Phases. Offline Trajectories
Online Trajectories per Iteration (Ours)
Pick Plush Toy
60
10
Lift the plush toy completely off the tabletop surface and maintain a stable grasp.
Pick & Place
100
10
Successfully pick up the package, transfer it, and place it entirely inside the basket.
Open Box
100
10
Fully rotate and flip open one side of the box lid to a completely open state.
Pull Tissue
60
10
Securely pinch and extract a single whole sheet of tissue completely out of the box container.
Press Button
60
10
Actuate the button by applying stable downward force to fully depress it without finger slippage.
Task
21
Success Criteria
G
Failure Mode and Quantitative Analysis
Table 5 provides a qualitative summary of the typical failure modes observed across tasks, evaluation settings, and method families. The comparison shows how failure patterns change from pure imitation learning to offline RL baselines and BORA-Full. Table 5: Qualitative Analysis of Typical Failure Modes Across Settings. Task
Setting
Pure IL Baselines
Offline RL Baselines
BORA-Full (Ours)
Pick Plush Toy
Standard
Minor rotation; unconfident grasp closure. Severe loose grasp under shape variations.
Minor object rotation during approach.
—
Minor object rotation (same as Standard).
—
Basket collision; placement hesitation. Grasp failure driven by novel shape/color.
Basket collision during transfer. Basket collision (same as Standard).
—
Insufficient lifting; incomplete lid opening. Insufficient lid opening (same as Standard).
Missed lid contact; localized hesitation.
—
Missed lid contact under layout shifts.
Collision with inner contents.
Insecure pinch, leading to tear/slip. Missed pinch due to box dimension shifts.
Off-center pinching; partial extraction. Off-center pinching (same as Standard).
Finger trembling; loose grip.
Insufficient downward pressing force. Insufficient force (same as Standard).
Off-center contact, causing finger to slip. Off-center contact and slipping.
—
Unseen
Pick & Place
Standard Unseen
Open Box
Standard Unseen
Pull Tissue
Standard Unseen
Press Button
Standard Unseen
—
Finger trembling; loose grip.
—
References [1] T.-Y. Xiang, A.-Q. Jin, X.-H. Zhou, M.-J. Gui, X.-L. Xie, S.-Q. Liu, S.-Y. Wang, S.-B. Duan, F.-C. Xie, W.-K. Wang, et al. Parallels between vla model post-training and human motor learning: Progress, challenges, and trends. arXiv preprint arXiv:2506.20966, 2025. [2] Y. Han, Z. Chen, Y. Zhao, C. Xu, Y. Shao, Y. Peng, Y. Mu, and W. Lian. Dexhil: A human-inthe-loop framework for vision-language-action model post-training in dexterous manipulation. arXiv preprint arXiv:2603.09121, 2026. [3] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martı́n-Martı́n. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298, 2021. [4] D. Zhang, J. Sun, C. Hu, X. Wu, Z. Yuan, R. Zhou, F. Shen, and Q. Zhou. Pure vision language action (vla) models: A comprehensive survey. arXiv preprint arXiv:2509.19012, 2025. [5] P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, ∗ J. Darpinian, K. Dhabalia, J. DiCarlo, et al. π0.6 : A vla that learns from experience. arXiv preprint arXiv:2511.14759, 2025. [6] Z. Zhou, A. Peng, Q. Li, S. Levine, and A. Kumar. Efficient online reinforcement learning finetuning need not retain offline data. In International Conference on Learning Representations, volume 2025, pages 32343–32368, 2025. 22
[7] A. Prasad, K. Lin, J. Wu, L. Zhou, and J. Bohg. Consistency policy: Accelerated visuomotor policies via consistency distillation. arXiv preprint arXiv:2405.07503, 2024. [8] G. Lu, Z. Gao, T. Chen, W. Dai, Z. Wang, W. Ding, and Y. Tang. Manicm: Real-time 3d diffusion policy via consistency model for robotic manipulation. arXiv preprint arXiv:2406.01586, 2024. [9] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. [10] O. Mees, D. Ghosh, K. Pertsch, K. Black, H. R. Walke, S. Dasari, J. Hejna, T. Kreiman, C. Xu, J. Luo, et al. Octo: An open-source generalist robot policy. In First Workshop on VisionLanguage Models for Navigation and Manipulation at ICRA 2024, 2024. [11] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. [12] Q. Li, Y. Deng, Y. Liang, L. Luo, L. Zhou, C. Yao, L. Zeng, Z. Feng, H. Liang, S. Xu, et al. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. arXiv preprint arXiv:2510.21571, 2025. [13] M. Chen, Y. Wang, Z. Li, H. Bharadhwaj, Y. Chen, C. Qin, Z. Kou, Y. Tian, E. Whitmire, R. Sodhi, et al. Flowing from reasoning to motion: Learning 3d hand trajectory prediction from egocentric human interaction videos. arXiv preprint arXiv:2512.16907, 2025. [14] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. [15] Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. π0.5 : A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. [16] H. Luo, Y. Wang, W. Zhang, S. Zheng, Z. Xi, C. Xu, H. Xu, H. Yuan, C. Zhang, Y. Wang, et al. Being-h0. 5: Scaling human-centric robot learning for cross-embodiment generalization. arXiv preprint arXiv:2601.12993, 2026. [17] Y. Zhong, X. Huang, R. Li, C. Zhang, Z. Chen, T. Guan, F. Zeng, K. N. Lui, Y. Ye, Y. Liang, et al. Dexgraspvla: A vision-language-action framework towards general dexterous grasping. arXiv preprint arXiv:2502.20900, 2025. [18] J. He, D. Li, X. Yu, Z. Qi, W. Zhang, J. Chen, Z. Zhang, Z. Zhang, L. Yi, and H. Wang. Dexvlg: Dexterous vision-language-grasp model at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14248–14258, 2025. [19] H. Luo, Y. Feng, W. Zhang, S. Zheng, Y. Wang, H. Yuan, J. Liu, C. Xu, Q. Jin, and Z. Lu. Being-h0: vision-language-action pretraining from large-scale human videos. arXiv preprint arXiv:2507.15597, 2025. [20] D. Huang, Z. Fang, T. Zhang, Y. Li, L. Zhao, and C. Xia. Co-rft: Efficient fine-tuning of vision-language-action models through chunked offline reinforcement learning. arXiv preprint arXiv:2508.02219, 2025. [21] Y. Chen, S. Tian, S. Liu, Y. Zhou, H. Li, and D. Zhao. Conrft: A reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.05450, 2025.
23
[22] C. Xu, J. T. Springenberg, M. Equi, A. Amin, A. Esmail, S. Levine, and L. Ke. Rl token: Bootstrapping online rl with vision-language-action models. arXiv preprint arXiv:2604.23073, 2026. [23] K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, Q. Zhang, Z. Yu, G. Fan, et al. πrl: Online rl fine-tuning for flow-based vision-language-action models. arXiv preprint arXiv:2510.25889, 2025. [24] M. Nakamoto, S. Zhai, A. Singh, M. Sobol Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:62244–62269, 2023. [25] P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577–1594. PMLR, 2023. [26] E. Su, T. Westenbroek, A. Nagabandi, and A. Gupta. Rfs: Reinforcement learning with residual flow steering for dexterous manipulation. In The Fourteenth International Conference on Learning Representations, 2026. [27] H. Ma, T. Chen, K. Wang, N. Li, and B. Dai. Efficient online reinforcement learning for diffusion policy. arXiv preprint arXiv:2502.00361, 2025. [28] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018. [29] J. Luo, Z. Hu, C. Xu, Y. L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine. Serl: A software suite for sample-efficient robotic reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16961–16969. IEEE, 2024.
24