UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning

UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning

arXiv:2610.03620v1 [cs.LG] 2 Oct 2026

Yudong Lin1∗ , Haoyuan Deng1∗ , Zhuoxuan Yuan1 , Zaijia Yang1 , Yuanjiang Xue1 , and Ziwei Wang1† Abstract— Online reinforcement learning (RL) enables robot policies to improve through physical interaction, but the assistance they require changes as their competence evolves. Existing intervention strategies based on offline estimates or fixed decision rules can therefore become mismatched to the current policy. To address this, we propose UniIntervene++, an adaptive intervention agent that learns to allocate control between autonomous execution and heterogeneous assisted behaviors during online RL. Specifically, UniIntervene++ first formulates the evolving RL policy, trajectory correction, and a task-structured CodePolicy as Options in a unified semi-Markov decision process and learns their relative values online. Building on this, competence-adaptive intervention periodically probes the RL policy through unassisted execution, keeping control allocation responsive to its evolving capability. Finally, coupled experience learning allows assisted behaviors to improve the RL policy, whose evolving outcomes in turn reshape future intervention decisions. In this way, UniIntervene++ jointly determines when to intervene, how to intervene, and when to return control as the RL policy improves. Across five real-world manipulation tasks, UniIntervene++ achieves an average success rate of 89.67%, outperforming all baselines by at least 6 percentage points, while reducing human intervention to 0.77%, a relative reduction of at least 94.6% from the best baseline. Code is available in our GitHub repository.

I. I NTRODUCTION Online reinforcement learning (RL) enables robot manipulation policies to improve through physical interaction and adapt to contact dynamics, perception errors, and task variation [1], [2], [3]. However, real-world online RL remains costly: rewards are often sparse, exploration can be unsafe or unproductive, and failed rollouts consume robot time [4], [5], [6]. Human-in-the-loop RL (HiL-RL), such as HIL-SERL, mitigates these issues through timely human corrections during online learning [7]. Yet current HiL-RL systems require a skilled operator to continuously monitor training and repeatedly allocate control between autonomous execution and assisted behavior, limiting real-world scalability [8], [9], [10]. Recent work has begun to automate this role. AutoSERL reuses a single demonstration to guide exploration and recover from predefined failure regions [11], while UniIntervene detects sustained value degradation and invokes a goalconditioned recovery policy learned offline [12]. These methods provide useful automated assistance, but the assistance required by the RL policy changes as its competence evolves during online RL. Early in learning, intervention can prevent ∗ Equal contribution 1 School of Electrical and Electronic Engineering, Nanyang Technolog-

ical University, Singapore † Corresponding author: ziwei.wang@ntu.edu.sg

Online training

RL policy competence increases

UNIINTERVENE++

Select the next Option

Adaptive control allocation from online outcomes

Early Training

Late Training

Option occupancy within one rollout

Option occupancy within one rollout

Coupled experience learning

Adaptive RL Probe

Policy improves

B-step RL-only window

Autonomous evidence

Q-value update Adjust B

Trajectory Correction

CodePolicy

RL policy

Fig. 1. Competence-adaptive control allocation in UniIntervene++. Illustrative Ring Transfer rollouts show a shift from early failures with substantial use of Trajectory Correction and CodePolicy to later autonomous success with predominantly RL policy control. Colored bars indicate Option occupancy within each rollout (teal: RL policy; orange: Trajectory Correction; purple: CodePolicy). Assisted experience improves the RL policy, while Adaptive RL Probe reserves unassisted windows to assess its evolving competence and guide subsequent allocation.

repetitive failures and provide useful experience; later, excessive assistance can limit the autonomous execution needed to further improve the RL policy [13], [14], [15]. Intervention strategies based on offline estimates or fixed decision rules can therefore become mismatched to the current policy. What is missing is an intervention agent that learns to allocate control between autonomous execution and heterogeneous assisted behaviors as the task policy improves online. This requires learning their relative values online while keeping control allocation responsive to the policy’s evolving competence. We therefore ask: Can an intervention agent learn a unified control-allocation policy that adapts as the RL policy improves during online RL? To address this gap, we propose UniIntervene++, an adaptive intervention agent that learns to allocate control between autonomous execution and heterogeneous assisted behaviors during online RL. Specifically, to compare autonomous execution and different forms of assistance within a unified decision process, Unified Intervention Policy Learning places the evolving RL policy, demonstration-guided trajectory correction, and a task-structured CodePolicy in a shared decision space and learns their relative values online.

To prevent initially effective assistance from obscuring the RL policy’s evolving competence, Competence-Adaptive Intervention, as illustrated in Fig. 1, periodically probes the RL policy through unassisted execution, providing direct evidence of its current competence and enabling control allocation to progressively shift from assistance-heavy to autonomy-heavy execution. Finally, to ensure that intervention also contributes to improving the RL policy, Coupled Experience Learning uses assisted experience to train the task policy, whose evolving outcomes in turn reshape future intervention decisions. In this way, UniIntervene++ jointly determines when to intervene, how to intervene, and when to return control as the RL policy improves, without requiring a separate offline intervention-learning stage. Across five sparse-reward real-world manipulation tasks, UniIntervene++ achieves an average success rate of 89.67%, exceeding the best baseline by 6 percentage points, while reducing the human intervention rate to 0.77%. Its control allocation progressively shifts toward the RL policy as autonomous competence improves. The main contributions are: • We formulate online intervention as a unified controlallocation problem, placing autonomous execution and heterogeneous assisted behaviors in a shared decision space and learning their relative values online. • We develop competence-adaptive intervention with coupled experience learning, which keeps control allocation responsive to the RL policy’s evolving competence while using assisted experience to improve the policy and reshape subsequent intervention decisions. • Extensive real-world experiments across five manipulation tasks show that UniIntervene++ achieves 89.67% average success, outperforming the best baseline by 6 percentage points, while reducing the human intervention rate to 0.77%. II. R ELATED W ORK A. Human-in-the-Loop Reinforcement Learning Human guidance can make physical policy learning tractable by providing targeted interventions when autonomous exploration becomes unsafe or unproductive. Interactive imitation learning established this data-collection principle: DAgger queries expert actions on learner-induced states [16]; HG-DAgger gives the supervisor control of the takeover gate, whereas ThriftyDAgger formalizes intervention using novelty and risk [17], [18]. For robot deployment, Sirius treats takeovers as both corrective labels and evidence of human trust, then reweights the collected experience for subsequent policy updates [19]. These supervised approaches mitigate covariate shift but primarily optimize imitation of corrective behavior. Reward-driven methods use intervention data beyond behavioral cloning. RLIF turns the onset of an intervention into a negative reward and applies off-policy RL, relaxing the assumption that corrective actions are near-optimal [20]. In parallel, RLPD and SERL provide sample-efficient offpolicy learning that mixes prior data with online interaction

[21], [22]. HIL-SERL combines this foundation with human takeovers, storing corrective transitions in both demonstration and online replay buffers [7]. E2HiL instead selects replay samples by their estimated effect on policy entropy to use costly interventions more efficiently [23]. Together, these systems show that intervention can shape where data are collected, which corrective actions are executed, and how the resulting transitions influence learning. They nevertheless generally keep the operator responsible for monitoring, intervention timing, and corrective action generation. Our work retains human takeover as an external safety backstop, but transfers intervention timing, corrective action selection, and control return to an intervention agent that learns online together with the RL policy. B. Automated Intervention and Recovery Robust execution is often obtained by pairing a nominal controller with a localized fallback. Plan patching learns a model-free policy from demonstrations only where an approximate planner’s transition model fails [24], while manifold-return methods guide a policy from outof-distribution observations back toward its demonstration distribution [25]. In safe RL, Recovery RL and LS3 learn recovery zones or safe sets from offline experience to constrain subsequent exploration [26], [27]. MetaReSkill instead discovers failure modes in simulation and uses model-predictive meta-reasoning to allocate a limited training budget among failure-specific recovery policies [28]. Contact-mode transition models have also been used to select among nominal skills and heuristically generated recoveries [29]. RecoveryChaining learns a local model-free recovery policy in a hybrid action space containing primitive actions and terminal nominal Options [30]. AutoSERL automates training-time recovery from a single successful demonstration [11]. UniIntervene is the closest value-based intervention formulation: it learns intervention behavior offline and uses value degradation to trigger recovery [12]. Its learned values and supporting modules are fixed before the target online learning process, however, and it does not formulate control return as a decision that must adapt to the improving RL policy. UniIntervene++ moves the intervention learner into the online loop and learns entry, corrective behavior, and exit from the same evolving value comparison. III. M ETHOD A. Problem Formulation and Framework Overview We formulate the robotic manipulation task as a Markov decision process (MDP) M = (S, A, ρ, P, r, γ), where st ∈ S denotes the state, ot the policy observation available at st , at ∈ A the action, rt the task reward, and γ the discount factor. We build on HIL-SERL [7], retaining its off-policy RL objective and update rule unchanged. Let πϕ (at | ot ) denote the RL policy trained within this framework. Without altering the RL policy objective, we introduce UniIntervene++, a unified online framework that learns an adaptive intervention agent to dynamically allocate control among the evolving RL policy and two complementary sources of automated

Fig. 2. Overview of UniIntervene++. At each Option boundary, the availability-masked intervention agent selects among the evolving RL policy, trajectory correction, and CodePolicy. Robot-control transitions update the RL policy, while Option-completion transitions update the intervention agent. Adaptive RL Probe preserves direct evidence of the RL policy’s current competence.

assistance, thereby determining when to intervene, which form of assistance to invoke, and when to return control to the policy as it improves. As illustrated in Fig. 2, at each high-level decision boundary, the agent observes the current scene and recent execution outcomes, selects one of the currently available execution modes, and allows the selected controller to interact with the environment until its termination condition is reached. The resulting robot-control transitions are used to continue training the RL policy, while the result of each completed execution mode is used to update the high-level intervention agent. We represent the available modes of execution as the Option set

or an external safety override. Because perception and primitive execution are fallible, completion of a CodePolicy plan does not necessarily imply task success. Selecting ωRL is an abstention decision that leaves control with the RL policy, whereas selecting ωTC or ωCP invokes the corresponding automated intervention. At high-level decision n, an Option begins at robot-control time tn , executes for kn control steps, and terminates at tn+1 = tn + kn . The resulting controlallocation problem is therefore a semi-Markov decision process (SMDP) [32], rather than a step-wise controller switch. Human takeover remains an external safety override and is not treated as a fourth Option.

Ω = {ωRL , ωTC , ωCP },

B. Unified Intervention Policy Learning We implement the intervention agent as a Double-DQN scheduler with parameters ψ. The agent observes the current scene and recent consequences of its control choices through the scheduler state   xn = fϕ̄ (otn ), hn−H:n−1 , vn , (2)

(1)

where each ω ∈ Ω is a closed-loop, temporally extended controller with controller-specific initiation and termination rules [31]: ωRL executes the current policy πϕ for a bounded horizon; ωTC feedback-controls the TCP toward a reachable point on a successful demonstration trajectory; and ωCP composes task-conditioned motion, gripper, and hold primitives using estimated object poses. The latter two Options are available only when a valid connection target or feasible plan exists, respectively. Each Option terminates upon controllerspecific completion, budget exhaustion, episode termination,

where fϕ̄ is a frozen snapshot of the RL policy’s visual encoder, hn−H:n−1 contains the most recent H completedOption summaries, and vn indicates which history entries are valid. Freezing the encoder prevents the scheduler input from drifting with RL policy updates. Each history entry

records the selected Option, its duration and scheduler reward, end-effector translation and rotation, path length, and force statistics. Not every Option is available in every state. We define an availability mask mn ∈ {0, 1}|Ω| , where mn (ω) = 1 indicates that Option ω satisfies its initiation condition. The RL Option is normally available; trajectory correction requires a valid local reference target, and CodePolicy requires valid perceptual inputs and a feasible task-conditioned plan. The mask is used for exploration, greedy selection, and bootstrap target computation. The greedy selection is ωn = arg max Qψ (xn , ω),

(3)

∆ℓRL ≥ η∆ℓE (B),

ω: mn (ω)=1

where Qψ is the scheduler action-value function; an availability-aware ϵ-greedy rule is used during training. After Option ωn executes for kn robot-control steps, the scheduler receives Rnsched =

kX n −1

γ j rtn +j − cTC 1{ωn = ωTC }

j=0

  kn ,1 , − cCP 1{ωn = ωCP } − cdur min Knorm (4) where 1{·} is the indicator function, cTC and cCP penalize calls to the two assisted Options, cdur penalizes prolonged execution, and Knorm normalizes Option duration. These costs discourage unnecessary or prolonged assistance while preserving task return. We learn Qψ using Double DQN [33]. At the successor state, the online network selects ωn∗ =

arg max Qψ (xn+1 , ω),

(5)

ω: mn+1 (ω)=1

and the target network Qψ̄ evaluates it using the SMDP target yn = Rnsched + bn γ kn Qψ̄ (xn+1 , ωn∗ ),

standard SMDP transition with the reward in Eq. (4), and is stored in scheduler replay. Thus, one probe window can produce multiple consecutive RL Option transitions. Because the executed Option is ωRL , its scheduler reward contains no trajectory-correction or CodePolicy cost. The probe adapts B from measured execution evidence rather than equating assisted episode success with the RL policy’s competence. Let ∆ℓE (B) be the demonstration’s reference arc-length increment over the probed segment. We project the measured TCP poses monotonically onto the reference path and denote their projected progress by ∆ℓRL . For a segment containing sufficient reference motion, a completed probe passes when

(6)

where ψ̄ denotes the target-network parameters and bn = 0 for a terminal transition and bn = 1 otherwise. Each completed Option produces an SMDP transition stored in scheduler replay for the Double-DQN update. The durationdependent discount γ kn places Options of different durations on a common temporal scale. We optimize the scheduler with the Huber temporal-difference loss. C. Competence-Adaptive Intervention A scheduler can initially favor a reliable automated controller and thereby deprive the improving RL policy of both execution opportunities and current competence estimates. As an internal behavior-selection mechanism of the intervention agent, Adaptive RL Probe periodically reserves a contiguous unassisted window of B robot-control steps. At each Option boundary within this window, it temporarily overrides the learned selection rule in Eq. (3) and forces ωRL . The probe introduces neither an additional scheduler action nor a separate reward or value function. Each forced RL Option retains the ordinary Option horizon, produces a

dmax ≤ dallow ,

(7)

where η is the required relative progress and dmax is the maximum deviation from the reference path. Projected arclength progress is used instead of unconstrained end-effector travel, so motion in an irrelevant direction cannot satisfy the probe. When the reference segment is nearly stationary, the probe instead evaluates a tighter local-deviation condition and does not require artificial motion. Task completion also passes the current probe. Human takeover, safety interruption, excessive path deviation, or stalled progress can end it early. After a configured number of consecutive passes, the unassisted window is increased; deviation or safety failures shorten it, while other failures clear the pass streak. Probe-specific progress, deviation, and pass/fail signals are used only to update this curriculum and are not added to Rnsched . A probe pass is not counted as an independent task success. D. Coupled Experience Learning The scheduler and RL learner are coupled through experience while retaining separate networks, optimizers, and replay buffers. Let ξt = (ot , at , rt , ot+1 ) denote a robotcontrol transition and zt its execution source. All transitions enter the online replay buffer, while transitions generated by automated assistance or human override are additionally routed to the demonstration buffer: Donline ← Donline ∪ {ξt }, (8) Ddemo ← Ddemo ∪ {ξt }, zt ∈ {TC, CP, Human}. Here, Ddemo is initialized with the task demonstrations, and TC and CP denote Trajectory Correction and CodePolicy, respectively. The unchanged HIL-SERL learner draws mixed off-policy batches from Donline and Ddemo . Thus, assisted experience directly contributes to RL policy learning without introducing an additional behavior-cloning or policydistillation objective. The RL policy and scheduler are not coupled through shared gradients. Instead, they form a coupled data loop across robot-control and Option timescales: automated Options generate useful experience, off-policy replay improves the RL policy, and the resulting policy produces new execution outcomes that update the scheduler’s value estimates. Adaptive RL Probe preserves direct evidence of autonomous competence, keeping this loop responsive as the RL policy evolves.

TABLE I R EAL - WORLD MANIPULATION RESULTS ACROSS FIVE TASKS . B OLD AND UNDERLINED VALUES INDICATE THE BEST AND SECOND - BEST RESULTS , RESPECTIVELY; TIES SHARE THE SAME FORMATTING . H IGHER SR AND LOWER HIR ARE BETTER .

Pick and Place Method

Contact rich

Deformable Object

Average

Ring Transfer

Tube Insertion

USB Insertion

Wipe Whiteboard

Towel Folding

SR↑

HIR↓

SR↑

HIR↓

SR↑

HIR↓

SR↑

HIR↓

SR↑

HIR↓

SR↑

HIR↓

HIL-SERL AutoSERL UniIntervene

83.33 53.33 85.00

39.15 17.07 25.32

85.00 80.00 73.33

39.65 0.10 15.80

90.00 88.33 90.00

51.68 3.91 9.90

85.00 80.00 90.00

63.11 16.80 10.90

75.00 66.67 80.00

76.57 33.55 32.30

83.67 73.67 83.67

54.03 14.29 18.84

UniIntervene++

88.33

0.30

91.67

2.58

96.67

0.96

86.67

0.01

85.00

0.00

89.67

0.77

Fig. 3.

UR7e setup for five manipulation tasks.

IV. E XPERIMENTS We conduct real-world robotic manipulation experiments to answer three key questions: (1) Can UniIntervene++ improve task success while reducing human interventions compared with HiL-RL methods (Sec. IV-B)? (2) Does UniIntervene++ adapt its intervention timing, Option selection, and control handoff as the RL policy evolves, enabling progressive autonomy (Sec. IV-B)? (3) Does UniIntervene++ outperform fixed-prior stochastic selection, and how does each source of automated assistance contribute to the final RL policy (Sec. IV-C)? A. Experiment Setup Real-world benchmark. As shown in Fig. 3, all experiments are conducted on a single UR7e robotic arm equipped with a Robotiq gripper. Human corrective actions and external safety overrides are issued through a 3Dconnexion SpaceMouse. We evaluate UniIntervene++ on Ring Transfer, Tube Insertion, USB Insertion, Wipe Whiteboard, and Towel Folding. The benchmark spans precision manipulation, contact-rich insertion and interaction, and deformable-

object manipulation. Collectively, these tasks test constrained extraction and placement, precise alignment, contact-aware correction, sustained tool–surface interaction, and manipulation under deformable-object dynamics. Baselines. We compare UniIntervene++ with three baselines for human-in-the-loop manipulation. HIL-SERL [7] uses an operator to provide corrective actions during online SAC/RLPD training. AutoSERL [11] derives trajectory guidance and automated recovery from a single demonstration. UniIntervene [12] is the offline, value-based counterpart: it detects sustained value degradation and invokes a memory-guided, goal-conditioned recovery policy using modules learned before online deployment. All methods use the same perception inputs, task rewards, action limits, prior data, and robot-interaction budget wherever applicable. UniIntervene++ retains the same low-level learner, but learns the intervention agent jointly with it online. Human override remains available as an external safety layer. Evaluation metrics. We report success rate (SR) and human intervention rate (HIR). SR is measured over three evaluation runs of 20 episodes per task after training. HIR counts human-controlled steps. In addition, we evaluate the RL policy at multiple checkpoints throughout training and report the duration-weighted control share of each Option, including the RL policy, CodePolicy, and Trajectory Correction. These diagnostics assess whether changes in control allocation track the evolving performance of the RL policy and whether control progressively shifts from assisted Options to autonomous execution. B. Experiments Result and Analysis Task performance. As shown in Table I, UniIntervene++ achieves the highest average success rate (SR), reaching 89.67% across the five real-world tasks under matched training budgets. This is 6 percentage points higher than the best baseline average of 83.67%, achieved by HIL-SERL and UniIntervene. At the task level, UniIntervene++ outperforms all baselines on Ring Transfer, Tube Insertion, USB Insertion, and Towel Folding by 3.33, 6.67, 6.67, and 5.00 percentage points respectively. On Wipe Whiteboard, UniIntervene++ achieves 86.67%, which is 3.33 percentage points below UniIntervene while remaining higher than HIL-SERL and

Total Intervention IR ↓

Human Intervention HIR ↓

Autonomy Share ↑

Ring

Ring

Ring

80

80

60 40

Towel

20

Wipe

USB

HIL-SERL

100

60 40

Tube Towel

20

Wipe

AutoSERL

75 50

Tube Towel

USB

UniIntervene

Tube

25

Wipe

USB

UniIntervene++

Fig. 5. Task-wise intervention decomposition across five real-world manipulation tasks. IR denotes the fraction of training steps executed under automated or human intervention, while HIR denotes the subset executed under human control. The automated-intervention share is (IR − HIR)/IR. Arrows indicate the preferred direction.

Fig. 4. TCP trajectories stored in the replay buffers during the USB Insertion task. Blue and red curves denote successful and failed trajectories, respectively; green circles and black crosses indicate the start and end points. Numbers in parentheses report the number of trajectories in each outcome category.

AutoSERL. These results demonstrate that the proposed adaptive intervention agent improves policy learning across precision, contact-rich, and deformable-object manipulation tasks. Across the five tasks, UniIntervene++ reaches the reported performance using 18,000–82,000 robot-interaction steps, corresponding to 29–131 minutes of effective training time. USB Insertion requires the shortest run, whereas Ring Transfer requires the longest. This places the reported task performance within practical real-robot interaction budgets. Using USB Insertion as a representative example, Fig. 4 visualizes the replay-buffer trajectories collected by the four methods. Rather than suppressing autonomous exploration, UniIntervene++ interleaves RL execution with structured corrective behaviors. The resulting replay buffer therefore combines exploratory experience from the evolving RL policy with spatially consistent successful trajectories generated through automated assistance. This balance may provide more informative training data for off-policy learning while preserving the policy’s opportunity to explore. Human and overall intervention. As shown in Fig. 5, UniIntervene++ achieves the lowest average human intervention rate (HIR) in the five tasks, only at 0.77%, compared to 54.03% for HIL-SERL, 14.29% for AutoSERL, and 18.84% for UniIntervene. Relative to AutoSERL, which has the lowest HIR among the baselines, UniIntervene++ reduces human-controlled steps by 13.52 percentage points or 94.6% relative. Although its average overall intervention rate (IR) is 56.04%, 95.62–100% of its intervention steps are

executed by automated Options. Thus, the overall IR reflects the active use of automated assistance rather than the reliance on human takeover. The differences in HIR reflect how the methods handle failures during online training. HIL-SERL relies on an operator for routine correction, resulting in the highest humancontrol requirement. AutoSERL is designed to automate intervention using a single successful demonstration [11]. However, in our benchmark, its trajectory-based guidance does not resolve all task-specific failures. On some tasks, automated intervention alone fails to generate additional positive-reward online trajectories, making human correction necessary during training. UniIntervene reduces reliance on human control through a learned recovery policy, but its intervention modules are learned offline and remain fixed during target-task online training. Its remaining human intervention suggests that the fixed recovery behavior does not cover all failure states encountered by the evolving task policy. For UniIntervene++, human takeovers occur primarily during early RL exploration, when the immature policy may enter unsafe states requiring an external safety override. Routine correction and recovery are instead handled by Trajectory Correction and CodePolicy. Thus, UniIntervene++ reduces human effort by assigning routine interventions to automated assistance, rather than by minimizing the total amount of assisted execution. Qualitative comparison of recovery behavior. Fig. 6 illustrates a failure mode that AutoSERL does not resolve. After a misaligned approach, the gripper closes without grasping the ring. In this rollout, AutoSERL returns the end effector to the demonstrated path but does not reopen the gripper or retry the grasp. The robot then follows the nominal transport motion without the object; without human takeover, the episode fails. By contrast, UniIntervene++ uses task-structured assistance with explicit gripper actions to reopen, realign, and regrasp the ring before continuing the transfer. This qualitative comparison highlights a specific recovery advantage of UniIntervene++ over AutoSERL in Ring Transfer: after an empty grasp, its task-structured assistance reopens the gripper and retries the grasp, enabling the transfer to succeed without human intervention. Progressive autonomy. Fig. 7 shows that, as RL policy success on Ring Transfer increases from zero to 88.33%, the scheduler raises the

TABLE II A BLATION S TUDY ON R ING T RANSFER AND T OWEL F OLDING Ring Transfer SR(%)

Towel Folding SR(%)

SERL w/o CodePolicy w/o Trajectory Correction Fixed-prior stochastic scheduler

0 0 50 75

0 0 47 80

UniIntervene++

88.33

85

Training configuration

Fig. 6. Visualization and comparison on the Ring Transfer task. From the same initial state, AutoSERL fails to complete the transfer, whereas UniIntervene++ completes the task through adaptive control allocation between autonomous execution and assisted behaviors. This illustrates how competence-adaptive intervention improves robust execution.

RL Policy

CodePolicy

Trajectory Correction

Fig. 7. Duration-weighted control allocation among the three Options and RL policy success evaluated at multiple checkpoints during Ring Transfer training. Within each time window, the control share of an Option is computed as its accumulated execution duration divided by the total execution duration of all three Options. The lower rows show representative execution sequences of the RL policy (circle), CodePolicy (square), and Trajectory Correction (triangle).

RL policy’s control share from 35.78% to 90.51%. Meanwhile, the combined allocation to CodePolicy and Trajectory Correction decreases from 64.22% to 9.49%. This shift demonstrates that the scheduler progressively returns control to the RL policy as its autonomous competence improves. The lower rows qualitatively illustrate the three control modes: the RL Option executes the current learned policy, CodePolicy follows a perception-grounded task plan, and Trajectory Correction steers toward a reachable point on a successful demonstration trajectory. These sequences illustrate controller behavior rather than a matched-condition comparison of their performance. C. Ablations and Diagnostics We evaluate five configurations on Ring Transfer and Towel Folding to examine the contribution of each automated assistance Option and the benefit of learned Option

allocation. SERL disables both the intervention agent and human correction, serving as the fully unassisted baseline. w/o CodePolicy removes ωCP , while w/o Trajectory Correction removes ωTC but retains the trajectory reference required by Adaptive RL Probe. Fixed-prior stochastic scheduler replaces learned Q-based selection with fixed sampling probabilities of 0.3, 0.5, and 0.2 for RL, Trajectory Correction, and CodePolicy, respectively, while retaining the same Options, feasibility masks, and Adaptive RL Probe. UniIntervene++ uses all three Options and learns their allocation with the availability-masked SMDP Double-DQN scheduler. Table II reports final success rates under the same training time. The full method achieves 88.33% SR on Ring Transfer and 85% on Towel Folding. Replacing learned Option selection with the fixed-prior scheduler reduces SR to 75% and 80%, respectively, supporting the benefit of adaptive online allocation when the controller set, feasibility masks, and probing mechanism are held fixed. Without CodePolicy, final SR falls to zero on both tasks, matching the unassisted SERL baseline, as these ablated policies are largely unable to acquire positive samples without relying on human intervention. In comparison, removing Trajectory Correction reduces SR to 50% and 47%. Under the evaluated budgets, CodePolicy therefore has the larger measured contribution, with Trajectory Correction providing complementary assistance. These are system-level ablations because removing an Option changes both executed assistance and the experience routed to the task policy. All scheduler-enabled variants retain Adaptive RL Probe, whose independent contribution is not isolated here. V. C ONCLUSION We presented UniIntervene++, an adaptive intervention agent that learns when to intervene, which assistance to use, and when to return control as the task policy improves online. UniIntervene++ places the evolving RL policy, trajectory correction, and a task-structured CodePolicy in a shared decision space and learns their relative values online. Competence-adaptive intervention preserves direct evidence of the RL policy’s evolving competence, while coupled experience learning allows assisted experience to improve the RL policy and reshape subsequent intervention decisions. Across five real-world manipulation tasks, UniIntervene++ achieved 89.67% average success, exceeding the best baseline by 6 percentage points, with

only 0.77% human intervention. On Ring Transfer, control allocated to the RL policy increased from 35.78% to 90.51% as its independently evaluated success rose from zero to 88.33%; ablations on two tasks further support learned allocation over a fixed prior. These results show that adaptive intervention can reduce operator effort and progressively return control to the RL policy without a separate offline intervention-learning stage. ACKNOWLEDGMENT Generative AI tools were used for language editing and to generate selected illustrative icons in Fig. 2. These icons are used solely for visualization and do not represent experimental data or results. All technical content, methodology, experiments, analysis, and conclusions were developed and verified by the authors. R EFERENCES [1] K. Lei, H. Li, D. Yu, Z. Wei, L. Guo, Z. Jiang, Z. Wang, S. Liang, and H. Xu, “Performant robotic manipulation with real-world reinforcement learning,” Science Robotics, vol. 11, no. 116, p. eaed6267, 2026. [2] H. Deng, H. Liu, W. Guo, Y. Ling, Z. Yang, Y. Xue, H. Sun, L. Wang, and Z. Wang, “Facet-0: A robotic foundation model for contact-rich precise manipulation,” arXiv preprint arXiv:2609.01596, 2026. [3] H. Deng, Z. Wu, H. Liu, W. Guo, Y. Xue, Z. Shan, C. Zhang, B. Jia, Y. Ling, G. Lu et al., “A survey on reinforcement learning of visionlanguage-action models for robotic manipulation,” 2025. [4] A. Padalkar, G. Quere, A. Raffin, J. Silvério, and F. Stulp, “Guiding real-world reinforcement learning for in-contact manipulation tasks with shared control templates,” Autonomous Robots, vol. 48, no. 4, p. 12, 2024. [5] C. Schwarke, V. Klemm, M. Van der Boon, M. Bjelonic, and M. Hutter, “Curiosity-driven learning of joint locomotion and manipulation tasks,” in 7th Annual Conference on Robot Learning, 2023. [6] H. Li, K. Lei, S. Zang, K. Hu, Y. Liang, B. An, X. Li, and H. Xu, “Failure-aware rl: Reliable offline-to-online reinforcement learning with self-recovery for real-world manipulation,” arXiv preprint arXiv:2601.07821, 2026. [7] J. Luo, C. Xu, J. Wu, and S. Levine, “Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning,” Science Robotics, vol. 10, no. 105, p. eads5033, 2025. [8] Z. He, Y. Cao, and M. Ciocarlie, “Uncertainty comes for free: Human-in-the-loop policies with diffusion models,” arXiv preprint arXiv:2503.01876, 2025. [9] A. Sharma, A. M. Ahmed, R. Ahmad, and C. Finn, “Self-improving robots: End-to-end autonomous visuomotor reinforcement learning,” arXiv preprint arXiv:2303.01488, 2023. [10] W. Guo, P. Tang, Y. Bai, Z. Yan, H. Deng, G. Lu, S. Li, and Z. Wang, “Efficient online robotic reinforcement learning with explicit manifold alignment,” IEEE Robotics and Automation Letters, 2026. [11] Y. Liu, H. Yu, S. Liu, Y. Wang, J. Zhang, Y. Yang, Y. Chen, and C. Zhang, “One demonstration is enough for real-world robotic reinforcement learning,” arXiv preprint arXiv:2607.01651, 2026. [12] H. Deng, Y. Gao, Y. Lin, H. Liu, Z. Wu, and Z. Wang, “UniIntervene: Agentic intervention for efficient real-world reinforcement learning,” arXiv preprint arXiv:2606.12372, 2026. [13] Y. Mo, J. Li, Q. Wu, Y. Kang, and R. Xu, “Ohp-rl: Online human preference as guidance in reinforcement learning for robot manipulation,” arXiv preprint arXiv:2605.15971, 2026. [14] Y. Zhao, H. Jin, L. Jiang, X. Zhang, K. Wu, P. Ren, Z. Xu, Z. Che, L. Sun, D. Wu et al., “Real-world reinforcement learning from suboptimal interventions,” arXiv preprint arXiv:2512.24288, 2025. [15] Y. Xue, L. Xu, Z. Liu, Z. Wu, Z. Gu, X. Song, B. Jia, and Z. Wang, “Worldsample: Closed-loop real-robot rl with world modelling,” arXiv preprint arXiv:2607.02431, 2026.

[16] S. Ross, G. J. Gordon, and J. A. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, vol. 15. PMLR, 2011, pp. 627–635. [17] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer, “HG-DAgger: Interactive imitation learning with human experts,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8077–8083. [18] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg, “ThriftyDAgger: Budget-aware novelty and risk gating for interactive imitation learning,” in Proceedings of the 5th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 164. PMLR, 2022, pp. 598–608. [19] H. Liu, S. Nasiriany, L. Zhang, Z. Bao, and Y. Zhu, “Robot learning on the job: Human-in-the-loop autonomy and learning during deployment,” The International Journal of Robotics Research, vol. 44, no. 10-11, pp. 1727–1742, 2025. [20] J. Luo, P. Dong, Y. Zhai, Y. Ma, and S. Levine, “Rlif: Interactive imitation learning as reinforcement learning,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 36 329–36 351. [21] P. J. Ball, L. Smith, I. Kostrikov, and S. Levine, “Efficient online reinforcement learning with offline data,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 1577–1594. [22] J. Luo, Z. Hu, C. Xu, Y. L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine, “Serl: A software suite for sampleefficient robotic reinforcement learning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 16 961–16 969. [23] H. Deng, Y. Lin, Y. Xue, H. Du, Q. Wang, B. Zhou, Z. Wu, and Z. Wang, “E2hil: Entropy-guided sample selection for efficient realworld human-in-the-loop reinforcement learning,” IEEE Robotics and Automation Letters, 2026. [24] A. Lagrassa, S. Lee, and O. Kroemer, “Learning skills to patch plans based on inaccurate models,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 9441–9448. [25] A. Reichlin, G. L. Marchetti, H. Yin, A. Ghadirzadeh, and D. Kragic, “Back to the manifold: Recovering from out-of-distribution states,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 8660–8666. [26] B. Thananjeyan, A. Balakrishna, S. Nair, M. Luo, K. Srinivasan, M. Hwang, J. E. Gonzalez, J. Ibarz, C. Finn, and K. Goldberg, “Recovery rl: Safe reinforcement learning with learned recovery zones,” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 4915–4922, 2021. [27] A. Wilcox, A. Balakrishna, B. Thananjeyan, J. E. Gonzalez, and K. Goldberg, “Ls3: Latent space safe sets for long-horizon visuomotor control of sparse reward iterative tasks,” in Conference on Robot Learning. PMLR, 2022, pp. 959–969. [28] S. Vats, M. Likhachev, and O. Kroemer, “Efficient recovery learning using model predictive meta-reasoning,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 7258–7264. [29] A. S. Wang and O. Kroemer, “Learning robust manipulation strategies with multimodal state transition models and recovery heuristics,” in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 1309–1315. [30] S. Vats, D. K. Jha, M. Likhachev, O. Kroemer, and D. Romeres, “Recoverychaining: Learning local recovery policies for robust manipulation,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025, pp. 9776–9783. [31] R. S. Sutton, D. Precup, and S. Singh, “Between MDPs and semiMDPs: A framework for temporal abstraction in reinforcement learning,” Artificial Intelligence, vol. 112, no. 1–2, pp. 181–211, 1999. [32] J. Janssen, Semi-Markov models: theory and applications. Springer Science & Business Media, 2013. [33] H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016, pp. 2094–2100.

Related documents

• UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning#1188331
• Can Jev be Your Q or Policy in Reinforcement Learning?#1248972
• Planning for Change: Reinforcement Learning Combined with Bounded Extremum Seeking for Robotic Control under Distribution Shift#1200894
• Official Review · ICLR 2025#787022
• CAPEX: Efficiently Distilling Foundation Model Behavior into Deployable Robot Policies through Experience-Adaptive Reasoning#1117719