arXiv CS open access

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

arXiv:2608.21204v1 [cs.RO] 21 Aug 2026

Varun Giridhar Anant Khandelwal Jeremy A. Collins Ignat Georgiev Animesh Garg Georgia Institute of Technology varungiridhar.github.io/qplanning

Abstract: Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93 → 99%, RoboTwin 83.8 → 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40 → 90% and insert-wallet 25 → 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N , filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor. Keywords: Behaviour Cloning, Q-Learning, Model Predictive Control, SelfImprovement, Vision-Language-Action Models

Figure 1: Q-Planning overview. Left: at inference, the frozen BC policy samples N candidate action chunks; the Q-function scores them and the executed chunk is a single-step Q-weighted average. Right: rollouts (successful and failed) are appended to a replay buffer and used to fine-tune only the Q-function. The updated Qϕ returns to the next inference iteration; the BC policy is never updated.

Introduction

Robot manipulation has progressed rapidly through large-scale Behaviour Cloning (BC), with large ∗ visuomotor policies such as RT-2 [1], Octo [2], π0.6 [3], Diffusion Policy [4] and FastWAM [5] demonstrating impressive zero-shot generalisation on real-world tasks. This success, however, comes at a structural cost: BC policies are bounded by the data they were trained on. A trained BC policy that fails on a task cannot learn from that failure without additional human teleoperation, and there is no mechanism by which rollouts collected during deployment improve subsequent behaviour. As we scale toward models with billions of parameters trained on millions of demonstrations, this demo ceiling becomes the dominant bottleneck for further progress. The dominant response has been Reinforcement Learning fine-tuning of the policy itself, pursued aggressively in the past year [6, 7, 8, 9, 10, 11, 12, 13, 14]. Yet updating a multi-billion-parameter vision-language-action (VLA) model on sparse-reward, on-policy data is expensive, brittle, and can silently degrade the very BC prior that gave the policy its competence in the first place [15]. An alternative line of work, value-based planning [16, 17, 18, 19], sidesteps policy updates by reranking action candidates at inference time, but has not been demonstrated at VLA scale, where high-dimensional flow-matching action spaces and long horizons make planning over random proposals intractable. Our insight is that the actor and the critic can be decoupled and trained on different data. A BC policy can only be trained on successful demonstrations (failure trajectories are not what we want to imitate), but an off-policy Q-function has no such restriction: it can be trained on any trajectory, successful or failed, because it estimates value rather than imitates actions. This asymmetry is what makes the self-improvement loop efficient: we can keep the expensive BC policy frozen and absorb all deployment signal, including failures, into a much smaller off-policy Q-function. Figure 1 illustrates this loop. To combine the two components at inference, we draw N candidate action chunks from the frozen BC policy and execute a single-step Q-weighted average, with softmax weights taken over the Qscores of the candidates. The Q-function’s vision and language encoders are amortised once per planning step, so only its action-conditioned decoder scales with N . Together, an off-policy Qfunction and a Q-weighted average over BC draws turn a frozen BC policy into a self-improving system. Ten iterations of online self-improvement lift LIBERO-10 success from 93% to 99% and bimanual RoboTwin from 83.8% to 91.4%; on a real bimanual robot the same loop improves purely from deployment rollouts on two contact-rich tasks, stack-cups 40 → 90% and insert-wallet 25 → 80%, over five iterations. Our contributions are: 1. An off-policy Q-function for large BC policies. A Q-chunking architecture with HLGauss categorical outputs and its own DinoV2 [20] and T5 [21] encoders, trained on the same demonstrations used to train the BC policy. 2. A real-time value-guided action selector. A single-step Q-weighted average over N BC flow-matching draws, amortising the encoders across all candidates so a planning step costs 400 ms on RoboTwin (1.6× faster than a single 10-step BC inference and comfortably inside the 960 ms replan budget; App. E). 3. Self-improvement from failures without touching the BC. A loop that folds successful and failed deployment rollouts into Q-only updates, evaluated across every LIBERO suite, bimanual RoboTwin, and two contact-rich real-robot tasks.

Related work

Large-scale BC and VLA policies. The dominant paradigm for general-purpose robot manipulation is large-scale Behaviour Cloning on teleoperated demonstrations [22, 1, 2, 23, 4, 3, 5, 24]. These policies excel at imitating the demonstration distribution but inherit its ceiling: they cannot incorporate failure observations collected during deployment, and out-of-distribution states reliably degrade 2

Table 1: Positioning against the closest methods. Q-Planning is the only method that combines all five axes. Method V-GPS [29] DSRL [33] IBRL [41] DAWR [42] Q-Planning (ours)

BC frozen

Plug-and-play large VLA

Learn from failures

Offline-toonline Q

No aux. actor

✓ ✓ ✓ ✗ ✓

✓ ✓ ✗ ✗ ✓

✗ ✓ ✓ ✓ ✓

✗ ✓ ✗ ✗ ✓

✓ ✗ ✗ ✓ ✓

performance [15, 25]. Q-Planning treats such a BC policy as a frozen action proposal distribution and gives it a value function it never had. Value functions and planning for control. Q-learning [26, 27, 28] and value-based planners [16, 17, 18, 19] have demonstrated strong control performance in low-dimensional or simulated regimes, but extending them to VLA-scale manipulation has been hindered by the cost of online search over high-dimensional action spaces. Recent work has explored pre-trained values as guidance for robotic foundation models [29, 30, 31, 32, 33], but does not address how to use the value function to actively self-improve via online iteration. Q-Planning closes this gap. Self-improvement and RL fine-tuning of robot policies. An active body of work fine-tunes large robot policies with online [6, 7, 8, 9, 11, 10, 12, 14, 34] or offline [35, 13, 36, 37] RL, and several recent systems iteratively co-improve a policy with a world model [38, 39, 40, 24]. All of these methods update the policy parameters, often the entire VLA, which is expensive at scale and risks degrading the BC prior. Q-Planning departs from this consensus: only the Q-function is updated during self-improvement, while the BC policy is left untouched. Positioning against value-guidance and imitation-bootstrapped methods. Table 1 distinguishes Q-Planning along five axes. V-GPS [29] steers a frozen policy with an offline-trained value and never iterates online. DSRL [33] adapts online but through a latent-noise auxiliary actor. IBRL [41] keeps its IL policy frozen but trains a fresh RL actor from scratch, which at VLA scale means relearning behaviour the BC already encodes; DAWR, the advantage-weighted regression baseline from the DPPO paper [42], fine-tunes the policy weights directly, which is expensive at scale and risks degrading the BC prior. Concurrent BC-to-Q work [43] shares the frozen-BC + off-policyQ ingredients but trains an auxiliary RL actor and has not been demonstrated on a multi-billionparameter VLA. Q-Planning is the only entry that is simultaneously BC-frozen, plug-and-play with a multi-billion-parameter policy, self-improving from failure rollouts, and free of an auxiliary actor.

Method

Overview. Q-Planning consists of three components: an off-policy Q-function over action chunks (Sec. 3.2), a single-step Q-weighted average over BC action-chunk draws (Sec. 3.3), and a selfimprovement loop that fine-tunes only the Q-function (Sec. 3.4). At every stage the base BC policy is frozen. 3.1

Problem setting

We address language-conditioned manipulation in a partially-observed MDP with action at ∈ Rda , observation ot (RGB images from one or more cameras), natural-language instruction ℓ, and perstep terminal indicator dt ∈ {0, 1} together with sparse terminal reward rt ∈ {0, 1} that is 1 only on the timestep at which the task is completed successfully. Following recent VLA work [4, 3, 5], the BC policy π BC outputs an action chunk at:t+H ∈ RH×da of length H at each planning step. We assume access to a dataset DBC of successful demonstrations used to train π BC . Our goal is to 3

improve policy performance with minimal additional human supervision: we may collect rollouts with our own planner and update value-side parameters, but we never re-collect demonstrations and never update π BC . 3.2

Off-policy Q-function over action chunks

Motivation. Standard scalar Q-regression under sparse terminal rewards is unstable [44]: most transitions carry no reward signal, and the long horizons of manipulation tasks amplify bootstrapping error. We therefore (i) treat the length-H action chunk as a single super-action so the effective bootstrapping horizon shrinks by a factor of H (Q-chunking [45]), and (ii) replace scalar regression with HL-Gauss categorical regression [44], which projects each scalar target onto a fixed grid of bins via a Gaussian kernel and trains the Q-network with a cross-entropy objective. Module design. The Q-function has its own visual and language encoders, parameter-disjoint vis txt from the BC policy: zvis = Eϕtxt (ℓ) uses T5. The Q-network t = Eϕ (ot ) uses DinoV2 and z txt Qϕ (ot , ℓ, at:t+H ) is a transformer decoder that cross-attends to [zvis t ; z ] and takes the candidate action chunk as a sequence of query tokens. The decoder output is projected to B HL-Gauss logits ℓϕ ; the predicted scalar value is the expectation of the softmaxed categorical distribution over the bin grid {vb }B b=1 of bin centres uniformly spaced in [vmin , vmax ]: Qϕ (ot , ℓ, at:t+H ) =

B X

vb · softmax ℓϕ (ot , ℓ, at:t+H ) b .

(1)

b=1

Training. Following the Q-chunking formulation [45], we train Qϕ on chunked transitions (ot , at:t+H , rt:t+H , ot+H , at+H:t+2H ) sampled from a buffer D, initialised to the demonstration dataset DBC and later expanded with online rollouts (Sec. 3.4). At training time we do not query π BC for a bootstrap action: we use the next chunk at+H:t+2H directly from the buffer, and since a chunk drawn from DBC is exactly what π BC was trained to imitate, the target remains an unbiased BC estimate of Qπ . The chunked squared-error Bellman loss is " 2 # H−1 X ′ t H L(ϕ) = ED Qϕ (ot , ℓ, at:t+H ) − γ rt+t′ − γ Qϕ̄ (ot+H , ℓ, at+H:t+2H ) , (2) t′ =0

where ϕ̄ are exponential-moving-average target parameters. In practice we do not minimise this squared error directly: following HL-Gauss [44], we project the scalar target onto the bin grid and minimise the cross-entropy to the predicted bin distribution, which stabilises learning under sparse, bimodal returns. A failed rollout is simply an all-zero-reward trajectory whose terminal chunk does not bootstrap, pinning its target to zero. Critically, this loss applies unchanged when D later includes BC online rollouts containing failures (Sec. 3.4), because Qπ is well-defined for any state-action input regardless of where the data came from. This is the asymmetry that lets us train Qϕ on data π BC itself could not be trained on. App. C justifies the choice of the one-step-shifted target over a planner-selected bootstrap. Keeping the Q-function’s encoders parameter-disjoint from π BC (DinoV2 and T5 backbones, App. A) is what makes the self-improvement loop safe: failures in one network cannot corrupt the other, so the Q-function can be updated freely without ever touching the BC weights. 3.3

Q-weighted action selection

Motivation. Given Qϕ , the natural inference-time strategy is to sample N action chunks per planning step, score them with Qϕ , and execute a value-informed aggregation. Two properties of the modern BC head make a very simple selector sufficient at VLA scale: flow-matching draws are multi-modal, so the N candidates cover distinct action modes rather than sitting inside a Gaussian envelope around a single mean; and they lie on the manifold the BC already finds plausible, so every candidate is inside Qϕ ’s training support. Multi-modality is important for self-improvement: any behaviour the BC head can produce with non-negligible probability is a candidate the planner can 4

select and the loop can amplify, so exploration is not restricted to a single Gaussian ball around the BC mode. We therefore avoid iterative sample-based search entirely (e.g., MPPI; Sec. 4.2, App. D) and instead run a single-step Q-weighted average over the raw BC draws. (n)

Module design. At each planning step t we draw N candidate chunks at:t+H ∼ π BC (ot , ℓ) from a truncated 3-step flow-matching pass of the BC policy (the reduced denoising budget both lowers latency and preserves sample diversity relative to a full 10-step draw), score each with Qϕ , and execute a softmax Q-weighted average with temperature λ: (n) w(n) ∝ exp Qϕ (ot , ℓ, at:t+H )/λ ,

āt:t+H =

N X

(n)

w(n) at:t+H .

(3)

n=1

āt:t+H is executed in the environment up to a replanning interval. The Q-network is evaluated for all N candidates in a single batched forward pass: vision and text tokens are encoded once per planning step (about 25 ms combined) and the N candidates enter the decoder as N parallel query sequences, so only the ∼500M Q-decoder cost scales with N (roughly 2-3 ms per candidate). A full planning step, including BC draws, encoding, scoring, and aggregation, takes 400 ms on the bimanual RoboTwin setup with N =32. This is 1.6× faster than a single 10-step BC inference and comfortably inside the 960 ms replan budget, the time before the executed portion of the current chunk runs out; App. E breaks down the full latency profile. 3.4

Self-improvement loop

A BC policy trained only on successful demonstrations has never seen the failure modes that emerge under autonomous execution. Q-Planning closes this loop by deploying the planner, collecting both successful and failed rollouts, and refining only the Q-function on them (Algorithm 1). Each iteration runs the planner for M episodes per task, appends the collected transitions to a replay buffer D, and refines Qϕ for S gradient steps on D via Eq. 2. Because only Qϕ (∼ 1B parameters, App. B) is updated, one iteration is dramatically cheaper than a full-policy gradient step. Algorithm 1: Q-Planning Self-Improvement D ← DBC ; for iteration i = 1, 2, . . . do // (1) Collect rollouts under Q-Planning; for task k ∈ T , episode j = 1 . . . M do Reset env; receive o0 ; while episode not done do (n) Draw N chunks {at:t+H } from π BC and compute āt:t+H via Eq. 3; Execute āt:t+H ; observe rt:t+H , ot+H ; end Append the full episode to D; end // (2) Refine Q only (BC frozen); for s = 1 . . . S do Update ϕ on a minibatch from D via L(ϕ) (Eq. 2); EMA update ϕ̄ ← η ϕ + (1−η) ϕ̄; end end We use M =100 episodes per task per iteration in simulation (20 per task on hardware) and S=200 Q-only gradient steps per iteration; full hyperparameters are in Appendix B.

Experimental results

Our experiments target four questions: (Q1) What action source should the Q-function score, i.e., does the choice of proposal distribution matter? (Q2) Does the online self-improvement loop lift 5

performance? (Q3) Does Q-Planning enable that self-improvement relative to value-guidance and imitation-bootstrapped alternatives? (Q4) Does self-improvement translate to the real world? 4.1

Experimental setup

We evaluate on two simulated manipulation benchmarks: the four LIBERO [46] suites (Spatial, Object, Goal, and the 10 hard long-horizon tasks of LIBERO-10), and 47 RoboTwin [47] tasks (a subset of the full RoboTwin suite that we have working evaluations for). All reported success rates and episode lengths aggregate over 20 episodes per task. For the BC policy we use the publiclyreleased FastWAM [5] weights, frozen throughout. The Q-function is trained on the same successful demonstrations that FastWAM was trained on (no additional data), for 12k gradient steps on LIBERO and 45k on RoboTwin, on 4× H200 GPUs, until per-task Q-values converge. LIBERO is evaluated with the standard episode budget of 540 environment steps (rather than the 700 steps used in the FastWAM paper); absolute LIBERO success rates therefore sit a few percentage points below FastWAM’s reported numbers, and this protocol applies identically to both baseline and Q-Planning rows. 4.2

Q1: which action source for planning?

We compare three candidates for the N chunks Qϕ scores: MPPI [16] with zero-mean Gaussian noise around the BC mean (T =3 iterations); MPPI + temporal smoothing, the same variant with noise convolved by a 1-D Gaussian kernel so perturbed chunks stay temporally coherent (App. D); and our single-step Q-weighted average over multi-modal 3-step flow-matching draws (Sec. 3.3). All share the same Qϕ and N =64; the unguided FastWAM policy is included as a reference. Evaluated on LIBERO-10. Table 2: Action-source comparison on LIBERO-10 (10 tasks, 20 episodes/task). Rows below the reference share the same Qϕ and differ only in how candidates are drawn. Latency is median wallclock time per planning step on an NVIDIA L40S under LIBERO’s eval-parity settings (App. E). Temporally-smoothed MPPI and our single-step Q-weighted average match on success (93.0%), but ours runs 1.7× faster and has fewer hyper-parameters. Action source

Success ↑

Mean ep. length ↓

Latency (ms) ↓

90.0 89.5 93.0 93.0

274 270 261 261

646 1114 1114 640

FastWAM (reference, no Q) MPPI MPPI + temporal smoothing Ours (3-step FM draws + Q-weighted avg.)

Two observations. Vanilla MPPI sits below the unguided BC baseline: jagged noise produces chunks outside Qϕ ’s training support, so the planner over-weights candidates with spuriously high Q-values. Temporally-smoothed MPPI and our single-step Q-weighted average over multi-modal flow-matching draws both recover the same +3pp gain over the frozen BC; we adopt the latter for the rest of the paper because it removes three hand-crafted hyper-parameters and runs 1.7× faster on LIBERO and 3.2× faster on RoboTwin (App. E), which is what makes the self-improvement loop in Sec. 4.3 and the real-robot deployment in Sec. 4.4 real-time. 4.3

Online self-improvement with Q-Planning

Table 3 reports results across the four LIBERO suites and RoboTwin under three settings: the frozen FastWAM baseline, Q-Planning evaluated offline (Q-weighted selection over BC draws, no selfimprovement), and Q-Planning after 10 iterations of online self-improvement (Q-only updates, BC frozen throughout). The offline column already shows the +1.3pp mean lift value guidance buys over the frozen BC (4 of 5 benchmarks won), but the focus of this section is what happens after the self-improvement loop is switched on. 6

Table 3: Main results on the four LIBERO suites and RoboTwin. Per-row values are means of pertask success rates (↑) and per-task mean successful episode length (↓). Bold marks the best success per row. Q-Planning (offline) improves over the BC policy on 4 of 5 benchmarks by +1.3pp on average; 10 iterations of Q-only self-improvement then lift every benchmark, reaching 97.6% mean success (+5.5pp over the frozen BC). FastWAM [5] Q-Planning (offline) Q-Planning (self-improved) Benchmark

Succ.

Len.

Succ.

Len.

Succ.

Len.

LIBERO-Spatial 90.5 LIBERO-Object 100.0 LIBERO-Goal 97.0 LIBERO-10 90.0 RoboTwin (47 tasks) 83.2

106 138 107 274 220

91.5 99.5 99.0 93.0 83.8

115 139 110 261 231

98.5 100.0 99.0 99.0 91.4

107 120 99 224 232

Mean

–

93.4

–

97.6

–

92.1

(a) LIBERO-10: baselines under identical online budget

(b) Breadth: LIBERO suites and bimanual RoboTwin 100

Success (\%)

100

80 70

IBRL DSRL Filtered SFT

60 0

90 85

DAWR Best-of-N Q-Planning (ours)

RoboTwin (bimanual) LIBERO-Spatial

80 8

Self-improvement iteration

LIBERO-Goal LIBERO-Object

Self-improvement iteration

Figure 2: Q-Planning self-improvement in simulation. (a) LIBERO-10 vs. five online baselines under an identical rollout budget: Q-Planning lifts success from 93% to 99% while every baseline plateaus, oscillates, or collapses. (b) Per-iteration trajectories across every LIBERO suite and bimanual RoboTwin, visualising the endpoints reported in Table 3. The BC policy is frozen throughout; only Qϕ is updated.

Q2: The self-improvement loop lifts every benchmark. With M =100 episodes per task per iteration and S=200 Q-only gradient steps per iteration (Algorithm 1), the loop takes every benchmark to a higher success rate: LIBERO-Spatial 91.5 → 98.5%, LIBERO-10 93 → 99%, RoboTwin 83.8 → 91.4%, and mean 92.1 → 97.6% (+5.5pp over the frozen BC and +4.2pp over the offline Q-Planning column). On the two suites that were already near ceiling under the offline planner (LIBERO-Object, LIBERO-Goal), success has no room to grow, so the loop instead shortens successful episodes (139 → 120 and 110 → 99 steps). Figure 2(b) plots the per-iteration trajectory of these gains. Q3: Q-Planning is what enables this self-improvement. Figure 2(a) compares Q-Planning on LIBERO-10 against five online baselines under the same rollout budget: Best-of-N (same selfimprovement loop, argmax selection instead of the weighted average), Filtered SFT (re-imitating only successful rollouts), IBRL [41], DSRL [33], and DAWR [42]. Best-of-N plateaus at 95%, so value-guided selection alone captures part of the gain but only the Q-weighted average plus self-improvement carries Q-Planning to 99%. Filtered SFT plateaus at 93.5%, confirming that reimitating only successes cannot absorb the failure signal Qϕ can. IBRL collapses, DSRL swings widely between 69% and 91%, and DAWR hovers below the frozen BC. Under the same online budget Q-Planning is the only method that improves stably from failures. Figure 3 visualises Qϕ ’s predictions: on success it climbs monotonically from ∼ 0.26 to ∼ 0.63, on failure it oscillates in a lower range, and the candidate scatter shows it separating high- from low-value BC draws at every step, the signal both the Q-weighted average and the loop exploit. 7

Qϕ value

1.0

(a) Success: "put the alphabet soup in the basket"

(b) Failure: "turn on stove and put the moka pot on it"

1.0

0.8

0.6

0.4

0.2

0.0

100

150

200

0.0

250

200

Episode timestep

400

600

800

Episode timestep

Figure 3: Q-value over time on two representative LIBERO-10 episodes. Solid lines show Qϕ of the executed (Q-weighted average) chunk; faint scatter shows the N =64 BC candidates scored per planning step. Real robot: Q-Planning (solid) vs SFT on successes (dotted) 100

Success (\%)

stack-cups

insert-wallet

Stack cups Insert wallet

0 1

Stack cups (SFT) Insert wallet (SFT)

Self-improvement iteration

Figure 4: Real-robot self-improvement on two contact-rich bimanual tasks. From 100 base demos and 20 online episodes per iteration, with the BC frozen and no human intervention, QPlanning improves from its own failures: stack-cups 40 → 90% and insert-wallet 25 → 80% in five iterations (Q-weighted selection alone starts the curves at 65% and 40%). SFT on successes alone (dotted) stalls at 55% and 30%.

4.4

Real-robot self-improvement

Q4: Self-improvement translates to the real world. We deploy Q-Planning on two contact-rich bimanual tasks (Fig. 4; setup in App. F): stack-cups (stacking plastic cups) and insert-wallet (slotting a credit card into a wallet). From 100 base demonstrations and 20 online episodes per iteration for 5 iterations, with the BC frozen and no human intervention, Q-Planning lifts stack-cups from its 40% BC baseline to 90% and from 25% to 80% on the harder insert-wallet task. SFT on successful rollouts alone (dotted in Fig. 4) stalls at 55% and 30%: the gains come from the failure signal Qϕ absorbs and SFT discards.

Conclusion

We introduced Q-Planning, which makes a frozen BC policy self-improving by exploiting a single asymmetry: BC must train on successful demonstrations, while an off-policy Q-function can train on any rollout. In practice this asymmetry buys a lot: a ∼ 1B-parameter Q-function trained beside a multi-billion-parameter policy selects among BC draws in real time and absorbs every deployment rollout into Q-only updates that never touch the BC weights. Ten such iterations lift every simulated benchmark we tested (mean success 92.1 → 97.6%), shortening episodes where success was already at ceiling; five lift two contact-rich real-robot tasks (stack-cups 40 → 90%, insert-wallet 25 → 80%) with no human intervention. Because only Qϕ is updated, self-improvement scales with the critic’s size rather than the policy’s, an increasingly attractive property as VLA backbones grow toward 10B+ parameters. Nor is the framework intrinsically tied to imitation learning: any proposal distribution that produces diverse plausible action chunks (offline RL, reward-weighted imitation, world-model rollouts) could stand in for the BC head, a natural next step for Q-Planning. 8

Limitations

BC policy dependence and exploration boundary. Q-Planning cannot bootstrap from scratch: any behaviour the BC head cannot produce with non-negligible probability is a behaviour the planner cannot select and the loop cannot learn to exploit. Multi-modal flow-matching draws widen this envelope beyond a single Gaussian around the BC mode, but tasks where the base BC generates no successful chunks at all remain out of reach. Gains therefore scale with BC quality and diversity. Q-decoder scaling with N . The BC and Q encoders are amortised once per planning step, but the Q decoder scales linearly with the candidate count N (∼ 2-3 ms per candidate on an L40S). Our deployed configurations (N =64 on LIBERO, N =32 on RoboTwin) fit inside the frozen BC baseline’s latency budget (Table 2, App. E), but pushing N toward the hundreds to attack harder exploration would eventually make the decoder pass, not the BC draw, the bottleneck. Terminal-reward supervision. Our loop assumes a per-episode success detector: the environment success bit in simulation and a human-provided per-episode label on the real robot. Extending to open-ended tasks would require language-conditioned or learned success models.

Acknowledgments We are grateful to the Georgia Tech PACE Phoenix cluster for compute, and to the FastWAM [5] authors for releasing pretrained weights and evaluation code that made this study possible.

References [1] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V. Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y. Lu, S. Levine, L. Lee, T.-W. E. Lee, I. Leal, Y. Kuang, D. Kalashnikov, R. Julian, N. J. Joshi, A. Irpan, B. Ichter, J. Hsu, A. Herzog, K. Hausman, K. Gopalakrishnan, C. Fu, P. Florence, C. Finn, K. A. Dubey, D. Driess, T. Ding, K. M. Choromanski, X. Chen, Y. Chebotar, J. Carbajal, N. Brown, A. Brohan, M. G. Arenas, and K. Han. Rt-2: Vision-languageaction models transfer web knowledge to robotic control. In J. Tan, M. Toussaint, and K. Darvish, editors, Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2165–2183. PMLR, 06–09 Nov 2023. URL https://proceedings.mlr.press/v229/zitkovich23a.html. [2] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, L. Y. Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024. [3] P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, ∗ J. Darpinian, K. Dhabalia, J. DiCarlo, et al. π0.6 : a VLA that learns from experience. arXiv preprint arXiv:2511.14759, 2025. [4] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023. [5] T. Yuan, Z. Dong, Y. Liu, and H. Zhao. Fast-wam: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026. [6] K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, Q. Zhang, Z. Yu, G. Fan, et al. πrl: Online rl fine-tuning for flow-based vision-language-action models. arXiv preprint arXiv:2510.25889, 2025. [7] H. Li, Y. Zuo, J. Yu, Y. Zhang, Z. Yang, K. Zhang, X. Zhu, Y. Zhang, T. Chen, G. Cui, et al. Simplevla-rl: Scaling vla training via reinforcement learning. arXiv preprint arXiv:2509.09674, 2025. [8] G. Lu, W. Guo, C. Zhang, Y. Zhou, H. Jiang, Z. Gao, Y. Tang, and Z. Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719, 2025. [9] Y. Chen, S. Tian, S. Liu, Y. Zhou, H. Li, and D. Zhao. Conrft: A reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.05450, 2025. [10] Y. Li, X. Ma, J. Xu, Y. Cui, Z. Cui, Z. Han, L. Huang, T. Kong, Y. Liu, H. Niu, et al. Gr-rl: Going dexterous and precise for long-horizon robotic manipulation. arXiv preprint arXiv:2512.01801, 2025. [11] Y. Guo, J. Zhang, X. Chen, X. Ji, Y.-J. Wang, Y. Hu, and J. Chen. Improving vision-languageaction model with online reinforcement learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 15665–15672. IEEE, 2025. [12] S. Tan, K. Dou, Y. Zhao, and P. Krähenbühl. Interactive post-training for vision-languageaction models. arXiv preprint arXiv:2505.17016, 2025. 10

[13] M. S. Mark, T. Gao, G. G. Sampaio, M. K. Srirama, A. Sharma, C. Finn, and A. Kumar. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint arXiv:2412.06685, 2024. [14] J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y. Wu, C. Yu, and Y. Wang. What can rl bring to vla generalization? an empirical study. Advances in Neural Information Processing Systems, 38: 97121–97151, 2026. [15] A. Kumar, J. Hong, A. Singh, and S. Levine. Should i run offline reinforcement learning or behavioral cloning? In International conference on learning representations, 2021. [16] G. Williams, A. Aldrich, and E. A. Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):344–357, 2017. [17] N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. 2022. [18] N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world models for continuous control. 2024. [19] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. [20] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. [21] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. [22] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. [23] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023. [24] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davchev, Y. Zhou, A. Gupta, A. Raju, et al. Robocat: A self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.11706, 2023. [25] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pages 143–156. PMLR, 2017. [26] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [27] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018. [28] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. [29] M. Nakamoto, O. Mees, A. Kumar, and S. Levine. Steering your generalists: Improving robotic foundation models via value guidance. arXiv preprint arXiv:2410.13816, 2024. 11

[30] C. Bhateja, D. Guo, D. Ghosh, A. Singh, M. Tomar, Q. Vuong, Y. Chebotar, S. Levine, and A. Kumar. Robotic offline rl from internet videos via value-function pre-training. arXiv preprint arXiv:2309.13041, 2023. [31] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030, 2022. [32] Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pages 23301–23320. PMLR, 2023. [33] A. Wagenmaker, M. Nakamoto, Y. Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506.15799, 2025. [34] W. Xiao, H. Lin, A. Peng, H. Xue, T. He, Y. Xie, F. Hu, J. Wu, Z. Luo, L. Fan, et al. Selfimproving vision-language-action models with data generation via residual rl. arXiv preprint arXiv:2511.00091, 2025. [35] A. Kumar, A. Singh, F. Ebert, M. Nakamoto, Y. Yang, C. Finn, and S. Levine. Pre-training for robots: Offline rl enables learning new tasks from a handful of trials. arXiv preprint arXiv:2210.05178, 2022. [36] J. Yang, M. S. Mark, B. Vu, A. Sharma, J. Bohg, and C. Finn. Robot fine-tuning made easy: Pre-training rewards and policies for autonomous real-world reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4804–4811. IEEE, 2024. [37] Z. Hu, R. Wu, N. Enock, J. Li, R. Kadakia, Z. Erickson, and A. Kumar. Rac: Robot learning for long-horizon tasks by scaling recovery and correction. arXiv preprint arXiv:2509.07953, 2025. [38] Y. Guo, T. Lee, L. X. Shi, J. Chen, P. Liang, and C. Finn. Vlaw: Iterative co-improvement of vision-language-action policy and world model. arXiv preprint arXiv:2602.12063, 2026. [39] C. Liu, W. Tan, L. Zhu, F. Li, J. Li, G. Yang, and H. T. Shen. Self-correcting vla: Online action refinement via sparse world imagination. arXiv preprint arXiv:2602.21633, 2026. [40] E. Y.-T. Jia, W. Yuan, T. Shi, V. Guizilini, J. Mao, and Y. Wang. Dreamplan: Efficient reinforcement fine-tuning of vision-language planners via video world models. arXiv preprint arXiv:2603.16860, 2026. [41] H. Hu, S. Mirchandani, and D. Sadigh. Imitation bootstrapped reinforcement learning. arXiv preprint arXiv:2311.02198, 2023. [42] A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024. [43] L. Dodeja et al. When life gives you BC, make Q-functions: Extracting Q-values from behavior cloning for on-robot reinforcement learning. arXiv preprint arXiv:2605.05172, 2026. [44] J. Farebrother, J. Orbay, Q. Vuong, A. A. Taı̈ga, Y. Chebotar, T. Xiao, A. Irpan, S. Levine, P. S. Castro, A. Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950, 2024. [45] Q. Li, Z. P. Zhou, and S. Levine. Reinforcement learning with action chunking. Advances in Neural Information Processing Systems, 38:55518–55553, 2026. 12

[46] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791, 2023. [47] Y. Mu, T. Chen, Z. Chen, S. Peng, Z. Lan, Z. Gao, Z. Liang, Q. Yu, Y. Zou, M. Xu, et al. Robotwin: Dual-arm robot benchmark with generative digital twins. In Proceedings of the computer vision and pattern recognition conference, pages 27649–27660, 2025.

Q-function architecture

images ot

DinoV2

language ℓ

zvis t

Transformer decoder

ztxt

(cross-attn)

HL-Gauss head (B bin logits)

Qϕ

query tokens

action chunk at:t+H

Figure 5: Q-function architecture. The Q-function Qϕ has its own DinoV2 visual encoder and T5 language encoder (parameter-disjoint from the BC policy). The transformer decoder cross-attends txt to visual tokens zvis (keys/values) and takes the candidate action chunk t and language tokens z at:t+H as a sequence of query tokens. The HL-Gauss head outputs B bin logits over discounted returns and the scalar Qϕ is the expectation of the softmaxed bin distribution.

Implementation details

Hyperparameters. We use H = 32 action-chunk length, inherited from FastWAM’s action horizon [5], N = 64 candidate chunks per planning step on LIBERO and N = 32 on RoboTwin, Q-weighting temperature λ = 1, and discount γ = 0.99. The planner executes the first 10 steps of each chunk on LIBERO and the first 24 on RoboTwin before replanning (App. E). When sampling action chunks from FastWAM during Q-Planning, we use 3 flow-matching denoising steps (rather than the default 10), which lowers latency and preserves the sample diversity that a Q-weighted average benefits from. The Q-function output uses B = 101 HL-Gauss bins on [vmin , vmax ] = [0, 1] with kernel width matched to bin spacing. Optimisation uses AdamW with learning rate 3×10−4 and an EMA target rate of η = 0.005. Q-training runs on 4× H200 GPUs for 12k gradient steps on LIBERO and 45k on RoboTwin before evaluation. One self-improvement iteration consists of M = 100 collected episodes per task (20 per task on the real robot) followed by S = 200 Q-only gradient steps on the combined offline + online buffer, with each minibatch drawn half from the original demonstrations and half from the accumulated online rollouts; the BC policy is frozen for all S steps and across all iterations. LIBERO episodes are capped at 540 environment steps; RoboTwin episodes use the benchmark default. Model size. The Q-function totals approximately 1B parameters (DinoV2 visual encoder ∼ 300M, T5 language encoder ∼ 220M, transformer decoder ∼ 500M). This is small relative to the multibillion-parameter FastWAM policy, and accounts for why one self-improvement iteration (S=200 Q-only gradient steps) is dramatically cheaper than a full-policy gradient step. Training stability. Two ingredients keep Q-training stable under sparse terminal rewards: HLGauss categorical regression removes the scale sensitivity of scalar MSE regression to sparse, bimodal returns, and Q-chunking shrinks the effective bootstrapping horizon by a factor of H, mitigating long-horizon variance blow-up.

Bootstrap target: one-step-shifted vs planner-selected

The Bellman loss in Eq. 2 uses the next buffer chunk at+H:t+2H as the bootstrap action rather than an action selected by our current planner. Two considerations justify this choice. First, it removes a full planner call from every Bellman update, which makes training an order of magnitude cheaper than an explicit Q-learning update [45]. Second, as self-improvement proceeds the buffer fills with on-policy rollouts collected by the Q-guided planner, so the shifted target effectively bootstraps from 14

planner-selected actions and the value/policy gap closes without an explicit correction. We tried a planner-selected bootstrap variant and observed no gain; we keep the shifted target for simplicity and training speed.

Temporal-smoothed MPPI (baseline in Sec. 4.2)

Two of the rows in Table 2 use variants of Model Predictive Path Integral (MPPI) control [16] as the action source that the Q-function scores. We describe them here for completeness. At each planning step t we run T iterations of MPPI. At iteration i, the proposal distribution over (i) (0) action chunks has mean āt:t+H ∈ RH×da , initialised at iteration i=0 to the BC sample āt:t+H = π BC (ot , ℓ) (the BC warm-start), and is perturbed by additive Gaussian noise: (n,i)

(i)

at:t+H = āt:t+H + Gσ ∗k ξ (n) ,

(4) (n) ξ (n) ∈ RH×da , ξ k,j ∼ N (0, 1), n = 1, . . . , N, where Gσ is a 1-D Gaussian kernel of standard deviation σ convolved along the chunk axis k. When σ → 0 the kernel is the identity and we recover the “vanilla MPPI” row of Table 2: independent Gaussian perturbations at every timestep of the chunk, which produce jagged trajectories whose per-step actions can jump far from the BC’s mean. Setting σ > 0 (we use σ = 2 over H = 32) correlates the noise across neighbouring timesteps so perturbed chunks stay temporally coherent and lie closer to the BC’s own trajectory manifold, which is what recovers the +3pp gain in Table 2. MPPI weights are then computed from the Q-scores with temperature λ and used to update the proposal mean: N X (n,i) (i+1) (n,i) w(n,i) at:t+H . (5) āt:t+H = w(n,i) ∝ exp Qϕ (ot , ℓ, at:t+H )/λ , n=1 (T ) iterations, the converged mean āt:t+H

After T is executed in the environment. Our current QPlanning planner (Sec. 3.3) is the limit T =1 with σ made irrelevant by drawing the candidates directly from a 3-step flow-matching pass of the BC head, whose native multi-modality plays the role that temporally-smoothed noise plays here.

Planning-step latency profile

We profile a full Q-Planning planning step (one action chunk) on a single NVIDIA L40S GPU, batch 1, median + p95 over 60 timed steps after 10 warm-up steps, with torch.cuda.synchronize around every timed region. The strict setting uses lerobot-eval parity (deterministic algorithms, TF32 off, cudnn.benchmark off, BC in bf16, Q in fp32) which reproduces the reported success rates; the standard-inference setting turns TF32 on, enables cudnn.benchmark, and drops deterministic kernels (a deployment mode). The replanning cadence is the deadline a planning step must beat: the planner emits an H=32-step chunk, of which the first 10 (LIBERO, 30 Hz) or 24 (RoboTwin, 25 Hz) steps execute before replanning, giving budgets of 333 ms and 960 ms. Table 4: Planning-step latency (ms) on an NVIDIA L40S. Median over 60 timed steps; parenthesised numbers are p95. Percentages express the fraction of the replan budget (LIBERO 333 ms, RoboTwin 960 ms). Strict = eval parity settings. Rows marked ◀ are the deployed configurations. The final row is an earlier iterative sample-based variant we prototyped, retained here for comparison. Configuration

Q-evals LIBERO strict

RoboTwin strict

BC only, 3 denoise steps BC only, 10 steps (eval baseline) Q-Planning, N =16, 3 steps Q-Planning, N =32, 3 steps ◀ RoboTwin Q-Planning, N =64, 3 steps ◀ LIBERO Earlier iterative sample-based variant (3 iters, N =64)

0 0 16 32 64 192

82 194 101 111 192 334

273 (275) 640 (643) 352 (354) 400 (402) 716 (725) 1276 (1282)

28 67 37 42 75 133

273 (275) 646 (647) 337 (338) 371 (372) 640 (641) 1114 (1117)

Findings. (i) Under strict eval-parity settings, RoboTwin is comfortably real-time: the deployed N =32 planner uses only 42% of the 960 ms budget, and 3.2× faster than the earlier iterative variant which would have missed the budget at 133%. (ii) LIBERO is latency-neutral relative to its own frozen-BC baseline: the deployed N =64 configuration takes 640 ms per step, essentially matching the 646 ms cost of a single 10-step BC draw the baseline itself pays. Lowering to N =16 fits the 333 ms budget with comparable success. (iii) Encoders (DinoV2 + T5) run once per planning step at 23-27 ms; only the ∼ 500M Q decoder scales with N (roughly 2-3 ms per candidate), which is why doubling N does not double the total step cost.

Real-robot setup

Platform and control. Two 6-DoF YAM arms, table-mounted and forward-facing (Fig. 4). Observations are RGB images from three cameras, one overhead and one on each wrist, shared by the BC policy and the Q-function. Actions are joint-space commands for both arms; the BC emits H=30 action chunks (1 s at 30 Hz) and the planner executes the full chunk open-loop before replanning, giving a 1 s replan budget. Policy inference runs on a single RTX 5090. Data and evaluation protocol. Each task starts from 100 demonstrations. Initial conditions are randomised within the demonstration distribution: cup positions vary across episodes, and for insertwallet the card’s starting slot varies while the wallet stays fixed. Each self-improvement iteration collects 20 episodes per task, and the per-iteration success rates in Fig. 4 are measured on these collection episodes. The robot is never teleoperated during rollouts; human input during the loop is limited to scene resets between episodes and the per-episode success label that supplies the terminal reward (Sec. 3.1). SFT baseline. The SFT comparison in Fig. 4 fully fine-tunes the policy on the successful rollouts collected under the same episode budget, discarding failures; Q-Planning instead absorbs the full rollout set into Q-only updates with the hyperparameters of App. B.

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Analyze & explore related research in Conceptio

Related documents