arXiv CS open access

GSBR: Geometric Support Behavior Regularization for Online RL

Charles Xu et al. · arxiv_cs

arXiv CS · Papers

deep-learning machine-learning neural-networks

Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation l

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

GSBR: Geometric Support Behavior Regularization for Online RL Charles Xu1 , Jacob Berg1 , Liyiming Ke2 , Sergey Levine1,2 2

UC Berkeley

Physical Intelligence

Base policy likelihood

Abstract — Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies – typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets – capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy’s support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy’s support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code: https://charlesxu0124.github.io/gsbr/. Base Policy

a¤

TD3+BC RLT Residual Policy BFN

Regularizer loss

arXiv:2611.05295v1 [cs.RO] 3 Nov 2026

DSRL EXPO DiceRL

a (a) Behavior cloning  b » ¼¯ [(a ¡ b) 2 ]

GSBR (Ours) a a (b) Support constraint (c) GSBR 0 0.5 1.0 © ª [min ka ¡ bi k 2 ¡ r] 2+ ; bi » ¼¯ a 2 A j ¼¯ (a) > ² Normalized throughput i

Figure 1. Left: (a) Behavior cloning regularizer matches the base policy distribution, not allowing the RL actor to drop poor modes. (b) Support constraint allows the actor to select the best behavior in the base policy distribution, but excludes improvements beyond it. (c) GSBR applies distance-based regularization on deviation outside the base policy support, allowing the actor to discard suboptimal behaviors and guide exploration beyond the prior. Right: GSBR achieves 23% higher normalized throughput across 19 tasks and 4 benchmarks than previous methods.

GSBR: Geometric Support Behavior Regularization for Online RL

1 Introduction Online reinforcement learning (RL) can be costly when every interaction happens on a physical system, so practical methods often rely on a behavior prior that focuses exploration on plausible actions. In robotics, such priors are now available at scale: vision-language-action (VLA) and world-action models (WAMs) learn diverse behaviors from large demonstration datasets (Kim et al., 2024; Black et al., 2024; Ye et al., 2026), but often lack the robustness and throughput required for reliable deployment (Physical Intelligence et al., 2025; Xu et al., 2026). RL fine-tuning can close this gap by updating a policy that is regularized toward the pretrained base model. This raises a general question for RL: how should a behavior regularizer preserve the base policy’s guidance while giving the actor enough freedom to improve on it? Improving upon base policies typically has two requirements: mode selection and local refinement. Consider a scene with two objects and the task is to grasp a specified target. Foundation models trained on large-scale data often produce diverse behaviors - and such base policies might have two action modes, one to grasp each object. The actor should be updated to discard the mode that reaches for the wrong object; we call this mode selection. However, existing behavior constraints can hinder one of the requirements and restrict policy improvement. Action regression (Fujimoto & Gu, 2021; Park et al., 2025b) and distribution-matching regularizers (Li & Levine, 2026; Fang et al., 2025) tie the actor to the base policy’s full action distribution, penalizing the actor even for discarding poor modes (Figure 1(a)). Support constraints (Kumar et al., 2019; Wagenmaker et al., 2025; Li et al., 2025) remove this penalty and permit mode selection, but can be too restrictive and often exclude optimal actions outside the base policy’s support (Figure 1(b)). Our key insight is to allow both mode selection and local refinement by introducing a geometric relaxation of the support constraint. We introduce Geometric Support Behavior Regularization (GSBR), an RL algorithm that estimates the support of the base policy using finite samples, and regularizes actions differently based on whether they are inside or outside the support. Actions inside the estimated support incur no penalty, allowing the actor to discard poor modes and concentrate on better ones without preserving their relative frequencies. Outside the support, GSBR replaces the infinite penalty of a hard constraint with a finite one that grows with distance, reflecting that value estimates become less reliable far from the base policy. Because the support of a learned base policy is intractable to compute, we approximate it by the union of neighborhoods around actions sampled from the base policy, with a radius that adapts to the observed samples’ distances from each other. GSBR provides a practical sample-based geometric support regularizer fitting for the diverse distribution of large-scale robot foundation policies. Our contributions are (1) a geometric behavior regularizer that permits both mode selection and local refinement in RL finetuning, (2) a sample-based approximation of multi-modal action distribution with an adaptive radius, with error bounds and a guarantee that the error vanishes as the number of reference samples grows, (3) an evaluation across 19 tasks drawn from four benchmarks where GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Further analysis shows that our approximation closely matches the true support region and examines the effects of sample count and support radius.

2 Related Work Sample-efficient online RL. Learning from costly environment interaction is a central challenge in robotic RL. Off-policy actor–critic methods address this through experience reuse, high update-todata ratios, and critic regularization to improve data efficiency (Haarnoja et al., 2018; Chen et al., 2021; Hiraoka et al., 2022). Prior data and demonstrations can also help by initializing policies and values, supplementing replay buffer, and guiding exploration (Ball et al., 2023; Nakamoto et al., 2023; Hu et al., 2023). These algorithmic advances enable practical systems to learn contact-rich manipulation within hours of robot interaction (Luo et al., 2024; 2025; Dong et al., 2026a). GSBR is complementary to these advances: it further improves sample efficiency through a behavior

GSBR: Geometric Support Behavior Regularization for Online RL

regularizer that uses a frozen base policy as a prior, reducing unnecessary exploration while allowing policy improvement beyond the prior’s support. RL with generative models, diffusion and flow policies. Diffusion and flow policies, including VLAs, represent multimodal behavior, but they require iterative action generation and complicate RL optimization. One family updates the generative model itself, through policy gradients, critic gradients, or denoising supervision (Ren et al., 2024; Wang et al., 2023; Li & Levine, 2026; Lu et al., 2025), or through distillation and advantage conditioning (Park et al., 2025b; Chen et al., 2025; Mark et al., 2024; Physical Intelligence et al., 2025). Another keeps the model frozen and trains a lightweight actor that corrects its actions (Silver et al., 2018; Johannink et al., 2019; Ankile et al., 2024; Xiao et al., 2025; Xu et al., 2026), steers its latent noise (Wagenmaker et al., 2025), or selects among sampled candidates (Dong et al., 2026b;a; Sun & Song, 2026), sometimes distilling the result back into the base model (Xiao et al., 2025; Intelligence et al., 2026). GSBR is most similar to the second family; methods within it differ mainly in how the actor is tied to the base policy, which we compare next. Behavior constraints for RL. Behavior constraints guide policy improvement using available experience or a pretrained policy, but differ in how they restrict mode selection and departures from prior behavior. Distribution matching and action regression. KL regularization penalizes reallocating probability toward higher-value modes (Wu et al., 2019; Fang et al., 2025; Li & Levine, 2026), and regression toward sampled actions favors their mean (Fujimoto & Gu, 2021; Tarasov et al., 2023). FQL and RLT (Park et al., 2025b; Xu et al., 2026) alleviate this mode collapsing problem by explicitly conditioning on noise or reference-action, but their pairwise regularization still ties the actor to the modes of randomly drawn inputs. In contrast, GSBR allows the actor to focus on promising modes and drop poor behaviors (mode selection) without retaining a particular action pairing. Support constraints and approximations. Ideal support constraints allow the actor to choose freely among actions with non-negligible density under the behavior prior. They permit mode selection but block local refinement. BEAR, SPOT, and PRDC (Kumar et al., 2019; Wu et al., 2022; Ran et al., 2023) constrain the actor to dataset actions via sample-based MMD or a density estimator. However, they require access to the dataset and do not leverage a pretrained foundation model’s broader behavioral repertoire. DSRL (Wagenmaker et al., 2025) acts in the base policy’s latent-noise space, while best-of-N methods select the highest-value action among sampled actions (Nakamoto et al., 2024; Li et al., 2025; Hansen-Estruch et al., 2023). Both enable mode selection, but remain suboptimal when local refinement is needed. These limitations motivate GSBR, which constructs an unpenalized support region from the base policy distribution and applies a finite geometric penalty beyond it, allowing online exploration to depart from the prior’s support. Residual reinforcement learning. Residual policies allow local refinement, but are often bounded by a prescribed scale (Silver et al., 2018; Johannink et al., 2019; Ankile et al., 2024). BCQ, EXPO and EXPO-FT (Fujimoto et al., 2019; Dong et al., 2026b;a) combine bounded corrections with value-based candidate selection to have both mode selection and local refinement. However, they only allow local refinement up to a fixed magnitude set as a hyperparameter, bounding the maximum allowed deviation without any guidance from the base policy. They can be interpreted as an implicit support constraint that admits an enlarged support region by a fixed radius. DICE-RL replaces the fixed magnitude by using a selectively dropped out squared-distance regularizer on the edit policy (Sun & Song, 2026). While this method can be viewed as an implicit support constraint with a geometric penalty for deviation, it creates constant optimization pressure that pulls even small edits towards each base policy sample. In contrast, GSBR explicitly approximates the support region for the base policy distribution, permitting in-distribution action to have zero behavior penalty, enabling free mode selection and refinement within the support region while allowing departures at a finite, distance-dependent cost.

GSBR: Geometric Support Behavior Regularization for Online RL

3 Preliminaries Reinforcement learning. We consider a discounted Markov Decision Process (MDP) M = (S, A, P, r, ρ, γ) with states s ∈ S, actions a ∈ A ⊆ Rd , transition distribution P (· | s, a), reward r(s, a), initial-state distribution ρ, and discount factor γ ∈ [0, 1). The goal is to learn a policy πθ that maximizes the expected discounted return P∞ J(θ) = Eρ,P,πθ [ t=0 γ t r(st , at )]. (1) Online RL with a pretrained behavior prior. In designing RL algorithms to interact with the MDP environment and optimize for the return, we leverage a behavior prior to guide exploration and training. We study online policy improvement with a frozen pretrained base policy πβ and an offline transition dataset Doff of task-specific transitions, e.g., demonstrations or base-policy rollouts. The learner collects additional experience through environment interaction and trains on both offline and online transitions. The base policy stays frozen and is accessed only through samples b ∼ πβ (· | s); we require no access to its parameters or gradients, so πβ can be any generative model. Challenges. Three properties make this setting difficult: (1) πβ is typically multimodal and its modes can differ widely in value, which behaviors are worth keeping depends on the task. (2) the base action support is intractable: we can draw samples but cannot enumerate all plausible actions (e.g., too expensive to enumerate for every state). (3) the critic is trained on limited data, its estimates are less reliable when the input is out-of-distribution; an actor that maximizes Qϕ without restraint will exploit these errors. Support and optimality assumption about the base policy. Let πβ (a | s) denote the base policy’s conditional action density. We define the support of πβ at state s: (2) Sβ (s) = a ∈ A : πβ (a | s) > 0 , and call its elements in-distribution actions. We assume only that the in-distribution actions include reasonable behaviors for the task, not that they include the optimal action. We further assume that improvements over πβ can be obtained in two ways: by concentrating on the most valuable supported actions regardless of how often πβ samples them, which we call mode selection, and by refining actions near the support, which we call local refinement. Behavior-regularized actor–critic RL. Following classic actor-critic RL algorithms, we will learn two functions: actor πθ (· | s, B) and critic Qπθ (s, a). The actor πθ (· | s, B) conditions on the state and on a set of N reference actions B = {b1 , . . . , bN } ∼ πβ (· | s)N . The critic function Qπθ (s, a) is the expected discounted return after taking a in s and subsequently following πθ . We learn it with classic temporal-difference learning loss, h 2 i LTD (ϕ) = E(s,a,r,s′ )∼D, B′ ∼πβ (·|s′ )N ,a′ ∼πθ (·|s′ ,B′ ) Qϕ (s, a) − sg r + γ Qϕ̄ (s′ , a′ ) , (3) where D is the replay distribution over offline and online transitions, sg denotes stop-gradient, Qϕ̄ is a slowly updated target critic. The actor is trained with a behavior-regularized Q-maximization objective, Lactor (θ) = Es∼D, B∼πβ (·|s)N ,a∼πθ (·|s,B) −Qϕ (s, a) + α ℓbeh (a, B) , (4) where ℓbeh is a behavior regularization penalty and α ≥ 0 controls its strength. The choice of ℓbeh determines which improvements the actor can reach.

4 Geometric Support Behavior Regularizer (GSBR) GSBR uses the base policy to identify where exploration is promising while leaving the actor free to select and improve behaviors in that region. Section 4.1 formulates the desired behavior constraint (ℓbeh in Equation 4) as a geometric relaxation of support-constrained policy improvement. Section 4.2 approximates this penalty from base-policy samples and bounds the approximation error. Algorithm 1 describes the practical instantiation of our algorithm.

GSBR: Geometric Support Behavior Regularization for Online RL

4.1 Geometric relaxation of support constraints The assumption in Section 3 translates into two requirements on ℓbeh . For mode selection, the penalty should be zero for in-distribution actions, so that the actor can concentrate on the most valuable modes without preserving the base policy’s relative mode frequencies. For local refinement, actions outside the support should incur a finite penalty rather than being excluded, and this penalty should grow with distance: the assumption places improvements near the support, and the critic is least reliable far from it. Together, these requirements call for a region of zero penalty surrounded by a finite, distance-dependent penalty (Figure 2). Following the definition of support region in Equation 2, the true geometric relaxation of the support constraint is given by ℓsupp (s, a) = d(a, Sβ )2 , d(a, X ) = inf ∥a − x∥2 . (5) x∈X

The penalty is zero throughout Sβ , leaving value maximization free to determine which supported behaviors to favor. Outside the region, it penalizes the squared distance to the nearest supported action, allowing departures when their predicted value gains justify the cost. This geometry retains guidance from the prior without imposing a hard boundary on policy improvement. Geometric Interpretation of GSBR The ideal penalty also admits a Wasserstein interpretation. Let P2 (Sβ ) denote the set of distributions with finite second moments supported entirely within Sβ . For an actor distribution πθ (· | s) with finite second moment, h i 2 Ea∼πθ (·|s) d(a, Sβ ) = inf W22 (πθ (· | s), ν) . (6) ν∈P2 (Sβ )

The expected penalty is thus the squared Wasserstein distance from the actor distribution to the set of distributions supported in (Sβ ), that is, the minimum quadratic transport cost required to move the actor’s probability mass into the support. Because ν can be any distribution on the support, the penalty constrains where the actor places its mass but not how it divides that mass among modes, which is the formal counterpart of mode selection.

4.2 Approximating support with reference-action neighborhoods Computing ℓsupp requires the support Sβ (s), which is unavailable when the base policy provides only samples. We therefore estimate the support from the reference set B. However, the reference set alone is a poor estimate: it leaves gaps between plausible actions, and penalizing distance directly to this set would pull the actor toward individual samples. We address this sampling sparsity by introducing a support ball around each reference action, with an adaptive radius determined by the observed spaces among sampled actions (Figure 2). Given B = {b1 , . . . , bN }, we compute each reference action’s nearest-neighbor distance, and define the support radius as: ui := min ∥bi − bj ∥2 , rB := Quantileq {ui }N i=1 j̸=i

The distance ui measures the local spacing of reference actions within the same mode or between adjacent action modes; the quantile q ∈ (0, 1) is a hyperparameter that filters out large distances of isolated action samples. The estimated support is the union of closed balls of radius rB centered around each reference action: N n o [ (7) C := B(bi , rB ) = a ∈ A : min∥a − bi ∥2 ≤ rB . i

i=1

Given this estimated support region C, we approximate the geometric relaxation of the support constraint as h i2 ℓGSBR (a, B) := d(a, C)2 = min ||a − bi ||2 − rB (8) i

GSBR: Geometric Support Behavior Regularization for Online RL

(a) Sample reference actions

(b) Estimate local spacing

Isolated samples do not affect rB

Base policy πβ (· | s)

Local spacings determine rB

Optimal action

d(a, C)

Estimated support region C

Figure 2. GSBR. To implement the geometric support regularizer on a generative robot policy, draw samples from πβ (· | s) and compute the nearest-neighbor distance between them. Set rB to a quantile over these distances and construct the support region C as the union of balls of radius rB around each sample. The behavior regularizer loss is computed as the Euclidean distance to the closest point in C.

where [x]+ = max(x, 0). We train the actor with Equation 4 by setting ℓbeh = ℓGSBR . Because ℓGSBR depends on B only through the nearest reference, an actor that observes all of B can move to any sampled mode at zero cost. This distinguishes GSBR from regularizers that tie each action to a single sampled reference action (Park et al., 2025b; Xu et al., 2026), which retain the base policy’s relative mode frequencies. Approximation Guarantee. The following theorem shows that ℓGSBR + 2Dr upper bounds the true loss ℓsupp ; their expected difference is bounded and approaches zero as the number of reference samples grows. Theorem 4.1 (Expected approximation error of the GSBR loss). Assume the action space A is compact with diameter D, and that πβ (a | s) has continuous density. Fix any a ∈ A, and define our reference action set B as above. Then, for any radius r > 0, define a λr > 0 as:

λr := inf Pr ∥b − c∥2 ≤ r

c∈Sβ b∼πβ

representing the worst-case probability over our support set that a sampled action from πβ falls within distance r of a given action in Sβ . The following bounds then hold: ℓsupp (a) ≤ ℓGSBR (a, B) + 2Dr, 2

EB [|ℓGSBR (a, B) − ℓsupp (a)|] ≤ 2Dr + D (1 − λr ) .

(9) (10)

The upper bound in Equation 9 also holds with r = rB where rB is shown above as the quantile of the nearest-neighbor distances between base policy samples. Further, with this choice of radius, rB becomes arbitrarily small as N increases, and the approximation converges uniformly over the action space in expectation: EB sup |ℓGSBR (a, B) − ℓsupp (a)| −→ 0 as N → ∞. (11) a∈A

The full proof is in Appendix A.2. The term D2 (1 − λr )N accounts for the event that no reference lies within radius r of the in-distribution action nearest to a, and it vanishes as N → ∞; The term 2Dr bounds the error caused by subtracting the quantile radius. We note that for generative policies, such as flow matching, they theoretically assign nonzero likelihood to every action (Lipman et al., 2023). Instead, we can define our constraint with respect to some πϵ derived through filtering for actions such that πβ (a|s) ≥ ϵ, but we found this was not necessary in practice.

GSBR: Geometric Support Behavior Regularization for Online RL

Algorithm 1 Online RL with GSBR Require: πβ , Doff , N , q, α. 1 Initialize πθ , Qϕ , Qϕ̄ ,and Don . 2 for each environment interaction do 3 Sample B ∼ πβ (· | s)N ; execute a ∼ πθ (· | s, B) and store the transition in Don . 4 for each gradient update do 5 Sample minibatch (s, a, r, s′ ) ∼ Doff and Don . Train the critic 6 Sample B′ ∼ πβ (· | s′ )Nh and a′ ∼ πθ (· | s′ , B′ ). 2 i 7 Update ϕ to minimize E Qϕ (s, a) − r − γQϕ̄ (s′ , a′ ) . Train the actor with geometric support regularization N 8 Sample B = {bi }N and aθ ∼ i=1 ∼ πβ (· | s) πθ (· | s, B). rB ← Quantileq {minj̸=i ∥bi − bj ∥2 }N i=1 . 10 Update θ to minimize Ebatch −Qϕ (s, aθ ) + α [mini ∥aθ − bi ∥2 − rB ]2+ . 11 end for 12 end for 9

4.3 Practical Implementation Algorithm 1 summarizes GSBR. Training alternates between environment interaction and gradient updates. Training always samples from a replay buffer D that mixes online and offline transitions. The learned actor receives the state and a set of reference actions B as inputs. The algorithm has hyperparameters: N - number of sample actions, q - quantile that affects the size of the radius, and α - that controls the strength of the behavior regularizer.

5 Experimental Results In this section, we empirically evaluate the performance of GSBR against previous online RL methods that leverage a frozen base policy across benchmarks covering both state and pixel inputs with MLP and VLA base policies. Throughout our experiments, we answer the following questions: (1) Is GSBR an effective behavior regularizer for online RL, (2) how sensitive is GSBR to the choice of support radius and number of samples, and (3) how well does GSBR estimate the true support region? 5.1 Experimental Setup Benchmarks. We evaluate all methods on 19 tasks across four benchmarks, each having different properties that may change the distribution of base policies. In RoboMimic (Mandlekar et al., 2022), we select four challenging tasks: can-mh, square-mh, transport-mh, and tool-hang-ph. Each task contains demonstrations collected by human operators of various proficiency levels and captures challenges like precise alignment and bimanual coordination. We test five variants of the cube-triple-task{1-5} in OGBench (Park et al., 2025a). These long-horizon tasks contain a play-style exploration dataset that may lead to a base policy with many action modes and test the RL actor’s ability to drop bad actions while exploring better ones. The recently released benchmark OCBench (Park & Levine, 2026) provides a more realistic dataset that emulates the statistical properties of human demonstrations. We select block-lite-double-task{1-2} and chamber-lite-easy-task{1-3}. In these three benchmarks, the base policy is an MLP behavior cloning policy trained on only the task-specific dataset, and both the base and RL policies receive state information from the simulator. To test each method on more realistic settings, we also experiment on LIBERO (Liu et al., 2023) with the fine-tuned π0.5 -LIBERO base policy using pixel inputs. We hand-picked five tasks where the base VLA leaves room for improvement: task{14, 29, 54, 60, 64}. On these image-based tasks, the RL network trains a CNN from scratch. Since the π0.5 policy is slow to sample from, we cache a set of reference actions and sample from these instead of sampling from the policy at each training step.

GSBR: Geometric Support Behavior Regularization for Online RL

Normalized throughput

Figure 3. Benchmarks. We evaluate our method across the four benchmarks above. From left to right, the tasks are RoboMimic tool-hang, OGBench cube-triple-task4, OCBench chamber-lite-easy-task3, and LIBERO task29. RoboMimic (4 tasks)

OGBench (5 tasks)

OCBench (5 tasks)

LIBERO (5 tasks)

All tasks (19 tasks)

1.00 0.75 0.50 0.25 0.00 Base Policy

TD3+BC

RLT

Residual Policy

BFN

DSRL

EXPO

DiceRL

GSBR (Ours)

Figure 4. Normalized throughput across benchmarks. GSBR achieves the highest average normalized throughput across all 19 tasks and leads all baselines on RoboMimic, OGBench, and OCBench. Throughput (success rate / mean episode length) is normalized by the best method’s seed-average value per task, then averaged across tasks. Error bars show SEM across seeds.

Baselines. We compare GSBR with seven online RL baselines. TD3+BC (Fujimoto & Gu, 2021) trains a state-conditioned actor with a Q-maximization objective and a squared regression penalty to base policy samples. For expressive base policies, TD3+BC regularizes the action to the mean of the reference action distribution. RLT (Xu et al., 2026) conditions a Gaussian chunk actor on a single sampled base policy chunk and penalizes deviation from that specific reference. While the conditioning alleviates the mode averaging problem, RLT cannot drop action modes. Residual Policy (Silver et al., 2018; Johannink et al., 2019) scales and adds a learned correction to a sampled base policy action, so its deviation from the base policy proposal is limited. We follow the formulation of Dong et al. (2026a) and learn an action chunked residual policy conditioned on the base policy input. BFN (Li et al., 2025) does not learn an explicit actor, instead formulating it as the base policy sample that maximizes the Q value. This is analogous to a hard support constraint since the implicit actor cannot generate actions outside the base policy’s support. DSRL (Wagenmaker et al., 2025) learns to steer the frozen base policy through its latent noise rather than directly producing corrected actions, also limiting the possible actions to the support of the base policy. We use the DSRL-NA variant for the state-based tasks and the DSRL-SAC variant for LIBERO. EXPO-FT (Dong et al., 2026a) combines bounded residual edit policy with best-of-N sampling at test-time to achieve mode selection. For a fair comparison, we freeze the base policy. DICE-RL (Sun & Song, 2026) learns latent-conditioned residual corrections to base policy samples, applies selective BC regularization on the residual policy, and applies best-of-N sampling. All methods use the same offline data, online interaction budget, network size, and architecture. They also share training settings, including the action-chunk horizon, discount factor, and critic configuration. For each task, we evaluate each policy for 50 episodes and report the mean and standard error of the mean (SEM) across three seeds. 5.2 Results Is GSBR an effective behavior regularizer for online RL? GSBR demonstrates strong empirical performance across benchmarks compared to the baselines. As shown in Figure 4 and Table 1, across

GSBR: Geometric Support Behavior Regularization for Online RL

Method

RoboMimic (4 tasks)

OGBench (5 tasks)

OCBench (5 tasks)

LIBERO (5 tasks)

All tasks (19 tasks)

TD3+BC RLT Residual Policy BFN DSRL EXPO DiceRL

96 ± 2 96 ± 2 94 ± 2 86 ± 2 97 ± 1 97 ± 1 95 ± 1

73 ± 1 90 ± 5 68 ± 10 69 ± 5 67 ± 5 80 ± 1 40 ± 1

98 ± 0 98 ± 0 95 ± 0 88 ± 2 87 ± 2 98 ± 1 97 ± 1

54 ± 2 86 ± 2 78 ± 1 66 ± 1 81 ± 2 82 ± 2 74 ± 1

79 ± 1 92 ± 1 83 ± 2 77 ± 1 82 ± 2 89 ± 1 75 ± 1

GSBR (Ours)

98 ± 0

99 ± 1

99 ± 0

85 ± 1

95 ± 0

Base Policy

Table 1. Average success rate (%) across benchmarks. Uncertainty is SEM across three seed-wise task averages. Values are rounded to the nearest integer; bold marks the highest unrounded mean in each column. Base Policy SEM is omitted because complete per-seed evidence is unavailable.

19 tasks, GSBR achieves 95% success rate and 0.95 normalized throughput, while the next-best baseline, RLT, achieves 92% success rate and 0.77 normalized throughput. On RoboMimic, while most methods have a similar success rate at around 95%, GSBR achieves the highest throughput, surpassing the next-best method, DSRL, by 20%. On OGBench, GSBR achieves a near-perfect success rate across all five tasks, which is about 9 percentage points higher in success rate and 33% (relative) higher throughput compared to RLT, the strongest baseline. On OCBench, most methods achieve a near-perfect success rate yet GSBR has approximately 13% higher throughput than TD3+BC. Further, we evaluate the performance of our behavior regularizer on LIBERO with a modern VLA as the base policy. Using π0.5 -LIBERO, the strongest baseline is DSRL in terms of throughput at 0.84 and RLT in terms of success rate at 86%. GSBR achieves 85% success rate and 0.86 normalized throughput, closely matching or beating the best baselines on either metric. These results demonstrate that GSBR provides an effective behavior regularizer that empirically outperforms baselines in success rate and throughput, indicating that our regularizer allows the maximum deviation from the base policy while providing stabilizing guidance. For additional experimental detail and results, refer to Appendix A.4.1 and A.4.2 How sensitive is GSBR to the choice of support radius and number of samples? We sweep over a range of N , the number of base policy samples used to compute the behavior regularizer loss on cube-triple (Table 2) and find little effect on the performance of GSBR. The normalized throughput for N = 8, N = 16, and N = 32 is 1.0, 0.91, and 0.93, respectively. These results suggest that as few as eight samples from the base policy suffice for effective behavior regularization on these tasks. We also ablate the effect of computing the support radius rB (Table 3). With rB = 0, the throughput degrades significantly to only 0.66. This ablation supports the benefit of leaving neighborhoods around reference actions unpenalized, unlike in other prior works, allowing the actor to refine its actions within these neighborhoods without being pulled toward individual samples.

Reference count N

Normalized TP

8 (Ours) 16 32

1.00 ± 0.05 0.91 ± 0.02 0.93 ± 0.02

Table 2. Reference-count ablation. On cube-triple, the number of base policy samples, N , has little effect on performance.

Support radius ablation

Normalized TP

rB = 0 GSBR (Ours)

0.66 ± 0.07 1.00 ± 0.05

Table 3. Support-radius ablation. Removing the support radius, rB , significantly reduces normalized throughput on cube-triple.

GSBR: Geometric Support Behavior Regularization for Online RL

N=8

N = 16

N = 32

N = 64

N = 128

q = 0:9

N=3

Figure 5. Visualization of the true and estimated support region on cube-triple. The estimated support region (in bronze) closely matches the true support using only 8 samples. The RL actor learned an action just outside the support region (marked with star).

How well do we approximate the true support region? Figure 5 shows the empirical support of the base policy visualized on a bimodal state in cube-triple. We visualize the first two action dimensions using q = 0.9, and the zero-penalty estimated support region C is marked in golden bronze. We estimate the true support of the base policy with KDE cutoff at 95% from 5,000 basepolicy samples. With only N = 3 references, the sampled actions cover one mode and miss the other. At N = 8, the zero-penalty region already captures both modes while leaving the low-density gap between them penalized. As N increases, the support radius shrinks as the local spacing between samples becomes tighter. On this particular state, the trained RL actor predicts an action slightly beyond the true support, which incurs little loss as it is close to the support region.

6 Conclusions Foundation models offer an appealing behavior prior for sample-efficient online RL, which raises the question of how to regularize an actor toward such a prior while still allowing it to improve. We introduce GSBR, a geometric support behavior regularizer that provides sufficient guidance without overconstraining policy improvement. We provide a theoretically motivated implementation that estimates the ideal loss using base policy samples. GSBR demonstrated strong empirical performance: Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the best prior method. One limitation of the method is that it uses multiple base policy samples at test-time, which may impose compute overhead and introduce more inference latency. Further, GSBR introduces an extra hyperparameter, the quantile q, compared to other common methods. Although we tuned q for the best performance, fortunately, Figure 7 shows reasonable performance across the full range of possible values. Validating GSBR on real robots is a natural next step. Future work could also explore alternative ways to construct the support radius that could improve performance or reduce hyperparameter count.

Acknowledgments This research was supported by ONR N00014-25-1-2060. We would like to thank Seohong Park, Louis McConnell, Chongyi Zheng, and Zhiyuan Zhou for helpful discussion and feedback on the paper; we thank Jagdeep Bhatia and Qiyang Li for providing code prototype and early discussion. We thank the Google TPU Research Cloud (TRC) program for providing TPU compute resources. This research also used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley.

Author Contributions CX led the project, developed the core idea and method, designed and conducted experiments, and led manuscript writing and figure preparation. JB developed the theory and experimental analysis, helped iterate the method, contributed to the training infrastructure, conducted experiments, and

GSBR: Geometric Support Behavior Regularization for Online RL

contributed to writing. LK and SL provided valuable guidance and feedback throughout the project and contributed to manuscript writing and revision.

References Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torne, and Pulkit Agrawal. From imitation to refinement—residual RL for precise assembly, 2024. URL https://arxiv.org/abs/2407. 16677. Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 1577–1594. PMLR, 2023. URL https://proceedings.mlr.press/v202/ ball23a.html. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π0 : A vision-languageaction flow model for general robot control, 2024. URL https://arxiv.org/abs/2410.24164. Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model, 2021. URL https://arxiv.org/abs/2101.05982. Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. ConRFT: A reinforced fine-tuning method for VLA models via consistency policy, 2025. URL https: //arxiv.org/abs/2502.05450. Perry Dong, Kuo-Han Hung, Tian Gao, Dorsa Sadigh, and Chelsea Finn. EXPO-FT: Sample-efficient reinforcement learning finetuning for vision-language-action models, 2026a. URL https:// arxiv.org/abs/2605.25477. Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. EXPO: Stable reinforcement learning with expressive policies. In International Conference on Learning Representations, 2026b. URL https://arxiv.org/abs/2507.07986. Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing-Yi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. In International Conference on Learning Representations, 2025. URL https:// arxiv.org/abs/2405.20555. Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems, 2021. URL https://arxiv.org/abs/ 2106.06860. Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 2052–2062. PMLR, 2019. URL https://proceedings.mlr.press/v97/fujimoto19a. html. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861–1870, 2018. URL https://proceedings.mlr.press/v80/ haarnoja18b.html. Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies, 2023. URL https: //arxiv.org/abs/2304.10573.

GSBR: Geometric Support Behavior Regularization for Online RL

Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout Q-functions for doubly efficient reinforcement learning. In International Conference on Learning Representations, 2022. URL https://arxiv.org/abs/2110.02034. Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learning, 2023. URL https://arxiv.org/abs/2311.02198. Physical Intelligence, Bo Ai, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Greg Balke, Kevin Black, George Bokinsky, Shihao Cao, Thomas Charbonnier, Vedant Choudhary, Foster Collins, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Maitrayee Dhaka, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachlan Groom, Haroun Habeeb, Hunter Hancock, Karol Hausman, Gashon Hussein, Victor Hwang, Brian Ichter, Connor Jacobsen, Szymon Jakubczak, Rowan Jen, Tim Jones, Gregg Kammerer, Ben Katz, Liyiming Ke, Mairbek Khadikov, Chandra Kuchi, Marinda Lamb, Devin LeBlanc, Brendon LeCount, Sergey Levine, Xinyu Li, Adrian Li-Bell, Vladislav Lialin, Zhonglin Liang, Wallace Lim, Yao Lu, Enyu Luo, Vishnu Mano, Nandan Marwaha, Aikys Mongush, Liam Murphy, Suraj Nair, Tyler Patterson, Karl Pertsch, Allen Z. Ren, Gavin Schelske, Charvi Sharma, Baifeng Shi, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, Will Stoeckle, Jiaming Tang, Jimmy Tanner, Shalom Tekeste, Marcel Torne, Kyle Vedder, Quan Vuong, Anna Walling, Haohuan Wang, Jason Wang, XuDong Wang, Chris Whalen, Samuel Whitmore, Blake Williams, Charles Xu, Sukwon Yoo, Lili Yu, Wuming Zhang, Zhuoyang Zhang, and Ury Zhilinsky. π0.7 : a steerable generalist robotic foundation model with emergent capabilities, 2026. URL https://arxiv.org/abs/2604. 15483. Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In International Conference on Robotics and Automation, 2019. URL https: //arxiv.org/abs/1812.03201. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model, 2024. URL https://arxiv.org/abs/2406. 09246. Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019. URL https://arxiv.org/abs/1906.00949. Qiyang Li and Sergey Levine. Q-learning with adjoint matching, 2026. URL https://arxiv. org/abs/2601.14234. Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=XUks1Y96NR. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URL https://arxiv.org/abs/2210.02747. Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, volume 36, 2023. Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. VLA-RL: Towards masterful and general robotic manipulation with scalable reinforcement learning, 2025. URL https://arxiv.org/abs/2505.18719.

GSBR: Geometric Support Behavior Regularization for Online RL

Jianlan Luo, Zheyuan Hu, Charles Xu, You Liang Tan, Jacob Berg, Archit Sharma, Stefan Schaal, Chelsea Finn, Abhishek Gupta, and Sergey Levine. SERL: A software suite for sample-efficient robotic reinforcement learning, 2024. URL https://arxiv.org/abs/2401.16013. Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 2025. doi: 10.1126/scirobotics. ads5033. URL https://arxiv.org/abs/2410.21845. Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li FeiFei, Silvio Savarese, Yuke Zhu, and Roberto Martı́n-Martı́n. What matters in learning from offline human demonstrations for robot manipulation. In Proceedings of the 5th Conference on Robot Learning, volume 164, pp. 1678–1690. PMLR, 2022. Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic RL: Offline RL and online RL fine-tuning of any class and backbone, 2024. URL https://arxiv.org/abs/2412.06685. Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline RL pre-training for efficient online finetuning. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv. org/abs/2303.05479. Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance, 2024. URL https://arxiv.org/ abs/2410.13816. Seohong Park and Sergey Levine. Behavioral Cloning Mystery. ArXiv, abs/2610.07056, 2026. Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, 2025a. Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267, pp. 48104–48127. PMLR, 2025b. URL https://proceedings.mlr.press/v267/park25f.html. Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Szymon Jakubczak, Rowan Jen, Tim Jones, Ben Katz, Liyiming Ke, Chandra Kuchi, Marinda Lamb, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Yao Lu, Vishnu Mano, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Charvi Sharma, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, Will Stoeckle, Alex Swerdlow, James Tanner, Marcel Torne, Quan Vuong, Anna Walling, Haohuan Wang, Blake Williams, Sukwon Yoo, Lili Yu, ∗ Ury Zhilinsky, and Zhiyuan Zhou. π0.6 : A VLA that learns from experience, 2025. URL https://arxiv.org/abs/2511.14759. Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 28701–28717. PMLR, 2023. URL https: //proceedings.mlr.press/v202/ran23a.html. Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization, 2024. URL https://arxiv.org/abs/2409.00588. Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning, 2018. URL https://arxiv.org/abs/1812.06298.

GSBR: Geometric Support Behavior Regularization for Online RL

Zhanyi Sun and Shuran Song. From prior to pro: Efficient skill mastery via distribution contractive RL finetuning, 2026. URL https://arxiv.org/abs/2603.10263. Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning, 2023. URL https://arxiv.org/abs/ 2305.09836. Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning, 2025. URL https://arxiv.org/abs/2506.15799. Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2208.06193. Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2202.06239. Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2019. URL https://arxiv.org/abs/1911.11361. Wenli Xiao, Haotian Lin, Andy Peng, Haoru Xue, Tairan He, Yuqi Xie, Fengyuan Hu, Jimmy Wu, Zhengyi Luo, Linxi Fan, Guanya Shi, and Yuke Zhu. Self-improving vision-language-action models with data generation via residual RL, 2025. URL https://arxiv.org/abs/2511. 00091. Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, and Liyiming Ke. RL Token: Bootstrapping online RL with vision-language-action models, 2026. URL https://arxiv.org/abs/2604.23073. Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, Ayaan Malik, Kyungmin Lee, William Liang, Nadun Ranawaka, Jiasheng Gu, Yinzhen Xu, Guanzhi Wang, Fengyuan Hu, Avnish Narayan, Johan Bjorck, Jing Wang, Gwanghyun Kim, Dantong Niu, Ruijie Zheng, Yuqi Xie, Jimmy Wu, Qi Wang, Ryan Julian, Danfei Xu, Yilun Du, Yevgen Chebotar, Scott Reed, Jan Kautz, Yuke Zhu, Linxi ”Jim” Fan, and Joel Jang. World action models are zero-shot policies, 2026. URL https://arxiv.org/abs/2602.15922.

A Appendix A.1 Practical Implementation In this section, we discuss some of the practical implementation choices for our method. Action chunking. Both the base policy and RL policy learn over action chunks of H consecutive steps, which vary between three and ten depending on the domain. During rollout and evaluation, we execute the entire action chunk before querying the next one. Reference action conditioning. Following Xu et al. (2026), we condition the RL actor on N reference action samples to provide additional learning signal and guidance. During training, we apply 50% dropout to prevent the actor from copying the reference action. Using demo data. Following Ball et al. (2023), we store separate offline and online replay buffers Doff and Don to maintain a 50/50 sampling ratio between demo and online data. Critic ensemble. We train an ensemble of 10 Q networks {Qϕk }K k=1 and use the following pessimistic target value (12) a′ ) a′ ) − 21 σϕ̄ (s′ , e y = R + γ H m µϕ̄ (s′ , e 14

GSBR: Geometric Support Behavior Regularization for Online RL

where µϕ̄ (s′ , e a′ ) and σϕ̄ (s′ , e a′ ) are the mean and standard deviation over the 10 target Q networks. Here R is the discounted chunk reward (Equation 14), m is the bootstrap mask, and e a′ ∼ πθ (· | s′ , B ′ ) with next-state references B ′ ∼ πβ (· | s′ )N . The chunk discount is γ H for a per-step discount; for LIBERO, we use the configured per-chunk discount directly. The main-text critic objective and Algorithm 1 use single-step notation. A.2 Proof of Theorem 4.1 Theorem 4.1 restated: Assume the action space A is compact with diameter D, and that πβ (a | s) has continuous density. Fix any a ∈ A, and define our reference action set B as above. Then, for any radius r > 0, define a λr > 0 as:

λr := inf Pr ∥b − c∥2 ≤ r

c∈Sβ b∼πβ

EB [|ℓGSBR (a, B) − ℓsupp (a)|] ≤ 2Dr + D2 (1 − λr )N . The upper bound in Equation 9 also holds with r = rB where rB is shown above as the quantile of the nearest-neighbor distances between base policy samples. Further, with this choice of radius, rB becomes arbitrarily small as N increases, and the approximation converges uniformly over the action space in expectation: EB sup |ℓGSBR (a, B) − ℓsupp (a)| −→ 0 as N → ∞. a∈A

Proof. We first show that λr > 0. Since Sβ is compact, it can be covered by finitely many balls of radius r/2 centered in Sβ . Each has positive probability under πβ , because its center belongs to the support of πβ . For every c ∈ Sβ , some covering ball has its center within r/2 of c, so that entire ball lies within distance r of c. Thus λr is at least the smallest probability among these finitely many covering balls, which is positive. Now fix an action a ∈ A and write ρa := d(a, Sβ ),

X := d(a, B),

t := [X − r]+ .

Because ρa = d(a, Sβ ) and B ⊆ Sβ , we have ρa ≤ X. Since ρa ≤ X, X − t = min{X, r} ≤ r, and X + t ≤ 2D, we have ℓsupp (a) − ℓGSBR (a, B) = ρ2a − t2 ≤ X 2 − t2 = (X − t)(X + t) ≤ 2Dr. Thus, adding ℓGSBR (a, B) to both sides shows that the GSBR penalty plus 2Dr upper bounds the support penalty: ℓsupp (a) ≤ ℓGSBR (a, B) + 2Dr. Next for the error bound, since Sβ is compact, there is a closest point b⋆ ∈ Sβ with ∥a − b⋆ ∥2 = ρa . A reference within distance r of b⋆ is also within distance ρa + r of a, by the triangle inequality. Thus, Pr ∥b − b⋆ ∥2 ≤ r ≤ Pr ∥b − a∥2 ≤ ρa + r b∼πβ b∼πβ And using the definition of λr as the lower bound, λr ≤ Prb∼πβ ∥b − a∥2 ≤ ρa + r

Because the references are independent of one another and of a, the probability that all N references miss this neighborhood is at most Pr(X > ρa + r | a) ≤ (1 − λr )N . B

GSBR: Geometric Support Behavior Regularization for Online RL

We now consider a decomposition of the expectation by partitioning on the event that X ≤ ρa + r.

On the event X ≤ ρa + r, subtracting r from both sides gives X − r ≤ ρa , and since ρa ≥ 0, t = [X − r]+ ≤ ρa . Therefore, the support penalty is at least the GSBR penalty, so the bound from above gives |ℓGSBR (a, B) − ℓsupp (a)| = ρ2a − t2 ≤ 2Dr. On the complementary event X > ρa + r, both penalties lie in [0, D2 ], so their absolute difference is at most D2 . Combining the two events gives EB [|ℓGSBR (a, B) − ℓsupp (a)| ] ≤ 2Dr Pr(X ≤ ρa + r | a) + D2 Pr(X > ρa + r | a) ≤ 2Dr + D2 (1 − λr )N .

Finally, consider the quantile radius rB . The one-sided bound above is pointwise in r and B, so it also holds with r = rB . Define the largest distance from the support to the reference set by hN := sup d(c, B). c∈Sβ

For any a ∈ A, using the notation above with r = rB , ρa ≤ X ≤ ρa + hN ,

ρa − rB ≤ t = [X − rB ]+ ≤ ρa + hN .

Thus |t − ρa | ≤ max{hN , rB }. Since both t and ρa lie in [0, D], we have, for every a ∈ A, |ℓGSBR (a, B) − ℓsupp (a)| = |t2 − ρ2a | ≤ 2D max{hN , rB }.

To show that this error vanishes, fix any δ > 0. Cover Sβ by M closed balls of radius δ/2 centered in Sβ . Each ball has probability at least λδ/2 > 0. If every covering ball contains a reference, every support point is within distance δ of a reference. Therefore, Pr(hN > δ) ≤ M (1 − λδ/2 )N . B

The same cover controls the quantile radius. If ui > δ, any covering ball containing bi cannot contain another reference, since the distance between two points in that ball is at most δ. Hence at most M references have ui > δ. For all sufficiently large N , N − M ≥ ⌈qN ⌉, so rB ≤ δ.

For all sufficiently large N , we have rB ≤ δ. On the event hN ≤ δ, the absolute penalty error is therefore at most 2Dδ simultaneously for every a ∈ A. On the complementary event, it is at most D2 . Taking the supremum over actions and then the expectation over reference samples gives EB sup |ℓGSBR (a, B) − ℓsupp (a)| ≤ 2Dδ + D2 Pr(hN > δ) B a∈A (13) 2 ≤ 2Dδ + D M (1 − λδ/2 )N . Let η > 0 be any desired error tolerance. Choose δ > 0 small enough that 2Dδ < η/2. For this fixed δ, D2 M (1 − λδ/2 )N → 0 as N → ∞, thus it must be less than η/2 for all sufficiently large N . The expected uniform error is therefore less than η for all sufficiently large N . Since η > 0 is arbitrary, the expected uniform error converges to zero as N → ∞, proving Equation 11. A.3 Implementation Choices Action chunking. For robotic control, we instantiate the action space using chunks of H consecutive control actions. Each action chunk is flattened into a vector of dimension d = Hda , where da is the dimension of an individual control action. In this implementation, a, b, and the references in B denote action chunks. Training uses H-step replay transitions (s, a, R, s′ ), where a concatenates the recorded control actions at , . . . , at+H−1 , s = st , s′ = st+H and R=

H−1 X

γ h r(st+h , at+h ),

h=0

s = st ,

s′ = st+H .

(14)

GSBR: Geometric Support Behavior Regularization for Online RL

Reference action conditioning. We parameterize the actor as a tanh-squashed Gaussian policy conditioned on s. To reduce the burden on the actor on learning the full distribution of the base policy and provide early-training exploration signal, we adopt the reference action passthrough mechanism from Xu et al. (2026) and additionally condition the actor on all sampled reference actions B. During training, we zero the entire reference input with probability pdrop . In early testing, we found that this mechanism improves initial performance without having to sample from the base policy during deployment. Critic ensembles. To stabilize Q-learning, we train an ensemble of K = 10 Q-functions {Qϕk }K k=1 . We compute the mean and standard deviation of their target predictions as v u K K u1 X X 2 1 ′ ′ ′ ′ ′ ′ Qϕ̄k (s , a ), σϕ̄ (s , a ) = t Qϕ̄k (s′ , a′ ) − µϕ̄ (s′ , a′ ) . (15) µϕ̄ (s , a ) = K K k=1

k=1

For TD learning, we use the pessimistic target and the critic loss is h 2 i . LQ (ϕk ) = E (s,a,R,s′ )∼D Qϕk (s, a) − R − γ H m µϕ̄ (s′ , a′ ) − 12 σϕ̄ (s′ , a′ )

(16)

B′ ∼πβ (·|s′ )N a′ ∼πθ (·|s′ ,B′ )

The discount convention follows Equation 12: for LIBERO, the configured per-chunk discount replaces γ H . For the actor update, we simply take the mean of the Q-values. Buffer sampling ratio. We maintain separate offline and online replay buffers, Doff and Don . Following the symmetric replay sampling strategy of RLPD (Ball et al., 2023), each training minibatch contains equal numbers of H-step sequences from the two buffers. This retains access to prior experience as new interactions accumulate. Rollouts and deployment. During a rollout, we sample reference chunks and obtain an action chunk from the actor without reference dropout. During deployment, we can optionally skip querying the base policy and run the actor with reference actions set to zero. A.4 Training details We describe the shared training protocol for the state-based benchmarks, followed by method-specific settings and the configuration used for LIBERO. Unless specified otherwise, the state-based methods use the protocol below. Our training and evaluations were done on either A5000 GPUs or TPU V5E chips. Common training settings are described in Table 6. We also report the hyperparameter sweep range and the selected values of each method in Table 7 and Table 8. We select the hyperparameter value with priority on average success rate, and favor the configuration with higher average throughput if multiple runs have success rate within 2% of each other. A.4.1 State-based benchmarks Base policies and datasets. For each training seed, we use a seed-matched flow-matching behavior cloning policy trained offline for 106 training steps. The base policy produces action chunks and remains frozen during online RL. On RoboMimic and OGBench Cube Triple, it uses a four-layer MLP with 512 units per layer and 16 Euler integration steps. On the OCBench Block and Chamber tasks, it uses a residual MLP with four 1024-unit layers and 10 Euler integration steps. We clip sampled base-policy actions to the action bounds. For RoboMimic, we use the low-dimensional v1.5 datasets: multi-human (MH) demonstrations for Can, Square, and Transport, and proficient-human (PH) demonstrations for Tool Hang due to it not providing a MH dataset. For OGBench, we use the cube triple play dataset. For OCBench, we use the type 51 dataset shards and normalize each action dimension using its 1st and 99th percentiles.

GSBR: Geometric Support Behavior Regularization for Online RL

Replay and online training. The RL components receive no offline pretraining. We initialize online replay with 50 episodes collected by the base policy, then train with minibatches of 256 sequences, split equally between the offline dataset and online replay. We use two gradient updates per environment step, except for the DICE-RL where we take the max performance between a UTD 2 and the effective UTD used for their results in robosuite. We do not filter transitions by success and use the environment rewards. RoboMimic supplies a binary success reward, and episodes terminate on success; Cube Triple uses the OGBench reward. A.4.2 LIBERO Tasks and observations. We evaluate on LIBERO-90 tasks 14, 29, 54, 60, and 64, with an episode limit of 400 environment steps and action chunks of length H = 10. The frozen base policy is π0.5 with the pi05 libero configuration, which receives images from both cameras and proprioception. The RL actor and critic use CNNs with a 64 × 64 image and an eight-dimensional state as input. Training and evaluation. LIBERO training uses online data only, without demonstrations. Basepolicy reference chunks are cached and reused. Each unsuccessful chunk receives reward −1, and a successful chunk receives reward 0. The discount factor is 0.975 per chunk, and TD targets are not clipped. We train for 1,000 trajectories with 20 gradient updates per action chunk. Method-specific settings. For GSBR, we draw N = 8 reference chunks from the cache and do not apply reference dropout. We use a separate radius for each of the seven action dimensions due to the chunk horizon, with each radius shared across the chunk’s time steps. LIBERO regularizer. For chunks with H = 10 and da = 7, we sample N = 8 references B = {bi }N i=1 . Define !1/2 H 1 X (xt,k − yt,k )2 , (17) ρk (x, y) := H t=1 ji ∈ arg min ∥bi − bj ∥2F , rB,k := Quantileq {ρk (bi , bji )}N (18) i=1 , j̸=i

ℓLIBERO GSBR (a, B) := min i

a 1 X [ρk (a, bi ) − rB,k ]2+ . da

(19)

k=1

We compute quantiles using linear interpolation. Nearest neighbors are selected using full chunks, while each action dimension has its own radius. The minimum selects one reference for all dimensions; the penalty vanishes when all dimension-wise distances to that reference lie within their radii. We use this loss in the actor objective and sweep α and q. A.5 Full results Our full per task results are shown in table 4, and 5, and visualized in figure 6.

GSBR: Geometric Support Behavior Regularization for Online RL

can-mh

Square-mh

Transport-mh

300

100

150

300 150 0 0.0

0.5

0 0.0

1.0

0.5

1.0

Toolhang-ph

TD3+BC RLT Residual Policy BFN DSRL EXPO DiceRL GSBR (Ours) Base Policy

160

RoboMimic 1.0

0.5

0.0

Normalized throughput

Online environment steps (millions)

Cube Triple T1 1200

Throughput (successes/h)

Cube Triple T3

Cube Triple T4

OGBench 1.0

160

150

100

Cube Triple T5

300

200

150

600 0

Cube Triple T2 300

0.5

0.0

Normalized throughput

Online environment steps (millions)

Block T1

Block T2

Chamber T1

600

400

0 0.0

0.5

0 0.0

1.0

0.5

1.0

Chamber T3 600

250

0 0.0

0.5

300

0 0.0

1.0

OCBench 1.0

500

1500

300

200

Chamber T2

3000

0.5

1.0

0 0.0

0.5

1.0

0.0

Normalized throughput

Online environment steps (millions)

LIBERO Task 14

LIBERO Task 29

LIBERO Task 54

LIBERO Task 60

500 150

250 0

500

1000

LIBERO Task 64

LIBERO

300

500

1000

500

600

250

300

500

1000

1.0

150

500

1000

0.5

500

1000

0.0

Normalized throughput

Training episodes

Figure 6. Absolute throughput (successes/h) over training steps Task

TD3+BC

RLT

Residual policy

BFN

DSRL

EXPO-FT DICE-RL GSBR (ours)

73±1 37±1 15±2 50±2

99±0 96±1 88±7 97±0

99±0 96±1 90±6 97±0

97±1 98±0 83±7 97±2

94±2 78±3 81±4 90±2

99±0 98±0 92±2 96±2

99±0 98±0 93±1 95±1

99±0 96±2 84±3 98±0

98±1 99±0 96±1 98±0

100±0 100±0 98±0 99±0 72±7 92±1 91±5 100±0 0±0 58±29

100±0 62±31 80±16 98±1 0±0

100±0 100±0 80±9 98±0 82±4 85±1 66±5 46±23 15±15 4±2

98±1 96±2 96±2 97±1 8±4

100±0 89±5 6±3 1±1 0±0

100±0 99±0 100±0 98±1 95±1

33±5 95±1 95±1 36±2 99±0 98±0 72±3 100±0 100±0 36±5 99±0 96±2 26±3 98±1 98±1

91±1 90±4 100±0 96±2 97±1

92±1 90±2 97±0 76±4 84±5

89±1 95±2 96±2 72±7 80±4

97±1 94±1 100±0 98±1 98±1

93±2 98±0 100±0 94±1 99±0

97±1 98±0 100±0 98±1 100±0

74±4 59±5 93±0 96±3 66±3

30±4 49±2 96±1 80±4 70±2

87±2 70±7 96±2 100±0 51±2

70±5 73±4 94±1 98±0 76±2

30±2 51±4 97±0 100±0 90±3

86±2 60±6 94±2 98±1 84±2

RoboMimic can-mh square-mh transport-mh tool-hang-ph

OGBench cube-triple Task 1 Task 2 Task 3 Task 4 Task 5

0±0 0±0 0±0 0±0 0±0

OCBench Lite Block 1 Block 2 Chamber 1 Chamber 2 Chamber 3 LIBERO Task 14 Task 29 Task 54 Task 60 Task 64

16 52 84 54 40

8±4 46±8 56±3 94±3 66±6

80±2 65±7 95±1 98±0 87±3

Table 4. Final success rate (%) across all tasks. Values are mean ± SEM.

GSBR: Geometric Support Behavior Regularization for Online RL

Task

DSRL

EXPO-FT

DICE-RL GSBR (ours)

389±21 325±6 65±11 112±10

198±10 308±7 113±7 297±6 63±5 104±5 80±7 121±12

317±5 273±3 90±2 143±10

289±6 226±20 52±2 128±6

416±16 348±8 105±2 165±3

1569±18 1566±14 266±3 329±32 64±12 109±7 184±61 334±16 0±0 66±36

838±7 91±47 90±35 175±10 0±0

405±14 798±18 1065±371 1396±10 60±13 172±3 120±10 134±28 63±5 85±3 117±14 2±1 33±4 32±20 110±1 0±0 7±7 1±0 3±2 0±0

1582±8 328±28 262±14 344±57 174±15

32±6 37±2 80±5 28±4 20±2

293±8 327±17 1302±33 406±22 464±20

273±4 340±42 1269±13 316±47 460±47

228±11 184±37 1817±18 252±26 286±22

133±5 137±7 185±19 243±25 161±6 157±9 78±8 73±10 88±6 80±5

229±15 264±3 1805±18 300±2 317±39

152±13 426±42 695±3 180±8 324±11

279±10 364±24 2311±18 388±35 522±32

33 141 213 156 84

17±9 122±28 119±9 548±60 156±17

375±30 209±38 263±6 596±15 255±15

314±42 163±19 256±5 594±44 158±11

70±13 131±11 281±6 287±34 175±9

270±42 242±25 263±11 672±27 193±7

69±6 136±16 281±4 709±19 274±16

470±39 168±28 572±51 630±38 249±11

TD3+BC

RLT

Residual policy

329±6 247±6 80±11 133±3

339±8 273±13 86±11 127±5

0±0 0±0 0±0 0±0 0±0

BFN

RoboMimic can-mh 122±1 square-mh 40±1 transport-mh 7±1 tool-hang-ph 24±2 OGBench cube-triple Task 1 Task 2 Task 3 Task 4 Task 5 OCBench Lite Block 1 Block 2 Chamber 1 Chamber 2 Chamber 3 LIBERO Task 14 Task 29 Task 54 Task 60 Task 64

514±72 263±55 604±43 537±12 118±6

Table 5. Absolute throughput (successes/hour) across all tasks and methods assuming 10 Hz for RoboMimic, OGBench, and OCBench, and 20 Hz for LIBERO. Values are mean ± SEM.

A.6 Hyperparameters We include all hyper parameter details for our method and what we sweep over in table 7, the baseline sweeps in table 8, and the training setup shared parameters in table 6. UTD

TD clip

1M 2M 2M 1M

2 2 2 2

5 5 5 3

0.998 0.998 0.99 0.99

[0, 1] [0, 1] None [0, 1]

1,000 episodes

0.975 per chunk

None

Benchmark

Tasks

Online Env Steps

RoboMimic RoboMimic OGBench OCBench Lite

can, square transport, tool-hang cube-triple-T[1-5] block-double-T[1-2] chamber-easy-T[1-3] Task-{14, 29, 54, 60, 64}

LIBERO

Table 6. Training settings. H is the action chunk length. UTD is per environment step; the discount γ is per environment step except for LIBERO, where it is per chunk.

GSBR: Geometric Support Behavior Regularization for Online RL

Task RoboMimic can-mh square-mh transport-mh tool-hang-ph OGBench cube 1 cube 2 cube 3 cube 4 cube 5 OCBench Lite block 1 block 2 chamber 1 chamber 2 chamber 3 LIBERO LIBERO 14 LIBERO 29 LIBERO 54 LIBERO 60 LIBERO 64

BC weight α

Quantile q

{0.1, 0.3, 1, 3, 10} {3, 10, 30} {0.1, 0.3, 1, 3, 10, 30} {3, 10, 30}

{0.1, 0.3, 0.5, 0.7, 0.9} {0.7, 0.9} {0.1, 0.3, 0.7, 0.9} {0.1, 0.3, 0.5, 0.7, 0.9}

{1, 3, 10} {3, 10, 30, 100} {30, 100, 300, 1000, 3000, 10000} {3, 10, 30, 100, 300, 1000} {100, 300, 1000}

{0.5, 0.7, 0.9} {0.1, 0.3, 0.5, 0.7, 0.9} {0.7, 0.9} {0.7, 0.9} {0.1, 0.3, 0.5, 0.7, 0.9}

{0.1, 0.3, 1, 3, 10, 30} {0.1, 0.3, 1, 3, 10} {0.01, 0.03, 0.1} {0.01, 0.03, 0.1, 0.3, 1, 3} {0.01, 0.03, 0.1, 0.3, 1}

{0.1, 0.3, 0.5, 0.7, 0.9} {0.1, 0.3, 0.5, 0.7, 0.9} {0.1, 0.3, 0.5, 0.7, 0.9} {0.1, 0.3, 0.5, 0.7, 0.9} {0.1, 0.3, 0.5, 0.7, 0.9}

{30, 100, 300} {10, 30, 100, 300, 1000} {30, 100, 300} {30, 100, 300} {100, 300, 1000, 3000}

{0.1, 0.5, 0.9} {0.1, 0.5, 0.9} {0.1, 0.5, 0.9} {0.1, 0.5, 0.9} {0.1, 0.5, 0.9}

Table 7. GSBR Hyperparameter sweep range.

GSBR: Geometric Support Behavior Regularization for Online RL

Method

Tasks

DICE-RL can-mh square-mh transport-mh tool-hang-ph cube 1 cube 2 cube 3 cube 4 cube 5 block 1 block 2 chamber 1, 3 chamber 2 LIBERO 14, 29, 54 LIBERO 60, 64 EXPO-FT can-mh; LIBERO 64 square-mh; cube 2 transport-mh tool-hang-ph; cube 3–5 cube 1; chamber 1–2 block 1–2; chamber 3; LIBERO 29 LIBERO 14, 60 LIBERO 54 Residual can-mh; cube 1–4; chamber 2–3 square-mh transport-mh; tool-hang-ph cube 5 block 1; chamber 1 block 2; LIBERO 64 LIBERO 14, 29, 60 LIBERO 54 DSRL can-mh; square-mh; block 2; chamber 1–3 transport-mh; cube 3, 5; LIBERO 60, 64 tool-hang-ph; cube 1–2, 4; block 1; LIBERO 14, 29, 54 RLT can-mh square-mh; transport-mh; tool-hang-ph cube 1 cube 2, 4 cube 3, 5 block 1; chamber 1–3 block 2 LIBERO 14, 29, 64 LIBERO 54 LIBERO 60 TD3+BC can-mh square-mh; cube 1 transport-mh tool-hang-ph cube 2, 4 cube 3 cube 5 block 1; chamber 1–3 block 2 LIBERO 14, 29, 64 LIBERO 54 LIBERO 60

Hyperparameter sets α ∈ {3, 10, 30, 100, 300}; u ∈ {0.3125, 2} α ∈ {3, 10, 30, 100, 300}; u ∈ {0.3125, 2} α ∈ {3, 10, 30, 100, 300}; u ∈ {0.3125, 2} α ∈ {3, 10, 30, 100, 300}; u ∈ {0.3125, 2} α ∈ {1, 3, 10, 30, 100, 300}; u ∈ {0.3125, 2} α ∈ {1, 3, 10, 30, 100, 300}; u ∈ {0.3125, 2} α ∈ {1, 3, 10, 30, 100, 300}; u ∈ {0.3125, 2} α ∈ {1, 3, 10, 30, 100, 300}; u ∈ {0.3125, 2} α ∈ {1, 3, 10, 30, 100, 300}; u ∈ {0.3125} α ∈ {0.3, 1, 3, 10, 30, 100}; u ∈ {0.3125, 2} α ∈ {0.3, 1, 3, 10, 30, 100}; u ∈ {0.3125, 2} α ∈ {0.3, 1, 3, 10, 30, 100}; u ∈ {0.3125, 2} α ∈ {0.3, 1, 3, 10, 30, 100}; u ∈ {0.3125, 2} α ∈ {10, 30, 100, 300}; u ∈ {2} α ∈ {10, 30, 100, 300}; u ∈ {2} s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3} s ∈ {0.01, 0.03, 0.1, 0.3} s ∈ {0.01, 0.03, 0.1, 0.3} s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3} s ∈ {0.01, 0.03, 0.1, 0.3} s ∈ {0.01, 0.03, 0.1, 0.3} (all tied) s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3, 1} s ∈ {0.01, 0.03, 0.1, 0.3, 1} b ∈ {1, 1.5, 2} b ∈ {1, 1.5, 2} b ∈ {1, 1.5, 2} α ∈ {0.03, 0.1, 0.3, 1, 3} α ∈ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000} α ∈ {1, 3, 10, 30, 100, 300, 1000} α ∈ {1, 3, 10, 30, 100, 300, 1000} α ∈ {1, 3, 10, 30, 100, 300, 1000} α ∈ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000} α ∈ {1, 3, 10, 30, 100, 300, 1000} α ∈ {10, 30, 100, 300, 1000} α ∈ {10, 30, 100, 300, 1000} α ∈ {10, 30, 100, 300, 1000} α ∈ {0.03, 0.1, 0.3, 1, 3} α ∈ {1, 3, 10, 30, 100, 300, 1000} α ∈ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000} α ∈ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000} α ∈ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000} α ∈ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000} α ∈ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000} (all tied) α ∈ {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000} α ∈ {1, 3, 10, 30, 100, 300, 1000} α ∈ {10, 30, 100, 300, 1000} α ∈ {10, 30, 100, 300, 1000} α ∈ {10, 30, 100, 300, 1000}

Table 8. Baseline search grids. Bold values are selected. Here s is residual scale, b is the DSRL bound/noise scale, and u is UTD.

GSBR: Geometric Support Behavior Regularization for Online RL

A.7 Sensitivity Analysis We visualize the hyper parameter grid for our method in figure 7. Our method across most tasks under a specific quantile performs similarly across different magnitudes of behavior constraint weights.

Success rate (%)

Throughput (successes/h)

Block Double Lite T1

0.7

0.5

0.3

0.1

0.3

0.1

1 3 Regularizer weight ®

300

Quantile q

100

0.9

258

279

149

144

0.7

287

239

131

150

106

0.5

232

229

154

160

126

0.3

257

265

143

171

134

107

0.1

276

234

129

164

126

0.1

0.3

Block Double Lite T2

1 3 Regularizer weight ®

0.7

0.5

0.3

0.1

0.3 1 3 Regularizer weight ®

400

Quantile q

0.9

178

159

251

281

331

0.7

174

141

186

303

251

0.5

163

182

266

330

286

0.3

122

179

174

303

364

0.1

141

193

208

285

234

0.1

0.3 1 3 Regularizer weight ®

Chamber Easy Lite T1 100

0.7

100

0.5

100

0.3

100

0.01

0.03 Regularizer weight ®

0.1

Quantile q

0.9

1926

1847

1267

0.7

1916

1825

1314

0.5

2033

1654

1282

0.3

1834

1829

1281

0.1

2311

1789

1368

0.01

0.03 Regularizer weight ®

0.1

Chamber Easy Lite T2

1175

Chamber Easy Lite T2

0.7

0.5

0.3

100

0.01

0.03

0.1 0.3 Regularizer weight ®

400

Quantile q

100

0.9

0.1

0.9

205

329

266

317

208

164

0.7

122

288

335

293

201

146

0.5

125

260

364

276

208

156

0.3

144

301

335

305

199

138

388

308

214

144

0.1 0

151

262

0.01

0.03

Chamber Easy Lite T3

0.1 0.3 Regularizer weight ®

0.7

0.5

0.3

0.01

0.1

100

0.03 0.1 0.3 Regularizer weight ®

550

Quantile q

200

Chamber Easy Lite T3 100

0.9

2350

Quantile q

100

0.1

200

Chamber Easy Lite T1 100

0.9

Block Double Lite T2 100

0.9

0.1

150

0.9

159

313

485

394

320

0.7

262

290

447

393

312

0.5

297

366

522

418

295

0.3

183

371

417

374

306

119

497

439

418

277

0.01

0.03 0.1 0.3 Regularizer weight ®

0.1 0

275

Selected within displayed grid

Figure 7. This figure demonstrates how the success rate, and throughput vary across different hyper parameter configs. Our method performs well across multiple magnitudes of BC weights.

GSBR: Geometric Support Behavior Regularization for Online RL

Analyze & explore related research in Conceptio

Related documents