arXiv CS open access

Rose: Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

Wen Zeng et al. · arxiv_cs

arXiv CS · Papers

cloud distributed-computing parallel-computing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Wen Zeng

Ke Hong

Xu Fu

Tsinghua University, Infinigence AI zengw25@mails.tsinghua.edu.cn

Tsinghua University

Infinigence AI

Zixiao Huang

Zehao Wang

Zhuoran Zhu

Tsinghua University

Yuanqing Wang

Bo Dai

Zhen Guo

Infinigence AI

Hao Lin

Chunyang Zhu

Boxun Li

Tsinghua University

Infinigence AI

Chao Yu

Quanlu Zhang

Guohao Dai

Tsinghua University

Infinigence AI

Shanghai Jiaotong University

Zhenhua Zhu

Yu Wang✉

Tsinghua University

Tsinghua University Embodied RL

Abstract Simulator

Embodied reinforcement learning (RL) coordinates policy inference, simulation, and model training with highly heterogeneous resource demands. Although individual components frequently underutilize GPU compute during execution, existing systems only overlap work at coarse component boundaries (e.g., pipelining inference and simulation across simulator groups), leaving substantial lowactivity GPU windows within subcomponent phases and nested loops. Exploiting these fine-grained windows manually requires tedious dependency analysis and workload-specific coordination. Furthermore, even when independent work is scheduled concurrently, cross-process GPU time slicing and streaming multiprocessor (SM) contention can negate the benefits of overlap. We present Rose, a runtime-aware system that automates finegrained overlap and GPU resource management for embodied RL while preserving synchronous training semantics. Through a lightweight programmable interface that exposes execution hierarchy and state accesses, Rose tracks runtime data readiness and GPU activity to perform a two-stage optimization. First, a window-guided graph optimizer identifies low-activity windows and applies graph transformations to overlap complementary work across components and loop levels. Second, leveraging NVIDIA Multi-Process Service (MPS) for cross-process GPU concurrency, a slack-aware resource manager tunes process-level SM quotas by exploiting branch synchronization slack to mitigate interference on the critical path. Both stages combine local profiling with end-to-end iteration validation to guarantee throughput gains with minimal tuning overhead. Across diverse policies, simulators, RL algorithms, and GPU platforms, Rose achieves 1.20–1.92× training throughput improvement and increases average SM activity by 23.8–39.3% compared with the state-of-the-art systems.

Rollout

Policy Training

Trajectory Buﬀer

Heterogeneous Components Simulator CPU

CPU-based Simulation

Policy FLOP/s

arXiv:2611.02705v1 [cs.DC] 2 Nov 2026

Rose: Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

Action Head

Backbone

Memory-bound Compute-bound

Reward Simulator Reward

FLOP/Byte

GPU

GPU-based Simulation

Autoregressive

VLM Reward

Diﬀusion

Figure 1: Embodied RL system and heterogeneous components.

Introduction

Reinforcement learning is increasingly used to train embodied policies for complex robotic tasks. Through trial-and-error interaction with simulators, RL can improve task performance and execution robustness [5, 12, 14]. Training these policies with RL requires coordinating multiple components with heterogeneous resource demands across rollout and policy training. As illustrated in Figure 1, rollout combines policy inference, simulation, and reward computation, while policy training performs model updates. Policy training and backbone inference require substantial GPU computation, while simulation combines physics computation and rendering with CPU and GPU demands that vary across simulators. Moreover, a component’s execution can be divided into phases corresponding to its subcomponents. GPU demand varies considerably within these phases, which leave GPU resources underused during parts of their execution. This variety motivates placing components with complementary resource demands on the same GPUs to share compute resources, a 1

Zeng et al.

placement strategy known as colocation. Existing systems such as RLinf [31] support flexible component placement and pipeline policy inference with simulation across simulator groups to improve utilization for colocation [32]. Nevertheless, profiling RLinf training a 𝜋0.5 [2] policy on LIBERO [13] with this pipeline reveals substantial remaining windows. Device-wide SM activity stays below 50% for 38.8% of the policy-simulator interaction loop execution time and 59.7% of the complete iteration time (Figure 3). Nsight Systems traces locate the interaction-loop windows in the action-head phase at the end of policy inference. These pipelines schedule entire component invocations, leaving internal phases outside the scheduler’s visibility. Our key insight is that an independent phase that can exploit a low-activity window may belong to the same component, another component, or a different loop level. Here, independence means the phase can be moved into the target window without breaking data and state dependencies. For example, the backbone phase of one pipeline group may overlap with the action-head phase of another group once the former’s observations are ready. Exploiting such opportunities requires exposing phase boundaries, intermediate-data readiness, and state constraints. However, scheduling independent phases does not by itself ensure concurrent GPU execution. Kernels submitted by separate component processes may still execute through GPU time slicing. NVIDIA Multi-Process Service (MPS) enables concurrent kernel execution across processes [22], but resource contention can offset the benefits of overlap. The resulting slowdowns can affect downstream execution differently. An earlier-finishing branch may have scheduling slack: it can tolerate some slowdown without delaying dependent downstream phases. This slack can guide which process to constrain to reduce interference on the branch that determines downstream progress. Because quotas apply to entire processes, their effects extend beyond individual overlap regions and must be evaluated over the complete iteration. We present Rose, a runtime-aware system for optimizing execution graphs and GPU resource sharing in colocated embodied RL. Through a programmable interface, developers wrap their existing implementations to expose execution regions and loops and annotate state accesses. Rose records invocation-level dependencies, intermediate-data readiness, and annotated state accesses at runtime. The execution hierarchy provides the structure for locating transformation opportunities, while runtime records supply the constraints for checking their legality. Rose uses this information to construct overlap schedules without requiring developers to implement workload-specific scheduling or coordination. Rose performs optimization in two stages. First, it identifies lowactivity windows from GPU activity measurements and uses the execution hierarchy to locate candidate phases and enclosing loops where graph transformations may expose overlap. Generic transformations partition and reschedule the exposed regions while preserving data and state dependencies and training update boundaries. Local profiling checks whether candidates shorten the completion time of affected regions. Promising transformations are composed into candidate graphs. Second, with the selected graph fixed, measured concurrent execution costs and branch slack guide process-level SM quota tuning. Both stages retain changes only when completeiteration measurements confirm an improvement. Candidate limits

and measurement reuse reduce tuning cost. The evaluated policies use fixed inference steps per action chunk without variable-length autoregressive decoding. This regularity supports keeping the selected graph and quotas fixed during subsequent training. We evaluate Rose across models, simulators, algorithms, and GPU platforms. Against the strongest evaluated RLinf configuration and the asynchronous baseline RL-VLA3 , Rose achieves 1.20–1.92× higher end-to-end throughput, measured in simulator steps/s over complete training iterations. It also increases SM activity, averaged over time and across GPUs, by 23.8–39.3%. Our contributions are: • A hierarchical execution model with composable transformations. We combine developer-exposed execution structure and state annotations with invocation-level dependencies and data readiness to support graph transformations that preserve dependencies and training update boundaries. • Window-guided graph optimization. We use low-activity windows and execution hierarchy to discover overlap opportunities, check candidate transformations through local measurements, and select execution graphs through complete-iteration evaluation. • Slack-aware GPU resource optimization. We combine branch slack and measured concurrent execution costs to guide process-level SM quota tuning, using a coarse-to-fine search and accounting for quota effects beyond the targeted overlap region.

2 Background and Related Works 2.1 Embodied Reinforcement Learning Embodied RL combines three main components: (1) policy inference, (2) environment simulation, and (3) policy optimization, denoted as Policy, Simulation, and Training in Figure 2, respectively. The first two components jointly perform rollout collection, i.e., the policy generates actions, while the environment executes them and returns new observations and rewards. Repeated interactions produce trajectories, which policy optimization uses to update the policy toward maximizing rewards. Colocated placement shares GPUs among simulator, policy inference, and training. Disaggregated placement assigns components to separate GPU subsets. Hybrid placement assigns simulation and inference to separate GPU subsets during rollout and reuses GPU resources for training. Simulatorgroup pipelining interleaves inference and simulation across simulator groups. In an asynchronous configuration, components execute asynchronously on separate GPU subsets, allowing rollout and training to overlap.

2.2

Policy Inference.

Policy inference combines a vision-language backbone with an action generation module to map observations and task instructions to actions [2, 3, 9, 10, 20]. Two stages have different GPU demands, and action generation may contain repeated decoding or refinement steps. Fixed output sizes and generation budgets make the execution structure tend to be regular. 2

Rose : Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

2 3

Policy

Onload

S1 S2 S1 S2

P1 P2 P1 P2

Oﬄoad Oﬄoad Onload

Onload

GPU 0

Weight update

GPU 1

GPU 2

GPU 3

GPU 4

GPU 5

GPU 6

GPU 7

100

0 S1 S2 S1 S2 P2

2 3

(a) Collocated 0

Training

SM activity (%)

Onload Onload

Simulator

S1 S2 S1 S2 P1 P2 P1 P2 S1 S2 S1 S2 P1 P2 P1 P2 S1 S2 S1 S2 P1 P2 P1 P2 S1 S2 S1 S2 P1 P2 P1 P2

Oﬄoad Oﬄoad Onload

GPU rank

(b) Disaggregated

Low-activity windows 50 0

100

200

300

400

100

50 0 0.0

0.5

Time(s)

1.0

1.5

Figure 3: SM activity over a complete training iteration with simulator-group pipeline. The lower panel shows policysimulator interaction spread from the upper rollout.

2 3

(d) Asynchronous

Figure 2: Embodied RL execution modes.

2.3

3 Motivation 3.1 Low-Activity Windows Across Execution Levels

Environment Simulation.

Policy optimization performs repeated minibatch updates over collected trajectories [12, 24, 25], involving GPU-intensive forward and backward computation. Trajectory availability and reuse, policyversion requirements, and parameter-update order constrain its overlap with rollout.

Whole-component scheduling leaves GPU resources underused within components and across loop levels, as illustrated in Figure 3. During one group’s action-head execution, another group’s backbone inputs may already be ready. Across rollout-epoch boundaries, training may overlap with environment reset when trajectory data are available and state accesses are independent. Exploiting these fine-grained overlap opportunities raises two key challenges: Challenge-1: Determining legal overlap plans. Reordering must preserve data and state dependencies, the policy versions used for rollout, and training update boundaries. Challenge-2: Selecting beneficial overlap plans. Partitioning and rescheduling introduce multiple candidate graphs. Downstream readiness can affect the benefits of combined transformations, and exhaustive evaluation is costly.

2.5

3.2

Environment simulation combines physics stepping, rendering, reward computation, and episode resets. LIBERO and RoboCasa use CPU physics with GPU rendering [13, 19, 35], while ManiSkill3 and BEHAVIOR-1K support GPU physics and rendering [11, 28]. Environment instances run in parallel, but interaction steps within each instance retain sequential data and state dependencies.

2.4

Policy Optimization.

Related Works

RL Systems. OpenRLHF [6], HybridFlow [26], and ReaL [16] improve RL efficiency through flexible placement and parameter resharding between rollout and training, while RLHFuse [34] and AReaL [4] overlap stages via stage fusion and asynchronous rollout, respectively. RLinf [31] extends such flexibility to embodied RL (Figure 2). These systems choose among manually designed execution modes, whereas Rose derives overlap from the workflow graph. Asynchronous Embodied RL Systems. RL-VLA3 [27] and EBRL [17] modify RL algorithms for asynchronous execution, overlapping simulation, inference, and training, with EBRL further adjusting CPU and GPU quotas at runtime. Such modifications potentially compromise convergence. In contrast, Rose preserves synchronous algorithms and derives overlap via graph transformations, without changing the algorithm’s semantics. Graph-based Execution and Scheduling. Dataflow systems [1, 7, 18] express computation as dependency graphs, and optimizers [8, 30, 33] and schedulers [15, 29] search for semantics-preserving transformations and map task DAGs onto heterogeneous resources. Rose applies these ideas to embodied RL, where nested loops and policy-version dependencies constrain valid transformations.

GPU Concurrency and Resource Interference

GPU time slicing. GPU execution may remain time-sliced even when processes overlap. Figure 4 compares serial and overlapping execution of policy inference and simulation. Without MPS, overlap reduces their joint completion time for LIBERO, but provides no reduction for ManiSkill. For ManiSkill, we observe time-sliced GPU execution across separate CUDA contexts despite process-level overlap, as illustrated in Figure 4(c). Interference under GPU concurrency. NVIDIA MPS enables cross-process GPU concurrency [22], but colocated workloads still share execution resources. Table 1 shows that, with MPS enabled, overlapping policy inference and ManiSkill simulation increases their execution time by 1.1% and 44.4%, respectively, relative to isolated execution. As simulation determines their joint completion time, the resulting speedup is limited to 1.07×. Inference finishes earlier than simulation, leaving branch slack before joint completion. To reduce interference raises the following challenge: Challenge-3: Tuning SM quotas under interference. Quota tuning must balance branch slack, concurrent execution costs, and effects beyond the targeted overlap region. 3

Zeng et al.

Table 1: Policy inference and simulator execution times. The reported speedup is 𝑇serial /max(𝑇inf,overlap,𝑇sim,overlap ).

Policy

Simulator

MPS

Inf. alone (s)

Sim. alone (s)

Serial (s)

Inf. overlap (s)

Sim. overlap (s)

Speedup

LIBERO ManiSkill

Off On

0.568 2.632

0.289 4.981

0.854 7.673

0.627 2.661

0.316 7.192

1.36× 1.07×

7.673 s

No actual overlap

Simulator

Data ﬂow

0.854 s

Serial

0.568 s

2.632 s

123

Separate

0.289 s 0.627 s Overlap

4.981 s Speedup

3.834 s

0.316 s

(a) LIBERO

1 2

3 Context switch

7.871 s

(b) ManiSkill

1 2

Proﬁle or validation

Output Optimal RL execution graph

Wrap with Rose Interface

No speedup

Graph ﬂow

Execution Hierarchy

Original RL implementation

Best SM quota

Data Plane

Runtime

Figure 4: Different execution modes and latency of CPUbased and GPU-based simulators. (a) 𝜋 0.5 and LIBERO, batch=16. (b) 𝜋0.5 and ManiSkill, batch=80. (c) Schematic illustration of time-sliced GPU execution across CUDA contexts without MPS.

State dependencies Execution Metrics

Invocation Hook

Data dependencies

Graph table

Graph registration

5 4

Tracing & Measurement

Validated buﬀer Validated graphs

Validated quotas

Window-guided graph optimizer Window Search

4 Design 4.1 Overview

Graph Transformation

As illustrated in Figure 5, Rose exposes execution hierarchy and forms execution graphs at runtime (§4.2). With the metrics of the execution graph, Rose performs a two-stage optimization:graphbased overlap and slack-aware resource management. The graph optimizer identifies low-activity nodes, searches for complementary work, and transforms the execution graphs to enable overlap (§4.3). With the execution graph fixed, the resource manager tunes the quota of a GPU process with slack to reduce interference between overlapping processes (§4.4).

4.2

Transformed Graphs

Slack-aware resource manager Slack process selection MPS Process 1 Quota: 100%

Process 2 Quota: 50%

SM SM

1 Code wrapping to expose Figure 5: Overview of Rose. ○ 2 The original graph registration. ○ 3 Graph exhierarchy. ○ 4 Collects dependenecution dispatch and measurement. ○ 5 The graph optimizer cies and metrics for optimization. ○ retrieves graphs, transforms them, and registers back. Validated graphs are retained and this may run for multiple 6 The resource manager retrieves the best graph, rounds. ○ tuning the selected process SM quota, and stores the best 7 The buffer outputs final plan for subsequent quota back. ○ RL training.

Programmable Execution Model

Rose provides a hierarchical execution graph representation of existing RL programs through a programming interface. Graph construction combines the exposed structure with state annotations and invocation records to establish dependencies and loop independence, enabling overlap across components and execution levels.

computation and callable entries, while invocation records associate each execution with its parent invocation, loop iteration, data references, and state accesses. 4.2.2 Execution Graph Construction. Rose constructs execution graphs by combining the exposed hierarchy with invocation-level bindings. Invocation recording. Figure 7 shows invocation recording through ExecutionTrace, using the hooks called in Figure 6(a). Each record identifies the structural node, parent invocation, and current iteration. The before hook binds inputs to data references and records state accesses, while after binds outputs. These records link invocations to the hierarchy and state dependency analysis. Data dependencies. Rose uses data references to identify which invocation produces each input, establishing producer-consumer dependencies. For loops, it records outputs at the end of each iteration, allowing consumers to proceed once their required data are ready without waiting for the entire loop to finish. For example, trajectory data from each rollout epoch can become available before the rollout returns the full collection. State dependencies. Beyond explicit inputs and outputs, nodes may access shared objects such as model weights and simulator

4.2.1 Hierarchical Execution Graph Representation. Rose provides ExecutionNode and LoopNode to represent execution as hierarchical execution graphs, as shown in Figure 6(a). An ExecutionNode wraps a code block in a component worker with a callable entry and may expose an internal graph of child nodes, data bindings, and execution constraints. Figure 6(b) illustrates this wrapping for policy code. Parent-child relations form an execution hierarchy rooted at the RL iteration loop, supporting analysis at different granularities. LoopNode extends ExecutionNode: its internal graph describes the loop body, while its iteration count specifies repetition. Loops with conditional termination, such as while loops, are wrapped with ExecutionNode to retain the native loop control. Accordingly, edges represent execution constraints that preserve required data and state dependencies. An invocation is a distinct execution of a node, which recurs across loop iterations or input groups. A node defines the exposed 4

Rose : Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

Complementary nodes

class ExecutionNode: class ExecutionNode: def __init__(self, parent=None, entry=None): def __init__(self, parent=None, entry=None): self.parent, self.entry = parent, entry self.parent, = parent, entry self.childrenself.entry = [] self.children = [] if parent is not None: if parent is not None: parent.children.append(self) parent.children.append(self)

(a) Rewire

def __call__(self, *args, **kwargs): def __call__(self, *args, **kwargs): call = trace.before(self, args, kwargs) call ==trace.before(self, args, kwargs) entry runtime.get_registered_entry(self) entry = runtime.get_registered_entry(self) if entry is not None: if entry is not=None: self.entry entry self.entry = entry result = self.entry(*args, **kwargs) result = self.entry(*args, trace.after(call, result) **kwargs) trace.after(call, result) return result return result

(b) Interleave

Path shortened Path shortened (c) Split & Merge Loop Node Count = 2

class LoopNode(ExecutionNode): class LoopNode(ExecutionNode): def __init__(self, count, parent=None, entry=None): def __init__(self, count, parent=None, entry=None): super().__init__(parent, entry) super().__init__(parent, entry) parent_loop = nearest_loop(parent) parent_loop = nearest_loop(parent) self.loop_level = parent_loop.loop_level + 1 self.loop_level = parent_loop.loop_level + 1 self.count = count self.count = count self.kind = None # Assigned by dependency analysis. self.kind = None # Assigned by dependency analysis.

Spread

Temporal Split

Spatial Split

Merge

Figure 8: Transformation APIs.

(a) Execution Node.

# Wrap native code block or new code for new component def vlm_entry(): Policy: # Wrap native code block or new codeclass for new component def def vlm_impl(): vlm_entry(): ... class Policy: def vlm_compute(self): def ... vlm_impl(): ... def ... vlm_compute(self): def ... denoise_entry(): Wrap with Node def def ... denoise_compute(self): def denoise_impl(): denoise_entry(): ... Wrap with Node def ... denoise_impl(): def denoise_compute(self): ... ... def ... policy_entry(input): def ... policy_compute(self, input): def def policy_impl(input): policy_entry(input): def policy_compute(self, input): x = self.vlm_compute(input) x = vlm(input) def policy_impl(input): x = vlm(input) return x = self.vlm_compute(input) self.denoise_compute(x) x = vlm(input) return denoise(x) x = vlm(input) return self.denoise_compute(x) return denoise(x) return denoise(x) return denoise(x) policy = ExecutionNode(rollout, policy_entry) policy = policy_impl) policy = ExecutionNode(rollout, ExecutionNode(rollout, policy_entry) vlm = ExecutionNode(policy, vlm_entry) policy = ExecutionNode(rollout, policy_impl) vlm vlm_impl) vlm = = ExecutionNode(policy, ExecutionNode(policy, vlm_entry) denoise = ExecutionNode(policy, denoise_entry) vlm = ExecutionNode(policy, vlm_impl) denoise = ExecutionNode(policy, denoise_impl) denoise = ExecutionNode(policy, denoise_entry) denoise = ExecutionNode(policy, denoise_impl)

A simulator reset annotated as refresh indicates that the new epoch’s episode state does not inherit the previous epoch’s state value. If the remaining data and state constraints also permit independent execution, the system identifies the loop as a free loop. A loop with dependencies across iterations is classified as a dependent loop. Loops with conditional termination, such as while loops, are classified as dependent loops without an iteration count. The resulting execution graphs and loop independence information provide the graph optimizer with the execution structure and constraints for reorganization.

(b) Wrap example.

Figure 6: (a) Execution node wrapper. (b) An example of wrapping policy code, can also wrap new component code. class ExecutionTrace: def before(self, node, args, kwargs): call = runtime.new_call_ref() self.calls[call] = CallRecord( node=node, parent_call=runtime.current_call(), iteration=runtime.current_iteration(), inputs=bind_inputs(call, args, kwargs), state=state_accesses(node, args, kwargs), ) runtime.invoke(call) profiler.begin(call) return call

4.3

Window-guided Graph Optimization

Building on the execution graphs, Rose identifies low-activity windows and searches for work that can execute during these windows while respecting data and state dependencies. It constructs overlap schedules through composable graph transformations (Figure 8), uses local measurements to guide candidate selection and composition, and retains only changes validated to reduce end-to-end iteration time. Graph optimization examples are illustrated in Figure 9. 4.3.1 Graph Transformation APIs. Rose provides four execution graph transformation APIs: Rewire, Split, Merge, and Interleave. The transformations reorganize execution graphs while reusing nodes’ callable entries and preserving required data and state dependencies. Rewire. Rewire changes the execution order of existing nodes while preserving required data and state dependencies. It applies to both loop and non-loop nodes whose inputs and state permit the new placement. Split. Split constructs separately schedulable executions of a node or its internal graph, each bound to a data partition or an independent group of loop iterations. Spatial split partitions batched computation into groups with separable data and state while preserving each group’s iteration order. Temporal split organizes independent iterations of a free loop into separate execution graphs while preserving the original batch sizes.

def after(self, call, result): record = self.calls[call] record.profile = profiler.end(call) record.outputs = bind_outputs(result, producer=call) runtime.leave(call)

Figure 7: Execution trace wrapper for invocation recording. state. Rose provides three annotations to describe such accesses: read existing state, update new state from its previous value, and refresh new state without inheriting its previous value. Each annotation applies to a specific state object. The system binds these annotations to invocation records to derive state dependencies. Loop independence. Rose analyzes data and state dependencies across iterations to determine loop independence. For a LoopNode, the classification is assigned by dependency analysis, as indicated by its kind field in Figure 6. 5

Zeng et al.

Rollout

Policy / VLM

Simulator / Reset

Update

Training

Communication

Batch K

Epochs N

Epoch 1

Epochs N

P Epoch 1

S1 S2 S1 S2 S1 S2

P1 P2 P1 P2 P1 P2

S1 S2

Epochs N/2

S1 S2

P1 P2

P1 P2 P1 P2 P1 P2

Split

θ1

S1 A1 O2 P2

Before

A1 O2

S2 O1

S1 O2 A1 P1

Time saved

θ2 θ3 Rollout weights θ₀ stay ﬁxed

A2 P2

After

D2 V2

After

(b) Temporal Split and Rewire across loop nodes

S2 A2 O1

Epoch 3

Rewire inside Policy

Before

Epoch 2 T1

(a) Split and Interleave on loop node O Send obs A Send action

T1 T2 T3

After

Time saved

P1 P2

T1 T2 T3

Epoch 3 Rewire

Epoch 1

S1 S2

P1 P2

Reward

Epoch 3

Epoch 2

Time saved

Temporal Split Batch K

Epoch 2

Data ready

Spatial Split Batch K/2 S1 S2 S1 S2 S1 S2

Denoise

Before

Original

V1 D2

Time saved

Rewire inside Simulator

A2 O1

O2 A1 P1

Before

A2 Time P2

S2 R1

saved

After

time

Time saved

R2 time

(d) Rewire inside non-loop node

Figure 9: Optimization examples. Black arrows indicate execution dependencies. (a) Split simulator-policy interact loop and interleave them. (b) Split training batch loop and rewire it to overlap simulator reset. (c) Rewire to overlap communication inside simulator-policy interaction node. (d) Rewire inside a component node between independent component graphs. Reward computation is a part of complete simulator computation. Merge. Merge collects results from the groups or execution graphs produced by Split to reconstruct the input required by the next unsplit node. Interleave. Interleave staggers nodes across the execution graphs produced by Split to enable overlap while preserving dependencies within each graph. For example, a legal Split of a policy-simulator step produces data groups A and B, and Interleave makes simulator(A) overlap policy(B), while each group retains its policy-beforesimulator ordering.

Candidate search. For each target node 𝑣 selected by this ranking, Rose searches for a candidate node 𝑤 whose work can overlap with 𝑣 through feasible graph transformations. When 𝑣’s parent is a LoopNode, Rose first searches its internal graph and tries Split followed by Interleave. Temporal split uses independent iterations from the parent loop or legal free ancestors, considered from nearest to farthest. If no free loop exists, Rose uses Spatial split. It then searches the loop’s parent’s internal graph in this order: other movable loops, splittable downstream non-loop consumers, and unsplittable non-loop nodes that permit Rewire. When 𝑣’s parent is non-loop, Rose first checks whether 𝑣 can be repositioned, then searches other nodes in the parent’s internal graph. If no feasible candidate exists, it searches the parent’s siblings. Feasibility requires checking transformation applicability, data and state constraints, and estimated capacity. Candidate ordering. Candidate ordering uses node rank 𝜋 (𝑣) followed by increasing duration mismatch 𝛿 (𝑣, 𝑤) = |𝑇 (𝑣) − 𝑇b(𝑤)|. Here, 𝑇 (𝑣) is the mean invocation duration of 𝑣, representing its average window duration, and 𝑇b(𝑤) estimates the candidate’s duration after any required Split. This ordering determines which alternatives receive measurements under the per-node trial budget.

4.3.2 Overlap Candidate Discovery. Rose uses the execution intervals of node invocations as candidate overlap windows. To limit candidate trials, it first ranks nodes by duration and SM activity over these windows, then searches for work that can overlap under data and state constraints, and finally orders feasible candidates by duration matching. Ranking low-activity nodes. For a schedulable node 𝑣, 𝐷 𝑣 is its total invocation duration within one training iteration, and 𝑎¯𝑣 is the time-weighted average SM activity observed over these invocations. Motivated by NVIDIA’s profiling metrics documentation [21], Rose excludes nodes with 𝑎¯𝑣 ≥ 0.8 and prioritizes those with 𝑎¯𝑣 < 0.5 over those with 0.5 ≤ 𝑎¯𝑣 < 0.8. Within each activity tier, Rose ranks nodes by decreasing 𝑆 (𝑣) = 𝐷 𝑣 (1 − 𝑎¯𝑣 ). The resulting rank 𝜋 (𝑣) incorporates both the activity-tier priority and the ordering within each tier. Eligible nodes must also satisfy 𝐷 𝑣 ≥ min{𝜏𝑎 , 𝜏𝑟 𝑇iter }, where 𝜏𝑎 is an absolute duration threshold and 𝜏𝑟 specifies a fraction of the observed iteration time 𝑇iter .

4.3.3 Candidate Evaluation and Graph Composition. Since local improvements do not guarantee reduced iteration time, Algorithm 1 combines local measurements for graph transformation and completeiteration validation to accept or reject the result. 6

Rose : Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

Algorithm 1 Window-guided graph optimization

Algorithm 2 Slack-aware GPU resource optimization

Require: Feasible graph 𝐺 0 ; settings M, 𝝉 Require: Target limit 𝐾 ≥ 1; alternative limit per target 𝐿 ≥ 1 Require: Round limit 𝑅 ≥ 1; improvement threshold 𝜖 ∈ (0, 1) Ensure: Graph 𝐺 selected for subsequent training 1: 𝐺 ← 𝐺 0 2: for 𝑟 = 1, . . . , 𝑅 do 3: 𝑚 ref ← RunTraining(𝐺 ); D ← [ ] 4: for all 𝑣 ∈ TopK (𝐺, 𝑆 (𝑣), 𝝉 ) do 5: Φ𝑣 ← Top𝐿 FeasibleTransform(𝐺, 𝑣); (𝜋, 𝛿 ) 6: if Φ𝑣 = ∅ then continue 7: 𝐴𝑣 ← CommonGraph(𝐺, 𝑣, Φ𝑣 ) 8: 𝑐 0 ← 𝐴𝑣 9: C𝑣 ← {𝑐 0 } ∪ {Transform(𝐴𝑣 , 𝜙 ) : 𝜙 ∈ Φ𝑣 } 10: for all 𝑐 ∈ C𝑣 do 11: 𝑚𝑐 ← MeasureLocal 𝑐 ) 12: end for 13: D [𝑣 ] ← 𝑐 ↦→ 𝑚𝑐 : 𝑐 ∈ C𝑣 14: end for 15: 𝐺 comb ← Combine(𝐺, D, M) 16: if 𝐺 comb = 𝐺 or ¬Valid(𝐺 comb ) then 17: break 18: end if 19: 𝑚 comb ← RunOrReuse(𝐺 comb ) 20: if 𝑚 comb .time ≥ (1 − 𝜖 )𝑚 ref .time then 21: break 22: end if 23: 𝐺 ← 𝐺 comb 24: end for 25: return 𝐺

Require: Fixed graph 𝐺; target process 𝑓 ; comparison graph 𝐴 Require: Initial numerator 𝑘 0 ; quota size 𝑄 Require: Coarse-step multiplier 𝜌 ≥ 2 Ensure: Selected quotas q∗ and measurement 𝑚 ∗ 1: 𝑘𝑏 ← 𝑘 0 2: H ← {𝑘𝑏 ↦→ 𝑡 (𝑘𝑏 ) } {Coarse search} 3: while 𝑘𝑏 > 1 do 4: 𝑘 ← max(1, 𝑘𝑏 − 𝜌 ); H [𝑘 ] ← 𝑡 (𝑘 ) 5: if H [𝑘 ] > H [𝑘𝑏 ] then break 6: 𝑘𝑏 ← 𝑘 7: end while {Fine search around the anchor 𝑘𝑏 } 8: for all 𝑑 ∈ { −1, +1} do 9: 𝑘 ← 𝑘𝑏 + 𝑑 10: while 1 ≤ 𝑘 ≤ 𝑄 and 𝑘 ∉ dom( H) do 11: H [𝑘 ] ← 𝑡 (𝑘 ) 12: if H [𝑘 ] > H [𝑘𝑏 ] then break 13: 𝑘 ← 𝑘 +𝑑 14: end while 15: end for {End-to-end validation} 16: 𝑘 ∗ ← arg min H [𝑘 ] 𝑘 ∈dom(H)

17: (q∗ , 𝑚 ∗ ) ← Validate(𝑘 ∗ ) 18: return (q∗ , 𝑚 ∗ )

MPS oﬀ

Policy

Simulator

3.834 s 7.871 s

MPS on 2.661 s

Little speedup 7.192 s

MPS 91/100

All targets in a round use the same reference graph 𝐺, the complete execution graph of an RL iteration. For each target 𝑣, FeasibleTransform(𝐺, 𝑣) performs the candidate search described above and returns feasible transformation alternatives 𝜙 (𝑣, 𝑤) for overlapping 𝑣 with 𝑤. The retained alternatives share a comparison graph 𝐴𝑣 : the internal graph of the lowest common ancestor enclosing all nodes affected by these alternatives. Transform(𝐴𝑣 , 𝜙) applies a retained alternative to 𝐴𝑣 , producing a candidate local graph 𝑐. Both the transformed graphs and the unchanged graph 𝑐 0 = 𝐴𝑣 are measured over this common comparison graph. Their completion times account for downstream readiness, reconstruction, and synchronization. Combine uses the measured reductions in completion time to greedily compose changes that can be applied together, selecting at most one candidate per target. Combined graphs are ranked by estimated cumulative benefit and checked against the memory settings 𝑀 before complete-iteration validation. A combined graph is accepted only when a valid observation shows a relative reduction in iteration time exceeding 𝜖 compared with 𝐺. RunOrReuse reuses only end-to-end observations of the same complete graph. Each complete-iteration trial advances training and updates the weights. Its execution trace, time, and validity guide subsequent optimization rounds.

5.563 s

More speedup

5.716 s

time

Figure 10: SM quota management and latency of overlapping execution on 𝜋0.5 and ManiSkill. “91/100” denotes the percentage quota of policy and simulator.

4.4

Slack-aware Resource Management

GPU contention can slow overlapping branches unequally while downstream work waits for both. The earlier-ready branch can tolerate some delay without postponing the original synchronization, and this tolerance is its branch slack. Rose tunes the earlier-ready process’s SM quota to reduce interference, using local measurements for quota search and complete-iteration validation for acceptance. For readiness times 𝐶 0 ≥ 𝐹 0 measured from a common origin, the earlier-ready branch has slack 𝜎 = 𝐶 0 − 𝐹 0 . After reducing its process quota, write the resulting readiness times as 𝐶 0 − 𝑥 and 𝐹 0 + 𝑦, respectively. The reduction in synchronization time is Δ𝑇sync = min{𝑥, 𝜎 − 𝑦}. Synchronization advances when the laterready branch accelerates (𝑥 > 0) and the earlier-ready branch’s added delay remains within its original slack (𝑦 < 𝜎), as illustrated in Figure 10. 7

1.40x

1.44x 1.32x

1.40x

1.51x

1.92x 1.42x

1.22x

2.16x

Rose

1.65x

1.80x

1.49x

1.82x

1.30x 1.40x

1.57x

1.48x

1.27x

1.53x

1.37x

1.39x 1.29x

1.46x

1.57x

2.16x

1.85x 1.41x

1.18x

1.60x

RL-VLA 3

Colocated+Pipe GRPO

1.69x

1.34x

1.31x 1.28x

1.45x

1.47x

1.86x 1.41x

1.21x

2.08x

Hybrid+Pipe

1.64x

1.92x

1.43x 1.30x

1.61x

1.56x

1.66x 1.43x

1.92x

Colocated

OpenVLA-OFT

π0.5

OpenVLA-OFT

1.33x

1.27x 1.26x

1.41x

1.74x 1.29x

2.18x

1.52x 1.19x

1.54x

1.67x

1.23x 1.36x

1.61x

1.44x 1.39x

1.28x 1.19x

1.68x

GRPO

1.73x

1.40x

1.41x

1.47x

1.61x

1.82x 1.36x

1.58x

2.39x

(b) A800

1.23x

1.26x 1.39x

GR00T

1.64x

1.33x 1.53x

1.73x 1.34x

2.17x

1.52x 1.24x

1.66x

1.78x

1.27x 1.35x

1.61x

1.49x 1.37x

1.76x

1.33x 1.20x

1.37x 1.48x

π0.5

1.84x

1.42x 1.50x

PPO

100 0

1.20x

1.85x

1.39x

1.42x

1.32x

1.43x

1.58x

1.93x 1.46x

2.17x 1.73x

1.20x 1.59x

1.82x 1.32x

200

2.31x

300

Hybrid PPO

(a) H100

1.29x

400 300 200 100 0

1.92x

LIBERO ManiSkill LIBERO

500 400 300 200 100 0 400 300 200 100 0

ManiSkill

Throughput (steps/s)

Zeng et al.

GR00T

Figure 11: Training throughput. Table 2: GPU execution efficiency on A800. With 𝐺 fixed to the graph selected by Algorithm 1, Rose uses a feasible observation 𝑚 0 under quotas 𝑞 0 to select two overlapping Average SM activity branches on distinct processes sharing a GPU. It tunes only the Model Simulator Baseline Rose earlier-ready process 𝑓 , keeping this target and all other quotas fixed throughout the search. If no eligible pair exists, Rose retains 𝑞 0 . 𝜋0.5 LIBERO 42.45% 59.15% The comparison graph 𝐴 is the internal graph of the lowest common ManiSkill 46.38% 58.01% ancestor enclosing both branches and their synchronization. OpenVLA-OFT LIBERO 54.98% 68.08% Uniform MPS provisioning caps the SM resources available to ManiSkill 43.59% 56.57% client CUDA contexts. The backend provides 𝑄 validated quota levels (§B). 𝑞(𝑘) retains 𝑞 0 except that 𝑓 ’s quota is set to 𝑘/𝑄, for 𝑘 ∈ {1, . . . , 𝑄 }. Algorithm 2 records in 𝐻 the locally measured com80 GB memory, and each node provides 920 GB host memory. Scalpletion time 𝑡 (𝑘) of 𝐴 under (𝐺, 𝑞(𝑘)), covering the same effective ing experiments use 4 and 8 A800 GPUs within one node and 16 work across quotas. GPUs across two nodes, connected via NVLink within nodes and Algorithm 2 first reduces the quota in coarse steps of 𝜌 levels 8×200 Gbps RDMA NICs across nodes. until completion time increases or the minimum quota is reached. Workloads and metrics. We evaluate 𝜋0.5 [2], OpenVLA-OFT [9], It then searches in both directions with unit steps around the best and GR00T N1.6 [20] on LIBERO-Spatial [13] and ManiSkill [28] coarse-search point, held fixed as the anchor; a direction stops when (“PutOnPlateInScene25Main-v3”), using PPO [24] and GRPO [25]. its measured time exceeds the anchor’s. The measured quota with Across systems, we match rollout volumes and policy-update budthe smallest 𝑡 (𝑘) is passed to Validate. Since a process quota also gets using benchmark-recommended configurations. We report affects work outside 𝐴, Validate retains the selected quota only end-to-end throughput (simulator steps/s) and average SM activity if a complete iteration is faster than 𝑚 0 ; otherwise, it returns 𝑞 0 . across GPUs over each iteration. Measurements follow the execution semantics in §4.3.3. Baselines. We compare Rose against four RLinf configurations and RL-VLA3 [27]. The RLinf configurations are explained in section 2 5 Evaluation and denoted as Colocated, Colocated+Pipe, Hybrid, and Hybrid+Pipe. We evaluate Rose in terms of throughput, scalability, optimization effectiveness and overhead, as well as its impact on policy learning. 5.2 End-to-End Performance

5.1

Training throughput. Figure 11 reports throughput for the twelve workload configurations on H100 and A800. (1) H100. Across PPO and GRPO, Rose achieves up to 1.39×, 1.43×, and 1.44× the RLinf throughput with the best configuration on LIBERO for 𝜋 0.5 , OpenVLA-OFT, and GR00T N1.6, respectively. On ManiSkill, the corresponding speedups are 1.20×, 1.27×, and 1.22×. (2) A800. Across PPO and GRPO, Rose achieves up to 1.42×, 1.27×, and 1.33× the RLinf throughput on LIBERO for the same models. On ManiSkill, the corresponding maximum speedups are

Experimental Setup

Rose is built on RLinf [31] with 5K lines of Python code, adding ExecutionNode and ExecutionTrace wrappers, and graph transformation implementation. It applies GPU quotas through MPS provisioning at a granularity of two SMs, as detailed in §B. Testbeds. We evaluate throughput on two 8-GPU platforms: NVIDIA H100 GPUs with Intel Xeon Platinum 8481C CPUs, and NVIDIA A800 GPUs with Intel Xeon Platinum 8358P CPUs. Each GPU has 8

Rose : Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

GPU 0

GPU 1

GPU 2

GPU 3

GPU 4

GPU 5

GPU 6

Colocated

GPU 7

50 0

100

200

300

100 50 0 0.0

0.5

1.0

Time(s)

Hybrid+Pipe

Colocated+Pipe

200 0

Colocated

Number of GPUs (a) LIBERO

π0.5

1.22x

MPS-on

OpenVLA-OFT (a) ManiSkill

1.24x

GR00T

Rose(one round)

1.42x

Rose 1.06x 1.10x

1.10x 1.13x 1.20x

Colocated+Pipe

400 300 200 0

Rose

96 92 88

500 400 300 200 100 4

Graph-only

1.29x

Rose 1.20x

1.33x

1.15x 1.27x

π0.5

OpenVLA-OFT (b) LIBERO

GR00T

Figure 14: Training throughput ablation. Latency/s

Throughput (steps/s)

Hybrid

1.08x 1.12x

1.5

Figure 12: SM activity after Rose optimization over a training iteration. The lower panel is spread from part of the upper rollout. 800 600 400 200

300

Throughput (steps/s)

SM activity (%)

100

π0.5 30-100% + Off

150 110 80

100 Off

Min 84.59 s (95%)

GR00T

89 140 30-100% + Off 110 86 8030 60 100 Off Min 80.40 s (91%) 83

80 89 93 95 97 100 80 89 91 93 95 100

Quota (%) (a) π0.5

Quota (%) (b) GR00T

OpenVLA-OFT 30-100% + Off

128 160 140 120 100 Off 124 30 60 Min 116.96 s (89%) 120 80 84 86 89 91 95 100

Quota (%) (c) OpenVLA-OFT

Figure 15: Latency during quota tuning. 4

Number of GPUs (b) ManiSkill

5.3

Ablation Study

We evaluate MPS and quota optimization on ManiSkill using the same workload and component placement: (1) Colocated, the best RLinf configuration; (2) Graph-only, with graph optimization but no MPS; (3) MPS-on, with the optimized graph and default MPS quotas; and (4) Full Rose, which additionally tunes the selected process’s SM quota. Figure 14 (a) reports the throughput of the four variants. Graph optimization alone achieves 1.08×, 1.10× and 1.06× the throughput of Colocated. With MPS enabled, the speedup becomes 1.12×, 1.13× and 1.10×, increasing by 3.4%, 2.3% and 3.9%. Finally, SM quota optimization brings the overall speedup of Full Rose to 1.29×, 1.20× and 1.24×, increasing by 15.6%, 6.7% and 12.6%. All speedups use Colocated as the baseline. The speedup breakdown of three methods is 33.1%, 13.5%, 53.4% for 𝜋 0.5 , 54.9%, 12.1%, 33.0% for OpenVLA-OFT, and 29.2%, 18.3%, 52.5% for GR00T. Figure 15 shows how local latency varies while tuning SM quota for all models. Local latency is measured over an overlap execution window spanning four action chunks. The x-axis denotes the SM quota of the selected process with slack. The full-range sweep shows that overly restrictive quotas substantially increase latency, while the default quota (100%) is suboptimal. Zooming into the 80–100% range reveals a U-shaped latency curve, with minima at 95%, 91%, and 89%, respectively. We additionally evaluate the benefit of using multiple graph optimization rounds. There are three variants: (1) Colocated+Pipe. The best configuration for RLinf on LIBERO. (2) Rose (one round). Execution graph after optimization for one round. (3) Full Rose. Execution graph after completing two rounds of optimization. Figure 14 (b) reports the throughput of the variants. The first and second rounds contribute 60.7% and 39.3% of the total reduction in iteration time for 𝜋0.5 , 62.9% and 37.1% for OpenVLA-OFT, and 67.6% and 32.4% for GR00T, respectively.

Figure 13: Training throughput on 4-16 GPUs. 1.29×, 1.20×, and 1.24×. (3) Rose also achieves up to 1.92× and 1.61× the throughput of RL-VLA3 on LIBERO and ManiSkill across platforms respectively. We observe that enabling simulator-group pipelining reduces throughput in RLinf’s colocated configuration on ManiSkill. Grouping environments reduces the batch size processed by each component. Policy inference time decreases approximately linearly with batch size, whereas the less compute-intensive GPU simulation sees only a modest reduction. Executing the additional simulator group therefore substantially increases cumulative simulation time, making simulation the bottleneck. As discussed in Section 3, this configuration still time-slices GPU execution across policy and simulator processes, thus providing insufficient concurrent GPU execution to offset the increased simulation cost, resulting in lower throughput. GPU execution efficiency. We plot the SM activity for a complete training iteration after automatic optimization in figure 12. SM activity remains below 50% for 8.4% of the interaction-loop execution time and 21.3% of the complete iteration time, corresponding to relative reductions of 78.4% and 64.3% in these time fractions, respectively. The remaining low-activity windows persist because no complementary and dispatchable work is available to fill them. Table 2 compares average SM activity between Rose and RLinf with the best configuration for 𝜋 0.5 and OpenVLA-OFT on LIBERO and ManiSkill. Across configurations, Rose improves SM activity by 23.8–39.3% on average. Scaling. We evaluate scaling on four, eight, and sixteen A800 GPUs on 𝜋0.5 PPO training with two simulators. The workload for each rank remains fixed, so the total simulator and training workloads increase with the GPU count under data parallelism. Figure 13 compares Rose with different RLinf configurations for each simulator and GPU count. On LIBERO, Rose improves throughput over the best configuration by 46.5%, 42.2%, and 40.2% on four, eight, and sixteen GPUs, respectively. On ManiSkill, the corresponding improvements are 31.5%, 29.2%, and 29.8%.

5.4

Optimization Overhead

We measure optimization overhead with the target limit set to 𝐾 = 3. For both workloads, tuning includes an end-to-end reference profile and two rounds of graph optimization, each followed by endto-end profiling. ManiSkill additionally requires seven short MPS 9

Zeng et al.

quota trials and a final validation. We distinguish total optimization time from net overhead by subtracting training execution whose progress is retained. For 𝜋 0.5 with PPO on LIBERO, two rounds of graph optimization take 1,998 s, including 1,336 s of training execution, leaving 662 s of net overhead. The policy reaches a 90.2% success rate after 252 iterations in 85,727 s, including optimization. Total time and net overhead therefore account for 2.3% and 0.8% of the complete run, respectively. For ManiSkill, the total optimization time is 4,615 s, including 2,290 s of retained training execution and 2,325 s of net overhead. Relative to its complete training time of 100,716 s to reach a 90.0% success rate, total time and net overhead correspond to 4.6% and 2.3%, respectively.

Systems 2007 (Lisbon, Portugal) (EuroSys ’07). Association for Computing Machinery, New York, NY, USA, 59–72. doi:10.1145/1272996.1273005 [8] Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. 2019. TASO: optimizing deep learning computation with automatic generation of graph substitutions. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (Huntsville, Ontario, Canada) (SOSP ’19). Association for Computing Machinery, New York, NY, USA, 47–62. doi:10.1145/3341301. 3359630 [9] Moo Jin Kim, Chelsea Finn, and Percy Liang. 2025. Fine-Tuning Vision-LanguageAction Models: Optimizing Speed and Success. In Proceedings of Robotics: Science and Systems. LosAngeles, CA, USA. doi:10.15607/RSS.2025.XXI.017 [10] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2025. OpenVLA: An Open-Source VisionLanguage-Action Model. In Proceedings of The 8th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 270), Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard (Eds.). PMLR, 2679–2713. https://proceedings.mlr. press/v270/kim25c.html [11] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan VillaRenteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. 2023. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. In Proceedings of The 6th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 205), Karen Liu, Dana Kulic, and Jeff Ichnowski (Eds.). PMLR, 80–93. https://proceedings.mlr.press/v205/li23a.html [12] Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, Dehui Wang, Dingxiang Luo, Yuchen Fan, Youbang Sun, Jia Zeng, Jiangmiao Pang, Shanghang Zhang, Yu Wang, Yao Mu, Bowen Zhou, and Ning Ding. 2025. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. arXiv:2509.09674 [cs.RO] https: //arxiv.org/abs/2509.09674 [13] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 44776–44791. doi:10.52202/075280-1939 [14] Jijia Liu, Feng Gao, Bingwen Wei, Xinlei Chen, Qingmin Liao, YI WU, Chao Yu, and Yu Wang. 2025. What Can RL Bring to VLA Generalization? An Empirical Study. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, Main Conference. Curran Associates, Inc., 97121–97151. doi:10.52202/0857133248 [15] Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, 881–897. https://www.usenix.org/conference/osdi20/presentation/ma [16] Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. 2025. ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys. [17] Liang Mi, Weijun Wang, Bowen Gao, Tianze Yu, Zixu Hao, Han Xiao, Xin Ding, Mingzhe Huang, Xin He, Lu Shi, Hao Wu, Haipeng Dai, Guihai Chen, Yunxin Liu, and Ting Cao. 2026. EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management. arXiv:2609.27547 [cs.LG] https://arxiv.org/abs/2609. 27547 [18] Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martín Abadi. 2013. Naiad: a timely dataflow system. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (Farminton, Pennsylvania) (SOSP ’13). Association for Computing Machinery, New York, NY, USA, 439–455. doi:10.1145/2517349.2522738 [19] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. 2024. RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots. In Proceedings of Robotics: Science and Systems. Delft, Netherlands. doi:10.15607/RSS.2024.XX.050 [20] NVIDIA, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. 2025. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv:2503.14734 [cs.RO] https://arxiv.org/abs/2503.14734

Conclusion

We presented Rose, a runtime-aware system for automatic overlap optimization in embodied RL. Combining a programmable interface with runtime measurements, Rose performs two-stage optimization through a window-guided graph optimizer and a slack-aware resource manager while preserving training semantics. Evaluations demonstrate substantial improvements in training throughput and GPU utilization.

References [1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 265–283. https: //www.usenix.org/conference/osdi16/technical-sessions/presentation/abadi [2] Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, brian ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian LiBell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. 2025. 𝜋 0.5 : a Vision-Language-Action Model with Open-World Generalization. In Proceedings of The 9th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 305), Joseph Lim, Shuran Song, and Hae-Won Park (Eds.). PMLR, 17–40. https://proceedings.mlr.press/v305/black25a.html [3] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Laura Smith, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. 2025. 𝜋 0 : A Vision-Language-Action Flow Model for General Robot Control. In Proceedings of Robotics: Science and Systems. LosAngeles, CA, USA. doi:10.15607/RSS.2025.XXI.010 [4] Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and YI WU. 2025. AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, Main Conference. Curran Associates, Inc., 36256–36282. doi:10. 52202/085713-1218 [5] Jiaheng Hu, Rose Hendrix, Ali Farhadi, Aniruddha Kembhavi, Roberto MartínMartín, Peter Stone, Kuo-Hao Zeng, and Kiana Ehsani. 2025. FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning. In 2025 IEEE International Conference on Robotics and Automation (ICRA). 3617–3624. doi:10.1109/ICRA55743.2025.11127934 [6] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. 2024. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv preprint arXiv:2405.11143 (2024). [7] Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly. 2007. Dryad: distributed data-parallel programs from sequential building blocks. In Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer 10

Rose : Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

[21] NVIDIA Corporation. 2026. NVIDIA DCGM User Guide: Profiling Metrics. Accessed: 2026-10-27. https://docs.nvidia.com/datacenter/dcgm/latest/userguide/feature-overview.html#profiling-metrics [22] NVIDIA Corporation. 2026. NVIDIA Multi-Process Service. Accessed: 2026-10-27. https://docs.nvidia.com/deploy/mps/latest/index.html [23] NVIDIA Corporation. 2026. NVIDIA Nsight Systems user guide. Accessed: 2026-10-27. https://docs.nvidia.com/nsight-systems/UserGuide/index.html [24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https: //arxiv.org/abs/1707.06347 [25] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402.03300 [26] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 1279–1297. doi:10.1145/3689031. 3696075 [27] Haoran Sun, Yongjian Guo, Zhong Guan, Shuai Di, Xiaodong Bai, Jing Long, Tianyun Zhao, Mingxi Luo, Hongke Zhao, Likang Wu, Xiaotie Deng, Xu Chu, Xi Xiao, Sheng Wen, Yicheng Gong, and Junwu Xiong. 2026. RL-VLA3 : A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training. arXiv:2602.05765 [cs.AI] https://arxiv.org/abs/2602.05765 [28] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-Kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. 2025. Demonstrating GPU Parallelized Robot Simulation and Rendering for Generalizable Embodied AI with ManiSkill3. In Proceedings of Robotics: Science and Systems. LosAngeles, CA, USA. doi:10.15607/RSS.2025.XXI.021 [29] Haluk Topcuoglu, Salim Hariri, and {Min You} Wu. 2002. Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Transactions on Parallel and Distributed Systems 13, 3 (March 2002), 260–274. doi:10.1109/71.993206 [30] Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. 2022. Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 267–284. https://www.usenix.org/conference/osdi22/ presentation/unger [31] Chao Yu, Yuanqing Wang, Zhen Guo, Hao Lin, Si Xu, Hongzhi Zang, Quanlu Zhang, Yongji Wu, Chunyang Zhu, Junhao Hu, Zixiao Huang, Mingjie Wei, Yuqing Xie, Ke Yang, Bo Dai, Zhexuan Xu, Jiakun Du, Xiangyuan Wang, Xu Fu, Letong Shi, Zhihao Liu, Kang Chen, Weilin Liu, Gang Liu, Boxun Li, Jianlei Yang, Zhi Yang, Guohao Dai, and Yu Wang. 2026. RLinf: Flexible and Efficient Large-Scale Reinforcement Learning via Macro-to-Micro Flow Transformation. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, Seattle, WA, 829–846. https://www.usenix.org/ conference/osdi26/presentation/yu-chao [32] Hongzhi Zang, Mingjie Wei, Si Xu, Yongji Wu, Zhen Guo, Yuanqing Wang, Hao Lin, Peihong Wang, Hua Yuan, Yixian Zhang, Liangzhi Shi, Yuqing Xie, Zhexuan Xu, Zhihao Liu, Kang Chen, Wenhao Tang, Quanlu Zhang, Weinan Zhang, Chao Yu, and Yu Wang. 2026. RLux-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models. In Proceedings of Robotics: Science and Systems. Sydney, Australia. doi:10.15607/RSS.2026.XXII.089 [33] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 559–578. https://www.usenix.org/conference/osdi22/presentation/zhenglianmin [34] Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, and Xin Jin. 2025. Optimizing RLHF Training for Large Language Models with Stage Fusion. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, 489–503. https://www.usenix.org/ conference/nsdi25/presentation/zhong [35] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Kevin Lin, Abhiram Maddukuri, Soroush Nasiriany, and Yifeng Zhu. 2025. robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. arXiv:2009.12293 [cs.RO] https://arxiv.org/abs/2009.12293

Zeng et al.

Runtime

MPS provisioning and local quota trials. Rose applies quotas through uniform MPS provisioning within each GPU worker process [22]. This ensures quota to take effect for simulator backend that manages CUDA contexts internally such as PhysX, which a Python-level Green Context wrapper cannot reliably constrain. Before launching a worker, Rose sets CUDA_MPS_ACTIVE_THREAD_ PERCENTAGE. The resulting limit remains fixed for the client’s lifetime, so quota changes require replacing the affected test worker. Quota trials use short workloads on separately initialized workers. We characterize effective quota levels by querying cudaDevAttr MultiProcessorCount in fresh MPS clients under different percentage settings. On the tested Hopper and Ampere GPUs, the smallest observed SM-count increment was two. Profiling and timing. Local validation measures execution time without Nsight Systems profiling. Within each comparison, reference and candidate runs use the same workload, timing boundaries, and profiling mode. During profiled iterations, Nsight Systems [23] collects devicewide SM activity throughout the iteration. NVTX ranges mark exposed invocations only in the first and last few steps of inner loops to reduce recording overhead. Invocation identifiers map these ranges to structural nodes, allowing sampled execution intervals to be aligned with the SM-activity trace. For unsampled invocations, device-associated durations are estimated from sampled invocations of the same node. Completion time of the comparison graph is the wall-clock span from its designated start to its completion event, including computation, data transfer and synchronization. End-to-end timing uses complete-iteration boundaries. Optimization overhead. Optimization overhead includes worker and model initialization, graph transformation and installation, data and state preparation, local trials and complete iteration validation. Quota tuning additionally incurs worker replacement and reinitialization, while compatible graph trials reuse initialized workers. These costs are reported in Section 5.

Runtime characteristics. The data plane records host-call duration for every invocation and additionally records device time when available. Separately, it maintains a device-wide SM activity ([21]) trace throughout the iteration for the GPU associated with each rank. For an invocation with device activity, the duration extends from the host start to the later of host return and device completion. Parent and child nodes provide alternative observation scopes but their measurements are not added as independent work. Replacement of callable graph. The runtime implements a node’s transformed internal graph as an alternative callable. A transformed graph must execute the node’s complete effective work while preserving its output interface and state effects. A hook at the node’s invocation entry selects this callable when registered. Each invocation executes either the native graph or its replacement, never both. Dependency-driven execution. An invocation is submitted only when its input, state, and scheduling requirements are satisfied. Output readiness accounts for required device computation and communication. A host return alone does not establish GPU output readiness. Intermediate buffers remain live until their required consumers complete. Local validation. Local validation compares the reference and transformed graphs over the same workload with comparable input and mutable state prepared. Preparation is excluded from regional timing but included in optimization overhead. Local trials do not commit their mutations to the ongoing training state.

Implementation

Rose extends RLinf [31] with 5K lines of Python, adding executionnode wrappers, the ExecutionTrace data plane, graph transformations and runtime support. Execution integration and bindings. Rose adds an invocation hook to the common ExecutionNode wrapper for synchronous and asynchronous execution. At each invocation, the hook selects the alternative callable registered for the node or its native entry if no alternative is registered. The driver installs these callables and their scheduling metadata on participating workers and launches invocations through RLinf’s distributed execution interface. Each replacement callable returns results through the original output interface. Spatial split binds invocations to subsets of the original inputs. Temporal split preserves data and state bindings for separate graphs. Fragment outputs retain their positions in the complete result, which Merge uses to reconstruct downstream inputs. Execution scheduling and completion. Independent execution graphs run as separate asynchronous tasks within each worker. Before execution, tasks wait for the required input, state and scheduling dependencies while allowing other runnable tasks to progress. Cross-worker data transfers reuse RLinf’s routed communication. For asynchronous entries, the wrapper invokes trace.after from the returned future’s completion callback. Output readiness is tracked separately through backend completion events covering device work and required data transfers. 12

Rose: Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

Analyze & explore related research in Conceptio

Related documents