JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
1
Knowledge Reutilization in Meta-Reinforcement Learning
arXiv:2606.18132v1 [cs.AI] 16 Jun 2026
Yuan Meng1,2,∗ , Bo Wang1,2,∗ , Juan de los Rios Ruiz1 , Xiangtong Yao1 , Zhenshan Bing2,† , Fuchun Sun3 , Fellow, IEEE, and Alois Knoll1 , Fellow, IEEE
Abstract—Meta-reinforcement learning enables fast adaptation by extracting shared structure from related tasks, but existing end-to-end methods often couple task inference with embodiment-specific control. This coupling can obscure nonparametric task semantics, reduce sample efficiency, and limit cross-agent reuse. We propose a meta-knowledge reutilization framework that learns task-level knowledge on a dynamicssimplified agent and transfers it to heterogeneous agents. The framework uses a Bayesian non-parametric prior to organize latent task modes and a high-level policy to generate tasklevel magnitude guidance. To bridge reusable task knowledge with different embodiments, we introduce a semantic-magnitude interface and a lightweight temporal adaptor, which convert frozen meta-knowledge into temporally aligned subgoals for embodiment-specific low-level controllers. Experiments on multiple locomotion agents show that our framework reduces finalstep tracking error by 94.75%–99.79% compared with recent state-of-the-art baselines and achieves comparable deployment performance with about 23.8% of their interaction data. Index Terms—Meta-reinforcement learning, cross-embodiment learning, knowledge reutilization, robotic locomotion, Bayesian non-parametric models.
I. INTRODUCTION Rapid adaptation to unseen tasks with limited interaction is a key challenge in robotic deep reinforcement learning (RL). Meta-reinforcement learning (Meta-RL) [1]–[4] addresses this challenge by combining outer-loop meta-learning with innerloop policy adaptation, enabling agents to learn how to adapt from limited experience. However, existing Meta-RL approaches still face limitations in how meta-knowledge is represented, acquired, and reused. In terms of representation, representative methods such as PEARL [5] adopt an end-toend framework in which task inference and policy learning are jointly optimized under a fixed Gaussian latent prior. Although this design is computationally convenient, it imposes a strong parametric assumption on the latent task structure. In robotic tasks, task semantics are often independent of a specific agent’s dynamics and may exhibit multi-modal and non-parametric structures [6]. Such structures are difficult to describe with a single Gaussian distribution, causing heterogeneous task modes to be compressed into a smooth latent space, 1 School of Computation, Information and Technology, Technical University of Munich, Munich, Germany 2 State Key Laboratory for Novel Software Technology, Nanjing University, Suzhou, China 3 Department of Computer Science and Technology, Tsinghua University, Beijing, China ∗ Equal contribution. The work was done during the research visit at Nanjing University. † Corresponding author: [email protected]
Magnitude-triggered Disentangled Learning
Meta-Knowledge Acquisition
Task Prediction yt Encoder
Decoder DPMM Prior
Semantic-Magnitude Interface
Simplified Agent
Complex Agent
Magnitude mt
Meta-Knowledge Reuitilization
Fig. 1: Overview of our proposed framework. The framework learns task-level meta-knowledge on a dynamics-simplified agent, trains target-agent low-level policies via magnitudeguided adaptation, and reuses the frozen meta-knowledge as semantic-magnitude subgoals for efficient deployment across heterogeneous agents. which can obscure task boundaries and increase the interaction required for reliable task inference. Recent studies have shown that more expressive Bayesian parametric or Bayesian nonparametric priors can improve the representation capability of latent spaces [4], [7], [8]. Nevertheless, a richer prior alone does not solve the acquisition and reuse problem. When task inference and policy learning are still optimized jointly under the same embodiment-specific dynamics, the learned latent representation may encode not only task semantics but also agent-specific action patterns. As a result, the obtained metaknowledge can be entangled with control-dependent behaviors, limiting its transferability across agents with different embodiments. This coupling further leads to an exploration dilemma in end-to-end Meta-RL. The inference module requires informative interaction data to identify the current task, while collecting such data depends on a policy that can already execute meaningful behaviors under the inferred task context. During early learning, neither component is sufficiently developed: the policy explores inefficiently, and the inference module receives limited or biased context for task identification. This chickenand-egg problem makes end-to-end Meta-RL highly sampleinefficient, especially for agents with complex morphology and dynamics. These observations reveal three key gaps. First, commonly used fixed Gaussian latent priors are insufficient for representing non-parametric task structures. Second, endto-end training tightly couples task inference with low-level control, leading to inefficient meta-knowledge acquisition. Third, the learned meta-knowledge is often entangled with
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
agent-specific dynamics, making cross-embodiment transfer and reutilization difficult. Addressing these gaps requires redesigning both the representation and acquisition processes for meta-knowledge. Based on these limitations, we identify three fundamental questions: Q1: How can we learn expressive meta-knowledge that captures non-parametric task semantics beyond a fixed Gaussian prior? • Q2: How can we decouple task inference from agentspecific dynamics and low-level control? • Q3: How can the learned meta-knowledge be effectively reused across agents with different embodiments? •
To address these questions, we propose ReMAP: Reusable Meta-knowledge for Adaptive Policy Transfer. An overview is shown in Fig. 1. First, to learn transferable meta-knowledge beyond a fixed Gaussian prior (Q1), we train the task inference module and high-level policy on a dynamics-simplified agent. This agent preserves task-relevant variables such as position and velocity, while removing high-dimensional embodimentspecific control complexity. We further impose a Bayesian non-parametric prior, specifically a Dirichlet Process Mixture Model (DPMM), on the latent task space, allowing the model to adaptively organize non-parametric task modes without predefining the number of components. Second, to decouple task inference from low-level control (Q2), we split training into two independent branches with different objectives. Unlike standard end-to-end Meta-RL, where one policy must both interpret task context and learn complex embodimentspecific control, ReMAP learns task inference and high-level magnitude generation only on the dynamics-simplified agent. Each target agent then trains its low-level policy separately through magnitude-guided curriculum learning, where the objective is not to infer task semantics but to track progressively generated task-level magnitude commands and acquire stable embodiment-specific motion patterns. This design avoids the chicken-and-egg problem between task inference and meaningful exploration, and prevents the latent task representation from being optimized through complex robot dynamics. Third, to reuse meta-knowledge across heterogeneous embodiments (Q3), we introduce a Semantic-Magnitude Alignment Interface (SMAI) as the bridge between task-level reasoning and embodiment-specific execution. The frozen task inference module predicts the semantic task context, and the frozen highlevel policy generates a scalar magnitude. SMAI embeds this magnitude into the corresponding semantic guidance channel, producing a shared semantic-magnitude subgoal that can be interpreted by different low-level controllers. Since different embodiments may require different execution durations to realize the same task-level command, we further introduce a lightweight stride predictor for few-shot online adaptation. It estimates how long each SMAI subgoal should be executed by the target agent, thereby temporally aligning the frozen tasklevel guidance with embodiment-specific motion dynamics. Thus, the same meta-knowledge module can be reused across agents, while each low-level policy only needs to translate the shared SMAI command into feasible motor actions under its own dynamics. We evaluate ReMAP on multiple locomotion
2
agents with different embodiments and compare it with representative Meta-RL baselines, including RL2 [1], PEARL [5], CEMRL [4], and MELTS [7]. Experiments show that ReMAP learns semantically separable task representations, reuses the frozen meta-knowledge module across agents, and achieves accurate downstream execution. At the final time step, ReMAP reduces the overall tracking MSE by 94.75%–99.79% compared with the baselines. Moreover, it achieves state-of-the-art deployment performance with 38M interactions across four embodiments, which is about 23.8% of the 160M interactions required by MELTS. The main contributions of this work are summarized as follows: • (Addressing Q1 & Q2) We propose a disentangled MetaRL framework that learns task-level meta-knowledge on a dynamics-simplified agent with a DPMM-regularized latent task space, encouraging the representation to encode task semantics rather than embodiment-specific dynamics. • (Addressing Q3) We introduce SMAI and a lightweight stride predictor to convert frozen task-level metaknowledge into temporally aligned semantic-magnitude subgoals, enabling reuse across heterogeneous agents with embodiment-specific low-level controllers. • We provide comprehensive experiments on multiple locomotion embodiments, demonstrating improved tracking accuracy, higher sample efficiency, and lower crossembodiment transfer cost than state-of-the-art Meta-RL baselines, including RL2 , PEARL, CEMRL, and MELTS. II. RELATED WORK A. Meta-Reinforcement Learning Meta-RL aims to enable agents to rapidly adapt to unseen tasks by leveraging experience collected from a distribution of related tasks. Existing methods can be broadly categorized into gradient-based, recurrence-based, and context-based approaches. Gradient-based methods learn an initialization or update rule that can be efficiently adapted to a new task through a few optimization steps [3], [9], [10]. Although effective for fast within-embodiment adaptation, they mainly update policy parameters under the same agent dynamics, making the learned prior difficult to reuse across different morphologies. Recurrence-based methods encode adaptation into a historydependent policy by aggregating past observations, actions, and rewards with recurrent or sequential models [1], [2], [11], [12]. However, the resulting task representation is typically implicit and jointly optimized with the control policy, which limits interpretability and cross-embodiment reuse. Contextbased methods explicitly infer a latent task representation from recent experience and condition the policy on this representation [4], [5], [7], [13]–[15]. Recent advances improve task inference through probabilistic modeling, metric learning, self-supervised objectives, mixture-based representations, or Bayesian non-parametric priors, showing that richer latent structures are beneficial for non-stationary and non-parametric task variations. Nevertheless, most existing methods still learn task inference and policy control in an end-to-end manner under embodiment-specific dynamics. Consequently, the learned
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
meta-knowledge may entangle task semantics with agentspecific motion patterns, leading to inefficient meta-knowledge acquisition and limited transferability when the target agent has different dynamics or morphology. In contrast, our work explicitly decouples task inference from low-level control, learns expressive meta-knowledge on a dynamics-simplified agent with a DPMM prior, and reuses the frozen task-level module across different embodiments. B. Deep Clustering and Non-Parametric Task Representation Task inference in Meta-RL is closely related to deep clustering, which combines representation learning with latent structure discovery [16]. Existing generative deep clustering methods, such as VaDE and GMVAE [17], [18], commonly employ Gaussian mixture priors [19] but require a predefined number of clusters, limiting their applicability when task structure is unknown or evolving. Bayesian non-parametric models address this limitation by adapting model complexity to the data [8], [20]–[23]. In particular, DPMM infers the number of latent components automatically and explicitly, making them well suited for heterogeneous and unstructured task distributions [8], [24]. In Meta-RL, Bayesian parametric and non-parametric latent priors have been explored for robotic task inference [4], [7]. However, they are typically embedded in end-to-end pipelines where task representation and policy control are learned under the same embodiment-specific dynamics, which may entangle task semantics with agent-specific motion patterns. Consequently, their potential as reusable meta-knowledge across embodiments remains limited. Our work addresses this gap by incorporating a Bayesian nonparametric prior into a disentangled framework that separates task-level inference from low-level control. C. Cross-Embodiment Learning Cross-embodiment learning is challenging because agents may differ in state spaces, action spaces, morphology, and dynamics. Existing few studies mainly focus on policy transfer or morphology-conditioned controller learning, using hierarchical decomposition or behavior alignment to improve control reuse across agents [25], [26]. However, these methods primarily transfer executable policies that remain coupled with embodiment-specific dynamics, while the reuse of task-level meta-knowledge in Meta-RL remains less explored. End-toend Meta-RL further entangles task inference with policy control, making latent task representations difficult to reuse when the embodiment changes. ReMAP addresses this issue by learning DPMM-regularized task-level meta-knowledge on a dynamics-simplified agent and using it to guide embodimentspecific low-level controllers, thereby separating task reasoning from physical execution. III. BACKGROUND This section briefly introduces the decision-making foundations used in our framework. We review Markov Decision Processes (MDPs) and Partially Observable MDPs (POMDPs) as standard formulations for fully and partially observable control [27], and then relate them to Meta-RL, where task identity
3
can be treated as a latent variable inferred from interaction data. This perspective motivates our use of Bayesian nonparametric inference for reusable task-level meta-knowledge acquisition. A. From MDPs and POMDPs to Meta-RL An MDP provides a standard formulation for sequential decision-making in fully observable environments [27]. It is defined by the tuple ⟨S, A, P, R, γ⟩, where S and A denote the state and action spaces, P(s′ |s, a) is the transition probability, R(s, a) is the reward function, and γ ∈ [0, 1) is the discount factor. At each step, the agent observes st , samples an action at ∼ π(·|st ), receives reward rt , and transitions according to P(st+1 |st , at ). The learning objective is to find a policy that maximizes the expected discounted return: "∞ # X t J(π) = Eτ ∼π γ R(st , at ) , (1) t=0
where τ denotes the trajectory induced by the policy and environment dynamics. In many robotic tasks, the full state is not directly observable. This setting is modeled as a POMDP, which augments an MDP with an observation space O and an observation model Z(o|s). The agent receives observations ot generated from latent states rather than observing st directly. Therefore, the policy must depend on the interaction history ht or a learned belief representation that summarizes hidden state information. From this view, decision-making under partial observability requires both inference and control: the agent must infer latent information from past interactions while selecting actions to maximize long-term return. RL learns policies through interaction with the environment, typically by optimizing parameters θ of a policy πθ to maximize J(πθ ). In POMDP settings, the policy is often conditioned on history or latent representations that approximate the belief state. While standard RL usually focuses on a single task, Meta-RL aims to generalize across a distribution of related tasks p(T ). During meta-training, the agent learns action patterns from training tasks, whereas during metatesting, it is evaluated on non-overlapping or even unseen tasks that are disjoint from the training set and require adaptation with limited interaction [5]. B. Dirichlet Process and Stick-Breaking The Dirichlet process (DP) is a Bayesian nonparametric prior over probability measures [8]. It is commonly used when the number of latent components is unknown in advance. Unlike parametric priors with a fixed number of components, a DP can support a countably infinite set of components, while only a finite subset is typically used for any finite dataset. Formally, let H be a base distribution over a parameter space Θ, and let α > 0 be a concentration parameter. A random probability measure G drawn from a DP is denoted as G ∼ DP(α, H). The base distribution H defines the expected location of the atoms, while α controls how strongly G concentrates around H.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
4
A key property of the DP is that a draw G is discrete with probability one, even when H is continuous. As a result, multiple samples from G may share the same parameter value, which naturally induces a clustering structure. A constructive representation of the DP is given by the stick-breaking process: βk ∼ B(1, α),
πk = βk
k−1 Y
(1 − βi ),
posterior with a simpler distribution [8]. Following the stickbreaking construction, the joint distribution can be written as p(x, v, θ, β) =
N Y
F (xn | θvn )Cat(vn | π(β))
n=1 ∞ Y
×
(2)
(5) B(βk | 1, α)H(θk | λ),
k=1
i=1
where βk determines the fraction assigned to component k from the remaining stick. The resulting weights {πk }∞ k=1 sum to one and follow the Griffiths-Engen-McCloskey (GEM) distribution. A draw from the DP can then be written as ∞ X G= (3) πk δθk∗ , k=1
where θk∗ ∼ H are atoms and δθk∗ denotes a point mass at θk∗ . This representation shows that the DP provides a flexible prior for modeling an unknown number of latent components.
where vn denotes the mixture assignment Qk−1of xn , βk is the stick-breaking variable, πk (β) = βk ℓ=1 (1 − βℓ ) is the corresponding mixture weight, and θk denotes the parameter of component k. We approximate the posterior p(v, θ, β | x) with a factorized variational distribution q(v, θ, β). The variational objective is to minimize the KL divergence q ∗ (v, θ, β) = arg min KL (q(v, θ, β)∥p(v, θ, β | x)) . q
(6)
Since log p(x) is constant with respect to q, this is equivalent to maximizing the evidence lower bound (ELBO) objective L:
C. Dirichlet Process Mixture Model The DPMM [8] provides a Bayesian non-parametric formulation for mixture modeling, where the number of active components is inferred from data rather than fixed in advance. This flexibility is particularly useful for latent task modeling, since the number of task modes may be unknown or may grow with the observed experience. Using the stick-breaking representation, a DPMM can be written as θk∗ | H ∼ H, π | α ∼ GEM(α), vi | π ∼ Cat(π),
(4)
xi | vi ∼ F (· | θv∗i ), where H is the base distribution, α is the concentration parameter, π denotes the mixture weights, vi is the component assignment of observation xi , and θk∗ parameterizes component k. Although the model defines countably infinite candidate components, only a finite subset is activated for a finite dataset, allowing the latent structure to adapt to the observed data. In this work, we interpret each active component as a latent task mode. Different components correspond to different patterns in the underlying task distribution. Compared with Gaussian mixture models (GMMs), which require the number of components to be specified beforehand, DPMM can adapt the number of active task modes according to the observed trajectories. This property is well-suited to Meta-RL, where tasks may be heterogeneous, non-parametric, or previously unseen. Therefore, DPMM provides a principled prior for modeling nonparametric latent task structures, which supports task inference and meta-knowledge reuse. D. Variational Inference for DPMM Exact posterior inference in a DPMM is generally intractable because the model contains a countably infinite number of possible mixture components. Variational inference provides a tractable approximation by replacing the true
L(q) = Eq [log p(x, v, θ, β)] − Eq [log q(v, θ, β)].
(7)
To make the inference finite-dimensional, we use a truncated mean-field approximation: N Y
q(v, θ, β) =
n=1
q(vn )
K Y
q(βk )q(θk ),
(8)
k=1
where K is a sufficiently large truncation level. This truncation is applied to the variational posterior, while the underlying DP prior remains non-parametric. The variational factors are defined as q(vn ) = Cat(r̂n1 , . . . , r̂nK ), q(βk ) = B(α̂k1 , α̂k0 ),
(9)
q(θk ) = H(θk | λ̂k ), where r̂nk denotes the responsibility of component k for data point xn . With ˆ· indicates variational factor. The local assignment factors are updated by r̃nk = exp (Eq [log πk (β)] + Eq [log F (xn | θk )]) , r̃nk . r̂nk = PK ℓ=1 r̃nℓ
(10)
The expected log mixture weight is computed from the Beta variational factors as Eq [log πk (β)] =ψ(α̂k1 ) − ψ(α̂k1 + α̂k0 ) +
k−1 X
[ψ(α̂ℓ0 ) − ψ(α̂ℓ1 + α̂ℓ0 )] ,
(11)
ℓ=1
where ψ(·) is the digamma function. For exponential-family likelihoods, the sufficient statistics of component k are N̂k =
N X n=1
r̂nk ,
sk (x) =
N X n=1
r̂nk t(xn ),
(12)
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
5
where t(xn ) denotes the sufficient statistic of the observation model. The global variational parameters are then updated as α̂k1 = 1+ N̂k ,
α̂k0 = α+
K X
N̂ℓ ,
λ̂k = λ+sk (x).
ℓ=k+1
(13) Under this formulation, the ELBO can be expressed using the sufficient statistics [8], [24]: L(q) =
K X Eq [θk ]⊤ sk (x) − N̂k Eq [a(θk )] k=1
+ N̂k Eq [log πk (β)] −
N X
r̂nk log r̂nk
n=1
(14)
B(βk | 1, α) + Eq log q(βk | α̂k1 , α̂k0 ) # " H(θk | λ) , + Eq log q(θk | λ̂k ) where a(θk ) is the log-normalizer of the exponential-family likelihood. Optimization alternates between updating the local responsibilities in Eq. (10) and updating the global parameters in Eq. (13). To improve scalability, Memoized Variational Bayes (MemoVB) [8] maintains cached sufficient statistics for each data batch and updates the full-dataset statistics incrementally. For a batch b, let Skb = [N̂kb , sbk ] denote the cached sufficient statistics of component k, and let Sk0 denote the fulldataset statistics. When batch b is revisited, MemoVB updates Sk0 ← Sk0 − Skb,old + Skb,new . This allows global parameters to be updated using full-dataset sufficient statistics while only processing one mini-batch at a time. MemoVB also introduces birth and merge moves to adapt the number of active components. A birth move proposes new components from data that are poorly represented by the current mixture, allowing the model to capture newly observed patterns. A merge move combines redundant components and accepts the proposal only when it improves the fulldata ELBO. Thus, MemoVB provides a practical mechanism for dynamically adjusting the active mixture structure while retaining a variational objective. In our framework, the DPMM prior is applied to the latent representations produced by a variational autoencoder (VAE). The VAE maps trajectory contexts into a structured latent space, while the DPMM clusters these latent variables into task modes. This design allows the model to infer compact and adaptive latent task variables, providing a principled basis for task inference and meta-knowledge reuse in Meta-RL. IV. METHOD A. Preliminary Assumptions and Applicability The proposed framework learns task-level meta-knowledge on a dynamics-simplified agent and reuses it across different embodiments. This requires task semantics to be separable from embodiment-specific execution. We therefore define the following assumptions that characterize the applicability of our method.
a) Task-level objective invariance: We focus on locomotion-oriented tasks whose objectives can be defined independently of robot morphology. The task goal should depend on environmental variables, such as position or velocity, rather than robot-specific variables such as joint angles, actuator commands, or contact configurations. Thus, reaching a spatial goal or tracking a target velocity can be shared across different agents, whereas tasks defined by morphology-specific manipulation interfaces are outside the intended scope. b) Reward consistency: Since the inference module uses rewards as part of the task context, reward signals should preserve consistent task semantics across agents. The reward functions need not be numerically identical, but they should measure the same task-related progress, such as distance to a goal or velocity-tracking error. Embodiment-specific terms, including joint-energy penalties or morphology-dependent action costs, should be normalized to avoid leaking agent-specific dynamics into the representation. c) Dynamics abstraction feasibility: The task-relevant behavior of each target agent should be representable through a simplified abstraction. Although target agents may have complex contacts, nonlinear actuation, or high-dimensional morphology, these factors should mainly affect task execution rather than task identification. Under these assumptions, the simplified agent preserves task-relevant variables while removing unnecessary control complexity. Consequently, the learned latent representation can capture task semantics rather than embodiment-specific motion patterns, enabling reuse across heterogeneous agents. B. Meta-knowledge Acquisition and Disentangled Learning As illustrated in Fig. 2, the proposed framework comprises two main components: disentangled meta-learning (Addressing Q1 & Q2) and meta-knowledge reuse during inference (Addressing Q3). The training part includes two independent parallel learning branches. The first branch learns a task inference module and a high-level policy on a dynamicssimplified agent, aiming to acquire reusable task-level metaknowledge. The second branch disentangled train low-level policies for target complex agents, enabling them to obtain basic movement and subgoal-following capabilities. After training, the task inference module, the high-level policy, and corresponding low-level policies are frozen and reused during inference. This section describes the training part, including simplified agent modeling, meta-knowledge acquisition, and disentangled low-level policy warm-up. 1) Dynamics Abstraction and Simplified Agent Modeling: To acquire task-level meta-knowledge without encoding embodiment-specific control patterns, we introduce a dynamics-simplified agent (Mujoco instantiation see Fig. 3a) as a controlled abstraction. The simplified agent is designed to preserve task-relevant variables, such as position, velocity, and orientation, while removing unnecessary control complexity caused by high-dimensional actuation, contact dynamics, and morphology-specific constraints. As discussed in Section IV-A, our framework primarily focuses on locomotion tasks;
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
6
I. Meta-knowledge Acquisition & Disentangled Learning a
b Magnitude s′ s
State Prediction
Lstate
Reward Generation
Lreward
Task Confidence
Ltask
Task Confidence SMAI
slow t
low slow t , rt
r DPMM Prior
Actor
Decoder
Encoder
Critics Llow π
alow t
DKL Simplified Agent
ct = {(s, s′ , r)}H t
ahigh t
Curriculum Schedule
z̄t = sg(zt )
st
Magnitude↑
Horizon↑
Switch↑
Distribution↑
Lhigh π
High-level Policy
Branch II: Disentangled Adaptive Learning Branch I: Meta-Knowledge Acquisition
··· c s′
State Prediction
s
Agent 1
Reward Generation
r
ŷt Task Confidence
LP
DPMM Prior
kt , gt
Decoder
Encoder
task 1
SMAI
task 2
z̄t
task 3
Agent 2
Stride Predictor
Meta task inference:
High-level Policy
ahigh t
few shot online adaptation
··· Agent i
st
task n
Observation Mapping
low slow t , rt
II. Meta-Knowledge Reutilization Fig. 2: Overview of ReMAP. a, The framework first learns reusable task-level meta-knowledge on a dynamics-simplified agent through DPMM-based task inference and high-level policy learning, while b, independently warming up embodiment-specific low-level policies with SMAI-guided curriculum adaptive learning. c, During deployment and meta-knowledge reutilization, the frozen meta-knowledge module generates semantic-magnitude subgoals, which are temporally aligned by a stride predictor and executed by different target agents with lightweight online adaptation. therefore, for tasks involving velocity or simple dynamic constraints, we use a mass-damper model (Fig. 3b): d2 x dx = Fext − c − mg, (15) dt2 dt where Fext is the external force, c is the damping coefficient, and g denotes gravity. The damping term ensures stable and physically plausible motion, while gravity introduces consistent environmental effects. For position-based tasks, the agent can be further modeled as a point mass with bounded kinematic dynamics s′ = s+a, with a ∈ [−amax , amax ], where s and s′ denote the current and next states, and a denotes the abstract action. The bounded action space prevents unrealistic transitions and aligns the exploration statistics with those of more complex agents. This abstraction keeps the motion physically meaningful while avoiding complex embodimentspecific dynamics. As shown in Fig. 3, the simplified agent serves as an intermediate task space in which task semantics can be learned more cleanly than in a high-dimensional robot
a
b
m
Fext
Simplified Agent
G = mg ff riction
Fig. 3: Simplified agent modeling. a, Illustration of the simplified agent instantiation in the Mujoco environment. b, Abstracted mass–damper modeling with external, gravity, and friction forces corresponding to Eq. (15). body. Therefore, for the context-based Meta-RL, the latent representation learned on this agent is encouraged to capture task-level structure rather than low-level motion patterns.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
7
2) Meta-Knowledge Acquisition: The upper-left branch of Fig. 2 shows the meta-knowledge acquisition process. The goal is to learn a variational task inference module and a corresponding high-level policy on the simplified agent introduced in Section IV-B1, so that the resulting task-level knowledge can later be reused by complex agents. For each training task Ti ∼ p(T ), the simplified agent interacts with the environment and collects trajectories using the current inference model and high-level policy. The collected data are used to jointly learn a latent task representation, a non-parametric task prior, and a high-level policy for the simplified agent. At each time step t, the task inference module receives a context window ct = {(st−τ , rt−τ , st−τ +1 )}H−1 τ =0 , where H is the context length. Unlike previous work [5] that uses the entire trajectory as context, our framework assumes that sufficient context can be inferred from a finite window of recent history. Therefore, we use a context length of H = 48 to improve sample efficiency. This design encourages the latent representation to capture global task-induced transition and reward structures. The task inference module is implemented as a VAE variant with a Bayesian non-parametric prior. The encoder qϕ (zt |ct ) maps the context window to a latent task variable zt ∈ Rdz : qϕ (zt |ct ) = N zt ; µϕ (ct ), diag(σϕ2 (ct )) , (16) where the encoder is implemented using a sequence model (In our case GRU module). The latent variable is sampled using the reparameterization trick [28]. The decoder pξ (st+1 , rt , yt | zt , at , st ) reconstructs the next state, reward, and task confidence from the latent variable and the current state. By jointly predicting the transition, the reward, and the task category, the decoder provides complementary supervision signals that encourage zt to encode task-level information. (ŝt+1 , r̂t , ℓt ) = pξ (zt , ahigh , st ), t
ŷt = softmax(ℓt ), (17) where ℓt ∈ RKy denotes the task-prediction logits, ŷt is the predicted task-confidence distribution, Ky is the number of task categories, and ahigh is the high-level action from simplit fied agent with minimum dynamic complexity. We define the decoder reconstruction losses as h i 2 Lstate = Ezt ∼qϕ (·|ct ) ∥st+1 − ŝt+1 ∥2 , h i 2 Lreward = Ezt ∼qϕ (·|ct ) ∥rt − r̂t ∥2 , (18) Ky X Ltask = −Ezt ∼qϕ (·|ct ) yt,k log ŷt,k , k=1
where yt denotes the ground-truth task label, represented as a one-hot vector. The overall decoder reconstruction objective is then Lrec = λs Lstate + λr Lreward + λy Ltask ,
(19)
where λs , λr , λy are factors that balance the reconstruction weighting. To model non-parametric task structures, we impose a DPMM prior pθ (z) on the latent space. This prior encourages similar tasks to form compact clusters and allows new
task modes to emerge when observed trajectories cannot be explained by existing components. In our framework, we asynchronously update the neural inference model and the DPMM prior iteratively. When updating the neural inference model, the DPMM prior is kept fixed and serves as a structured latent regularizer. When updating the DPMM, the encoder is fixed and the latent samples generated from recent context windows are treated as observations for the mixture model. z Specifically, given a latent buffer Z = {zn }N n=1 , we update the local component responsibilities and global variational parameters according to Eq. (10) and Eq. (13). For mini-batch training, we further follow the MemoVB update in Sec. III-D, which updates the full-dataset sufficient statistics by replacing the old cached statistics of the current batch with the newly computed ones. This allows the DPMM prior to track the evolving latent space without refitting the mixture model from scratch. In addition, we use birth and merge moves to adapt the number of active components: birth moves introduce new components for latent samples that are poorly explained by the current mixture, while merge moves remove redundant components when the full-data ELBO (Eq. 14) improves. After the DPMM update, its parameters are fixed again and used to regularize the next neural update. To avoid unstable hard assignments, we compute the latent regularization using a soft component-wise KL term. Specifically, the KL regularization is approximated as a responsibility-weighted average between the encoder posterior and the DPMM components. To summarize, the task inference objective is X LI = Lrec + βKL ρtk DKL (qϕ (zt |ct )∥pθk (zt )) , (20) k
where βKL balances reconstruction fidelity and latent prior regularization, ρtk denotes the responsibility of component k for zt , and pθk (zt ) is the k-th DPMM component. This soft weighting reduces the training noise caused by assigning a latent sample to an incorrect component. As shown in Fig. 2, the inferred latent variable zt is then used to condition the high-level policy. The policy operates on the augmented task state (st , zt ), where st denotes the taskrelevant state of the simplified agent. It outputs a continuous high-level action ahigh ∼ πhigh (· | st , zt ) , t
(21)
where ahigh is an abstract action in the simplified state space. t Since the simplified agent is defined over task-level variables, this high-level action can be used to generate task-aligned subgoals, such as a desired intermediate position or velocity. The high-level policy is trained with an entropy-regularized actor-critic objective [29]. To prevent the policy update from modifying the task-inference module, the latent task variable zt is detached before being used by the high-level policy: z̄t = sg(zt ), where sg(·) denotes the stop-gradient operation. We use two high-level critics Qχ1 and Qχ2 , which are trained with the standard clipped double-Q soft Bellman objective. The high-level actor is updated by h high Lhigh = E η log π a | s , z̄ high high t t t π at ∼πhigh i (22) − min Qχj st , z̄t , ahigh , t j=1,2
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
8
where ahigh is sampled from the high-level policy, η is the t entropy coefficient, and the minimum over the two critics mitigates overestimation bias. The actor and critics are optimized with their respective losses, and the target critics are updated by Polyak averaging. Since z̄t is detached during policy optimization, gradients from the high-level actor-critic objective are not propagated to the encoder, decoder, or DPMM prior. These modules are therefore updated only by the reconstruction and latent-structure objectives in Eq. (20). This design reduces the influence of control-specific optimization on the latent space and encourages zt to encode task-level information from transitions and rewards. After convergence, the learned task inference module and high-level policy form the reusable meta-knowledge module: Mmeta = (I, πhigh ),
I = (qϕ , pξ ),
(23)
where qϕ infers latent task variables, pξ reconstructs taskrelevant transition and reward information, and pθ provides the DPMM task prior. This module is frozen after training and reused during cross-embodiment deployment. 3) Magnitude-triggered Disentangled Learning: In parallel with meta-knowledge acquisition, the upper-right branch of Fig. 2 trains an embodiment-specific low-level policy for each complex agent. This learning branch is independent of the task inference module and the high-level policy. Its goal is not to learn task semantics, but to provide each target agent with basic behaviors, stable exploration, and the ability to adapt its motion according to task-level subgoal magnitudes. This is necessary because complex agents may fail to perform meaningful exploration if they cannot first maintain stable movement. We leverage Soft Actor-Critic (SAC) [29] as the optimization backbone for training the low-level policy. However, instead of applying vanilla SAC directly, we introduce two modifications to make the low-level controller suitable for cross-agent meta-knowledge reuse. First, we introduce a Semantic-Magnitude Alignment Interface (SMAI), which can later bridge the task-level guidance from the trained metaknowledge module Mmeta to the low-level controller through magnitude signals. This interface does not require the lowlevel policy to imitate a fixed high-level trajectory. Instead, it enables the agent to adapt its action pattern according to the magnitude and direction of the issued task-level command. Second, we use a curriculum-based adaptation schedule in a non-stationary Meta-RL setting, where the Low-level policies must rapidly and dynamically adapt their behavior within a single episode in response to the magnitude signal, while preserving sufficient exploration under non-stationary conditions. The curriculum gradually increases the target magnitude, switching frequency, trajectory horizon, and initial-state diversity within a rollout episode, so that the low-level policy first learns stable locomotion and then acquires the ability to adjust its behavior in response to changing subgoal magnitudes. For each target agent i, we train a low-level controller as a magnitude-guided action policy: (i) (i) (i) at ∼ πlow · | st , gt , (24) (i)
(i)
where st is the state of complex agent i, at is the motor action, and gt is the task-level guidance signal provided
through the SMAI. The role of SMAI is to represent a tasklevel command by coupling its semantic guidance channel with its scalar magnitude. Let ut ∈ {1, . . . , Kg } denote the active guidance channel, and let bt = e(ut ) ∈ {0, 1}Kg be its channel selector. Given a scalar magnitude mt ∈ R, the SMAI signal is defined as gt = mt bt ∈ RKg ,
(25)
where Kg is the number of task-level guidance channels. The active channel specifies the semantic type of the command, while the value assigned to that channel specifies the corresponding magnitude. Importantly, gt does not contain morphology-dependent quantities such as joint torques, contact forces, or gait patterns. Therefore, the same SMAI signal can be shared across different agents, while each low-level controller learns how to adapt its motion scale and action pattern according to its own dynamics. During low-level policy warm-up, the guidance signal is not generated by the meta-knowledge module. Instead, its semantic channel and magnitude are sampled from a curriculumdependent task-level distribution: (ut , mt ) ∼ pe (U, M),
gt = mt e(ut ),
(26)
where pe denotes the curriculum-dependent distribution over task-level guidance channels and their associated magnitudes at training epoch e. U denotes the set of semantic guidance channels in the SMAI representation, and M denotes the scalar magnitude space. The low-level reward encourages the agent to match the task-relevant quantity associated with the active SMAI channel, while regularizing excessive control effort: (i)
(i)
rlow = − Φui t (st ) − mt
2 2
(i)
− λa at
2 2
,
(27)
where Φui t (·) extracts the task-relevant quantity of agent i associated with the active guidance channel ut , such as centerof-mass position for goal-reaching commands or center-ofmass velocity for velocity-tracking commands, and λa controls the action regularization term. To stabilize SMAI-conditioned low-level policy learning under the non-stationary setting, we introduce a curriculum-based adaptation schedule that progressively increases the difficulty of the training distribution. The curriculum jointly controls the commanded target magnitude, within-trajectory task-switching frequency, rollout horizon, and initial-state diversity. Given a sampled guidance channel ut and its target magnitude m∗t , the curriculum first adjusts the commanded magnitude according to the task type: ( ∗ 1, e < E , αe mt , ut ∈ Uvel , vel m̃t = α = 2 e 1, e ≥ E , m∗t , ut ∈ Ugoal , vel
(28) where Uvel and Ugoal denote the velocity-tracking and goalreaching guidance channels, respectively. e is the current training epoch, and Evel is the epoch after which the full velocity range is enabled. Thus, velocity-tracking tasks are initially trained with reduced target magnitudes to avoid unstable coldstart exploration, whereas goal-tracking tasks preserve their
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
9
original target specification and are made progressively harder through the other curriculum factors. The temporal complexity of rollouts is increased by gradually introducing more frequent task changes. Let C(e) denote the prescribed number of command updates within one rollout at epoch e, and let H(e) denote the current rollout horizon. When C(e) > 0, the command update interval is defined as H(e) . (29) ∆(e) = C(e) At the beginning of a rollout and every ∆(e) steps thereafter, the active semantic guidance channel and its target magnitude are resampled from the curriculum-dependent distribution: (uτ , m∗τ ) ∼ pe (U, M),
if τ mod ∆(e) = 0,
(30)
where τ ∈ {0, . . . , H(e) − 1} is the time index within the current rollout. Between two consecutive command-update steps, the sampled pair (uτ , m∗τ ) is kept unchanged, so that the low-level controller has sufficient time to respond to the current task-level command. The sampled magnitude m∗τ is then adjusted according to the curriculum schedule in Eq. (28), and the resulting commanded magnitude is embedded into the active semantic channel through Eq. (26). This produces the executed SMAI signal g̃τ that is provided to the low-level policy. Early stages use no or few within-trajectory command updates, allowing the controller to first acquire stable tracking of a single guidance signal. As training progresses, increasing C(e) exposes the controller to more frequent transitions between semantic channels and target magnitudes, forcing it to recover from changing task-level commands and adapt online without losing stability. Meanwhile, the rollout horizon is progressively enlarged as H(e) = Hk ,
if Ek−1 ≤ e < Ek ,
k = 1, . . . , K,
(31)
where 0 = E0 < E1 < · · · < EK−1 < EK denote the curriculum transition epochs, and H1 < H2 < · · · < HK . Thus, the rollout horizon is explicitly increased as training progresses: early stages use short horizons to facilitate stabilization and error recovery, whereas later stages require the controller to maintain stable behavior over longer trajectories. Finally, the reset distribution is widened after a predefined curriculum stage: ( pdefault (s0 ), e < Erand , s0 ∼ (32) prand (s0 ), e ≥ Erand , where prand randomizes task-relevant initial quantities such as position and velocity. In addition to this staged schedule, we employ a performance-triggered task switch: when the taskrelevant quantity associated with the active guidance channel, (i) Φui t (st ), is sufficiently close to the current commanded magnitude m̃t , a new subgoal is sampled with a fixed probability. This prevents the agent from remaining in already solved easy conditions and encourages continuous adaptation to new subgoals. Overall, the proposed curriculum first enables the controller to learn stable locomotion under simple targets and then progressively exposes it to larger velocity commands, more frequent task transitions, longer horizons, and more
diverse initial states. Consequently, the warmed-up low-level controller learns not only reliable embodiment-specific motion generation, but also stable translation of task-level magnitude guidance signal from the frozen meta-knowledge module into executable actions. C. Meta-Knowledge Reutilization After meta-knowledge acquisition, the task inference module and the high-level policy are frozen and reused for agents with heterogeneous embodiments. As illustrated in Fig. 2 bottom, deployment for meta-knowledge reuse follows a closed-loop procedure in which the target agent first maps its raw embodiment-specific state into the shared abstract state space, then infers the task context using the frozen inference module, generates a task-level magnitude signal through the frozen high-level policy, and finally executes the resulting structured subgoal through its own frozen low-level controller. In this process, the reusable components only determine what task-level behavior should be achieved, whereas embodimentspecific execution is handled by the low-level policy and a lightweight temporal adaptor. 1) Meta-Knowledge Reuse and Magnitude guidance Gen(i) eration: For a target agent i, let st denote its full state at time step t. Since the meta-knowledge module is trained in the simplified agent space, the raw state is first projected into a shared task-relevant representation: (i)
(i)
s̄t = R(i) (st ),
(33)
where R(i) is an agent-specific abstract state remapping function. It removes morphology-dependent quantities, such as joint torques, actuator states, and contact-specific variables, while retaining task-aligned variables such as position, velocity, and orientation. Based on this representation, we construct a recent interaction context oH−1 n (i) (i) (i) (i) , (34) c̄t = s̄t−τ , rt−τ , s̄t−τ +1 τ =0
which has the same semantic structure as the context used during meta-knowledge acquisition. The frozen inference module then produces a latent task representation (i) (i) zt ∼ qϕ z | c̄t , (35) (i)
where zt , as described in Sec. IV-B2, captures the current task semantics in an embodiment-independent manner. The DPMM prior associated with the inference module provides the latent task-mode structure, while the task prediction head (i) provides a semantic task-confidence vector ŷt . Conditioned on the remapped abstract state and the inferred latent task variable, the frozen high-level policy generates an abstract high-level action: high,(i) (i) (i) at ∼ πhigh · | s̄t , zt . (36) This action is not directly used as the final subgoal magnitude. Directly interpreting the high-level action as a subgoal may produce overly aggressive or overly conservative commands, especially when transferred to agents with different dynamics.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Instead, we use it as a supervisory signal that controls the progression rate of the generated subgoal magnitude. Specifically, the high-level action with values in [−1, 1] is first transformed into a normalized progression coefficient: 1 high,(i) (i) (i) αt = at +1 , αt ∈ [0, 1]. (37) 2 (i)
Given the target magnitude m⋆,t in the shared task space and (i) the previously issued magnitude signal mt−1 , the new scalar magnitude is generated through a residual update: (i) (i) (i) (i) (i) mt = mt−1 + αt m⋆,t − mt−1 , (38) (i)
where mt may correspond to a desired position magnitude, velocity magnitude, or other task-dependent scalar command. For velocity-tracking tasks, the generated magnitude is further constrained within the valid velocity range. This residual formulation has an adaptive contraction effect: when the current magnitude is far from the target, the high-level action can induce a large progression step; as the magnitude approaches the target, the residual term naturally decreases, preventing abrupt or oscillatory guidance. The final task-level subgoal provided to the low-level controller is constructed by embedding the generated magnitude into the semantic channel indicated by the predicted task con(i) fidence. Specifically, the task prediction ŷt is first converted into a semantic selector (i) (i) (i) (i) ût = arg max ŷt,k , bt = onehot ût ∈ {0, 1}Kg , k (39) (i) The generated scalar magnitude mt is then injected into the selected semantic channel, so that the deployment-time subgoal follows the same SMAI representation as defined in Eq. 25. In this representation, the active channel specifies the semantic type of the subgoal, while its value specifies the corresponding task-level magnitude. This design provides the low-level controller with a compact and unified guidance signal, where task identity and quantitative command strength are encoded in the same task-aligned vector space. 2) Stride Prediction for Temporal Alignment: Directly (i) updating gt at every low-level control step can be unstable for complex agents because of inertia, contact dynamics, actuation delay, and morphology-dependent response time. To compensate for this mismatch, we introduce an agent-specific (i) stride predictor Pζ , where ζ denotes its parameters. Given the current subgoal, the stride predictor estimates an execution (i) stride kt ∈ Z+ , which specifies how many low-level control steps the current subgoal should be held fixed. (i) After executing the low-level policy for kt steps, the achieved state is compared with the issued magnitude signal in the task-relevant dimension associated with the active semantic channel. The stride predictor is updated online by minimizing 2 (i) û (i) (i) (i) LP (ζ) = Φi t s (i) − mt , (40) t+kt
2
(i)
û where Φi t
(·) extracts the controlled task-relevant quantity (i) of agent i corresponding to the active semantic channel ût ,
10
such as its center-of-mass position or velocity. The distinction (i)
û
between R(i) and Φi t is important: R(i) maps raw states into the abstract state space used for task inference, whereas (i)
û
Φi t evaluates how well the physical agent has achieved the issued magnitude signal under the selected task channel. This temporal adaptation operates only at the lightweight stride predictor and does not update the frozen inference module, high-level policy, or low-level policies. 3) Policy Adaptive Deployment and Abstract State Remap(i) (i) ping: Given gt and the predicted kt , the agent-specific low-level policy executes the task-level guidance signal over multiple control steps: (i) (i) (i) (i) a(i) , τ = t, . . . , t + kt − 1. (41) τ ∼ πlow · | sτ , gt The low-level policy translates the abstract task-level command into executable motor actions under the dynamics of agent i. After execution, the generated transition is mapped back to the abstract state space through R(i) . Specifically, R(i) extracts global task-relevant quantities from the complex agent, such as the position and velocity of the body center of mass, while filtering out embodiment-specific joint states, actuator variables, and contact-related details. This observation mapping aligns the deployed agent’s state representation with the state space used by the high-level encoder, allowing the frozen inference module to process interaction contexts from heterogeneous agents in a consistent abstract space. The resulting interaction tuple is then appended to the context window for the next inference step. This closes the loop between abstract task inference, high-level magnitude generation, temporal alignment, and embodiment-specific control. During deployment, the frozen meta-knowledge module remains unchanged. Online adaptation is restricted to the lightweight stride predictor. This separation preserves the embodimentindependent nature of the learned task representation while allowing each agent to adapt its physical execution to the generated subgoals. Consequently, the same meta-knowledge module can be reused across heterogeneous agents: the inference module and high-level policy operate in the shared ab(i) (i) stract task space, whereas πlow , Pζ , R(i) , and Φi absorb the morphology- and dynamics-specific differences. In summary, the overall learning and inference procedures of our framework are provided in Algs. 1 and 2 of Supplementary Sec. A. V. EXPERIMENTS A. Experimental Setup Embodiment. We evaluate the proposed framework in a cross-embodiment Meta-RL setting. The goal of this experiment is to examine whether the meta-knowledge learned from the simplified agent can be reused across agents with different embodiments. Specifically, we consider the Mujocobased task suite [30] with four embodiments: Hopper, Walker, Half-Cheetah, and Ant. As shown in Fig. 4, these agents exhibit distinct morphologies and dynamics, including singleleg hopping, bipedal walking, planar quadrupedal-like running, and full quadrupedal locomotion. This diversity provides a challenging benchmark for evaluating whether the learned
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
11
meta-knowledge is independent of embodiment-specific control dynamics.
Hopper
Walker
Half-Cheetah
Ant
Fig. 4: Cross-agent benchmark for meta-knowledge reuse. We evaluate four agents with different embodiments: Hopper, Walker, Half-Cheetah, and Ant. Each agent is tested on four locomotion tasks, including forward/backward goal reaching (flags) and forward/backward velocity tracking (arrows). Task configuration. For each agent, we evaluate four non-parametric locomotion tasks: Velocity Forward, Velocity Backward, Goal Forward, and Goal Backward. The velocitytracking tasks require the agent to match a target horizontal velocity, while the goal-reaching tasks require the agent to move toward a target horizontal position. Table I summarizes the task parameter configuration. For the velocity-tracking tasks, the target velocity is sampled from [1, 3] for forward running and from [−3, −1] for backward running. For the goal-reaching tasks, the target position is sampled from [0, 10] for goals in front and from [−10, 0] for goals behind the agent. To preserve the Meta-RL setting, task instances for the simplified-agent environment and those for each complex agent are independently sampled from the corresponding task distributions. This avoids a complete overlap between the reusable meta-knowledge acquired on the simplified agent and the task experience observed by complex agents during disentangled learning. Additionally, the reward is defined as the negative ℓ1 distance between the target state and the current task-relevant state. Following the reward consistency assumption in Sec. IV-A, this unified reward formulation is shared across environments, as it measures the shared tasklevel progress while leaving each agent to learn its own policy under embodiment-specific dynamics. TABLE I: Parameter configurations of the four non-parametric locomotion tasks Base Task
Target
Unit
Range
Reward
Velocity Forward
vx
m/s
1 ≤ vx∗ ≤ 3
−|vx∗ − vx |
Velocity Backward
v−x
m/s
∗ ≤ −1 −3 ≤ v−x
∗ −v −|v−x −x |
Goal Forward
px
m
0 < p∗x ≤ 10
−|p∗x − px |
Goal Backward
p−x
m
−10 ≤ p∗−x < 0
−|p∗−x − p−x |
Baselines. We compare ReMAP with four representative Meta-RL baselines that cover recurrent, probabilistic contextbased, mixture-based, and Bayesian non-parametric task inference paradigms: 2 • RL [1]: a recurrence-based Meta-RL method that encodes the adaptation process in the hidden state of a recurrent policy, allowing the agent to condition its behavior on previous observations, actions, rewards, and terminations. • PEARL [5]: an off-policy context-based Meta-RL method that performs probabilistic task inference from
recent experience and conditions the actor-critic policy on the inferred latent task variable. • CEMRL [4]: a context-based Meta-RL method that introduces a Gaussian Mixture Model (GMM) latent representation to model clustered task distributions and improve adaptation in non-stationary and dynamic environments. • MELTS [7]: a Bayesian non-parametric context-based Meta-RL method that combines DPMM-based task selfdiscovery with a VAE-style inference model to identify non-parametric task structures in a self-adaptive manner. For all Meta-RL baselines, the training and testing task sets are sampled independently from the corresponding task distributions to avoid complete overlap between meta-training and meta-testing tasks. These baselines represent strong state-ofthe-art Meta-RL methods for task inference and adaptation. However, recent methods are trained in an end-to-end manner under a specific agent embodiment, where task inference and control are optimized jointly. Once the embodiment changes, the task representation and policy must be trained from scratch or substantially adapted for the new dynamics. This embodiment-specific coupling greatly increases the training cost and limits the transferability of the learned metaknowledge. In contrast, ReMAP learns the task inference module and high-level policy once on a dynamics-simplified agent and reuses them across target agents. Therefore, only the embodiment-specific execution layer needs to be prepared for each agent, which improves cross-embodiment reutilization and transfer efficiency. In addition, we conduct ablation studies to evaluate the contribution of the Bayesian non-parametric prior used in ReMAP. We compare the ReMAP full model with three prior variants: (i) a single Gaussian prior, which assumes that all task embeddings are generated from one shared latent distribution; (ii) a Gaussian Mixture Model prior with K = 2, which introduces multiple latent components but relies on a predefined fixed and potentially improper assumption about the number of task modes; and (iii) a stick-breaking prior, which provides a Bayesian non-parametric formulation with a theoretically unbounded number of components. These variants are used to examine whether the DPMM-based latent organization is necessary for learning compact, semantically separable, and reusable task representations. B. Meta-Knowledge Acquisition Before transferring the learned meta-knowledge to complex agents, we first evaluate whether the task inference module and the high-level policy trained on the simplified agent can acquire executable, semantically structured task knowledge. The purpose of this stage is therefore to verify the acquired meta-knowledge from three aspects. First, we evaluate whether the optimization process of the task inference module and the high-level policy converges during training. Second, after convergence, we deploy the trained modules and validate whether they support continuous, reasonable task inference sufficient to guide downstream policies. Third, we analyze the latent space to determine whether the learned representation contains distinguishable task semantics rather than merely encoding trajectory-level variations.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
a
b Position (m)
Simplified Agent
−100
−150
Velocity (m/s)
Average Reward
−50
−200
−250
0
200
400
600
Epochs
800
1,000
12
c
Inference Trajectories 10
0
goal forward
−10
goal backward
2
velocity forward
0 velocity backward −2 0
20
40
60
Time Step
Fig. 5: Evaluation of the simplified agent on four non-parametric tasks. a, Average reward during meta-knowledge acquisition over four task scenarios, including goal tracking forward/backward and velocity tracking forward/backward. The curve reports the mean reward with standard deviation over at least five trials (µ ± σ). b, Inference trajectories on twelve task variations, where dashed horizontal lines denote the ground-truth targets. The inference module and high-level policy can accurately track both position (above) and velocity (below) targets in both directions. c, Visualization of the learned latent space, where different task types form separable clusters, indicating that the inference module captures the underlying task structure. Fig. 5a shows the training reward curve of the simplified agent in the given tasks. The reward increases steadily during training and gradually converges to a stable level at -50. This result indicates that the task inference module and the highlevel policy reach a stable optimization state under the simplified environment. Since the agent provides an abstraction with simplified embodiment dynamics, the convergence behavior suggests that the framework can consistently learn task-related behavioral patterns without being dominated by complex lowlevel control effects. However, reward alone does not reflect whether the learned latent representation contains informative task semantics. To further validate, we deploy the trained task inference module and high-level policy on the simplified agent to evaluate whether the learned meta knowledge can actually guide task execution. Fig. 5b shows example tracking trajectories under different task settings. The dashed lines denote the desired task specifications, while the solid lines represent the actual executed trajectories. The simplified agent continuously adjusts its position and velocity according to the inferred task representation, and the resulting trajectories closely follow the corresponding task targets. This result demonstrates that the learned latent representation contains actionable task-level semantic information rather than coupled latent variations, as in a single Gaussian latent space. Moreover, the high-level policy can successfully translate the inferred task semantics into executable subgoals that guide downstream locomotion behaviors. The close alignment between the executed trajectories and the desired task specifications indicates that the acquired meta knowledge is not only semantically meaningful but also practically usable for hierarchical task execution. To further analyze whether the learned meta-knowledge contains clear task semantics, we visualize the inferred latent variables using t-SNE projection. As shown in Fig. 5c, the latent space forms four clearly separated clusters corresponding to different task types. The strong separation between clusters indicates that our DPMM-regularized variational encoding process can organize the latent representation according to
underlying task semantics. Task inference results with similar objectives are clustered together, while tasks with different semantic meanings remain distinguishable in the latent space. This structured latent organization demonstrates that the learned meta-knowledge captures task-level information that generalizes beyond the simplified environment. Overall, these results answer Q1 by demonstrating that the proposed method learns task-discriminative representations for non-parametric task semantics. This property is essential for downstream deployment, since it enables the reused inference module to provide semantic guidance for generating appropriate highlevel subgoals across different embodiments. C. Disentangled Learning In parallel with meta-knowledge acquisition, we train lowlevel policies for complex agents through magnitude-triggered SMAI conditioning. This experiment addresses Q2 by testing whether task inference can be separated from embodimentspecific control. End-to-end Meta-RL suffers from a chickenand-egg problem: task inference requires informative trajectories, but such trajectories can only be collected when the policy already performs meaningful exploration. For complex locomotion agents, early exploration often fails due to unstable motion, making this coupling especially inefficient. Our disentangled learning strategy therefore trains low-level controllers independently from the inference module and high-level policy. With SMAI and the curriculum schedule, each controller first learns stable embodiment-specific motion and then acquires the ability to follow changing magnitude commands, without entangling task semantics with low-level control. Fig. 6 shows the training reward curves of the low-level controllers for four embodiments: Hopper, Walker, Cheetah, and Ant. Under the proposed curriculum, convergence indicates that the corresponding low-level policy has acquired both stable locomotion and subgoal-magnitude-following capability under its own dynamics and action space. Different embodiments exhibit substantially different learning difficulties due to
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
−3,000 0
100
200
Epochs
300
200
0
0
100
200
Average Reward
−2,000
Average Reward
−1,000
Hopper
Walker
Ant Average Reward
Average Reward
Half-Cheetah
13
600 400 200 0 0
300
1,000
Epochs
2,000
3,000
400
200
0 0
1,000
Epochs
2,000
3,000
Epochs
Fig. 6: Disentangled low-level policy adaptive learning reward curves of four agents with complex dynamics. The solid line represents the average reward, while the shaded region denotes the standard deviation across five trials (µ ± σ).
D. Cross-Embodiment Meta Knowledge Reutilization Case Study: Disentangling Task Semantics From Dynamics. Before evaluating cross-embodiment reutilization under the full non-parametric task setting, we first examine whether the learned meta-knowledge mainly encodes tasklevel semantics rather than embodiment-specific dynamics. This analysis is important for answering Q2, since reusable meta-knowledge should remain consistent across agents even when their physical structures and motion patterns differ. To avoid the additional ambiguity introduced by multi-task inference, we conduct a controlled single-task case study on the goal-forward task. The low-level policies are trained only for this task, while the task inference module and highlevel policy learned from the dynamics-simplified agent with full non-parametric tasks (Fig. 5) are kept frozen and reused following the deployment pipeline in Fig. 2c.
Goal Forward
a
Position (m)
10
5
0 0
10
20
30
40 50 60 Time Steps
70
80
90
100
Encoding vs. Target Goal Across Agents
b 0 Encoding
their morphology and stability characteristics. Cheetah and Ant have relatively stable support structures during locomotion, which allows them to maintain balance more easily during exploration and converge within about 300 epochs. In contrast, Walker is a bipedal agent and Hopper relies on a single support leg, making balance control considerably more difficult. For these agents, small deviations of the center of mass may destabilize the body and interrupt locomotion. As a result, Walker and Hopper require longer training, typically around 3000 epochs, before reaching stable locomotion behaviors. The importance of stable low-level control during cold-start exploration, together with representative collapse failures of unstable agents, is further discussed in Supplementary Sec. B. In our framework, the objective of low-level policy learning is therefore not to acquire full task semantics, which are learned by the meta-knowledge module, but to establish the embodiment-specific execution capability required for downstream deployment. This includes maintaining balance, generating stable locomotion patterns, and tracking the issued SMAI magnitudes. Once the low-level policy reaches a stable locomotion regime, it becomes a valid execution layer for the shared task inference module and high-level policy. The reusable meta-knowledge can then generate semanticmagnitude guidance, while the embodiment-specific low-level controller translates this guidance into executable motor actions. This separation between task reasoning and motor control provides a positive answer to Q2 and forms the basis for stable and efficient cross-embodiment deployment in ReMAP.
−5
0
2
Half-Cheetah
4 6 Target Goal Ant
Walker
8
10 Hopper
Fig. 7: Visualization of inference results trained on the goalforward task. a, Goal-tracking trajectories across four agents within the first 100 time steps. Solid curves denote the current goals, while dashed curves denote the corresponding target goals. b, Encoding-target relationship after convergence across four agents. Scatter points denote inferred latent encodings obtained from trajectory samples from the 50th time step onward, while solid lines show the linear regression trends. Fig. 7a shows the goal-tracking trajectories of four target agents, including Half-Cheetah, Ant, Walker, and Hopper. Although these agents have substantially different embodiments, ranging from quadrupedal and multi-legged structures to bipedal and single-legged locomotion, all agents can stably follow the goal-forward commands generated from the reused meta-knowledge module. This indicates that the frozen tasklevel module provides guidance that can be physically realized by different embodiment-specific low-level controllers. To further inspect whether the inferred representation is affected by agent dynamics, we sample 50 trajectories for each
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
agent and compare the inferred meta-knowledge encodings with the corresponding ground-truth target goals after the trajectories enter the stable execution stage, i.e., from the 50th time step onward. As shown in Fig. 7b, the inferred encodings exhibit a clear linear relationship with the target goals. Moreover, the regression trends across different agents are highly consistent, with no evident agent-specific deviation. This suggests that the inference module captures the parametric variation of the task itself, rather than the specific dynamics or action patterns used by each agent to execute the task. Together with the non-parametric task clustering results in Fig. 5, this case study shows that the proposed metaknowledge inference module can represent both categorical task semantics and within-task parametric variations. The embodiment-specific dynamics are instead absorbed by the downstream low-level policies. These results provide evidence for Q2: the proposed disentangled design separates task-level inference from low-level control, enabling the same metaknowledge to be reused across heterogeneous agents. Cross-Embodiment Meta-Knowledge Reutilization. We further evaluate whether the meta-knowledge acquired on the dynamics-simplified agent can be smoothly and stably reused across heterogeneous embodiments under the non-parametric task setting. The task space contains four task types: goal forward, goal backward, velocity forward, and velocity backward. During deployment, the task inference module, highlevel policy, and target agent low-level policies are kept frozen. After online adaptation for the stride predictor, each target agent executes the generated semantic-magnitude subgoals through its own low-level controller. Fig. 8 summarizes the deployment results on Half-Cheetah, Ant, Walker, and Hopper. Figs. 8a–d show the representative inference trajectories for the four agents. For each agent, the upper plot reports goalposition tracking and the lower plot reports velocity tracking. Solid curves denote the executed trajectories, while dashed horizontal lines denote the corresponding task targets. Across all embodiments, the agents robustly follow the generated goal- or velocity-level commands through SMAI conditioning despite their substantially different dynamics and morphologies. This indicates that reused meta-knowledge provides executable task-level guidance, while embodiment-specific lowlevel policies translate it into feasible actions. To analyze whether the transferred inference module preserves semantic task structure, we visualize the inferred latent representations after deployment using t-SNE. As shown in Figs. 8e–h, the latent embeddings form clearly separated clusters for the four task categories across all target agents. The cluster structure remains consistent even though the trajectories are collected from different embodiments. This suggests that the frozen inference module mainly captures task semantics, rather than agent-specific motion patterns, after being reused on complex agents. We further quantify the semantic decodability of the inferred latent variables using decoder prediction confusion matrices. Figs. 8i–l show that the predictions concentrate strongly along the diagonal for all four agents, indicating that the inferred latent representations remain informative enough to recover the corresponding task identities after cross-agent deployment.
14
Together with the t-SNE visualizations, these results show that the deployed latent space preserves both categorical task information and clear task separability across heterogeneous embodiments. An additional observation from the trajectory plots is that some agents may exhibit temporary overshoot or transient tracking errors during the early stage of deployment, especially in the goal-tracking tasks. This is expected because the inference module relies on a finite context window and may require sufficient trajectory history before accurately identifying the current task mode and magnitude. As more interaction data becomes available, the inferred task representation becomes more reliable, allowing the high-level policy to generate corrected semantic-magnitude subgoals. The agents, therefore, recover from early deviations and continue tracking the desired task objectives instead of permanently diverging. Overall, Fig. 8 demonstrates that ReMAP can reuse the same frozen task-level meta-knowledge across agents with different embodiments. The inference module preserves task semantics, the decoder maintains high semantic prediction consistency, and the generated subgoals remain executable through embodiment-specific low-level controllers. These results provide direct evidence for cross-embodiment metaknowledge reutilization and support the claim that task-level reasoning is effectively decoupled from low-level dynamics in the proposed framework. Baseline Comparison. We further compare ReMAP with four Meta-RL baselines, including RL2 , PEARL, CEMRL, and MELTS. Different from ReMAP, these methods jointly learn task inference and control under the target embodiment. Therefore, when the embodiment changes, both the task representation and the policy are coupled with the new dynamics and must be relearned or substantially adapted. This setting makes task inference less efficient, especially during early exploration, because the agent must simultaneously identify the task and discover feasible motor behaviors. Fig. 9 shows the inference trajectories of the baselines on four non-parametric tasks for the Half-Cheetah embodiment. RL2 and PEARL exhibit clear tracking limitations. RL2 lacks an explicit task representation and mainly relies on recurrent hidden states to encode adaptation history, which leads to weak separation between different task modes. PEARL improves task inference by using probabilistic context variables, but its single-Gaussian latent prior tends to compress different task modes into a smooth representation. As a result, PEARL often produces averaged behaviors and shows limited adaptation to task parameters that are away from the dominant training distribution. This can be observed from the goal-tracking trajectories, where the predicted behaviors either drift away from the desired targets or converge to intermediate responses rather than accurately matching each target. CEMRL and MELTS achieve stronger tracking performance than RL2 and PEARL, especially for velocity-tracking tasks. CEMRL benefits from its GMM-based task representation and can distinguish multiple task modes when the predefined number of mixture components matches the underlying task structure. However, this assumption is restrictive in taskunknown settings: an improper component number can easily
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
b
Half-Cheetah (Ours)
Hopper (Ours)
10 Position (m)
0
d
Walker (Ours)
10 Position (m)
10 Position (m)
c
Ant (Ours)
0
10 Position (m)
a
15
0
0
2
2
2
2
20
40
60
80
100
0
Time Step
20
40
60
80
100
100
60
80
100
velocity backward high
Walker (Ours)
goal_forward goal_backward vel_forward vel_backward
low
40
Time Step
high
Ground Truth
Ground Truth
goal_forward goal_backward vel_forward vel_backward
20
l
Ant (Ours)
Prediction
0
velocity forward high
vel_backward vel_forward goal_backward goal_forward
Ground Truth
80
k
high
Half-Cheetah (Ours)
low
60
−2
h
goal backward
j
goal_forward goal_backward vel_forward vel_backward
40
g
goal forward
i
20
0
Time Step
f
vel_backward vel_forward goal_backward goal_forward
0
Time Step
e
Prediction
−2
vel_backward vel_forward goal_backward goal_forward
0
−2
0
Ground Truth
−2
0
Prediction
vel_backward vel_forward goal_backward goal_forward
0
Velocity (m/s)
−10
Velocity (m/s)
−10
Velocity (m/s)
−10
Velocity (m/s)
−10
Hopper (Ours)
goal_forward goal_backward vel_forward vel_backward
low
Prediction
low
Fig. 8: Cross-agent deployment of our ReMAP DPMM-based task inference module. a–d, Inference trajectories on HalfCheetah, Ant, Walker, and Hopper, respectively. For each agent, the upper plot shows position tracking and the lower plot shows velocity tracking across twelve non-parametric tasks; dashed horizontal lines denote the corresponding ground-truth targets. e–h, t-SNE visualizations of the inferred latent representations after deploying the frozen task inference module to different target agents. Across all embodiments, different task types form clearly distinguishable clusters, indicating that the learned latent representation preserves task semantics across agents. i–l, Decoder prediction confusion matrices for HalfCheetah, Ant, Walker, and Hopper. The consistently high prediction accuracy shows that the inferred latent representations remain semantically informative after cross-agent deployment. lead to incorrect task partitioning and unstable inference, as further examined in the ablation study. MELTS introduces a Bayesian non-parametric prior and can obtain more flexible task discovery than fixed-component mixture models. Nevertheless, because it is still trained end-to-end on the complex embodiment, its early-stage inference can be inaccurate when the agent has not yet collected informative task-semantic trajectories. Only after sufficient rollout context is accumulated can the inferred task representation become reliable enough to guide accurate tracking. In contrast, as shown in Fig. 8, ReMAP decouples task-level meta-knowledge acquisition from embodiment-specific motor learning. The task inference module and high-level policy are learned on the dynamics-simplified agent and then reused as a
frozen task-level module, while the target agent only needs to execute the generated semantic-magnitude subgoals through its low-level controller. This design enables more efficient task exploration from the beginning of deployment and avoids entangling task semantics with the complex dynamics of the target embodiment. To quantitatively evaluate tracking accuracy, Fig. 10 reports the per-step MSE between the task targets and the executed trajectories. For each method, each task is evaluated using 50 sampled trajectories with its best-achieved model, and the MSE is computed at each time step. For goal tracking, ReMAP achieves an average MSE of 2.16 over the full rollout, which reduces the error by 95.9%, 94.8%, 89.0%, and 89.3% compared with PEARL (52.21), RL2 (41.35), CEMRL
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
b
RL2
0
d
CEMRL 10
Position (m)
10 Position (m)
Position (m)
10
c
PEARL
0
MELTS 10
Position (m)
a
16
0
0
2
2
2
2
0 −2 0
20
40
60
80
100
0 −2 0
20
40
60
80
100
Velocity (m/s)
−10
Velocity (m/s)
−10
Velocity (m/s)
−10
Velocity (m/s)
−10
0 −2 0
20
40
60
80
Time Step
Time Step
Time Step
goal forward
goal backward
velocity forward
100
0 −2 0
20
40
60
80
100
Time Step
velocity backward
Fig. 9: Baseline comparison of inference trajectories on four non-parametric tasks for Half-Cheetah embodiment. a, RL2 ; b, PEARL; c, CEMRL; and d, MELTS. For each baseline, the upper plot shows position tracking and the lower plot shows velocity tracking. Dashed horizontal lines denote the corresponding ground-truth targets. a
Goal Tracking MSE per Time Step
Goal Tracking MSE
150
100
50
0
0
10
Velocity Tracking MSE
b
20
30
40 50 60 Time Step
70
80
90
100
90
100
Velocity Tracking MSE per Time Step 15
10
5
0
0
10
20
ReMAP (Ours)
30
40 50 60 Time Step PEARL
RL2
70
80
CEMRL
MELTS
Fig. 10: Tracking MSE comparison between our ReMAP and baseline methods (Half-Cheetah 4 tasks). a, Goal tracking forward and backward MSE over time steps. b, Velocity tracking forward and backward MSE over time steps. Solid curves denote the mean MSE over at least 50 trials for each task, and shaded regions indicate the standard deviation (µ ± σ). Lower MSE indicates more accurate tracking performance. (19.68), and MELTS (20.24), respectively. At the final time step, ReMAP further reduces the goal-tracking MSE to 0.005, compared with 74.91, 40.28, 8.15, and 2.79 for PEARL, RL2 , CEMRL, and MELTS, respectively. This corresponds to absolute MSE reductions of 74.90, 40.27, 8.14, and 2.79, or relative reductions of 99.99%, 99.99%, 99.93%, and 99.81%.
These results indicate that ReMAP not only converges to more accurate goal tracking, but also maintains stable long-horizon execution after the task has been inferred. For velocity tracking, ReMAP obtains an average MSE of 0.34 over the full rollout, which is comparable to CEMRL (0.32) and MELTS (0.32), and substantially lower than PEARL (4.55) and RL2 (4.26). In the early stage of inference, ReMAP also shows faster error reduction: within the first 10 time steps, its velocity-tracking MSE is 1.58, compared with 1.81 for CEMRL, 1.89 for MELTS, 4.07 for PEARL, and 4.46 for RL2 . At the final time step, ReMAP achieves a velocity-tracking MSE of 0.157, compared with 3.73, 3.59, 0.190, and 0.301 for PEARL, RL2 , CEMRL, and MELTS, respectively. This yields absolute MSE reductions of 3.57, 3.43, 0.033, and 0.144, corresponding to relative reductions of 95.78%, 95.62%, 17.42%, and 47.80%. These results show that ReMAP achieves final tracking accuracy comparable to or better than the strongest baselines while reducing inference delay during deployment. Overall, ReMAP reduces the final-step tracking MSE by 94.75%–99.79% over the baselines under joint goal and velocity tracking. These results show that separating task-level meta-knowledge from embodiment-specific control enables more accurate goal tracking, comparable velocity tracking, and faster early-stage deployment than end-to-end Meta-RL baselines. Learning Cost for Cross-Embodiment. Finally, we evaluate the training cost required to achieve cross-embodiment deployment. Although the previous tracking results show that ReMAP achieves comparable or better execution accuracy than the strongest baselines, its main advantage lies in the amount of interaction data required to reach this performance. As shown in Fig. 11, ReMAP requires only approx. 38M environment interactions across the four embodiments, which is about 23.8% of the 160M interactions required by MELTS. This indicates that ReMAP can achieve state-of-the-art deployment performance while incurring substantially lower training costs. The reduction in training cost comes from two sources.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
17
Training Cost Comparison Across Four Target Agents
a
444.0
Additional ablation studies on alternative latent priors are provided in Supplementary Sec. C.
Env steps (M)
400
VI. CONCLUSION 241.0
200
180.0
160.0
38.1
0 b
ReMAP (Ours)
Deploy Adaptation
RL2
PEARL
1.1%
14.5%
CEMRL
MELTS
Simplified Agent 42.0%
Hopper
Half-Cheetah 18.0% 8.7% 15.7%
Walker
Ant
Fig. 11: Training cost comparison between ReMAP and baseline methods across four target agents. a, Total environment steps, measured in millions, required for training convergence. ReMAP achieves substantially lower total cost by reusing high-level meta-knowledge and training only embodimentspecific low-level controllers for target agents. b, Cost decomposition of ReMAP, showing the relative contribution of simplified-agent high-level training, target-agent low-level controller training, and online deployment adaptation. First, task-level meta-knowledge is acquired on the dynamicssimplified agent, where task semantics can be learned more efficiently because the model does not need to solve highdimensional embodiment-specific control at the same time. Once trained, the task inference module and the high-level policy are frozen and reused across target agents, so this cost is amortized over multiple embodiments. In contrast, end-to-end Meta-RL baselines need to relearn or substantially adapt both task inference and control whenever the embodiment changes. Second, the objective of low-level learning in ReMAP is fundamentally different from that of end-to-end Meta-RL. In previous methods, the policy must simultaneously interpret the inferred task context and discover complex dynamicsdependent action patterns, which makes exploration and training highly sample-inefficient. In ReMAP, the low-level controller is trained independently through SMAI-conditioned curriculum learning. It only needs to track semantic-magnitude commands that gradually increase in difficulty, while acquiring embodiment-specific locomotion and exploration capabilities. Because task semantics are handled by the reusable metaknowledge module, the low-level policy can focus on executable motor behavior rather than jointly learning task reasoning and control. This decoupled design substantially reduces the required interaction data while preserving strong downstream execution performance. Overall, these results provide a positive answer to Q3: the learned task-level meta-knowledge can be effectively reused across heterogeneous embodiments.
This paper presented ReMAP, a meta-knowledge reutilization framework for cross-embodiment Meta-RL. ReMAP learns DPMM-regularized task-level meta-knowledge on a dynamics-simplified agent and transfers it to heterogeneous embodiments through SMAI-guided low-level controllers. By decoupling task inference from embodiment-specific execution, the frozen task inference module and high-level policy can be reused without retraining for each target agent. Experiments on multiple locomotion embodiments show that ReMAP learns semantically separable task representations, achieves accurate goal and velocity tracking, and substantially reduces transfer cost compared with end-to-end Meta-RL baselines. These results demonstrate the effectiveness of reusable tasklevel meta-knowledge for efficient cross-embodiment policy transfer. Future work will extend the semantic interface and investigate sim-to-real deployment. R EFERENCES [1] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, “Rl2 : Fast reinforcement learning via slow reinforcement learning,” arXiv preprint arXiv:1611.02779, 2016. [2] J. Beck, R. Vuorio, Z. Xiong, and S. Whiteson, “Recurrent hypernetworks are surprisingly strong in meta-rl,” Advances in Neural Information Processing Systems, vol. 36, pp. 62 121–62 138, 2023. [3] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International conference on machine learning. PMLR, 2017, pp. 1126–1135. [4] Z. Bing, D. Lerch, K. Huang, and A. Knoll, “Meta-reinforcement learning in non-stationary and dynamic environments,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3476– 3491, 2022. [5] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen, “Efficient offpolicy meta-reinforcement learning via probabilistic context variables,” in International conference on machine learning. PMLR, 2019, pp. 5331–5340. [6] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in Conference on robot learning. PMLR, 2020, pp. 1094–1100. [7] Z. Bing, Y. Yun, K. Huang, and A. Knoll, “Context-based metareinforcement learning with bayesian nonparametric models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 10, pp. 6948–6965, 2024. [8] M. C. Hughes and E. Sudderth, “Memoized online variational inference for dirichlet process mixture models,” Advances in neural information processing systems, vol. 26, 2013. [9] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” arXiv preprint arXiv:1803.02999, 2018. [10] T. Nguyen, T. Luu, T. Pham, S. Rakhimkul, and C. D. Yoo, “Robust maml: Prioritization task buffer with adaptive learning process for model-agnostic meta-learning,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 3460–3464. [11] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel, “A simple neural attentive meta-learner,” in International Conference on Learning Representations, 2018. [12] G. Shala, A. Biedenkapp, P. Krack, F. Walter, and J. Grabocka, “Efficient cross-episode meta-rl,” in The Thirteenth International Conference on Learning Representations, 2025. [13] L. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson, “Varibad: a very good method for bayes-adaptive deep rl via meta-learning,” Proceedings of ICLR 2020, 2020.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
[14] L. Li, R. Yang, and D. Luo, “Focal: Efficient fully-offline metareinforcement learning via distance metric learning and behavior regularization,” in International Conference on Learning Representations, 2020. [15] M. Wang, Z. Bing, X. Yao, S. Wang, H. Kai, H. Su, C. Yang, and A. Knoll, “Meta-reinforcement learning based on self-supervised task representation learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, 2023, pp. 10 157–10 165. [16] Y. Lu, H. Li, Y. Li, Y. Lin, and X. Peng, “A survey on deep clustering: from the prior perspective,” Vicinagearth, vol. 1, no. 1, p. 4, 2024. [17] Z. Jiang, Y. Zheng, H. Tan, B. Tang, and H. Zhou, “Variational deep embedding: An unsupervised and generative approach to clustering,” arXiv preprint arXiv:1611.05148, 2016. [18] N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan, “Deep unsupervised clustering with gaussian mixture variational autoencoders,” arXiv preprint arXiv:1611.02648, 2016. [19] D. A. Reynolds et al., “Gaussian mixture models.” Encyclopedia of biometrics, vol. 741, no. 659-663, p. 3, 2009. [20] Y. Meng, Z. Bing, X. Yao, K. Chen, K. Huang, Y. Gao, F. Sun, and A. Knoll, “Preserving and combining knowledge in robotic lifelong reinforcement learning,” Nature Machine Intelligence, vol. 7, no. 2, pp. 256–269, 2025. [21] E. Nalisnick and P. Smyth, “Stick-breaking variational autoencoders,” in International Conference on Learning Representations, 2017. [22] P. Goyal, Z. Hu, X. Liang, C. Wang, and E. P. Xing, “Nonparametric variational auto-encoders for hierarchical representation learning,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5094–5102. [23] M. Ronen, S. E. Finder, and O. Freifeld, “Deepdpm: Deep clustering with an unknown number of clusters,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9861–9870. [24] D. M. Blei and M. I. Jordan, “Variational inference for dirichlet process mixtures,” Bayesian Analysis, vol. 1, no. 1, pp. 121–144, 2006. [25] D. Hejna, L. Pinto, and P. Abbeel, “Hierarchically decoupled imitation for morphological transfer,” in International Conference on Machine Learning. PMLR, 2020, pp. 4159–4171. [26] A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei, “Metamorph: Learning universal controllers with transformers,” arXiv preprint arXiv:2203.11931, 2022. [27] R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” Artificial intelligence, vol. 112, no. 1-2, pp. 181–211, 1999. [28] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013. [29] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning. Pmlr, 2018, pp. 1861–1870. [30] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for modelbased control,” in 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033. Yuan Meng (Student Member, IEEE) received the B.Sc. degree in mechanical engineering in 2020 from RWTH Aachen University, Aachen, Germany, and the M.Sc. degree in mechatronics and robotics in 2023 from the Technical University of Munich, Munich, Germany, where he is currently working toward the Ph.D. degree in computer science with the Informatics 6. His current research interests include robotic lifelong reinforcement learning and Vision-Language-Action model systems.
Bo Wang received the B.Sc. degree in computer science from Ostfalia University of Applied Sciences, Wolfenbuettel, Germany, in 2024. He is currently working toward the M.Sc. degree in Robotics, Cognition, and Intelligence at the Technical University of Munich, Munich, Germany. His current research interests include robotic meta-reinforcement learning, humanoid manipulation, and Vision-LanguageAction model systems.
18
Juan de los Rios received the B.Sc. degree in Engineering Science and the M.Sc. degree in Robotics, Cognition and Intelligence from the Technical University of Munich. His master’s thesis focused on meta reinforcement learning. His research interests include autonomous robotic systems, multi-agent reinforcement learning, and the coordination and orchestration of robotic agents.
Xiangtong Yao (Student Member, IEEE) received the B.Sc. and M.Sc. degrees in computer science from Sun Yat-sen University, Guangzhou, China, in 2017 and 2020, respectively. He is currently working toward the Ph.D. degree in computer science with Informatics 6, Technical University of Munich, Munich, Germany. His current research interests include language control robotics and sensor-based control for manipulation.
Zhenshan Bing (Member, IEEE) received the B.S. degree in mechanical design manufacturing and automation and the M.Eng degree in mechanical engineering from the Harbin Institute of Technology, Harbin, China, in 2013 and 2015, respectively, and the doctorate degree in computer science from the Technical University of Munich, Munich, Germany, in 2019. He worked as a Postdoctoral Researcher with Informatics 6, Technical University of Munich, from 2019 to 2025. He is currently an Associate Professor with the State Key Laboratory for Novel Software Technology, Nanjing University (Suzhou Campus), Suzhou, China
Fuchun Sun (Fellow, IEEE) received the Ph.D. degree in computer science from Tsinghua University, Beijing, China, in 1997. He is currently a Full Professor with the Department of Computer Science and Technology, Tsinghua University. His current research interests include robotic perception and cognition. Dr. Sun was the recipient of the National Science Fund for Distinguished Young Scholars.
Alois Knoll (Fellow, IEEE) received the diploma (M.Sc.) degree in electrical/communications engineering from the University of Stuttgart, Stuttgart, Germany, in 1985, and the Ph.D. (summa cum laude) degree in computer science from the Technical University of Berlin (TU Berlin), Berlin, Germany, in 1988. He served on the Faculty of the Computer Science Department, TU Berlin, until 1993. He joined the University of Bielefeld, Germany, as a Full Professor and served as the Director of the Technical Informatics research group until 2001. Since 2001, he has been a Professor with the Department of Informatics, Technical University of Munich, Munich, Germany