ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer Ajsal Shereef Palattuparambil 1 Thommen George Karimpanal 2 Santu Rana 1
arXiv:2604.08355v1 [cs.AI] 9 Apr 2026
Abstract
typically require costly retraining even when faced with minor changes to a task’s goals or semantics (Zhang et al., 2018).
Reinforcement Learning (RL) agents often struggle to generalize knowledge to new tasks, even those structurally similar to ones they have mastered. Although recent approaches have attempted to mitigate this issue via zero-shot transfer, they are often constrained by predefined, discrete class systems, limiting their adaptability to novel or compositional task variations. We propose a significantly more generalized approach, replacing discrete latent variables with natural language conditioning via a text-conditioned Variational Autoencoder (VAE). Our core innovation utilizes a Large Language Model (LLM) as a dynamic semantic operator at test time. Rather than relying on rigid rules, our agent queries the LLM to semantically remap the description of the current observation to align with the source task. This source-aligned caption conditions the VAE to generate an imagined state compatible with the agent’s original training, enabling direct policy reuse. By harnessing the flexible reasoning capabilities of LLMs, our approach achieves zero-shot transfer across a broad spectrum of complex and truly novel analogous tasks, moving beyond the limitations of fixed category mappings. Code and videos are available here.
A promising direction for closing this gap is to equip agents with a mechanism for “imagination” (Hafner et al., 2019; Nair et al., 2018). Recent work MAGIK (Palattuparambil et al., 2025) demonstrated a novel framework for zeroshot knowledge transfer by training an agent to imagine analogous goals. It learns to disentangle an observation’s task-agnostic features (e.g., an object’s position) from its task-specific features (e.g., its identity as an “apple”) into a discrete class latent c. By programmatically swapping the latent class of a target object (e.g., “orange”) with that of a source object (“apple”), the authors demonstrated how an agent can generate a source-aligned observation and reuse its original policy. However, relying on a predefined, discrete set of classes fundamentally constrains the agent’s analogical capabilities. A critical limitation of this framework is its inability to extrapolate to unseen tasks that share the same action affordance as the source. For instance, while prior work can map a known “orange” to an “apple,” it fails to transfer the policy for picking an “apple” to a previously unseen “banana.” This rigidity severely limits deployment in real-world environments where novel objects are ubiquitous. Such scenarios demand a semantic understanding that extends beyond fixed classification systems. To overcome this bottleneck, we introduce Analogical Semantic Policy Execution via Language Conditioned Transfer (ASPECT), a method grounded in natural language. By leveraging the rich semantic context provided by language, our framework can map a known source policy to any target task, including those involving unseen objects, provided they share the same core action affordance. This significantly improves the generalisability of RL policies, allowing them to adapt to open-world settings without costly retraining.
1. Introduction Humans possess a remarkable capacity for analogical reasoning, enabling us to adapt existing knowledge to new, structurally similar tasks with minimal/zero relearning. If we learn to pick an apple, we can intuitively apply that same skill to pick an orange without extensive trial and error. This fluid transfer of knowledge remains a fundamental challenge for Deep Reinforcement Learning agents, which
Specifically, we generalize the imagination-based transfer framework by replacing the discrete class latent c with a continuous latent space conditioned on natural language. Rather than training a VAE on predefined classes, we utilize a text-conditioned VAE trained on image-text pairs. This approach enables the model to disentangle structural features
1
Applied Artificial Intelligence Initiative, Deakin University, Waurn Ponds, Geelong 2 School of IT, Deakin University, Waurn Ponds, Geelong. Correspondence to: Ajsal Shereef Palattuparambil <[email protected]>.
1
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
z from high-dimensional semantic content derived from textual descriptions.
to accelerate transfer between tasks with different goals. However, these and other traditional transfer learning methods still require a phase of online interaction and fine-tuning, especially when task structures or dynamics change.
Our approach uses a Large Language Model (LLM) as a dynamic semantic operator at test time. In contrast to the original MAGIK framework, which relied on researcherdefined, fixed transformations (e.g., “map red ball to green ball”), our method leverages the LLM’s reasoning capabilities to autonomously determine semantic remappings. By prompting the LLM with a description of the current observation, the agent identifies how to align the novel target task with its existing knowledge base. For instance, the LLM might deduce that, within the context of the agent’s goal, a target “orange” is semantically analogous to a source “apple.”
A related field, domain adaptation, attempts to learn policies that are robust to shifts in observations, often by learning domain-invariant features (Gamrian & Goldberg, 2019). While effective, these methods typically require access to data from the target domain, limiting transferability, and do not explicitly model the semantic analogies between task components. 2.2. Imagination, Planning, and World Models The concept of “imagination” is most prominently featured in model-based RL. Works like Dreamer (Hafner et al., 2019) learn a latent-space world model and then train a policy by imagining future trajectories entirely within this learned model, leading to high sample efficiency. Other approaches have used imagination to generate and select goals (Nair et al., 2018).
This source-aligned description conditions the generative model, allowing the agent to reconstruct the current state by combining the latent feature z of the current state with the LLM-provided remapped captions. Effectively, the agent “reimagines” the novel target scene in the familiar terms of its source task, enabling direct, zero-shot application of the original policy. Integrating generative imagination with the flexible reasoning of LLMs enables our approach to transcend simple one-to-one mappings, facilitating complex, compositional, and truly novel analogical knowledge transfer.
The imagination scheme used by (Palattuparambil et al., 2025), and adopted by our approach, is distinct. It does not imagine future states based on environment dynamics; instead, it “imagines” a translation, an analogical mapping between the components of two different environments. Our method generalizes this by conditioning the translation on natural language, rather than the confined, predefined discrete latents of the original work.
Our contributions are as follows: • We propose a novel framework that leverages a textconditioned VAE and an LLM to achieve flexible, zeroshot policy transfer.
2.3. Relational and Analogical Reasoning
• We introduce the concept of an LLM as a ”semantic operator” to dynamically map target observations to source-task analogues, replacing the rigid, discrete class system of prior work.
The method is fundamentally inspired by human analogical reasoning. This has been most closely studied in Relational Reinforcement Learning (RRL) (Zambaldi et al., 2018), which aims to learn policies that operate over objects and their relations, rather than raw pixel data. This relational structure is a powerful prior for generalization. Our work differs by focusing on the transfer problem: rather than learning a single, general relational policy from scratch, we assume a high-performing policy exists in a source domain and focus on translating a new task into the source policy’s language.
• We demonstrate that our approach can generalize to a significantly wider and more complex range of semantic tasks, including those involving unseen objects.
2. Related Work Our work, ASPECT, is positioned at the intersection of several key research areas in reinforcement learning, including transfer learning, zero-shot generalization, relational reasoning, and imagination-based policy execution.
3. Preliminaries and Background This section provides an overview of the core concepts that form the foundation of our work: Reinforcement Learning (RL), the training of a source policy, Variational Autoencoders (VAEs), and the principles of language-conditioned generative models.
2.1. Transfer Learning and Domain Adaptation in RL The challenge of transferring knowledge across tasks is a long-standing problem in reinforcement learning. Prominent approaches include Successor Features (SFs) (Barreto et al., 2017; Chua et al., 2024), which learn representations that decouple environment dynamics from reward functions 2
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
3.1. Reinforcement Learning (RL)
3.4. LLMs as Reasoning Engines
We formulate our problem within the Reinforcement Learning (RL) framework. An RL environment is typically modeled as a Markov Decision Process (MDP), defined by the tuple (S, A, T , R, γ). Here, S is the space of all possible states, A is the set of actions, T (s′ |s, a) is the state transition function defining the probability of transitioning to state s′ from state s after taking action a, R(s, a) is the reward function, and γ ∈ [0, 1] is the discount factor.
LLMs (Brown et al., 2020) are Transformer-based models trained on web-scale text corpora. While LLMs excel at text generation, their most significant capability for our work is their emergent capacity for in-context learning and zero-shot reasoning. Given a prompt containing a few examples or a structured set of instructions (a “context”), LLMs can perform novel tasks without any gradient updates. This allows them to function as flexible “semantic operators,” capable of translating concepts, performing analogical reasoning, and rephrasing information from one domain to another based on the provided context. We leverage this capability to build our mapping function MLLM (More details in Section 4.2).
The goal of an RL agent is to learn a policy, π(a|s), which is a mapping from states to a distribution over actions. The policy is optimized to maximize P∞ the expected cumulative discounted return Gt = E[ k=0 γ k Rt+k+1 ], which represents the total accumulated reward over time.
4. Methodology 3.2. Source Policy Learning
Our objective is to enable an RL agent, pre-trained on a source task, to perform zero-shot transfer to novel, analogous target tasks (formally defined in Definition 5.2) using an imagination-based mechanism guided by natural language. Building on prior work (Palattuparambil et al., 2025), which used a discrete class system, we introduce a more flexible framework centered around a text-conditioned VAE and an LLM for semantic reasoning.
In our framework, the agent’s source policy πsource is trained in the source environment to solve a specific task, learning to maximize the expected cumulative reward Gt . The core of our methodology is agnostic to the specific algorithm used to train πsource . Any standard deep RL algorithm can be employed. To demonstrate this flexibility, our experiments utilize a variety of prominent algorithms, including on-policy methods like Proximal Policy Optimization (PPO) (Schulman et al., 2017) and off-policy methods like Deep Q-Networks (DQN) (Mnih et al., 2015) (for discrete actions) and Soft Actor-Critic (SAC) (Haarnoja et al., 2018) (for continuous actions).
Let πsource (a|s) denote the policy learned by an RL agent (e.g., using SAC (Haarnoja et al., 2018)) for a source task, operating on states s from the source environment S S . Our goal is to derive an effective policy πtarget (a|st ) for a target task in environment S T , where st ∈ S T , without any direct interaction or training in the target environment. We achieve this by learning a sophisticated imagination function ΨLLM (st , ct→s ) that maps a target state st and the LLMgenerated source-aligned caption to an imagined sourcealigned state simagined . The target policy is then defined as πtarget (a|st ) = πsource (a|ΨLLM (st , ct→s )). See Section 4.2 for the definition of ct→s . In this context, the term “zeroshot” refers to the fact that our method is applied directly to the target task without any re-interaction with the environment and without requiring any additional data collection or training in the target setting.
3.3. Language-Conditioned Generative Models Generative models, such as Variational Autoencoders (VAEs) (Kingma & Welling, 2013), learn a compressed, probabilistic latent representation that captures the underlying structure of data. While traditional VAEs generate data from this latent space alone, conditional variants allow for control over the generative process through auxiliary information. In the context of language conditioning, modern approaches leverage rich embeddings from pre-trained language models (like CLIP (Radford et al., 2021)) to condition generation on free-form text descriptions. This text conditioning is typically integrated into the model architecture using mechanisms like FiLM (Feature-wise Linear Modulation) (Perez et al., 2018) or Cross-Attention (Vaswani et al., 2017), enabling fine-grained semantic control over the synthesized output. Our work builds upon these foundations to enable language-guided imagination.
The core components of our methodology are: (1) a textconditioned VAE trained to reconstruct states based on semantic descriptions, and (2) an LLM acting as a semantic operator to translate target task descriptions into source task analogues at test time. 4.1. Text-Conditioned VAE for Imagination We employ a VAE architecture modified to condition the generative process on natural language descriptions. This VAE is trained offline on a dataset D = {(xi , cs,i )}N i=1 consisting of observations xi (e.g., image frames) paired with corresponding source captions cs,i describing the scene con3
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
tent. These pairs are collected during the agent’s interaction with the source environment while learning πsource . Note that cs,i describes the scene itself and does not inherently contain information about the agent’s task. 4.1.1. A RCHITECTURE AND L ATENT S PACE The VAE consists of an encoder network qϕ (z|x) and a decoder network pθ (x|z, cs ). The encoder maps an input observation x to a distribution over a continuous latent variable z ∈ Rk . The decoder reconstructs the observation x̂ by sampling from z and using the provided source caption cs as guidance for the scene’s semantic content. Specifically, the source caption cs is first embedded into a continuous vector ecs = Etext (cs ) using a pre-trained text embedding model, such as the text encoder from CLIP (e.g., ‘clip-vit-base-patch32’). This text embedding ecs then conditions the decoding process pθ (x|z, ecs ). We implement this conditioning using a combination of Feature-wise Linear Modulation (FiLM) and Cross-Attention layers within the decoder architecture. FiLM layers modulate the intermediate feature maps of the decoder based on the latent code z, while Cross-Attention layers allow the decoder features to attend to the text embedding ecs , integrating the semantic guidance provided by the caption into the reconstruction process. This ensures that the generated image x̂ reflects both the structural information from z and the semantic content specified by cs .
Figure 1. The overall idea of ASPECT. The agent uses an LLM to semantically remap the target observation (e.g., target task: “pick yellow duckie from blue room”) to a source-aligned description (e.g., source task: “pick blue box from grey room”), enabling the direct application of the pre-trained source policy to the target task.
externally, or follow a template based on object detectors. The function of the LLM is to translate this target caption ct into a source-aligned caption ct→s . The LLM manipulates the caption ct based on the source and target task goals, effectively finding a semantic analogy that aligns the current target situation with the agent’s prior experience from the source task. This process can be formulated as ct→s = MLLM (ct , C), where C represents the context provided to the LLM used to retrieve ct→s from ct . This context comprises a description of the environment, the target task (e.g., “Pick yellow duckie”), the source task the agent was trained on (e.g., “Pick blue box”), and the caption ct of the current observation (e.g., “A yellow duckie is on the table”). Along with this context, a directive query is provided (see Appendix B). The LLM then outputs the manipulated, source-aligned caption ct→s (e.g., “A blue box is on the table”).
4.1.2. T RAINING O BJECTIVE AND D ISENTANGLEMENT The VAE is trained to maximize a modified Evidence Lower Bound (ELBO) objective, incorporating both pixel-wise Mean Squared Error and perceptual loss for high-fidelity reconstruction, regularized by a β-weighted KL-divergence term (Higgins et al., 2017). Crucially, to ensure effective zero-shot transfer, we strictly disentangle the spatial latent z from the semantic text embedding ecs . We employ an adversarial training scheme with a Gradient Reversal Layer (GRL) (Ganin et al., 2016), where a discriminator Dψ learns to predict ecs from z via a contrastive InfoNCE loss. The encoder is simultaneously updated to maximize this loss, guarding against semantic leakage and ensuring z captures only structural features orthogonal to the text. The complete training objective is detailed in Appendix E, and we discuss the disentanglement in Appendix F.2.
This querying leverages the LLM’s ability to perform analogical reasoning on-the-fly. Significantly, this approach allows the agent to potentially handle target tasks involving objects or concepts (like the “yellow duckie”) that were entirely absent during the VAE training, relying on the LLM’s general semantic understanding to bridge the gap. The complete zero-shot transfer procedure is summarized in Algorithm 1 in Appendix A. The overall idea of our method is illustrated in (Figure 1).
4.2. LLM as a Semantic Operator for Analogical Mapping The key component enabling flexible, zero-shot transfer is the use of an LLM as a semantic operator, MLLM . At test time, when the agent encounters a state st in the target task, we first obtain a caption ct of this state. This caption can be generated by a Vision-Language Model (VLM), provided
5. Theoretical Analysis of Analogical Transfer In this section, we formally characterize the analogical transfer mechanism of ASPECT. We define analogous tasks using MDP homomorphisms, prove the existence of an ideal 4
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
Definition 5.2 (Analogous Tasks). The target MDP MT is analogous to the source MDP MS if there exists an affordance-preserving mapping Φ.
source-aligned state, and derive a performance degradation bound under approximate semantic mapping. 5.1. Preliminaries
5.3. MDP Homomorphism
Let the source and target tasks be modeled as Markov Decision Processes (MDPs):
Define the state mapping
MS = (SS , A, TS , RS , γ)
h : ST → SS ,
MT = (ST , A, TT , RT , γ)
h(u, v) = (u, Φ(v)).
Definition 5.3 (Exact Homomorphism). The mapping h defines an exact MDP homomorphism if for all s ∈ ST , a ∈ A:
where: • SS and ST are the source and target state spaces,
RT (s, a) = RS (h(s), a),
• A is the shared action space, • TS (s′ | s, a) and TT (s′ | s, a) are transition kernels,
TT (s′ |s, a) = TS (h(s′ )|h(s), a).
• RS (s, a) and RT (s, a) are reward functions,
Under the structural–semantic factorization above, this reduces to direct substitution via Φ.
• γ ∈ (0, 1) is the discount factor.
5.4. Existence of Ideal Source-Aligned State
State Factorization. We assume that states admit a structural–semantic decomposition:
Lemma 5.4 (Existence of Ideal Aligned State). If MT is analogous to MS , then for every sT ∈ ST , there exists a corresponding
s = (u, v)
sideal = h(sT ) ∈ SS .
where:
Proof. Let sT = (u, v). By definition of h:
• u ∈ U denotes structural features (e.g., spatial configuration),
h(sT ) = (u, Φ(v)).
• v ∈ V denotes semantic identity (e.g., object class or affordance role).
Since Φ(v) ∈ VS , we have h(sT ) ∈ SS . Existence follows directly from definition of h.
Thus: 5.5. Value Functions SS ⊆ U × VS ,
ST ⊆ U × VT .
For any policy π : S → ∆(A), define its value function in MDP M as:
5.2. Analogous Tasks via Affordance-Preserving Mapping
π VM (s) = E
Definition 5.1 (Affordance-Preserving Mapping). A mapping Φ : VT → VS
"∞ X
# t
γ RM (st , at ) s0 = s, π .
t=0
5.6. Value Preservation Under Exact Homomorphism
is called affordance-preserving if for all u, u′ ∈ U, v, v ′ ∈ VT , and a ∈ A:
Lemma 5.5 (Value Preservation). If h defines an exact homomorphism, then for any policy πS : SS → ∆(A), define the induced target policy:
RT ((u, v), a) = RS ((u, Φ(v)), a), πT (s) := πS (h(s)). TT ((u′ , v ′ ) | (u, v), a) = TS ((u′ , Φ(v ′ )) | (u, Φ(v)), a).
Then for all sT ∈ ST : 5
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
Value Function.
For any MDP M and policy π:
πT πS VM (sT ) = VM (h(sT )). T S π VM (s) = E
Proof. From reward preservation:
"∞ X
# t
γ R(st , at ) s0 = s .
t=0
Define the Bellman operator:
RT (s, a) = RS (h(s), a). From transition preservation:
π (TM V )(s) =
TT (s′ |s, a) = TS (h(s′ )|h(s), a).
π(a|s) R(s, a) A Z +γ T (s′ |s, a)V (s′ )ds′ da. Z
S
Thus the Bellman equations coincide: πT VM (s) = E T
Lemma 5.6 (Contraction Property). For any policy π, the Bellman operator is a γ-contraction:
πT RT (s, a) + γVM (s′ ) T
π π ∥TM V1 − T M V2 ∥∞ ≤ γ∥V1 − V2 ∥∞ . π π Hence TM has a unique fixed point VM .
πS πS = E RS (h(s), a) + γVM (h(s′ )) = VM (h(s)). S S
Proof. For any state s: π π |(TM V1 )(s) − (TM V2 )(s)| Z h Z i = π(a|s) γ T (s′ |s, a)(V1 (s′ ) − V2 (s′ ))ds′ da ZA Z S ≤γ π(a|s) T (s′ |s, a) V1 (s′ ) − V2 (s′ ) ds′ da ZA ZS ≤γ π(a|s) T (s′ |s, a)∥V1 − V2 ∥∞ ds′ da A S Z Z = γ∥V1 − V2 ∥∞ π(a|s) T (s′ |s, a)ds′ da.
5.7. Induced Policies and Value Functions Let the source and target MDPs be
MS = (SS , A, TS , RS , γ), MT = (ST , A, TT , RT , γ),
A
where γ ∈ (0, 1).
Since T (·|s, a) isRa probability distribution over the state space, we have S T (s′ |s, a)ds′ = 1. Similarly, since π(·|s) is a probability distribution over actions, we have R π(a|s)da = 1. Thus: A
Let h : ST → SS be an exact MDP homomorphism and
π π |(TM V1 )(s) − (TM V2 )(s)| ≤ γ∥V1 − V2 ∥∞ .
Ψ : ST → SS be the approximate state mapping produced by ASPECT. Source Policy.
S
Taking the supremum over all s yields the contraction property. By the Banach fixed-point theorem, since γ ∈ (0, 1), π TM has a unique fixed point.
Let πS : SS → ∆(A)
5.8. Performance Degradation Under Approximate Mapping
be a stochastic policy.
Let Induced Target Policies.
We define two target policies: sideal := h(sT ),
(πS ◦ h)(a|sT ) := πS (a|h(sT )),
ŝ := Ψ(sT ).
Assumption 5.7 (TV-Lipschitz Source Policy). There exists LT V > 0 such that for all s1 , s2 : DT V (πS (·|s1 ), πS (·|s2 )) ≤ LT V ||s1 − s2 ||
πΨ (a|sT ) := πS (a|Ψ(sT )). 6
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
Applying the LT V -Lipschitz assumption and the state approximation bound ||Ψ(s) − h(s)|| ≤ ϵ:
Assumption 5.8 (Bounded Action-Value). There exists Qmax such that |QπTS ◦h (s, a)| ≤ Qmax
≤ 2Qmax LT V ||Ψ(s) − h(s)|| ≤ 2LT V Qmax ϵ ∀s, a. Substituting both terms back into our original pointwise inequality yields:
Theorem 5.9 (Performance Degradation Bound). Suppose the source policy πS is LT V -Lipschitz continuous with respect to the Total Variation distance, meaning DT V (πS (·|s1 ), πS (·|s2 )) ≤ LT V ||s1 − s2 || for all s1 , s2 ∈ SS .
|V1 (s) − V2 (s)| ≤ γ||V1 − V2 ||∞ + 2LT V Qmax ϵ Since this inequality holds for every state s ∈ ST , it must also hold for the supremum over all states. Taking the supremum of the left side gives:
If the approximate state mapping Ψ produced by ASPECT satisfies ||Ψ(s) − h(s)|| ≤ ϵ for any target states s ∈ ST , then the maximum value function degradation across all states (measured in the supremum norm) is bounded by: πS◦h πΨ ||VM − VM ||∞ ≤ T T
||V1 − V2 ||∞ ≤ γ||V1 − V2 ||∞ + 2LT V Qmax ϵ Rearranging the terms completes the proof:
2LT V Qmax ϵ 1−γ
(1 − γ)||V1 − V2 ||∞ ≤ 2LT V Qmax ϵ
πS◦h πΨ and V2 := VM . Define the Proof. Let V1 := VM T T πS◦h πΨ Bellman operators T1 := TMT and T2 := TM . T
||V1 − V2 ||∞ ≤
2LT V Qmax ϵ 1−γ
Since V1 and V2 are fixed points of their respective Bellman operators: V1 (s) = (T1 V1 )(s)
and
6. Experiments
V2 (s) = (T2 V2 )(s)
In this section, we present the experimental evaluation of ASPECT. We describe the environments, the source and target tasks, and the baselines used to validate our natural language-conditioned imagination framework.
For any arbitrary target state s ∈ ST , we evaluate the absolute difference and apply the triangle inequality by adding and subtracting (T1 V2 )(s): |V1 (s) − V2 (s)| = |(T1 V1 )(s) − (T2 V2 )(s)|
6.1. Experimental Setup We evaluate ASPECT across three distinct environments designed to test complementary aspects of generalization: MiniGrid, MiniWorld, and a custom Fragile Object Manipulation environment. Visualizations of these environments are shown in Figure 2. These environments differ in observation modality (image-based vs. feature-based), action space (discrete vs. continuous), reward structure (sparse vs. dense), and the underlying RL algorithm used for the source policy (DQN, PPO, SAC), allowing for a comprehensive analysis of the proposed method’s agnosticism. All episodes terminate either upon successful task completion or when the maximum number of timesteps is reached. All experiments were run for 5 random seeds. Implementation details of the RL policies are provided in Appendix D.
≤ |(T1 V1 )(s) − (T1 V2 )(s)| + |(T1 V2 )(s) − (T2 V2 )(s)| For the first term, because the Bellman operator is a γcontraction in the supremum norm, the difference in expectations from state s is strictly bounded by the maximum possible difference across all states: |(T1 V1 )(s) − (T1 V2 )(s)| ≤ γ||V1 − V2 ||∞ For the second term, we evaluate the difference caused by the policy shift: |(T1 V2 )(s) − (T2 V2 )(s)| Z = (πS (a|Ψ(s)) − πS (a|h(s)))QVT2 (s, a)da A
6.1.1. M INI G RID
Applying the absolute value, bounding the action-value by Qmax (Assumption 3), and using the definition of Total Variation distance: Z ≤ Qmax |πS (a|Ψ(s)) − πS (a|h(s))| da
MiniGrid (Chevalier-Boisvert et al., 2023) is a 2D gridworld that provides pixel-based top-down observations. MiniGrid supports discrete action spaces, enabling rapid prototyping of navigation and object-interaction tasks. The source policy is trained using DQN. The environment uses a sparse reward structure, where the agent receives a positive reward only upon successful task completion.
A
= 2Qmax DT V (πS (·|Ψ(s)), πS (·|h(s))) 7
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
6.1.2. M INI W ORLD
6.2. Source and Target Task Definitions
MiniWorld (Chevalier-Boisvert et al., 2023) is a 3D firstperson simulator that provides egocentric, pixel-based visual observations and supports discrete control. The source policy is trained using PPO. Unlike MiniGrid, MiniWorld employs a dense reward setting, where the agent receives incremental shaping rewards for approaching goal objects in addition to a terminal success reward. This setting allows us to test visual generalization and affordance transfer under richer sensory inputs.
For each environment, we train a source policy πsource on source task, and subsequently evaluate its zero-shot transfer performance on multiple target tasks that introduce new objects, altered visual contexts, or both. 6.2.1. S OURCE TASK S ETTINGS • MiniGrid: Pick the red ball and avoid the green ball. • MiniWorld: Pick the blue box and avoid the green ball.
6.1.3. F RAGILE O BJECT M ANIPULATION E NVIRONMENT
• Custom Env: Pick both the light circle (low force threshold) and heavy square (high force threshold) without breaking either.
To explicitly evaluate affordance understanding and forcesensitive control, we developed a custom feature-based environment. The agent interacts with fragile objects, each characterized by a specific fragility threshold. If the applied force exceeds this threshold, the object breaks. The source policy is trained using SAC. The agent operates in a continuous, three-dimensional action space (turn angle, move distance, apply force). The reward function is dense: the agent is penalized for breaking an object, rewarded for successful pickups, and receives a positive shaping reward for approaching the target.
6.2.2. TARGET TASK S ETTINGS (E VALUATION ) We evaluate ASPECT on three categories of generalization challenges, as detailed in Table 1: 1. Case 1 (Unseen Objects): Tests whether the agent can reuse a learned skill for unseen objects (e.g., picking a “yellow duckie” instead of a “blue box”) by leveraging affordance-based analogies. 2. Case 2 (Combined Shift): Challenges the agent to handle both visual shifts (e.g., texture changes) and semantic shifts (unseen objects) simultaneously. 3. Case 3 (Reversed Task): Evaluates robustness to conflicting priors, where the object associated with reward in the source task becomes a distractor in the target task.
(a) MiniGrid
(b) MiniWorld
Table 1. Target Task Settings (Evaluation). The agent must generalize zero-shot to these novel scenarios.
(c) Manipulation
Figure 2. Visualizations of the three environments used in our experiments: (a) MiniGrid (2D navigation), (b) MiniWorld (3D egocentric), and (c) Manipulation (continuous control).
The observation is a 12-dimensional feature vector that encodes the agent’s orientation (as sin / cos of the heading angle), the object’s relative position and bearing (as sin / cos, normalized by environment size), one-hot encodings for object type (circle/square) and weight (light/heavy), and boolean flags indicating whether the object has been picked or broken.
Case
Env.
Target Task Description
1. Unseen Objects (Different reward object)
MiniGrid MiniWorld Custom
Pick purple box, avoid green ball. Pick yellow duckie, avoid green ball. Pick heavy circle & light square (inverted weights).
2. Combined Shift (Visual + Semantic)
MiniGrid MiniWorld
Blue Walls. Pick purple box. Wood/Brick textures. Pick yellow duckie.
3. Reversed Task (Prior as Distractor)
MiniGrid
Pick purple box, avoid red ball (source target). Pick yellow duckie, avoid blue box (source target).
MiniWorld
7. Experimental Results
Together, these environments allow us to demonstrate that ASPECT is agnostic to both observation modality (imagebased or feature-based), control type (discrete or continuous), and the RL algorithm (DQN, PPO, or SAC), while remaining effective across sparse and dense reward regimes.
In this section, we detail the experimental results. We conduct a series of experiments designed to evaluate the primary capabilities and advantages of ASPECT. Our evaluation aims to answer two key questions: 8
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer Table 2. Consolidated Experimental Results for Case 1 (Generalization to Unseen Objects). Metrics are reported as counts (out of 10) ± std dev. ”Tgt” = Target, ”Dstr” = Distractor, ”Lift” = Successful Lifts, ”Brk” = Broken Objects.
1. Can ASPECT generalize to truly novel tasks? We test its ability to perform zero-shot knowledge transfer to target tasks that share the same underlying affordance (e.g., ”picking”) but involve previously unseen objects or different, though semantically similar, observations.
M INI W ORLD M ETHOD
T GT
D STR
M INI G RID T GT
D STR
M ANIP. L IFT
B RK
S OURCE 2.80 ± 0.80 0.20 ± 0.20 0.00 ± 0.00 0.00 ± 0.00 2.80 ± 1.06 4.60 ± 0.04 FT (C ONV.) 10.0 ± 0.00 0.60 ± 0.40 9.60 ± 0.24 0.00 ± 0.00 9.80 ± 0.20 0.00 ± 0.00 FT (L IM .) 8.00 ± 1.04 0.40 ± 0.24 7.20 ± 0.86 0.40 ± 0.40 6.60 ± 0.87 2.40 ± 1.51 SF-S IMP. 0.20 ± 0.20 0.00 ± 0.00 5.60 ± 1.28 0.00 ± 0.00 SF-R EC . 0.00 ± 0.00 0.00 ± 0.00 9.00 ± 0.20 0.00 ± 0.00 ASPECT 8.40 ± 0.24 0.02 ± 0.02 9.40 ± 0.40 0.00 ± 0.00 9.60 ± 0.24 0.40 ± 0.24
2. How data-efficient is ASPECT compared to finetuning? We compare the zero-shot performance of ASPECT against the data efficiency of fine-tuning the source policy on the target task. This highlights the sample complexity advantage of our approach.
7.2. Zero-Shot Generalization Results 7.1. Baselines
We evaluate the zero-shot transfer performance of ASPECT across three generalization scenarios involving novel objects and observational changes.
We compare ASPECT against the following baselines: • SF-Simple (Chua et al., 2024): A successor featurebased method that learns transferrable representations without complex auxiliary tasks.
7.2.1. C ASE 1: G ENERALIZATION TO U NSEEN O BJECTS Results for this setting, where the agent transfers its skill to previously unseen objects, are summarized in Table 2.
• SF-Reconstruction (Zhang et al., 2017): An approach utilizing successor features combined with a reconstruction auxiliary task to learn robust state representations for navigation across similar environments.
In the MiniWorld environment (Table 2), where the target object is a “yellow duckie” (unseen during training), the standard PPO policy fails completely (2.80 success), as it relies on specific visual features of the source object (“blue box”). Similarly, even though the successor feature baselines (SF-Simple and SF-Reconstruction) are allowed to interact with the target environment, they fail to generalize (see Figure 6 in Appendix F.1). In contrast, ASPECT, which operates zero-shot without any target interaction, achieves a success rate of 8.40 ± 0.24, outperforming the PPO policy fine-tuned for 20K steps (8.00 ± 1.04). While it strictly underperforms the fully converged fine-tuned PPO upper bound (10.00 ± 0.00), it achieves this without any gradient updates on the target task. This demonstrates the effectiveness of the LLM-guided mapping in bridging the semantic gap between “blue box” and “yellow duckie”.
• PPO/DQN/SAC (Source): The original source policy evaluated directly on the target task (zero-shot) to measure the immediate transfer gap. • PPO/DQN/SAC (Fine-tuned): The source policy finetuned on the target task, providing an upper bound on performance (or a strong adaptive baseline) given access to target environment interactions. We denote fine-tuning until convergence as FT (Conv.) and finetuning with limited steps as FT (Lim.). While our approach is conceptually inspired by MAGIK (Palattuparambil et al., 2025), we do not include it as a baseline. MAGIK is designed to transfer skills between known objects using discrete, one-hot class representations. However, ASPECT focuses on generalization to unseen objects where such pre-defined one-hot vectors cannot be constructed, rendering MAGIK inapplicable to these experimental settings.
In the MiniGrid environment (Table 2), ASPECT again demonstrates rigorous zero-shot performance (9.40 ± 0.40), significantly outperforming the DQN policy fine-tuned for 50K steps (7.20 ± 0.86) and effectively matching the fully converged fine-tuned DQN baseline (9.60 ± 0.24). While SF-Reconstruction performs better here (9.00 ± 0.20) than in MiniWorld, potentially due to the simpler visual grid structure, it still lags behind ASPECT, despite having the advantage of environmental interaction. The baseline DQN policy completely fails to transfer.
First, we briefly describe our captioning process. To generate noise-free semantic descriptions of the environment, we employed a structured captioning module (see Appendix C). This module populates a predefined template with sensor data, such as object type and location, ensuring accurate and consistent input for the language-conditioned imagination process without requiring complex vision-language model inference or post-processing. For the semantic mapping MLLM , we utilized Grok-4.1-fast and Gemini 2.5 Flash as our Large Language Models.
Finally, in the Fragile Object Manipulation task (Table 2), the challenge involves inverting physical properties: picking a “heavy circle” and “light square” when trained on the opposite. The standard SAC policy struggles significantly, lifting only 2.80 ± 1.06 objects on average, and incurs a high failure rate, breaking an average of 4.60 objects (as 9
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer Table 4. Consolidated Experimental Results for Case 3 (Reverse Task). Metrics are reported as counts (out of 10) ± std dev. ”Old Target” refers to the object rewarding in source but distracting in target.
shown in the “Num Objects broken” column). ASPECT successfully navigates this affordance inversion, lifting 9.60 ± 0.24 objects, which is comparable to the fine-tuned SAC expert (9.80 ± 0.20) and significantly outperforms the SAC policy fine-tuned for 10K steps (6.60 ± 0.87), by correctly mapping the target objects to their source counterparts based on the “fragility” interactions described in text.
M INI W ORLD M ETHOD S OURCE FT (C ONV.) FT (L IM .) SF-S IMPLE SF-R ECONSTRUCTION ASPECT
7.2.2. C ASE 2: C OMBINED G ENERALIZATION Case 2 introduces a more difficult challenge: generalising to tasks that involve both a novel object and a shift in environmental observations (e.g., room colour or texture, See Figure 9 for more variations). The results are presented in Table 3.
T GT
D STR
M INI G RID T GT
TARGET
O LD TARGET
2.80 ± 0.66 9.80 ± 0.24 5.60 ± 1.36 0.00 ± 0.00 0.00 ± 0.00 8.40 ± 0.54
8.20 ± 0.37 0.00 ± 0.00 0.40 ± 0.24 0.00 ± 0.00 0.00 ± 0.00 0.20 ± 0.20
0.00 ± 0.00 9.80 ± 0.20 3.40 ± 0.87 3.80 ± 1.68 1.20 ± 0.96 9.00 ± 0.54
7.20 ± 0.86 0.00 ± 0.00 0.80 ± 0.37 0.00 ± 0.00 0.00 ± 0.00 0.40 ± 0.24
Results in MiniGrid (Table 4) follow a similar pattern. The DQN baseline is completely fixated on the “red ball” (source reward), picking it 7.20 ± 0.86 times and never picking the correct “purple box”. ASPECT successfully generalizes, picking the “purple box” 9.00 ± 0.54 times, dramatically outperforming the DQN policy fine-tuned for 50K steps (3.40 ± 0.87). These results demonstrate that the LLM guides the hallucination of the target environment according to the current and the known task, allowing the agent to effectively filter out obsolete reward signals.
Table 3. Consolidated Experimental Results for Case 2 (Combined Generalization). Metrics are reported as counts (out of 10) ± std dev. ”Tgt” = Target, ”Dstr” = Distractor.
M ETHOD
O LD TARGET
the “blue box” 8.20 ± 0.37 times, while rarely picking the correct target (“yellow duckie”, 2.80 ± 0.66). In contrast, ASPECT overcomes this bias, achieving 8.40 ± 0.54 success on the novel target, which is significantly higher than the PPO policy fine-tuned for 20K steps (5.60 ± 1.36). This is because the LLM hallucinate the rewarding object (“blue box”) as a distractor (“green ball”) in the source environment, and the agent ignores it.
In MiniWorld (Table 3), the agent faces a scene with a wooden floor and brick walls (vs. grass/concrete in source) and must pick a “yellow duckie”. The standard PPO baseline struggles significantly (3.20 ± 0.66). However, ASPECT achieves a success rate of 8.80 ± 0.37, outperforming the PPO policy fine-tuned for 20K steps (7.60 ± 0.50) and approaching the fully converged fine-tuned PPO baseline (9.20 ± 0.37).
M INI W ORLD
M INI G RID
TARGET
D STR
S OURCE 3.20 ± 0.66 0.20 ± 0.20 0.00 ± 0.00 0.00 ± 0.00 FT (C ONV.) 9.20 ± 0.37 0.40 ± 0.24 9.80 ± 0.20 0.00 ± 0.00 FT (L IM .) 7.60 ± 0.50 0.40 ± 0.24 6.20 ± 0.73 0.60 ± 0.24 SF-S IMP. 0.00 ± 0.00 0.00 ± 0.00 4.80 ± 1.62 0.00 ± 0.00 SF-R EC . 0.00 ± 0.00 0.00 ± 0.00 4.60 ± 2.09 0.00 ± 0.00 ASPECT 8.80 ± 0.37 0.04 ± 0.24 8.80 ± 0.58 0.40 ± 0.24
7.3. Data Efficiency and Sample Complexity A key advantage of our approach is its data efficiency. By leveraging the semantic priors of the LLM and the structural priors of the VAE, ASPECT enables immediate transfer without the need for environment interaction in the target domain.
Similarly, in the MiniGrid environment (Table 3), where the wall colour changes to blue, ASPECT maintains high performance (8.80 ± 0.58). It significantly outperforms the DQN policy fine-tuned for 50K steps (6.20 ± 0.73) and closely trails the fully converged fine-tuned DQN upper bound (9.80 ± 0.20). This result highlights the robustness of our text-conditioned imagination to compound distribution shifts. Qualitative visualizations of this process are shown in Appendix F.3.
To quantify this “zero-shot gap,” we compare ASPECT’s performance against the fine-tuning curves of standard RL baselines. In the MiniWorld environment, ASPECT outperforms the PPO baseline fine-tuned for 20K steps across all cases (Cases 1, 2, and 3) (see Appendix F, Figure 3). Similarly, in the MiniGrid environment, ASPECT consistently outperforms the DQN baseline fine-tuned for 50K steps across all cases (see Appendix F.1, Figure 4). Furthermore, in the Fragile Object Manipulation environment, ASPECT outperforms the SAC baseline fine-tuned for 10K steps (see Appendix F.1, Figure 4d).
7.2.3. C ASE 3: U NSEEN O BJECT AND R EVERSED TASK In this scenario, we evaluate the agent’s robustness to conflicting priors. The object that was rewarding in the source task is present in the target task but is now a distractor (or non-rewarding), while a novel object is the target.
This represents a significant saving in sample complexity, which is particularly critical for real-world applications where data collection is expensive or dangerous. While finetuning eventually yields slightly higher asymptotic perfor-
In MiniWorld (Table 4), the source rewarding object (“blue box”) acts as a distractor. The standard PPO policy exhibits a strong bias towards its training prior, mistakenly picking 10
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
mance (e.g., reaching 10.0 success in MiniWorld), ASPECT provides a “jump-start” that equates to tens of thousands of training steps, effectively bypassing the initial exploration and adaptation phase.
to “imagine” and solve analogous target tasks by relating them to prior experiences. Our experiments across diverse environments ranging from grid worlds to continuous manipulation demonstrated that ASPECT can robustly generalize to unseen objects, visual shifts, and even contradictory reward structures without any training in the target domain. Furthermore, we showed that this method remains effective even when relying on noisy, unstructured captions from vision-language models, highlighting its potential for realworld applications.
7.4. Robustness to Unstructured VLM Captions While structured captions ensure consistency, they rely on predefined templates that may not scale to open-ended scenarios. To evaluate ASPECT’s robustness to variable and unstructured language, we conducted experiments on the MiniWorld environment using captions generated by a VisionLanguage Model (VLM).
However, our approach has limitations. First, reliance on the generative capabilities of the VAE means the system is susceptible to imagination artifacts. As detailed in Appendix F.4, the model can struggle with extreme close-ups or fail to materialize remapped objects in the imagined scene, potentially leading to policy failure. Second, by leveraging Large Language Models (LLMs) as semantic operators, our system inherits the known limitations of these models, including hallucinations, biases, and unpredictability. An incorrect semantic mapping generated by the LLM could lead to agent behaviors that are misaligned with the intended user goals, posing safety risks in critical applications. Future work must address robust verification and safety constraints to mitigate these risks before deployment in sensitive domains.
We used the nvidia/nemotron-nano-12b-v2-vl model to generate captions from observation frames, prompting it with both the image and auxiliary sensor data (object type and location). Due to the low resolution of the observations (80 × 80), the VLM occasionally produced inaccuracies. To mitigate this, we implemented a verification step where captions were checked for completeness; inaccurate instances were re-generated using stronger models (Grok-4.1 and Gemini 2.5 Flash) or post-processed to remove hallucinations. The resulting captions were significantly more variable in length and structure compared to the template-based approach. To handle these longer, unstructured descriptions (see Appendix C for examples), we replaced the standard CLIP text encoder with LongCLIP (Zhang et al., 2024), which supports inputs exceeding the 77-token limit of standard CLIP.
Impact Statement This paper presents work whose goal is to advance the field of Reinforcement Learning, specifically focusing on zeroshot generalization through language-conditioned imagination. Our method, ASPECT, demonstrates the potential to create more adaptable agents capable of operating in diverse and novel analogous environments without extensive retraining. This has positive implications for the scalability of autonomous systems in real-world settings, such as robotics and personalized assistants.
Table 5 (Appendix F) presents the results of ASPECT using these noisy, unstructured captions across all three generalization cases in MiniWorld. Remarkably, the method maintains high performance, achieving success rates comparable to those obtained with clean, structured captions (e.g., 8.60 ± 0.50 for Case 1 vs. 8.40 ± 0.24 with structured text). In all cases, the agent successfully identifies and picks the rewarding object while completely avoiding the distractor (0.00 failure rate). These results underscore ASPECT’s ability to extract relevant semantic cues even from noisy, variable-length natural language descriptions, further validating the flexibility of the text-conditioned imagination framework. Crucially, this capability suggests that our method can be applied to real-world settings where structured captions are unavailable, a constraint that would otherwise severely limit real-world applicability.
References Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020.
8. Discussion and Limitations In this work, we introduced ASPECT, a novel framework for zero-shot policy transfer that leverages the semantic reasoning of Large Language Models and the generative capabilities of text-conditioned VAEs. By treating the LLM as a dynamic semantic operator, our approach enables agents
Chevalier-Boisvert, M., Dai, B., Towers, M., de Lazcano, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. Minigrid & miniworld: Modular & customizable 11
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023.
European Conference on Artificial Intelligence (ECAI 2025), volume 413 of Frontiers in Artificial Intelligence and Applications, pp. 2874–2881. IOS Press, 2025. doi: 10.3233/FAIA413.
Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PmLR, 2021.
Chua, R., Ghosh, A., Kaplanis, C., Richards, B. A., and Precup, D. Learning successor features the simple way. Advances in Neural Information Processing Systems, 37: 49957–50030, 2024. Gamrian, S. and Goldberg, Y. Transfer learning for related reinforcement learning tasks via image-to-image translation. In International conference on machine learning, pp. 2063–2072. PMLR, 2019.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL http: //jmlr.org/papers/v22/20-1364.html.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., March, M., and Lempitsky, V. Domainadversarial training of neural networks. Journal of machine learning research, 17(59):1–35, 2016.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861–1870. Pmlr, 2018.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017. Zambaldi, V., Raposo, D., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., Tuyls, K., Reichert, D., Lillicrap, T., Lockhart, E., et al. Relational deep reinforcement learning. arXiv preprint arXiv:1806.01830, 2018.
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. betavae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
Zhang, A., Ballas, N., and Pineau, J. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937, 2018. Zhang, B., Zhang, P., Dong, X., Zang, Y., and Wang, J. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision, pp. 310–325. Springer, 2024.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529–533, 2015.
Zhang, J., Springenberg, J. T., Boedecker, J., and Burgard, W. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371–2378. IEEE, 2017.
Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018. Palattuparambil, A. S., Karimpanal, T. G., and Rana, S. Magik: Mapping to analogous goals via imaginationenabled knowledge transfer. In Proceedings of the 28rd 12
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
A. Algorithm Pseudocode Algorithm 1 Zero-Shot Action Selection via LLM-Conditioned Imagination Input: Target observation st , VLM V (·), LLM MLLM (·, context), Text Encoder Etext (·), VAE Encoder qϕ (z|x), VAE Decoder pθ (x|z, ed ), Pre-trained source policy πsource (a|s) Output: Action a to execute in the target environment // Step 1 & 2: Observe and Describe Target State ct ← V (st ) // Step 3: Semantic Manipulation via LLM ct→s ← MLLM (ct , context) {Get source-aligned description} // Step 4: Encode Semantic-Orthogonal Features z ∼ qϕ (z|st ) // Step 5: Generate Imagined State ect→s ← Etext (ct→s ) {Embed source caption} simagined ∼ pθ (x|z, ect→s ) {Reconstruct with z and source text embedding} // Step 6: Execute Source Policy a ∼ πsource (a|simagined ) return a
B. LLM & VLM Prompts In this section, we provide the specific prompts and context used for the captioning module and the LLM semantic operator MLLM . B.1. System Prompt: Observation Captioning You are a high-precision computer vision annotator for the MiniWorld 3D environment for RL research. Your task is to convert low-resolution visual observations into structured, spatially accurate text descriptions. ### 1. OBJECT IDENTIFICATION The environment contains interactable objects of various types and colors. * **Standard Objects:** Common examples include "Blue Box," "Green Ball," "Yellow Duckie," "Blue Box," "Medkit," and "Key." * **General Rule:** If you see an object that is not in the list above, describe it using **[Color] + [Shape]** (e.g., "Purple Cylinder," "White Cube"). * **Multiple Objects:** If multiple objects are present, list them all (e.g., "A Blue Box and a Green Ball"). ### 2. ENVIRONMENT & SKY * **Walls & Floor:** Describe the specific color and texture visible (e.g., "Gray concrete wall," "Green grass floor"). * **Sky:** Include the phrase **"Blue sky"** ONLY if the sky is visible. If the wall obscures the sky completely (e.g., close-up view), do not mention the sky. ### 3. SENSOR DATA INTEGRATION You will be provided with "Additional Sensor Data (Ground Truth)". * **MANDATORY:** You MUST include the exact **distance** and **angle** numeric values provided in the sensor data for every visible object. * **Direction:** Explicitly state the direction (e.g., "to the left", "to the right") as given. **Constraints:** 13
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
* **NO META-COMMENTARY:** NEVER use the phrase "sensor data" or "indicated by sensor data" in the output caption. The sensor data is context for YOU, not text to be quoted. * **Length Limit:** Keep captions concise and small. For empty scenes, use a sentence describing the environment and state no objects are visible. * If no objects are visible, do NOT justify why. simply state the layout of the scene (Wall, floor and sky if available) and mention that there is no objects. * All the image will contain a wall and the floor. Caption should contain a description of the wall and the floor. Remember caption should be small and concise. * Give extra emphasis on the objects in the sensor data. ensure they are properly described and the distance and angle are correctly mentioned. * DO NOT use any starting phrases such as "The image depicts/shows". B.2. System Prompt: Semantic Alignment (Source Description) We used the same system prompt across all environments. The system prompt for MLLM is provided below: You are an imagination reasoning assistant that simulates how a reinforcement learning (RL) agent mentally hallucinates its observation to reuse known skills and solve new target tasks. The user prompt will contain the following: 1. 2. 3. 4.
A brief about the environment. What agent knows. This is the Context, C What is the target task? A description of the current observation
###ROLE AND PURPOSE You must: 1. Show detailed reasoning | step-by-step interpretation of how imagination occurs. 2. Output a final JSON containing the transformed or unchanged scene description. You are not just answering but *solving the target task through reasoning* | explain what changes are needed, why, and finally give the exact JSON result. ###HOW IMAGINATION WORKS - The agent performs strictly its known skills (source tasks). - When the target task differs (Agent absolutely can’t perform the target task even if there is a subtle differences such as object type/color or background. Give extra emphasis on the differences between source and target and map the target to source wherever required), the agent imagines | mentally alters its perception of the scene so the new goal is solvable using the same skill. - If you determine that there are changes between the target and source and imagination would be necessary, make the changed description as close as possible to the source task. - Imagination occurs strictly in **observation space**, not the physical world. 1. **Minimal Transformation** - Modify only what is necessary to make the target solvable. - Preserve spatial layout, geometry, and environment details. 2. **Affordance Reasoning** 14
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
- If two objects afford the same action (e.g., pick, push, open), they can be substituted. - Example: - Known: "pick red ball" - Target: "pick green ball" → Imagine the green ball as red. This will direct the agent to pick the red ball, as it is a known skill. In reality, it is picking the green ball. - If the target task requires dealing with objects that the agent doesn’t know, the agent must *imagine* that object as the closest recognizable object it knows provided affordance are matching. 3. **Multi-Object or Sequential Tasks** - Give proper attention to the spatial position if there are multiple object. Don’t mix up the positions. 4. **No Fabrication** - Never add or invent objects or properties not present in the input scene. 5. **Realism and Consistency** - Maintain the original tone, structure, style, and spatial wording. Do not invent additional texts. - Modify only essential object properties (color, shape, size, etc.).
###REASONING STYLE - Think step-by-step, like a human reasoning through perception. - Explain: 1. What the agent currently knows. 2. What the target task is. 3. What are the differences between the target and the source task. 4. How to map such differences to the source task. 5. Whether the subtask division is necessary. 6. What is visible in the current scene? 7. What minimal changes are needed and why? 8. Can the minimal changes help the agent to solve the target task by mentally imagining an altered scene?. Check against each of the subtasks. Discard if the change doesn’t affect the subtask. - Use clear, causal reasoning before outputting the JSON. - Stop reasoning once the decision is made.
###OUTPUT FORMAT After reasoning, always output only valid JSON in this format: { "imagine": true | false, "description": "<rewritten or unchanged scene description>" } - "imagine": true → if imagination was applied to alter the scene. - "imagine": false → if no change was necessary or possible. - "description": the final complete and realistic scene. 15
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
- Do **not** include extra commentary, code, or markdown after the JSON. B.3. Context Construction The context C provided to the LLM consists of four main components: 1. A brief description of the environment dynamics and rules. 2. The source task description (what the agent knows). 3. The target task description. 4. A description of the current observation. Example (MiniWorld): • Environment Context: - The agent operates in a partially observable 3D gridworld-like room. The agent sees a portion of the room. - At the start of each episode, the agent and objects are randomly initialised in the environment. - The agent can perform the following actions: rotate left/right, move forward/backward, and pick up objects. - The environment may contains different objects of different color. - The agent task is to pick/avoid the objects according to the mission string. - Since, the observation is partial, the agent can explore the environment by moving around to find the objects to pick. - Once one object is picked, the object dissapears from the scene and it is added to agent’s inventory, which it can hold forever. This doesn’t prevent picking another object later. - The agent can store multiple objects in it’s inventory. - Non-interactive elements (walls, floor, background) cannot be acted upon. - The agent receives a reward upon successfully completing the Target task (for example, picking the specified object from specified room). - Each episode ends once the Target task is completed or a maximum step limit is reached.
• What Agent Knows: “Pick the blue box and avoid green ball from the room with grass floor and concrete wall”. • Target Task: “Pick yellow duckie and avoid green ball from the room with grass floor and concrete wall”. • Current Observation Description: “A yellow duckie is visible at 1.0 units and 34.3 degrees to the left, and a green ball is visible at 3.1 units and 32.9 degrees to the left, both located on a green grass floor surrounded by grey walls under a blue sky.” B.4. Query Examples Below are examples of how the LLM transforms target observation descriptions into source-aligned descriptions to enable zero-shot transfer. The reasoning shown is a summary of the LLM’s internal thought process. Example 1: Unseen Object Adaptation • Input: “A yellow duckie is visible at 1.0 units and 34.3 degrees to the left, and a green ball is visible at 3.1 units and 32.9 degrees to the left, both located on a green grass floor surrounded by grey walls under a blue sky.” 16
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
• Output: “A blue box is visible at 1.0 units and 34.3 degrees to the left, and a green ball is visible at 3.1 units and 32.9 degrees to the left, both located on a green grass floor surrounded by grey walls under a blue sky.” • Reasoning Step: The agent reimagines the target object (yellow duckie) as the source reward object (blue box) to trigger the correct pickup policy. The environment context remains unchanged as it matches the source. Example 2: Distractor Persistence • Input: “A green ball is visible at a distance of 4.2 units and an angle of 24.6 degrees to the left, located on a green grass floor surrounded by gray walls under a blue sky.” • Output: “A green ball is visible at a distance of 4.2 units and an angle of 24.6 degrees to the left, located on a green grass floor surrounded by gray walls under a blue sky.” (No Change) • Reasoning Step: No semantic change is needed because the detected object (green ball) matches the source distractor, which the agent is already trained to avoid. Example 3: Complex Environmental & Semantic Shift • Input: “A yellow duckie is visible at 1.0 units and 34.3 degrees to the left, and a blue box is visible at 3.1 units and 32.9 degrees to the left, both located on a wodden floor surrounded by brick walls under a blue sky.” • Output: “A blue box is visible at 1.0 units and 34.3 degrees to the left, and a green ball is visible at 3.1 units and 32.9 degrees to the left, both located on a green grass floor surrounded by grey walls under a blue sky.” • Reasoning Step: This complex mapping requires three simultaneous adjustments: 1. Remapping the target reward (yellow duckie) to the source reward (blue box). 2. Remapping the old source reward (blue box), which is now a distractor, to a known distractor (green ball) to ensure avoidance. 3. Hallucinating the environment textures (wooden floor/brick walls) back to the training environment (grass floor/concrete walls) to ensure feature consistency.
C. Structured vs. Unstructured Captions C.1. Structured Captions The structured query is template filling. For example, in the MiniWorld environment: The agent is in a room with a grass floor <floor description> and concrete walls <wall description>. A blue box <object1> is found to the left at angle 10 at a distance of 3.5 units <location1>. A green ball <object2> is found to the right at angle 22.7 at a distance of 2.5 units <location2>. If no objects are visible: The agent is in a room with grass floor and concrete walls. No objects are visible in the current view. C.2. Unstructured Captions The unstructured query varies significantly. Here are a few examples: • “A green ball is visible at a distance of 3.9 units and an angle of 34.2 degrees to the left. It is located on a green grass floor surrounded by gray walls under a blue sky.” • “The image depicts a room with a green floor and gray walls. The ceiling is blue, indicating a clear sky. No objects are visible in the room.” • “The image depicts a 3D environment with a gray wall, green grass floor, and a blue sky visible above. A blue box is located at 2.6 units and 0.1 degrees to the right, while a green ball is positioned at 3.5 units and 30.7 degrees to the right within the scene.” 17
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
• “An empty room with a green grass floor and gray walls under a blue sky.” • “A blue box is visible at a distance of 4.1 units and an angle of 6.5 degrees to the left, while a green ball is located at a distance of 3.5 units and an angle of 29.9 degrees to the right. The scene is set on a green grass floor, with a grey wall in the background, under a blue sky.’
D. RL Implementation Details All Reinforcement Learning (RL) source policies—DQN, PPO, and SAC—were trained using the default hyperparameters and configurations provided by the Stable Baselines3 (SB3) library (Raffin et al., 2021). No extensive hyperparameter tuning was performed for the source tasks, demonstrating the robustness of the source policies.
E. Text-Conditioned VAE Architecture and Training E.1. Architecture Our Text-Conditioned Variational Autoencoder (VAE) is designed to separate visual content into a semantic, text-conditioned component and a spatially-structured latent variable z that captures residual details (e.g., layout, pose, background). The framework consists of four main components: a visual encoder, a text encoder, a conditional decoder, and a set of discriminators. Visual Encoder. The encoder Eϕ is a ResNet-style Fully Convolutional Network (FCN). It processes an input image x ∈ R3×H×W through a series of downsampling blocks (comprising strided convolution, LayerNorm, and LeakyReLU) followed by three residual blocks. The encoder outputs a spatial feature map parameterizing a diagonal Gaussian distribution qϕ (z|x) = N (µ(x), diag(σ 2 (x))), producing a spatial latent tensor z ∈ RCz ×h×w (where Cz = 8). Maintaining spatial dimensions allows the latent code to preserve localized visual structure. Text Encoder. Textual descriptions are encoded using a frozen pre-trained LongCLIP model (LongCLIP-GmP-ViT-L-14) for unstructured captions. For structured captions, we utilized the standard CLIP model. We extract the last hidden state sequence t ∈ RL×Dt to capture fine-grained semantic details. A learnable MLP adapter projects these embeddings to the decoder’s working dimension. Conditional Decoder. The decoder Dθ (z, t) reconstructs the image x̂ by progressively upsampling the latent z while conditioning on text t. It is composed of hierarchical Cross-Attention FiLM Spatial Blocks. In each block: 1. FiLM Modulation: The spatial latent z is upsampled to the features’ resolution and mapped to affine parameters (γ, β) for Feature-wise Linear Modulation, allowing the latent structure to spatially modulate the features. 2. Cross-Attention: A Multi-Head Cross-Attention layer allows the visual features to attend to the sequence of text embeddings t, injecting semantic information. Discriminators. The architecture includes two distinct adversarial modules used during training: • Caption Discriminator (Dtext ): To ensure the latent z captures only visual information orthogonal to the text, we employ an adversarial disentanglement module. This consists of a Multi-Layer Perceptron (MLP) that takes the flattened latent z and attempts to predict the corresponding text embeddings t. A Gradient Reversal Layer (GRL) is placed before this discriminator, causing the encoder to learn representations that are invariant to the text (i.e., maximizing the discriminator’s loss). • Image Discriminator (Dimg ): A PatchGAN discriminator is used to enforce photorealism. It operates on local image patches to distinguish between real images x and reconstructions x̂, encouraging the decoder to generate high-frequency textures. E.2. Training Formulation The model is trained to minimize a composite objective function. The core foundation is the text-conditioned ELBO: LV AE (x, cs ) = Eqϕ (z|x) [log pθ (x|z, ecs )] − β · DKL (qϕ (z|x)||p(z)) 18
(1)
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
where p(z) ∼ N (0, I) and β is the annealing term. Adversarial Disentanglement (Ldis ): To ensure z is orthogonal to the text, we use a contrastive adversarial loss. A discriminator Dψ predicts the text embedding ecs from the latent z (via GRL). It minimizes the InfoNCE loss: B
Ladv (Z, E) = −
exp(sim(êcs,i , ecs,i )/τ ) 1 X log PB B i=1 j=1 exp(sim(êcs,i , ecs,j )/τ )
(2)
where êcs,i = Dψ (GRL(zi )) and τ is the temperature. The encoder maximizes this loss. Total Objective: The full training objective combines these with perceptual and GAN losses for high-fidelity generation: L = LMSE + λLPIPS LLPIPS + λKL LKL + λdis Ladv + λgan Lgan
(3)
Reconstruction (LMSE ): We calculate the pixel-wise Mean Squared Error (MSE) between the input x and reconstruction x̂ to ensure structural fidelity. Perceptual Loss (LLPIPS ): We optimize a VGG-based LPIPS perceptual loss to capture high-level semantic similarity and textural details that pixel-wise metrics may miss. KL Divergence (LKL ): The posterior is constrained towards a standard normal prior, DKL (qϕ (z|x)||N (0, I)), using cyclical annealing to prevent posterior collapse. GAN Loss (Lgan ): The Image Discriminator and Decoder are optimized via a Hinge loss adversarial objective to improve generation quality. E.3. Implementation Details We train the model using the AdamW optimizer with a OneCycleLR scheduler (max learning rate 2 × 10−4 , weight decay 0.01). The implementation utilizes ‘LongCLIP-GmP-ViT-L-14‘ as the text backbone for natural scenes, while standard CLIP is used for structured environments. Hyperparameters are set to λKL = 1.5, λdis = 15.0, and λgan = 10.0. Training is performed with a batch size of 32 for 500 epochs.
F. Additional Qualitative Results F.1. Learning curves We present the detailed fine-tuning learning curves for the MiniWorld, MiniGrid and Fragile Object Manipulation environments in Figure 3 and 4 respectively. 15
15.0
15
12.5
10 10.0
5
0
7.5
Score
Score
Score
10
5.0 2.5
5
0
5
0.0 2.5
Source Target
5 0
25000
50000
75000
100000
125000
Step size
(a) Case 1
150000
175000
200000
Target 2 Source
5.0 0
25000
50000
75000
100000
125000
Step size
150000
175000
200000
(b) Case 2
10
Source Target 0
25000
50000
75000
100000
125000
Step size
150000
175000
200000
(c) Case 3
Figure 3. PPO Fine-tuning in MiniWorld in each case.
The learning curve for SF in the MiniGrid environment is shown in Figure 5. The SF agent is allowed to interact with different environments sequentially. Each dip in the learning curve indicates an environmental change. The learning curve for SF in the MiniWorld environment is shown in Figure 6. As indicated by the reward curve, the agent fails to learn any meaningful policy and the performance never improves across tasks. 19
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
C ASE
R EWARDING O BJECT P ICKED
D ISTRACTOR O BJECT P ICKED
C ASE 1 C ASE 2 C ASE 3
8.60 ± 0.50 8.40 ± 0.40 8.60 ± 0.40
0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00
1.25
1.00
1.00
0.75
0.75
0.50 0.25 0.00
1.0
Score
1.25
Score
Score
Table 5. Experimental results of ASPECT for MiniWorld environment using unstructured, VLM-generated captions. The method demonstrates robustness to noise and variability in the conditioning text.
0.50 0.25
0.5
0.0
0.00
0.25
0.25
0.50 0.0
0.2
0.4
0.6
Step size
0.8
0.5
0.50
Source Target 1.0 1e6
Source Target
Source Target 0.0
(a) MiniGrid Case 1
0.2
0.4
0.6
0.8
Step size
1.0 1e6
(b) MiniGrid Case 2 400
0.0
0.2
0.4
0.6
Step size
0.8
1.0 1e6
(c) MiniGrid Case 3
Source Target
Score
300
200
100
0
100
0
25000
50000
75000
100000
125000
Step size
150000
175000
200000
(d) Manipulation Figure 4. Fine-tuning curves for MiniGrid and Fragile Object Manipulation.
F.2. Disentanglement of Layout and Semantics Figure 8 demonstrates the text-guided image generation capabilities of our VAE, highlighting the disentanglement between structural and semantic features. The first column displays the original observation, and the second column shows its reconstruction. Subsequent columns show generations conditioned on different text descriptions while keeping the spatial latent z fixed. As observed, the structural layout (e.g., background geometry, walls, floor) remains consistent across all generations, captured by the unchanged latent z. Meanwhile, the semantic content (textures, colors, object identities) adapts to the varying text prompts. This confirms that our architecture effectively disentangles the spatial layout (encoded in z) from the semantic attributes (controlled by the text embedding). To further validate this disentanglement, we visualized the spatial latent features. Figure 7 displays the original and remapped images alongside their corresponding 8-channel spatial latents. Notably, the first two rows of the latent visualization (representing specific feature channels) remain identical between the original and remapped states. These channels correspond to the structural background features, which are preserved by the architecture, while the text-conditioned channels adapt to the new semantic prompts. This visual evidence reinforces that our model successfully isolates background structure from semantic object identity. F.3. Imagination in different unseen settings We provide visual examples of the imagination process in Figure 9. These qualitative results demonstrate how the agent hallucinates the target observation (containing unseen rooms and objects) to match the source task, enabling zero-shot 20
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
Score
1.0
0.5
0.0
0.5 SF_Simple SF_Reconstruction 0.0
0.5
1.0
Step size
1.5
2.0
1e6
Figure 5. Learning curve for Successor Features (SF) in MiniGrid. The agent interacts with environments sequentially, with performance dips indicating task transfers. SF_Simple SF_Reconstruction
8 6
Score
4 2 0 2 4 6 0.00
0.25
0.50
0.75
1.00
1.25
Step size
1.50
1.75
2.00 1e6
Figure 6. Learning curve for Successor Features (SF) in MiniWorld. The flat reward curve indicates a failure to learn or transfer to the target tasks.
transfer. F.4. Failure Cases While ASPECT demonstrates robust zero-shot generalization, we identify specific failure modes in the imagination process. Figure 10 illustrates these cases. Common issues include artefacts in the generated background or object when the target object is positioned too close to the camera, and occasional instances where the remapped object fails to materialize in the imagined scene.
21
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
(a) Original Image
(b) Original Latent
(c) Remapped Image
(d) Remapped Latent
Figure 7. Visualization of spatial latent channels. The first two rows of the latent feature maps remain consistent between (b) and (d), capturing the preserved background structure, while other channels shift to reflect the semantic change from (a) to (c).
Figure 8. Text-guided image generation demonstrating latent disentanglement. Structural features captured in z remain fixed (preserving the background layout) while the text modifies the semantic appearance. Left: MiniWorld, Right: MiniGrid.
22
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
Original
Imagined
Original
Imagined
(a) Duckie in grass floor and concrete wall
(b) Blue box in source setting
(c) Medkit in grass floor and concrete wall
(d) Blue box in source setting
(e) Duckie in slime floor and metal grill wall
(f) Green ball in source setting
(g) Medkit in slime floor and metal grill wall
(h) Green ball in source setting
(i) MedKit in wodden floor and brick wall
(j) Blue box in source setting
(k) Duckie in slime floor and metal grill wall
(l) Blue box in source setting
(m) Duckie and MedKit in slime floor and metal grill wall
(n) Blue box and Green ball in source setting
(o) Duckie in wooden floor and brick wall
(p) Green ball in source setting
(q) Duckie in lava floor and metal grill wall
(r) Blue box in source setting
(s) MedKit in lava floor and metal grill wall
(t) Green ball in source setting
Figure 9. Comparison of real target observations and the corresponding source-aligned imagined states generated by ASPECT. We display 10 examples demonstrating the robust visual remapping capability.
23
ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer
Original
Imagined
(a) Object too close
(b) Background artefacts
(c) Partial object close up
(d) Artefacts in object/background
(e) Target visible
(f) Remapped object missing
Figure 10. Visualisation of failure cases. The model struggles with extreme close-ups, leading to generation artefacts, and occasionally fails to generate the source object even when the target is visible.
24