arXiv CSopen access

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Published as a conference paper at ICLR 2026

A R EWARD -F REE V IEWPOINT ON M ULTI -O BJECTIVE R EINFORCEMENT L EARNING Ying-Tu Chen1,† Wei Hung1,† Bing-Shu Wu1,† Zhang-Wei Hong2 Ping-Chun Hsieh1 National Yang Ming Chiao Tung University, Hsinchu, Taiwan 2 Massachusetts Institute of Technology {shu0924.cs13,pinghsieh}@nycu.edu.tw, [email protected]

arXiv:2604.24532v1 [cs.LG] 27 Apr 2026

A BSTRACT Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multiobjective reinforcement learning (MORL), one widely studied approach addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL’s challenge of handling unknown user preferences. We propose using the RFRL’s training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.*

I NTRODUCTION

Many sequential decision-making tasks require optimizing multiple, often conflicting objectives. For example, in robot control, there is a trade-off between minimizing energy consumption and maximizing speed. One common approach to find a Pareto optimal policy is to maximize a weighted sum of the objectives, where the weights represent user preferences. User preferences depend on context—for instance, prioritizing speed in emergencies and energy efficiency in routine operations. Since user preferences are unknown in advance, solving multi-objective decision-making requires learning a set of policies for different preferences before testing. Reinforcement learning (RL) (Sutton & Barto, 2018) has achieved strong performance in sequential decision-making, making multi-objective RL (MORL) a widely studied approach for learning policies for different user preferences (Hayes et al., 2022). A naive but inefficient solution is to train a separate policy for each preference. Another more scalable approach is to train a single policy network (Yang et al., 2019; Basaklar et al., 2023; Hung et al., 2023) conditioned on preferences, enabling parameter sharing and generalization across preferences. During training, the policy is optimized over a range of sampled preferences, each defining a reward function weighted by the preference. At test time, users can specify a preference to obtain the corresponding policy. Another approach to handling unknown user preferences at test time is reward-free reinforcement learning (RFRL) (Jin et al., 2020; Touati et al., 2023), which has historically been developed independently of MORL despite addressing a similar problem. In RFRL, the agent explores the environment without receiving reward signals during training and instead learns a set of optimal policies †

Equal contribution.

* https://rl-bandits-lab.github.io/MORL-FB/

Published as a conference paper at ICLR 2026

for any possible reward function in the environment. MORL can be seen as a special case of RFRL (Alegre et al., 2022), as RFRL does not restrict the reward function to be a weighted sum of predefined reward functions. However, despite their similarities, no prior work has explicitly adapted RFRL methods to solve MORL problems. In this paper, we ask: Can RFRL inform MORL? We hypothesize that the objective of RFRL to learn optimal policies for any reward function could serve as a useful auxiliary task for MORL (Jaderberg et al., 2016; Rafiee et al., 2022; Veeriah et al., 2019). Although MORL under linear scalarization only needs optimal policies for linear combinations of known objectives, learning beyond these combinations could accelerate MORL via effective knowledge sharing. To investigate this question, we adapt a state-of-the-art RFRL algorithm (Touati et al., 2023) to the MORL setting, treating the preference-weighted reward as the test-time reward function given to RFRL. However, this naive approach performs poorly compared to existing MORL methods, likely because purely rewardfree exploration does not prioritize states that are important for optimizing the preference-weighted reward in the given MORL task. As a result, the policies learned by RFRL for these reward functions could be suboptimal. To address this, we propose guiding exploration using sampled preferences and mini-batch sampling, directing the agent to visit states that maximize the corresponding preferenceweighted reward function. This ensures that learning is focused on policies most relevant to MORL. We highlight the main technical novelty of this paper: (1) A new perspective of solving MORL: We identify the close connection between MORL and RFRL, which have evolved independently despite tackling similar challenges of unknown user preferences at test time. This insight motivates new MORL algorithms by rethinking policy learning with multiple objectives from the perspective of RFRL. (2) Algorithmic enhancements for adapting RFRL to MORL: Even though RFRL and MORL are closely related, vanilla RFRL can perform poorly in the MORL setting (see Section 4). To address this, we introduce three key enhancements: (i) Preference-guided exploration: We propose to use the preference vector to sample latent vectors aligned with the target rewards to facilitate exploration in the latent space; (ii) Training on latent vectors computed by mini-batch sampling from replay buffer as auxiliary tasks: Our approach trains the policy network on latent vectors computed from mini-batch transitions sampled from the replay buffer. This design learns a broader range of policies than required for MORL and can be beneficial by providing auxiliary tasks; (iii) Auxiliary Q loss: To better adapt RFRL to MORL, we further facilitate the learning of representations from the observed reward vectors (instead of pseudo rewards as in RFRL) via an auxiliary Q loss as an additional learning signal. Our experimental results demonstrate that our approach is both simple and effective. First, our method significantly outperforms the state-of-the-art MORL algorithms across various tasks in the MO-Gymnasium benchmark suite (Felten et al., 2023), including discrete and continuous control. Second, when trained with a limited number of preference samples, our method achieves substantially higher performance than other MORL approaches. This highlights that decoupling environment knowledge from reward information enhances generalization, particularly in scenarios with limited preference samples. To the best of our knowledge, this is the first work to adapt RFRL for MORL and present a practical algorithm that performs well across diverse deep RL tasks.

P RELIMINARIES

This section provides a brief review of MORL, along with the notation used throughout the paper. We use boldface symbols for vectors and matrices. For any n ∈ N, we use [n] as a shorthand for {1, · · · , n}. For a set Z, we let ∆(Z) denote the set of all probability distributions over Z. We formulate the MORL problem as an Multi-Objective Markov Decision Process (MOMDP) defined by the tuple (S, A, P, R, γ, µ), where S and A are the state and action spaces, P : S × A → ∆(S) is the transition function, R : S × A → Rd is a vector-valued reward function of d objectives, γ ∈ [0, 1) is the discount factor, and µ is the initial state distribution. Let Π denote the set of all stationary randomized policies. Let st , atP , rt be the state, action, and reward received at time t. For ∞ a policy π ∈ Π, define Vπ := Eπ,s0 ∼µ [ t=0 γ t R(st , at )] as the expected total discounted return vector achieved by π. Let Viπ denote the i-th entry of Vπ . For a pair of policies π and π ′ , we say ′ that π Pareto-dominates π ′ (denoted by π ≻ π ′ ) if Viπ ≥ Viπ for all i ∈ [d] and there exists some ′ j ∈ [d] such that Vjπ > Vjπ . 2

Published as a conference paper at ICLR 2026

The general goal of MORL is to discover the Pareto front, which is defined as the set of nondominated policies. That is, for each policy π in the Pareto Front, there exists no other policy π ′ ∈ Π such that π ′ ≻ π. To search for the Pareto front, one common approach is to leverage a scalarization utility function fλ : Rd → R under a user preference vector λ ∈ Λ, where Λ denotes the preference set. In this paper, we focus on the linear scalarization setting where fλ (r) = λ⊤ r, as commonly adopted in the MORL literature (Abels et al., 2019; Yang et al., 2019; Basaklar et al., 2023; Hung et al., 2023; Lu et al., 2023). Without loss of generality, we presume that Λ is the ddimensional unit simplex. Notably, it has recently been shown by Lu et al. (2023) that any point on the Pareto front can be achieved by training a policy using linear scalarization due to the convexity of the policy-induced value function’s range. Since the preference λtest at test time is unknown during training, our goal is to learn a preference-conditioned π : S ×Λ → ∆(A) that can maximize P∞ t ⊤ policy γ λ r the total discounted scalarized reward E , for any λ ∈ Λ. t t=1

R EWARD -F REE RL FOR M ULTI -O BJECTIVE RL

This section explains why MORL can be seen as a special case of RFRL. We then discuss how this perspective enhances MORL by improving generalization and sample efficiency. MORL as a special case of RFRL. The goal of RFRL is to compute an optimal policy for any scalar reward function R : S × A → R provided at test time, without observing any reward signal during training (i.e., “reward-free”) nor requiring additional environment interaction at test time. Formally, P∞ t RFRL solves the following optimization problem at test time: arg maxπ Eπ,s0 ∼µ t=0 γ rt , where rt is the reward realization that corresponds to the test-time reward function. MORL under linear scalarization addresses a similar problem, but presumes the vector-valued reward signal from R(s, a) can be observed during training, and assigns R(s, a) = λ⊤ R(s, a), where λ (a user-specified preference vector) defines a linear combination of multiple reward components in R. Both RFRL and MORL aim to retrieve an optimal policy for a given reward function at test time, but their approaches differ. While MORL typically focuses on finding the Pareto front by learning a set of optimal policies for various preferences λ, RFRL learns policies for all possible reward functions, potentially including optimal policies for scalarized MORL rewards. RFRL achieves this by training a conditional policy network (Touati & Ollivier, 2021) or leveraging a precollected dataset through planning or batch RL (Jin et al., 2020), providing a broader policy set than traditional MORL. Key idea: RFRL as a source of auxiliary tasks. RFRL learns policies for a broader class of reward functions than required for MORL, but this can be beneficial by providing auxiliary tasks. Prior research has shown that incorporating auxiliary tasks improves sample efficiency and generalization in RL (Jaderberg et al., 2016; Veeriah et al., 2019; Rafiee et al., 2022). Since RFRL naturally trains policies across a spectrum of reward functions, it provides a structured way to design these auxiliary tasks. However, directly applying RFRL to MORL can be data-inefficient since rewardfree exploration may not prioritize states that are crucial for learning the Pareto front in MORL. The key challenge is: How to utilize the auxiliary tasks offered by RFRL effectively to accelerate the learning of optimal policies in MORL? In the sequel, we describe how RFRL can be adapted to effectively improve training in MORL. 3.1

F ORWARD -BACKWARD MORL (MORL-FB)

In this section, we formally present MORL-FB by describing the implementation of RFRL and the key components that adapt RFRL to MORL by enhancing its learning efficiency. RFRL Implementation. We implement RFRL for MORL using the state-of-the-art ForwardBackward (FB) RL algorithm (Touati & Ollivier, 2021). The FB method learns a set of policies optimized for different reward functions by decomposing the Q-value of an optimal policy for a scalar reward function R into two neural networks: Fθ (forward representation) and Bω (backward representation), where θ and ω denote their parameters. This decomposition allows the Q-function for a given reward function R to be expressed as: Q(s, a, zR ) = Fθ (s, a, zR )⊤ zR ,

(1)

Published as a conference paper at ICLR 2026

where zR ∈ Rdz is an dz -dimensional latent vector, and both Fθ and Bω are neural networks producing dz -dimensional outputs. Intuitively, zR is meant to encode the optimal policy that corresponds to the current reward function. Once a reward function R is revealed, zR is defined as: zR = E(s,a)∼D [Bω (s, a)R(s, a)] ,

(2)

where D represents an arbitrary state-action distribution. In our implementation, we use D as the distribution induced by the replay buffer collected by the agent during training. Using this formulation, the greedy policy for a given reward function R is defined as: π(s, zR ) = arg max Fθ (s, a, zR )⊤ zR . a

(3)

Now we are ready to present the design of MORL-FB at both training time and test time. Algorithm 1 MORL-FB

Figure 1: A motivating experiment on Deep Sea Treasure. (a)(b) Training performance (UT and HV defined in the sequel) of MORL-FB under different batch sizes for ẑλ . (c) KDE contour of return vector distributions of π(·, z) induced by ẑλ (with various batch sizes b) and ẑ ∼ N (0, Idz ). This shows that ẑλ corresponds to learning for more diverse and relevant behavior in MORL than zλ and the z sampling strategy of the original FB. The detailed configuration is provided in Appendix C.

1: Input: z dimension dz , sample number ns 2: Initialize replay buffer M ← ∅ 3: for each iteration i do 4: Sample preference λ uniformly from Λ 5: z ← PG-E XPLORE(λ) 6: Generate rollouts using z 7: Sample ns transitions D ∼ M 8: Update FB networks Fθ , Bω and policy π 9: end for 10: function PG-E XPLORE(λ) 11: Sample ns transitions D ∼ M ⊤ P λ 12: z ← (s,a,r,s′ )∈D Bω (s,a)r √ ns z 13: Normalize z to z ← dz ∥z∥2 14: return z 15: end function

Test Time: At test time, we can easily adapt Equation (2) by replacing R(s, a) with a user-specified scalarized multi-objective reward based on a preference vector λ as R(s, a) = λ⊤ R(s, a). Next, given the learned Fθ and Bω , we compute the corresponding zR and use it in the policy defined by Equation (3). This effectively retrieves an optimal policy for the given preference λ. Training with Preference-Guided Exploration: During training, Fθ , Bω , and π must be trained by sampling z and conditioning the networks on these sampled values. Since the test-time user preference λtest is unknown at this stage, we cannot directly compute z using Equation (2). At a high level, training on a diverse set of z samples is equivalent to training the agent on a variety of reward functions, since z is inherently linked to rewards through Equation (2). In principle, z can be sampled from any distribution without restriction. In (Touati & Ollivier, 2021), z is drawn from a standard normal distribution N (0, Idz ) in a dz -dimensional space. However, we found that this approach leads to poor sample efficiency when testing the agent on MORL tasks. We hypothesize that sampling z from a normal distribution produces representations that differ significantly from the actual zR obtained from a preference-weighted multi-objective reward function (see Figure 5 for a visualization of empirical z distributions). To address this issue, we propose Preference-Guided Exploration (PG-Explore), which constructs a more relevant z distribution via sampling guided by preference-weighted rewards. The design of PG-Explore builds on the following insights: • Using {zλ }λ∈Λ only leads to limited exploration of z: Recall that in MORL, we can observe multi-objective rewards R(s, a) (or its noisy version) during training. One direct approach is to compute z as: 4

Published as a conference paper at ICLR 2026

(a)

(b)

zλ = E[Bω (s, a)λ⊤ R(s, a)] = E[Bω (s, a)R(s, a)⊤ λ] = (E[Bω (s, a)R(s, a)⊤ ]) λ, | {z }

(4)

=:H

where (a) holds due to the fact that λ⊤ R(s, a) is a scalar and can be swapped in the matrix multiplication with Bω (s, a) and (b) follows from that λ can be moved out of the expectation. Equation (4) suggests that zλ is in the span of d preference-agnostic column vectors of the dz × d matrix H, for any preference λ. In practice, since the number of objectives d is usually much smaller than dz , the coverage of {zλ }λ∈Λ in Rdz can be extremely small. This leads to limited exploration of z during training such that the agent can commit to a set of improper z, especially in the early training stage when Fθ and Bω are not well trained. • Constructing ẑλ by mini-batch sampling for exploration: To encourage exploration of z relevant to MORL, we propose a conceptually simple and yet effective technique that leverages mini-batch sampling to construct ẑλ . Specifically,Pwe sample a batch of ns data samples (denoted by D) from the replay buffer and compute ẑλ = (s,a,r,s′ )∈D Bω (s, a)r⊤ λ/ns . Figure 1 shows a comparison of training with zλ , ẑλ under various batch sizes, and z drawn from N (0, Idz ) as in the original FB method, in Deep Sea Treasure (DST), which is a goal-oriented navigation task with two-dimensional rewards as (treasure value, step cost). As the true zλ is not available, we use ẑλ with a large ns as a surrogate for zλ . We can see that: (i) ẑλ indeed corresponds to learning more diverse behavior than just learning for zλ . (ii) ẑλ ’s are more relevant to the reward functions in MORL encountered at test time than the z sampled from N (0, Idz ), improving sample efficiency. • Learning as auxiliary tasks: Recall that in PG-Explore, we construct P induced by ẑλ serves ⊤ ẑλ = (s,a,r,s′ )∈D Bω (s, a)r λ/ns by mini-batch sampling. This means that for any given preference λ, the agent can learn beyond zλ and, moreover, from multiple values z from different batches of transitions, providing richer learning signals. This approach is closely related to the auxiliary tasks in deep RL, where training objectives that are not directly or totally aligned with the target objective have been shown to accelerate learning (Jaderberg et al., 2016; Veeriah et al., 2019; Rafiee et al., 2022). Training Objective Functions: (i) Measure loss: To train the Fθ and Bω networks in MORL-FB, we use the standard measure loss LM (Fθ , Bω ; zλ ), which minimizes the Bellman residual on the successor measure (Touati et al., 2023). Fθ̄ and Bω̄ are target networks. This loss is defined as: LM (Fθ , Bω ; zλ ) = E(st ,at ,st+1 )∼D Fθ (st , at , zλ )⊤ Bω (s′ , a′ ) (s′ ,a′ )∼D

2 i −γFθ̄ (st+1 , π(st+1 , zλ ), zλ )⊤ Bω̄ (s′ , a′ ) − 2E(st ,at ,st+1 )∼D [Fθ (st , at , zλ )⊤ Bω (st+1 , at+1 )].

(5)

(ii) Auxiliary Q loss: In the context of MORL, we propose to employ an auxiliary Q loss to facilitate the learning of FB representations from the observed reward vectors, instead of the pseudo rewards in the original FB (also see the ablation study in Section 4.1). Specifically, the Q-loss is constructed as the squared temporal difference error represented in Fθ and Bω , and the transitions are sampled from the replay buffer to compute zλ via our preference-guided function: LQ (Fθ ; zλ ) = E(s,a,r,s′ )∼D (Fθ (s, a, zλ )⊤ zλ − (λ⊤ r + γFθ̄ (s′ , π(s′ , zλ ), zλ )⊤ zλ ))2 .

(6)

We summarize the implementation in Algorithm 1. The detailed pseudo code (Algorithm 2) and further details about loss functions and implementation are provided in Section B. Note that the FB framework can use either state-dependent or state-action-dependent backward representation, and both perform well in practice (see Appendix D.4). As the original FB (Touati et al., 2023) presumes a state-dependent design, we focus mainly on state-dependent ones in the subsequent experiments.

E XPERIMENTS

Evaluation Domains. We leverage the MO-Gymnasium benchmark suite (Felten et al., 2023) and consider various discrete and continuous control tasks. For the continuous control tasks, we consider 5

Published as a conference paper at ICLR 2026

robot locomotion tasks in Multi-objective MuJoCo with up to 5 objectives, including Walker2d, Halfcheetah2d, Ant3d, Hopper3d, Humanoid2d, and Humanoid5d. Each environment presents a unique set of objectives, e.g., the goal of Ant3d is to optimize both x-axis and y-axis speeds while minimizing energy consumption. The experiments of the discrete control tasks are provided in Appendix D.3. The detailed configurations for all tasks are provided in Appendix C. Benchmark Methods. To evaluate the effectiveness of our proposed approach, we compare MORLFB against various benchmark methods, including: (i) Single preference-conditioned policy methods: PD-MORL (Basaklar et al., 2023), Q-Pensieve (Hung et al., 2023), CAPQL (Lu et al., 2023), Envelope Q-Learning (EQL) (Yang et al., 2019), and PCN (Reymond et al., 2022); (ii) Multi-policy MORL: PG-MORL (Xu et al., 2020), SFOLS (Alegre et al., 2022), MORL/D (Felten et al., 2024), GPI-LS, and GPI-PD (Alegre et al., 2023) ; (iii) Reward-free RL: We take the original FB approach (Touati et al., 2023) as a baseline. Regarding PD-MORL, Q-Pensieve, and FB, we leverage their official implementations from (Basaklar et al., 2023; Hung et al., 2023; Touati et al., 2023). To ensure a fair comparison among all the benchmark methods, we adopt the standard PD-MORL without the auxiliary pre-trained preference interpolator, which essentially requires a substantial amount of additional data for pre-training and could bias the comparison. Regarding CAPQL, EQL, PCN, PG-MORL, MORL/D, GPI-LS, and GPI-PD, we leverage the implementation of MORL-Baselines (Felten et al., 2023) for better reproducibility. As the PG-MORL in MORL-Baselines can only support two-objective tasks, we extend this code base to accommodate those tasks beyond two objectives. On the other hand, the original SFOLS only focuses on discrete control tasks by default. For a more thorough comparison, we utilize its official implementation for discrete problems and extend SFOLS with a TD3 backbone for evaluation on continuous control. Moreover, we apply hyperparameter optimization to the benchmark algorithms and MORL-FB. Appendix C.2 details the selection range of hyperparameters and the final selected values. For all the tasks, we run each algorithm for 3 million environment steps, which is comparable to most of the existing MORL studies. Below we report the average performance and the empirical standard deviation over 5 random seeds for each task. More detailed configurations of the experiments and benchmark methods are provided in Appendix C.

Figure 2: Evaluation of MORL-FB and several MORL benchmark algorithms on diverse continuous control tasks within the MO-Gymnasium suite, assessing performance using key metrics. These results demonstrate the clear advantage of MORL-FB across all tested benchmarks. Evaluation Metrics. We evaluate the performance of each algorithm using three metrics that are widely used in the MORL literature (Van Moffaert & Nowé, 2014; Yang et al., 2019; Kyriakis & Deshmukh, 2022; Basaklar et al., 2023; Hung et al., 2023; Lu et al., 2023): • Utility (UT): To evaluate the scalarized total reward P across different preferences at an aggregate level, we employ the utility metric defined as Eλ [ t λ⊤ rt ], where the expectation is taken with respect to the uniform distribution over the preference set Λ (i.e., d-dimensional unit simplex). 6

Published as a conference paper at ICLR 2026

• Hypervolume (HV): As a standard metric in the literature of general multi-objective optimization, hypervolume naturally captures the inherent trade-off among different objective functions using one aggregate scalar value (Zitzler & Thiele, 1999). Specifically, given a reference point uref ∈ Rd and any collectionfor return vectors U ⊆ Rd, the hypervolume of U can be formally defined as S HV(U; uref ) := µ y u ⪰ y ⪰ uref , where µ(·) denotes the d-dimensional Lebesgue u∈U

measure. In practice, uref is selected based on the range of possible total return and is taskdependent. The configuration of uref for each MORL task is provided in Appendix C. • Episodic Dominance (ED): As a metric complementary to HV and UT, ED is meant to capture the relative strength of a pair of algorithms under different preferences. Specifically, given any two algorithms ALG1 , ALG2 , we define ED(ALG1 , ALG2 ) := Eλ [I{λ⊤ g(τALG1 ) ≥ λ⊤ g(τALG2 )}], where g(·) denotes the trajectory-wise cumulative return vector, τALG1 and τALG2 are the trajectories generated under the policies of ALG1 and ALG2 , and λ is drawn uniformly from Λ. Note that we use 500 uniformly sampled preference vectors and evaluate across 5 distinct random seeds for each preference vector for statistical robustness. To ensure rigorous evaluation, we further follow the guidelines of (Agarwal et al., 2021) by taking the normalized UT scores and reporting the aggregated performance across tasks in median, mean, and interquartile mean (IQM). Regarding the normalized scores, we follow the procedure in (Fu et al., 2020), which (i) employs a random policy—where actions are selected uniformly at random— as the baseline with normalized score of 0 and (ii) an expert policy trained by singleobjective SAC—as the topline with normalized score of 100. The above normalization is done on a per-preference basis. Does the reward-free viewpoint of MORL-FB improve sample efficiency over the benchmark methods? Figure 2 shows the performance of all the methods in UT, HV, and ED for continuous control tasks. Regarding ED, for each baseline algorithm ALG, we report ED(ALG, MORL-FB) to show the pairwise comparison. We can make the following observations: (i) MORL-FB achieves either the best or close to the best UT and HV among all methods on all the tasks, regardless of the number of objectives. This showcases that MORL-FB is indeed sample-efficient in the sense that it can discover a diverse collection of high-performing policies across various preferences using only as few as 3 million samples as used by the expert policy. (ii) Given that ED(ALG, MORL-FB) are consistently smaller than 0.5 for all baselines, we see that MORL-FB outperforms all benchmark methods (including the state-of-the-art methods like PD-MORL and Q-Pensieve), under most preferences. (iii) PD-MORL and Q-Pensieve perform well on two-objective tasks (e.g., Halfcheetah2d and Walker2d) but underperform when the number of objectives is larger (e.g., Ant3d and Humanoid5d). Additional results on discrete control environments can be found in Section D.3. Moreover, regarding the aggregated results, Figure 3 shows that MORL-FB reliably outperforms the benchmark methods both in conventional statistics (e.g., mean and median) and robust metrics like IQM. MORL-FB achieves the best IQM scores by a large margin vis-à-vis other methods, confirming the significant improvements over the state-of-the-art MORL. Recall that MORL-FB leverages PG-Explore to address the fundamental exploration issue of vanilla FB, which suffers from sample inefficiency in MORL. Remarkably, the per-task results in Figure 2 and aggregated results in Figure 3 show that MORL-FB enjoys significantly better UT and HV across tasks. Accordingly, the ED scores ED(FB, MORL-FB) remain nearly zero in all tasks.

0HGLDQ

,40

0HDQ

025/)%2XUV )% *3,3' *3,/6 6)2/6 3&1 025/' 3*025/ &$34/ 43HQVLHYH 3'025/

1RUPDOL]HG:HLJKWHG5HZDUG

Figure 3: Evaluation of MORL-FB and several Does MORL-FB achieve effective general- MORL benchmark algorithms using aggregate ization across preferences? To better assess metrics, including median, mean, and interquarthe generalization capabilities of MORL-FB tile mean (IQM). These results show the superior across preferences, we further evaluate the al- performance of MORL-FB across all metrics. gorithms in a stylized setting where they are trained only on a small set of preference vectors Λtrain (rather than the whole Λ) and aim for generalization over Λ at test time. Specifically, for a d-objective task, we let Λtrain include only the standard basis preferences, i.e., d-dimensional one-hot vectors, and the uniform preference vector 7

Published as a conference paper at ICLR 2026

Figure 5: Empirical z distribution under t-SNE for Humanoid2d with MORL-FB (preferenceguided sampling, blue) and original FB (standard normal, red): The multi-modal distribution observed with MORL-FB suggests a more diverse set of latent representations compared to unimodal nature of original FB.

Figure 4: Evaluation of MORL-FB and benchmark methods (PD-MORL and Q-Pensieve) under a reduced preference set Λtrain during training: These demonstrate the generalization capability of MORL-FB across preferences.

[1/d, · · · , 1/d]. The testing setup is exactly the same as that for Figure 2. Here we focus on comparing MORL-FB to PD-MORL and Q-Pensieve, which are the top two benchmark methods in Figure 2 and utilize conditioned networks of structures different from MORL-FB. As shown in Figure 4, both PD-MORL and Q-Pensieve exhibit a notable decline in performance across all three metrics compared to those in Figure 2. In contrast, MORL-FB maintains consistent performance across the evaluated tasks, with only minimal degradation in UT and HV values compared to Figure 2. These findings showcase that MORL-FB can generalize more effectively over the entire preference set, even when trained on a limited set of preference vectors. More detailed results, such as the numerical values and the aggregated performance (e.g., IQM) are in Appendix D.

Figure 6: Evaluation of MORL-FB and its ablated versions on the Ant3d task. The results highlight the importance of PG-Explore and auxiliary losses, as removing these components leads to performance degradation. 4.1

Figure 7: Zero-shot cross-objective transfer from Hopper2d to Hopper3d and Hopper4d using vanilla FB and MORL-FB: Results demonstrate effective transfer by MORL-FB, supporting the efficacy of its proposed enhancements.

A BLATION S TUDY

Preference-Guided Exploration (PG-Explore). To investigate the benefits of sampling z from a preference-guided distribution, we perform an ablation study on comparing the proposed MORLFB and a variant of MORL-FB that samples z from N (0, Idz ), i.e., the distribution adopted by the vanilla FB. Figure 6 (specifically the bars in gray and orange) shows that MORL-FB indeed benefits significantly from a preference-guided distribution across the tested task, highlighting its importance in enabling directed exploration and sample-efficient policy learning. Moreover, we visualize the empirical distributions of the sampled z of MORL-FB and the original FB. Specifically, we record the z vectors used throughout training and apply t-SNE (Van der Maaten & Hinton, 2008) for visualization in a two-dimensional space. The results on Humanoid2d in Figure 5 show that sampling z from a normal distribution results in a unimodal empirical distribution (contours in red). By contrast, MORL-FB with the preference-guided sampling exhibits a multi8

Published as a conference paper at ICLR 2026

modal distribution (contours in blue), indicating a richer and more diverse set of z distributions. This multi-modality allows MORL-FB to better capture the underlying reward structure, achieving improved generalization and adaptation to various objectives. More visualization of z distributions for other tasks can be found in Appendix D. Auxiliary Q loss. To corroborate the efficacy of the auxiliary Q loss, we further conduct an ablation study on this term. From Figure 6 (specifically the bars in black and orange), the Q loss can facilitate the learning of forward and backward representations in MORL-FB and thereby boost the performance in both UT and HV. More ablation results across environments are in Appendix D.2. 4.2

Z ERO -S HOT C ROSS -O BJECTIVE T RANSFER

Recall that one salient feature of MORL-FB is to use z to encode the λ-dependent scalarized reward function. Accordingly, MORL-FB is endowed with the ability to achieve zero-shot transfer even across tasks of different number of objectives. Such zero-shot cross-objective transferability allows us to add new factors to the reward function without the need for retraining and hence is a very useful feature in practice. To validate this, we use MORL-FB to learn the F and B networks on Hopper2d and directly evaluate them on Hopper3d and Hopper4d, which involve additional reward terms like “jump height” and “z-axis speed”, at test time. Details of the environment configurations are in Appendox C. We conducted the same evaluation for vanilla FB (Touati et al., 2023) as a baseline. From Figure 7, vanilla FB cannot achieve effective cross-objective transfer given that vanilla FB already suffers in the standard MORL setting (cf. Figure 2). By contrast, MORL-FB achieves effective transfer across objectives in a zero-shot manner, corroborating the proposed enhancements.

R ELATED W ORK

5.1

M ULTI -O BJECTIVE RL

Single preference-conditioned policy methods. Single preference-conditioned policy methods learn one policy network that adapts to different objective trade-offs by conditioning on preference. Many of these methods employ scalarization techniques (Van Moffaert et al., 2013; Yang et al., 2019), transforming multi-objective problems into weighted single-objective problems. They allow policies to dynamically adjust at inference time. However, relying solely on linear reward aggregation without proper representation learning can lead to suboptimal solutions. To address this, CAPQL (Lu et al., 2023) introduced the concave reward terms for better optimization landscapes, while CN-DER (Abels et al., 2019) proposed a preference-conditioned Q-network with an experience replay mechanism to handle dynamic weights and mitigate non-stationarity. Q-Pensieve (Hung et al., 2023) improved sample efficiency by reusing past policy snapshots. Without using scalarization, Abdolmaleki et al. (2020) proposed to learn action distributions per objective and fitted a parametric policy via supervised learning. To improve adaptability to diverse and unseen preference vectors, methods like PCN (Reymond et al., 2022) formulated MORL as a classification problem, and MOAC (Zhou et al., 2024) finds Pareto-stationary points by adapting multi-gradient descent to MORL without scalarization. PD-MORL (Basaklar et al., 2023) also trains a single preferenceconditioned network but directly incorporates preference vectors, for example, through cosine similarity measures within its value-function update rule to efficiently learn a comprehensive set of policies across the continuous preference space. Multi-policy methods. Multi-policy methods explicitly learn multiple policies to cover the Pareto front, capturing diverse trade-offs in the training process. A key challenge in multi-policy MORL is efficiently constructing a coverage set that represents the full Pareto front while maintaining scalability. To refine policy selection and handle dominated actions, Lizotte et al. (2012) introduced a structured approach using linear value function approximation. Subsequent methods (Kyriakis & Deshmukh, 2022; Van Moffaert & Nowé, 2014; Xu et al., 2020) focused on improving exploration efficiency across the preference space but lacked structured learning mechanisms to generalize across diverse preferences. Building on the idea of incorporating structure into policy learning, Felten et al. (2024); Mossalam et al. (2016) extended structured learning for multi-objective RL, employing decomposition and sequential single-objective optimization to enhance efficiency. However, scalability and adaptability remained challenges. DG-MORL (Lu et al., 2024) leveraged demonstrations and a self-evolving mechanism to improve scalability. As for improving adaptability, Mossalam et al. (2016) extended Optimistic Linear Support (OLS) to deep RL, constructing a 9

Published as a conference paper at ICLR 2026

convex coverage set through a sequence of single-objective, providing a structured way to represent diverse trade-offs, but lacked effective transferability. 5.2

S UCCESSOR F EATURES FOR T RANSFER ACROSS R EWARD F UNCTIONS

One related transfer setting in RL is to learn policies for all reward functions that are linear combinations of a finite set of known features. Barreto et al. (2017) proposed the successor feature (SF), which reflects the state-action occupancy of a policy and can be viewed as an extension of the classic successor representation (Dayan, 1993). By design, SFs achieves transfer across reward functions in two ways: (i) For a fixed policy π, SFs can enable fast policy evaluation of π across different reward functions; (ii) Given a set of policies, SFs can be combined with the generalized policy improvement (GPI) update to generate a new policy that is no worse than the given set of policies and has a sub-optimality gap characterized by the task differences, for any reward function (Barreto et al., 2017; 2018). Accordingly, SFs has been combined with deep neural networks and applied to various tasks like subgoal extraction (Kulkarni et al., 2016) and robot navigation (Zhang et al., 2017). Subsequently, Borsa et al. (2019) extended the SFs by decoupling the policy and task description for better representational flexibility. Moreover, Alegre et al. (2022) applied SFs and GPI to solve MORL by enabling transfer across different scalarized reward functions. Chua et al. (2024) proposed a practical method for learning the SFs directly from pixels for image-based control. More recently, Zhang et al. (2024) established the convergence analysis with generalization guarantees for SFs under neural function approximation. Despite the transfer capability, the limitations of SFs and its variants are mainly two-fold: (i) SFs require a set of pre-defined reward features, which could miss important aspects of the environment and limit adaptability. (ii) While SFs can achieve fast policy evaluation across reward functions, SFs cannot directly induce an optimal policy when given an arbitrary new reward function at test time. 5.3

R EWARD -F REE RL

Among the studies in the RFRL literature, (Touati & Ollivier, 2021; Touati et al., 2023) are the most relevant to our work. To address the aforementioned issues of SFs, Touati & Ollivier (2021) adopted the reward-free MDP formulation and proposed a low-rank model termed the forwardbackward (FB) representations, which capture the state-action occupancy of the optimal policies for all the reward functions by learning the required features directly from data. The FB framework has been implemented and validated for both reward-free discrete control (Touati & Ollivier, 2021) and continuous control (Touati et al., 2023). Subsequently, the framework of FB representations has been extended to various settings, such as offline RL with low-quality data (Jeen et al., 2024), online unsupervised RL (Sun et al., 2025), imitation learning (Pirotta et al., 2024; Tirinzoni et al., 2025), and partially-observable MDPs (Jeen et al., 2025). Another line of RFRL research focuses on achieving provably efficient exploration without using any reward information, in both tabular settings (Jin et al., 2020; Kaufmann et al., 2021; Ménard et al., 2021; Wu et al., 2022) and under function approximation (Qiu et al., 2021; Wagenmaker et al., 2022; Wang et al., 2020; Zanette et al., 2020; Zhang et al., 2021). Inspired by the RFRL literature, we propose to rethink MORL via RFRL and adapt the FB method to boost the sample efficiency and generalization in MORL.

C ONCLUSION , L IMITATIONS , AND F UTURE W ORK

We propose MORL-FB, which offers a new perspective by rethinking MORL through the lens of RFRL. By using RFRL as auxiliary tasks, MORL-FB provides the first systematic adaptation of RFRL to MORL and enhances its sample efficiency with critical algorithmic enhancements, including preference-guided exploration with mini-batch sampling and an auxiliary Q loss based on the observed reward vectors. MORL-FB achieves strong performance in a variety of discrete and continuous control tasks, offering superior efficiency, better generalization, and zero-shot cross-objective transfer. A key limitation, inherited from FB-based RFRL, is the need for more advanced exploration, especially in complex or sparse-reward environments, where dedicated strategies are crucial. Future work includes exploring RFRL methods beyond FB, such as representation learning of successor measures (Agarwal et al., 2025; Farebrother et al., 2023) and learning distance-preserving state representations (Park et al., 2024), to further reveal RFRL’s advantages in MORL. 10

Published as a conference paper at ICLR 2026

ACKNOWLEDGMENT This research is partially supported by the National Science and Technology Council (NSTC) of Taiwan under Grant Numbers 114-2628-E-A49-002 and 114-2634-F-A49-002-MBK. We also thank the National Center for High-performance Computing (NCHC) for providing computational and storage resources.

E THICS STATEMENT Our work develops and evaluates reinforcement learning methods purely in simulated environments, without involving human subjects or sensitive data. This submission follows the code of ethics.

R EPRODUCIBILITY STATEMENT We release our code in the supplementary material and describe the commands needed to execute the code in a Readme file attached in the supplementary material. Additionally, we attach the list of package dependencies which can be used to build the environment.

T HE U SE OF L ARGE L ANGUAGE M ODELS (LLM S ) Large language models (LLMs) were employed exclusively for language editing and polishing of the manuscript. They were not used for designing methods, conducting experiments, or analyzing results.

B IBLIOGRAPHY Abbas Abdolmaleki, Sandy H. Huang, Leonard Hasenclever, Michael Neunert, H. Francis Song, Martina Zambelli, Murilo F. Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In International Conference on Machine Learning, 2020. Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International Conference on Machine Learning, 2019. Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021. Siddhant Agarwal, Harshit Sikchi, Peter Stone, and Amy Zhang. Proto successor measure: Representing the behavior space of an RL agent. In International Conference on Machine Learning, pp. 566–586, 2025. Lucas Nunes Alegre, Ana Bazzan, and Bruno C Da Silva. Optimistic linear support and successor features as a basis for optimal policy transfer. In International Conference on Machine Learning, 2022. Lucas Nunes Alegre, Ana LC Bazzan, Diederik M Roijers, Ann Nowé, and Bruno C da Silva. Sample-efficient multi-objective learning via generalized policy improvement prioritization. In International Conference on Autonomous Agents and Multiagent Systems, 2023. André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, 2017. Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501–510, 2018. 11

Published as a conference paper at ICLR 2026

Toygun Basaklar, Suat Gumussoy, and Umit Ogras. PD-MORL: Preference-driven multi-Objective reinforcement learning algorithm. In International Conference on Learning Representations, 2023. Diana Borsa, Andre Barreto, John Quan, Daniel J Mankowitz, Hado van Hasselt, Remi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In International Conference on Learning Representations, 2019. Raymond Chua, Arna Ghosh, Christos Kaplanis, Blake A. Richards, and Doina Precup. Learning successor features the simple way. In Advances in Neural Information Processing Systems, 2024. Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 1993. Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them. In International Conference on Machine Learning, 2023. Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Marc G. Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. In International Conference on Learning Representations, 2023. Florian Felten, Lucas Nunes Alegre, Ann Nowe, Ana Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In Advances in Neural Information Processing Systems, 2023. Florian Felten, El-Ghazali Talbi, and Grégoire Danoy. Multi-objective reinforcement learning based on decomposition: A taxonomy and framework. Journal of Artificial Intelligence Research, 2024. Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018. Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers. A practical guide to multi-objective reinforcement learning and planning. In International Conference on Autonomous Agents and Multiagent Systems, 2022. Jifeng Hu, Yanchao Sun, Hechang Chen, Sili Huang, haiyin piao, Yi Chang, and Lichao Sun. Distributional reward estimation for effective multi-agent deep reinforcement learning. In Advances in Neural Information Processing Systems, 2022. Wei Hung, Bo-Kai Huang, Ping-Chun Hsieh, and Xi Liu. Q-Pensieve: Boosting sample efficiency of multi-objective RL through memory sharing of Q-snapshots. In International Conference on Learning Representations, 2023. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016. Scott Jeen, Tom Bewley, and Jonathan Cullen. Zero-shot reinforcement learning from low quality data. Advances in Neural Information Processing Systems, 37:16894–16942, 2024. Scott Jeen, Tom Bewley, and Jonathan Cullen. Zero-shot reinforcement learning under partial observability. In Reinforcement Learning Conference, 2025. Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning, 2020. Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive reward-free exploration. In Algorithmic Learning Theory, 2021. 12

Published as a conference paper at ICLR 2026

Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396, 2016. Panagiotis Kyriakis and Jyotirmoy Deshmukh. Pareto policy adaptation. In International Conference on Learning Representations, 2022. Xi Lin, Xiaoyuan Zhang, Zhiyuan Yang, Fei Liu, Zhenkun Wang, and Qingfu Zhang. Smooth tchebycheff scalarization for multi-objective optimization. In International Conference on Machine Learning, 2024. Daniel J Lizotte, Michael Bowling, and Susan A Murphy. Linear fitted-Q iteration with multiple reward functions. Journal of Machine Learning Research, 2012. Haoye Lu, Daniel Herman, and Yaoliang Yu. Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality. In International Conference on Learning Representations, 2023. Junlin Lu, Patrick Mannion, and Karl Mason. Demonstration guided multi-objective reinforcement learning. arXiv preprint arXiv:2404.03997, 2024. Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning, 2021. Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. arXiv preprint arXiv:1610.02707, 2016. Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with Hilbert representations. In International Conference on Machine Learning, pp. 39737–39761, 2024. Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier. Fast imitation via behavior foundation models. In International Conference on Learning Representations, 2024. Shuang Qiu, Jieping Ye, Zhaoran Wang, and Zhuoran Yang. On reward-free RL with kernel and neural function approximations: Single-agent MDP and Markov game. In International Conference on Machine Learning, 2021. Shuang Qiu, Dake Zhang, Rui Yang, Boxiang Lyu, and Tong Zhang. Traversing Pareto optimal policies: Provably efficient multi-objective reinforcement learning. arXiv preprint arXiv:2407.17466, 2024. Banafsheh Rafiee, Jun Jin, Jun Luo, and Adam White. What makes useful auxiliary tasks in reinforcement learning: investigating the effect of the target policy. arXiv preprint arXiv:2204.00565, 2022. Mathieu Reymond, Eugenio Bargiacchi, and Ann Nowé. Pareto conditioned networks. In International Conference on Autonomous Agents and Multiagent Systems, 2022. Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet, and Joelle Pineau. Reward estimation for variance reduction in deep reinforcement learning. In Conference on Robot Learning, 2018. Jingbo Sun, Songjun Tu, Haoran Li, Xin Liu, Yaran Chen, Ke Chen, Dongbin Zhao, et al. Unsupervised zero-shot reinforcement learning via dual-value forward-backward representation. In International Conference on Learning Representations, 2025. Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. A Bradford Book, 2018. Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the cvar via sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015. 13

Published as a conference paper at ICLR 2026

Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control via behavioral foundation models. In International Conference on Learning Representations, 2025. Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. In Advances in Neural Information Processing Systems, 2021. Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In International Conference on Learning Representations, 2023. Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine learning, 84 (1):51–80, 2011. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 2008. Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of Pareto dominating policies. Journal of Machine Learning Research, 2014. Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In Adaptive Dynamic Programming and Reinforcement Learning, 2013. Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L Lewis, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. Advances in Neural Information Processing Systems, 32, 2019. Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Rewardfree RL is no harder than reward-aware RL in linear markov decision processes. In International Conference on Machine Learning, 2022. Ruosong Wang, Simon S Du, Lin Yang, and Russ R Salakhutdinov. On reward-free reinforcement learning with linear function approximation. In Advances in Neural Information Processing Systems, 2020. Jingfeng Wu, Vladimir Braverman, and Lin Yang. Gap-dependent unsupervised exploration for reinforcement learning. In International Conference on Artificial Intelligence and Statistics, 2022. Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In International Conference on Machine Learning, 2020. Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems, 2019. Andrea Zanette, Alessandro Lazaric, Mykel J Kochenderfer, and Emma Brunskill. Provably efficient reward-agnostic navigation with linear value iteration. In Advances in Neural Information Processing Systems, 2020. Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371–2378, 2017. Shuai Zhang, Heshan Devaka Fernando, Miao Liu, Keerthiram Murugesan, Songtao Lu, Pin-Yu Chen, Tianyi Chen, and Meng Wang. SF-DQN: Provable knowledge transfer using successor feature for deep reinforcement learning. In International Conference on Machine Learning, pp. 58897–58934, 2024. Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Reward-free model-based reinforcement learning with linear function approximation. In Advances in Neural Information Processing Systems, 2021. 14

Published as a conference paper at ICLR 2026

Tianchen Zhou, FNU Hairi, Haibo Yang, Jia Liu, Tian Tong, Fan Yang, Michinari Momma, and Yan Gao. Finite-time convergence and sample complexity of actor-critic multi-objective reinforcement learning. In International Conference on Machine Learning, 2024. Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Computation, 1999.

Published as a conference paper at ICLR 2026

A PPENDICES A Additional Background: Successor Measure and Forward-Backward Representations 16 A.1 Successor Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.2 Forward-Backward (FB) Representations . . . . . . . . . . . . . . . . . . . . . .

B Detailed Pseudo Code of MORL-FB

C Detailed Configurations of Experiments

C.1 Evaluation Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.3 Compute Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D Additional Experimental Results

D.1 Visualization of z Distribution in Different Environments . . . . . . . . . . . . . .

D.2 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.3 Performance Comparison of MORL-FB Under Different Q-loss Coefficients . . . .

D.4 Performance Comparison of MORL-FB Under State-Action-Based Rewards and State-Based Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.5 Performance Comparison of MORL-FB Under Stochastic Rewards . . . . . . . . .

D.6 Performance Comparison of MORL-FB With Nonlinear Scalarization . . . . . . .

D.7 Sample Efficiency of MORL-FB . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.8 Cross-Objective Transfer Capability of MORL-FB . . . . . . . . . . . . . . . . .

D.9 RFRL as a source of auxiliary tasks . . . . . . . . . . . . . . . . . . . . . . . . .

D.10 Comparison of Pareto Fronts . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.11 Robustness under Worst-Case Preferences . . . . . . . . . . . . . . . . . . . . . .

D.12 Computational Cost Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.13 Detailed Experimental Results of Section 4 . . . . . . . . . . . . . . . . . . . . .

A DDITIONAL BACKGROUND : S UCCESSOR M EASURE AND F ORWARD -BACKWARD R EPRESENTATIONS

A.1

S UCCESSOR M EASURE

Recall that in standard single-objective RL, the Q-function under a policy π with respect to a reward function R : S × A → R is defined as "∞ # X Qπ (s, a) := Eπ γ t R(st , at ) | s0 = s, a0 = a , (7) t=0

which captures the long-term expected discounted reward under the policy π. One way to interpret the Q function is through the lens of successor measure Mπ : S × A → ∆(S × A), which reflects the discounted, expected future occupancy of the state-action pairs in X when starting from (s, a) and following policy π. Formally, the successor measure Mπ is defined as follows: For any subset 16

Published as a conference paper at ICLR 2026

X ⊂ S × A, define Mπ (s, a, X) := Eπ

"∞ X

# γ t · I{(st , at ) ∈ X} s0 = s, a0 = a ,

(8)

t=0

where I{·} is the indicator function. The key property is that this measure Mπ is agnostic to the reward function R as it only depends on the environment dynamics and the policy π. Note that viewing Mπ as a measure can deal with both the discrete and continuous cases. For ease of exposition, in the sequel, we focus on the case of discrete state and action spaces. With that said, we also let Mπ (s, a, s′ , a′ ) denote the successor measure of the state-action pair (s′ , a′ ) when starting from (s, a) and following policy π. We can connect the Q function in Equation (7) with the successor measure in Equation (8) as follows: X Qπ (s, a) = Mπ (s, a, s′ , a′ ) · R(s′ , a′ ). (9) (s′ ,a′ )∈S×A

Crucially, we can make two observations: (i) If we can directly learn the successor measure of a policy π, then we can derive the corresponding Q function in a zero-shot manner given any reward function. (ii) For any reward function R, if we can learn the successor measure of an optimal policy ∗ ∗ ), then we can learn the optimal Q function QπR and thereafter derive an for R (denoted by πR ∗ in the successor optimal policy for R. Therefore, if we can directly encode the information about πR measure, then we can learn the optimal Q function for any possible reward function.

A.2

F ORWARD -BACKWARD (FB) R EPRESENTATIONS

Our MORL-FB method is built on the Forward-Backward (FB) representation approach (Touati & Ollivier, 2021; Touati et al., 2023), which provides a compact, low-rank factorization of the successor measure. The FB framework decomposes the successor measure into the product of two vector-valued functions, namely a Forward function F : S × A × Rdz → Rdz and a Backward function B : S × A → Rdz . Let πz be a policy that depends on some vector z ∈ Rdz . Specifically, under the FB framework, the successor measure Mπ is represented by: Mπz (s, a, s′ , a′ ) = F(s, a, z)⊤ B(s′ , a′ ),

(10)

where z acts as some latent vector that captures the required information about the policy πz . Using this factorization, the Q-function for any policy πz can be expressed as X Qπz (s, a) = F(s, a, z)⊤ B(s′ , a′ )R(s′ , a′ ), (11) (s′ ,a′ )∈S×A



 X

= F(s, a, z)⊤ 

B(s′ , a′ )R(s′ , a′ ) .

(12)

(s′ ,a′ )∈S×A

=:zR

}

Note that Equation (12) holds for any z (and hence any policy πz ). Therefore, we have the freedom to set zR to be the latent vector that results in an optimal policy for the reward function R, i.e., πzR ≡ ∗ πR . To achieve this, we shall simply enforce the following Bellman optimality equation (Touati & Ollivier, 2021), i.e., ∗ πR (s) = arg max F(s, a, zR )⊤ zR . a∈A

(13)

The final optimal policy for the given preference is then directly constructed by maximizing the approximated Q-function: Therefore, if F and B are well learned, then one can directly retrieve an optimal policy for any reward function R given at test time by computing zR and apply Equation (13). This factorization provides the foundation of our MORL-FB algorithm to efficiently explore and generalize across preferences during training. 17

Published as a conference paper at ICLR 2026

Algorithm 2 MORL-FB 1: Input: Network parameters θ, θ̄, ω, ω̄, η, η̄, preference sampling distribution Pλ , preference set

Λ, actor learning rates µπ , FB presentation learning rate µFB , z dimension dz , sample number ns , update frequency nu , warm up steps nw , and target smoothing coefficient τ 2: Initialize networks Fθ , Bω , πη and target networks Fθ̄ , Bω̄ , πη̄ 3: Initialize replay buffer M ← ∅ 4: for each iteration i do 5: Sample a preference vector λ ∼ Pλ 6: if i ≤ nw then ▷ Warm-up stage 7: Sample z from a multivariate√normal distribution N (0, Idz ) z 8: Normalize z such that z ← dz ∥z∥ 2 9: else 10: z ← PG-Explore(λ) 11: end if 12: for each environment step t do 13: at ∼ πη (·|st ; λ) 14: st+1 ∼ P(·|s S t , at ) 15: M ← M {(st , at , rt , λt , st+1 )} 16: end for 17: for each gradient step j do 18: Sample a batch of transitions {(s, a, r, λ, s′ )} from the replay buffer M 19: zj ← PG-Explore(λ) 20: θ ← θ − µFB ∇θ (LQ (θ; λ) + LM (θ, ω; λ)) 21: ω ← ω − µFB ∇ω (Ln (ω; λ) + LM (θ, ω; λ)) 22: end for 23: if i % nu == 0 then 24: η ← η − µπ ∇η Lπ (η; λ) 25: θ̄ ← τ θ + (1 − τ )θ̄ 26: ω̄ ← τ ω + (1 − τ )ω̄ 27: η̄ ← τ η + (1 − τ )η̄ 28: end if 29: end for 30: function PG-E XPLORE(λ) 31: Sample a batch D of ns non-terminal transitions {(s, a, r, s′ )} from M ⊤ P λ 32: z ← (s,a,r,s′ )∈D Bω (s,a)r ns √ z 33: Normalize z such that z ← dz ∥z∥ 2 34: return z 35: end function

D ETAILED P SEUDO C ODE OF MORL-FB

Algorithm 2 details the proposed MORL-FB method. Initially, during the warm-up phase (lines 6-8), the latent vector z is sampled from a standard multivariate normal distribution. After the warm-up, z is determined using the preference-guided sampling scheme (line 10). This z is then used to generate trajectories within the environment, which are stored in the replay buffer M (lines 12-16). Model updates are performed by sampling transitions from M (lines 17-22). A delayed actor update mechanism is employed for the actor model (lines 23-24), and target networks are updated via a soft update scheme (lines 25-27). The Preference-Guided Exploration function (lines 30-35) normalizes √ z the sampled latent vector z (line 33) as z ← dz ∥z∥ . This normalization step, motivated by the 2 prior work (Touati et al., 2023), has been observed to improve performance. As shown in Algorithm 2, the training of MORL-FB involves the following loss functions: Measure Loss. The Measure loss, LM (θ, ω; zλ ), is central to learning a task-agnostic representation of environment dynamics, Fθ (st , at , zλ ), encoding command-conditioned successor measures. It enforces Bellman consistency for these measures when projected onto a learned basis Bω (s′ , a′ ), as shown in Equation (14). This mechanism, drawn from (Touati et al., 2023), aims to separate the 18

Published as a conference paper at ICLR 2026

environment structure from specific rewards. This disentanglement is crucial for enabling zero-shot generalization, allowing the agent to understand “what happens next” irrespective of the immediate goal, forming a reusable foundation for various tasks. LM (Fθ , Bω ; zλ ) =E(st ,at ,st+1 )∼D (Fθ (st , at , zλ )⊤ Bω (s′ , a′ ) (s′ ,a′ )∼D

− γFθ̄ (st+1 , π(st+1 , zλ ), zλ )⊤ Bω̄ (s′ , a′ ))2

− 2 E(st ,at ,st+1 )∼D [Fθ (st , at , zλ )⊤ Bω (st+1 , at+1 )].

(14)

where ρ denotes the underlying distributions of the dataset. Auxiliary Q Loss. To ensure the learned representation Fθ is relevant for decision-making, the Auxiliary Q Loss, LQ (θ; zλ ), connects it to task-specific values. When explicit reward signals rt and corresponding preferences λ are available, Equation (15) minimizes a standard temporal difference error. This is vital for MORL contexts, effectively teaching Fθ to support optimizing diverse rewards. h 2 i LQ (θ; zλ ) = E(st ,at ,rt ,st+1 )∼D Fθ (st , at , zλ )⊤ zλ − λ⊤ rt + γFθ̄ (st+1 , πη̄ (st+1 ), zλ )⊤ zλ . (15) Orthonormality Loss. The Orthonormality Regularization Loss, Ln (ω), acts as a crucial regularizer for the learned basis functions Bω (s, a). Its purpose, as reflected in Equation (16), is to promote a well-conditioned and non-degenerate basis. By encouraging properties such as orthogonality between basis vectors and unit norm, this loss helps prevent representational collapse and redundancy within Bω . This, in turn, ensures that the successor measures are projected onto a stable and diverse set of features, enhancing the robustness and quality of the learned representations Fθ . Ln (ω) = E(s,a)∼D,(s′ ,a′ )∼D

i 2 Bω (s, a)⊤ Bω (s′ , a′ ) − ∥Bω (s, a)∥22 − ∥Bω (s′ , a′ )∥22 .

(16)

Policy Loss. The agent’s behavior is refined through the Policy Optimization Loss, Lπ (η; zλ ), which trains the policy πη within an actor-critic paradigm. The actor’s objective is to maximize the Q-values estimated by the critic, where these Q-values are derived from the learned representation as Q(s, a; zλ ) = Fθ (s, a, zλ )⊤ zλ (Equation (17)). This loss drives the policy to select actions that are optimal for the task specified by the current command zλ . It thus enables the agent to translate its universal understanding of the environment into effective, task-adaptive behavior. Lπ (η; zλ ) = Es∼D [−Q(s, πη (s); zλ )] ,

where Q(s, a; zλ ) = F(s, a, zλ )⊤ zλ .

(17)

D ETAILED C ONFIGURATIONS OF E XPERIMENTS

In this section, we describe the experimental setup used to evaluate the performance of our approach. We detail the hyperparameters used in our experiments, as well as the reference points chosen for HV evaluation across different environments. C.1

E VALUATION E NVIRONMENTS

We evaluate the performance of our proposed method, MORL-FB, across a diverse set of multiobjective reinforcement learning environments. These environments, detailed below, encompass both established MuJoCo-based locomotion tasks and discrete problems, allowing us to assess its adaptability across distinct settings with varying state spaces, action spaces, and objective numbers. MuJoCo-Based Continuous Control: • Halfcheetah2d: The state space and action space are defined as S ⊆ R17 and A ⊆ R6 , respectively. The two objectives for this environment are maximizing moving speed along the x-axis and minimizing energy cost. 19

Published as a conference paper at ICLR 2026

• Walker2d: The state space and action space are defined as S ⊆ R17 and A ⊆ R6 , respectively. The two objectives for this environment are maximizing moving speed along the x-axis and minimizing energy cost. • Hopper3d: The state space and action space are defined as S ⊆ R11 and A ⊆ R3 . The three objectives include maximizing moving speed along the x-axis, maximizing jumping height along the z-axis, and minimizing energy cost. • Ant3d: The state space and action space are defined as S ⊆ R27 and A ⊆ R8 . The three objectives are maximizing moving speed along the x-axis, maximizing moving speed along the y-axis, and minimizing energy cost. • Humanoid2d: The state space and action space are defined as S ⊆ R376 and A ⊆ R17 . The two objectives are maximizing moving speed along the x-axis and minimizing energy cost. Additionally, we set the healthy reward parameter to 1.0 to encourage exploration and stability. • Humanoid5d: The state space and action space are defined as S ⊆ R376 and A ⊆ R17 . This environment has five objectives: maximizing moving speed along the x-axis, maximizing moving speed along the y-axis, maximizing angular velocity on the left elbow, maximizing angular velocity on the right elbow, and minimizing energy cost. Similar to Humanoid2d, the healthy reward parameter is set to 1.0 to ensure meaningful evaluation. Classic Control: • Deep Sea Treasure (DST): The Deep Sea Treasure environment is a classic MORL problem in which the agent controls a submarine in a 2D grid world (Vamplew et al., 2011). The two conflicting objectives are typically maximizing collected treasure value and minimizing the time cost of collection. • Fruit Tree Navigation (FTN): This environment is structured as a full binary tree of depth d = 5, 6, or 7 (Yang et al., 2019). The agent navigates from the root to a leaf node. Every leaf contains a fruit with values for six objectives: Protein, Carbs, Fats, Vitamins, Minerals, and Water. C.2

E XPERIMENTAL S ETUP

To begin with, we describe the hyperparameters of the benchmark MORL methods and the proposed MORL-FB for better reproducibility. Hyperparameters for Experiments. To ensure a fair comparison, for those benchmark methods that already provide tuned task-specific hyperparameters on MuJoCo, we primarily refer to their original papers for the hyperparameter configurations, including PGMORL (Xu et al., 2020) and CAPQL (Lu et al., 2023). Table 1 and Table 3 list the detailed hyperparameters used in our experiments. For PGMORL, the hyperparameters reflect its evolutionary population-based design. The parameter n defines the number of parallel reinforcement learning tasks in each generation. Each task includes mw warm-up iterations and mt evolutionary iterations. Pnum and Psize define the number and size of the performance buffers. The PPO parameters used across all environments are summarized in Table 2. For CAPQL, the hyperparameter α controls the strength of a concave regularization term added to the reward. The general hyperparameters shared by CAPQL are listed in Table 4. Table 1: Hyperparameters of PGMORL. Environments

Pnum

Psize

HalfCheetah2d Walker2d Hopper3d Ant3d Humanoid2d Humanoid5d

6 6 15 15 6 35

80 80 200 200 200 200

20 20 40 40 40 40

100 100 210 210 100 550

2 2 2 2 2 2

7 7 7 7 7 7

−1 −1 −106 −106 −1 −106

Published as a conference paper at ICLR 2026

Table 2: PPO hyperparameters used in PGMORL. Parameter

Value

Timesteps per actor batch Processes number Learning rate Discount factor (γ) GAE lambda Batch size PPO epochs Entropy coefficient Value loss coefficient

2,048 4 3 × 10−4 0.995 0.95 32 10 0 0.5

Table 3: Augmentation strength of CAPQL. Environments

HalfCheetah2d Walker2d Hopper3d Ant3d Humanoid2d Humanoid5d

0.1 0.05 0.2 0.2 0.005 0.005

Table 4: Hyperparameters of CAPQL. Parameter

Value

Optimizer Learning rate Discount factor (γ) Number of hidden units per layer Replay buffer size Batch size Nonlinearity Target smoothing coefficient (τ )

Adam 3 × 10−4 0.99 256 106 256 ReLU 0.005

For algorithms without officially tuned or specified hyperparameters, we perform hyperparameter optimization (HPO) by following (Eimer et al., 2023). Specifically, we employ Bayesian optimization by using the Weights & Biases Sweeps. Each episode refers to a simulation run for 1M environment steps, and the optimization is performed over 10 episodes. Tables 5 to 11 summarize the search ranges and the final selected hyperparameters. • MORL-FB: For MORL-FB, which is built on the implementation of the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm (Fujimoto et al., 2018), we prioritized the tuning of parameters inherent to TD3. Specifically, this included critical elements such as learning rates for actor and critic networks, the target network update rate, and policy delay, which are known to significantly influence the stability and performance of TD3based agents. • Q-Pensieve: Our tuning efforts centered on parameters associated with its core Q-Snapshot mechanism (Hung et al., 2023). Given that Q-Pensieve’s efficacy in improving sample efficiency stems from the storage and utilization of these Q-function snapshots, parameters governing the snapshot buffer, the frequency of snapshot capture, and their influence on the policy update were key areas of focus during hyperparameter optimization. • PD-MORL: For PD-MORL, which can be viewed as the multi-objective extensdion of TD3 (Fujimoto et al., 2018), we prioritized the tuning of parameters inherent to TD3. This 21

Published as a conference paper at ICLR 2026

included critical elements such as learning rates for actor and critic networks, the target network update rate, and policy delay, which are known to significantly influence the stability and performance of TD3-based agents. • MORL/D: For MORL/D, which is built upon SAC, we tuned the standard hyperparameters of SAC. In addition, we focused on four components that are central to the MORL/D framework: population size, neighborhood size, scalarization method, and weight adaptation method. These elements directly affect how the algorithm decomposes the multiobjective space and maintains policy diversity. • PCN: For PCN, since the hyperparameter settings for continuous control tasks such as MuJoCo environments were not specified in (Reymond et al., 2022), we tuned three critical parameters: learning rate, batch size, and hidden dimension, which affect training stability, sample efficiency, and the model’s ability to generalize across diverse preference vectors. • SFOLS: For SFOLS, since we utilize its official implementation for discrete problems and extend it with a TD3 backbone for evaluation on continuous control tasks, we tuned the parameters inherent to TD3, including learning rates for the actor and critic networks, policy noise, and target policy smoothing noise. These components are known to significantly influence the stability and performance of TD3-based agents. • GPI-PD: For GPI-PD, since GPI-PD relies on a learned dynamics model for sample generation, we tuned four key parameters: dynamics rollout length, rollout frequency, model training frequency, and real data ratio. These factors critically affect model accuracy, stability of planning updates, and the overall effectiveness of Dyna-style training. Table 5: Hyperparameter tuning for MORL-FB. Parameter

Value

Optimal Value

Learning rate

[0.0001, 0.01]

0.0001

Policy update delay

{1, 2, 5, 10}

Steps per update

{1, 2, 5, 10}

Latent dimension (z dimension)

{50, 150, 300}

300

Exploration noise std.

[0.1, 1.0]

0.1

Target policy smoothing noise std.

[0.1, 1.0]

0.2

Table 6: Hyperparameter tuning for Q-Pensieve. Parameter

Value

Optimal Value

Learning rate

[0.0001, 0.01]

0.0001

Q replay buffer size

{1, 2, 4}

Preference set size

{1, 2, 4}

Table 7: Hyperparameter tuning for PD-MORL. Parameter

Value

Optimal Value

Learning rate

[0.0001, 0.01]

0.0003

Exploration noise std.

[0.1, 1.0]

0.15

Target policy smoothing noise std.

[0.1, 1.0]

0.25

Policy update delay

{1, 2, 5, 10}

Hyperparameters of MORL-FB. The hyperparameters for the MORL-FB experiments were chosen to ensure fair evaluation and stable learning, guided by prior research and the HPO results. The 22

Published as a conference paper at ICLR 2026

Table 8: Hyperparameter tuning for PCN. Parameter

Value

Optimal Value

Learning rate

[0.0001, 0.01]

0.0023

Batch size

{64, 128, 256}

Number of hidden units per layer

{256, 512, 1024}

1, 024

Table 9: Hyperparameter tuning for MORL/D. Parameter

Value

Optimal Value

Learning rate

[0.0001, 0.01]

0.0013

Batch size

{64, 128, 256}

256

Number of hidden units per layer

{256, 512, 1024}

1, 024

Pop size

{4, 6, 8}

Neighborhood size

{0, 1, 2}

Scalarization method

{none, ws}

Weight adaptation method

{none, P SA}

P SA

Table 10: Hyperparameter tuning for SFOLS. Parameter

Value

Optimal Value

Learning rate

[0.0001, 0.01]

0.0006

Batch size

{64, 128, 256}

256

Number of hidden units per layer

{256, 512, 1024}

1, 024

Target smoothing coefficient (τ )

[0.001, 0.02]

0.0061

Exploration noise std.

[0.1, 0.3]

0.1736

Target policy smoothing noise std.

[0.1, 0.5]

0.4232

Table 11: Hyperparameter tuning for GPI-PD. Parameter

Value

Optimal Value

Learning rate

[0.0001, 0.01]

0.0003

Batch size

{64, 128, 256}

256

Number of hidden units per layer

{256, 512, 1024}

1, 024

Dynamics rollout length

[5, 20]

Dynamics rollout freqency

[50, 500]

312

Dynamics train freqency

[100, 500]

139

Dynamics real ratio

[0.05, 0.2]

0.0845

hyperparameters tuned via HPO are presented in Table 5, while the remaining values were primarily drawing from the default settings of original FB and TD3 algorithms. The complete configuration is detailed in Table 12. Reference Points for HV Evaluation. We compute the HV indicator using predefined reference points (Ref. Point) for each environment. These reference points serve as baselines to measure the 23

Published as a conference paper at ICLR 2026

Table 12: Hyperparameter configuration for MORL-FB experiments. Parameter

Value

Total number of environment steps Replay buffer size Latent dimension (z dimension) Interface batch size Number of hidden units per layer Preprocessing feature dimension Batch size Target smoothing coefficient (τ ) Discount factor (γ) Learning rate Policy update delay Steps per update Exploration noise std. Target policy smoothing noise std. Clipping parameter Value loss coefficient

3 × 106 1 × 106 300 5,120 1,024 512 1,024 0.01 0.99 1 × 10−4 10 1 0.1 0.2 0.5 1

coverage of the Pareto front obtained during training. Table 13 provides the specific reference points used in different environments. Table 13: Reference points used for HV calculation in different environments. Environment

Ref. Point

Deep Sea Treasure Fruit Tree Navigation Halfcheetah2d Walker2d Hopper3d Hopper4d Ant3d Humanoid2d Humanoid5d

C.3

(0, -50) (0, 0, 0, 0, 0, 0) (0, -8000) (0, -8000) (0, 0, -8000) (0, 0, -8000, 0) (0, 0, -8000) (0, -8000) (0, 0, 0, 0, -8000)

C OMPUTE R ESOURCES

All models were trained on a workstation featuring a single NVIDIA RTX 4090 GPU, an Intel Core i7-13700K CPU, and 64 GB of system memory.

Published as a conference paper at ICLR 2026

A DDITIONAL E XPERIMENTAL R ESULTS

D.1

V ISUALIZATION OF z D ISTRIBUTION IN D IFFERENT E NVIRONMENTS

We analyze the learned latent representations of MORL-FB by visualizing the distribution of sampled z across various environments using t-SNE (Van der Maaten & Hinton, 2008). These visualizations offer insight into the method’s ability to capture the underlying structure of the multi-objective tasks. Consistent with the observations on Humanoid2d (Figure 5), the visualizations for Walker2d, Hopper3d, and Ant3d (Figure 8, Figure 9, and Figure 10, respectively) demonstrate that preferenceguided sampling yields distinct distributions compared to sampling from a standard normal distribution. These results further support the hypothesis that preference-guided sampling promotes the exploration of a more diverse set of latent representations, which may contribute to improved generalization and adaptation on various objectives.

Figure 8: Empirical z distribution under MORL-FB with preference-guided sampling versus Original FB with simple normal distributions on Walker2d.

Figure 9: Empirical z distribution under MORL-FB with preference-guided sampling versus Original FB with simple normal distributions on Hopper3d. To shows the multi-modality of MORL-FB, we visualized the positions of latent variables z inferred from different preferences on the t-SNE plot in Figure 11. This demonstrates that MORL-FB effectively encodes different preferences into separate regions of the latent space, leading to more diverse policies. To further illustrate this, we provide a demo of the policies learned by MORL-FB and vanilla FB in this link: https://imgur.com/a/ehx1v7q, where the z’s are selected from different positions on the t-SNE plot. 25

Published as a conference paper at ICLR 2026

Figure 10: Empirical z distribution under MORL-FB with preference-guided sampling versus Original FB with simple normal distributions on Ant3d.

Figure 11: Empirical z distribution under MORL-FB with preference-guided sampling (blue) versus Original FB with simple normal distributions (red) on Humanoid2d. This figure is slightly different from Figure 5 due to the additional preference points and the inherent randomness of t-SNE.

D.2 D.2.1

A BLATION S TUDY E XPERIMENT ON P REFERENCE -G UIDED E XPLORATION

To assess the data efficiency of our proposed MORL-FB, we compared its performance against PDMORL (Basaklar et al., 2023). PD-MORL necessitates three to five times more training samples to train its preference interpolator. Can MORL-FB maintain superior performance while reducing data requirements? Table 14 reveals that MORL-FB consistently outperforms PD-MORL across various environments. Specifically, MORL-FB achieves higher UT values in five out of six tasks, demonstrating its effec26

Published as a conference paper at ICLR 2026

tiveness in most scenarios. In more complex settings, such as MO-Humanoid, how does MORL-FB compare in HV results? The results indicate that MORL-FB remains competitive, underscoring its significant data efficiency gains. By eliminating the need for additional data to pretrain an interpolator, MORL-FB achieves competitive or superior performance while requiring significantly fewer training samples in multi-objective environments. Table 14: Performance comparison between MORL-FB and PD-MORL (with interpolator) across key metrics (UT, HV, and ED) on various multi-objective tasks.

D.2.2

Environments

Metrics

PD-MORL (w/i interpolator)

MORL-FB

Halfcheetah2d

UT(× 103 ) HV(× 108 ) ED

5.62 ± 0.05 1.08 ± 0.00 0.07 ± 0.01

7.69 ± 0.08 1.24 ± 0.00 -

Walker2d

UT(× 103 ) HV(× 107 ) ED

2.18 ± 0.02 5.45 ± 0.01 0.56 ± 0.00

2.23 ± 0.03 4.32 ± 0.02 -

Hopper3d

UT(× 103 ) HV(× 1011 ) ED

2.26 ± 0.01 1.08 ± 0.00 0.20 ± 0.01

2.36 ± 0.01 1.15 ± 0.00 -

Ant3d

UT(× 103 ) HV(× 1011 ) ED

3.59 ± 0.06 4.20 ± 0.04 0.60 ± 0.00

3.43 ± 0.22 4.18 ± 0.04 -

Humanoid2d

UT(× 102 ) HV(× 107 ) ED

2.93 ± 0.07 1.06 ± 0.01 0.33 ± 0.01

8.13 ± 0.01 1.75 ± 0.02 -

Humanoid5d

UT(×103 ) HV(×1015 ) ED

0.93 ± 0.04 6.64 ± 0.09 0.43 ± 0.01

1.11 ± 0.00 6.99 ± 0.06 -

E XPERIMENTS ON z D IMENSION

We investigate the impact of the z dimension in the Hopper3d environment. As shown in Table 15, the performance metric increases with the z dimension. However, when the z dimension reaches 300, performance declines, likely due to insufficient training steps for the larger network. Table 15: Empirical study on z dimension on Hopper3d. z dimension

D.2.3

Metrics

MORL-FB

UT(× 10 ) HV(× 1011 )

2.21 ± 0.01 1.03 ± 0.05

100

UT(× 103 ) HV(× 1011 )

2.25 ± 0.00 1.13 ± 0.01

150

UT(× 103 ) HV(× 1011 )

2.36 ± 0.01 1.15 ± 0.00

300

UT(× 103 ) HV(× 1011 )

2.01 ± 0.01 0.97 ± 0.02

E XPERIMENTS ON AUXILIARY L OSS

While Touati et al. (2023) also includes a similar auxiliary loss, there is one salient difference between theirs and our Q loss Equation (6): As the original FB is designed for RFRL, it does not have the reward signal available at training and hence needs to construct a pseudo reward as 27

Published as a conference paper at ICLR 2026

B ⊤ E[BB ⊤ ]−1 z (cf. Equation (9) in (Touati et al., 2023)). On the other hand, as MORL-FB addresses MORL and can observe vector reward signals at training, we propose to use the actual scalarized reward λ⊤ r in the Q loss. While this algorithmic difference appears seemingly subtle, this design makes a huge difference in the performance. Below we show an ablation study that compares MORL-FB with our Q loss (denoted as “Original”) and MORL-FB with the auxiliary loss using pseudo reward in FB (denoted as “Pseudo Q Loss”). The results are summarized in Table 16 and Fig. 12. This shows that the auxiliary loss of FB cannot be directly applied and needs to be adapted properly in the context of MORL. Note that we use the term “auxiliary” since the original FB is directly built on the measure loss and hence the Q loss is auxiliary for learning FB representation, rather than being unimportant for MORL. Table 16: Ablation study of MORL-FB on preference-guided exploration. Environments Ant3d

Metrics

MORL-FB (w/o PG-Explore)

MORL-FB (Pseudo Q Loss)

MORL-FB (Ours)

UT(× 103 ) HV(× 1011 )

1.45 ± 0.02 1.25 ± 0.02

1.27 ± 0.16 1.91 ± 0.02

3.93 ± 0.04 3.85 ± 0.01

Figure 12: Evaluation of MORL-FB and its ablated versions across different environments. The results highlight the importance of preference-guided sampling and the auxiliary Q-loss for MORL performance.

D.2.4

E XPERIMENTS ON M EASURE L OSS

Q-loss is surely important in our method as it guides our representation to learn the reward function used in the environment. On the other hand, both the preference-guided exploration and the measure loss are also essential to the success of MORL-FB based on Figure 12 and an additional ablation study on the measure loss shown below. Table 17: Ablation study of MORL-FB on measure loss and Q loss. Environments Ant3d

Metrics

MORL-FB (w/o measure loss)

MORL-FB (w/o Q loss)

MORL-FB (Ours)

UT(× 103 ) HV(× 1011 )

1.37 ± 0.28 2.74 ± 0.02

-1.53 ± 0.00 0.00 ± 0.00

3.93 ± 0.04 3.85 ± 0.01

Published as a conference paper at ICLR 2026

D.2.5

E XPERIMENTS ON Q- LOSS C OEFFICIENT

In our implementation, the measure loss LM and the Q loss LQ are directly added together without an additional weighting term. We chose this default configuration because it already yields strong and stable performance across all evaluated tasks, and we did not observe a need for further hyperparameter tuning during development. Moreover, we have conducted an additional sensitivity analysis to better understand how the balance between these two losses affects performance. Specifically, we varied the Q-loss coefficient αQ (with αQ = 1 as the default setting) and reported the corresponding results in Ant3d. Table 18 shows that MORL-FB’s performance is largely insensitive to the choice of αQ , indicating that directly adding LM and LQ is a reasonable and robust design choice. Table 18: Performance of MORL-FB with different Q-loss coefficients. Environments

Ant3d

D.3

MORL-FB

Metrics UT(×10 ) HV(×1011 )

αQ = 0.25

αQ = 0.5

αQ = 1 (Default)

αQ = 2

3.76 ± 0.41 4.03 ± 0.22

3.79 ± 0.32 4.13 ± 0.19

3.77 ± 0.11 3.95 ± 0.13

3.95 ± 0.13 4.23 ± 0.04

P ERFORMANCE C OMPARISON OF MORL-FB U NDER D IFFERENT Q- LOSS C OEFFICIENTS

We evaluate MORL-FB and several baselines on two classic discrete control tasks: Deep Sea Treasure (DST) and Fruit Tree Navigation (FTN). DST features the fundamental trade-off between two conflicting objectives. FTN, on the other hand, involves a tree search process with 6-dimensional terminal rewards, as previously detailed in the evaluation environments section. The detailed configurations for these experiments are provided in Section C. Figure 13 shows the performance of all the methods in UT, HV, and ED for discrete control tasks. Regarding ED, for each baseline algorithm ALG, we report ED(ALG, MORL-FB) to show the pairwise comparison. We can observate that MORL-FB consistently achieves competitive or superior performance across all three metrics on the discrete control tasks Deep Sea Treasure and Fruit Tree Navigation.

Figure 13: Competitive Results of MORL-FB on Discrete Control Tasks. We evaluate MORLFB and several benchmark MORL algorithms on classic discrete control tasks in MO-Gymnasium. Performance is measured using UT, HV and ED. MORL-FB demonstrates competitive results against specialized discrete MORL algorithms. 29

Published as a conference paper at ICLR 2026

D.4

P ERFORMANCE C OMPARISON OF MORL-FB U NDER S TATE -ACTION -BASED R EWARDS AND S TATE -BASED R EWARDS

In this paper, we primarily focus on state-based rewards. However, as the original FB supports both state-based (Touati et al., 2023) and state-action-based rewards (Touati & Ollivier, 2021), MORL-FB can also be extended to state-action-based rewards by replacing B(s) with B(s, a). Since some MO MuJoCo rewards depend on both states and actions, we compare MORL-FB and the extended one. As shown in the Figure 14, the state-action-based variant yields slight performance improvements on several tasks.

Figure 14: Evaluation of MORL-FB with different reward function representations. This figure presents the performance of MORL-FB when reward functions depend on states only (i.e., R(s)) versus state-action pairs (i.e., R(s, a)).

D.5

P ERFORMANCE C OMPARISON OF MORL-FB U NDER S TOCHASTIC R EWARDS

As vanilla FB naturally handles stochastic rewards, MORL-FB inherits this capability. To further demonstrate this, we evaluated MORL-FB under stochastic rewards by adding zero-mean Gaussian noise N (0, σ 2 ), similar to prior work (Romoff et al., 2018; Hu et al., 2022). The result is shown in Figure 15.

Figure 15: Evaluation of MORL-FB under stochastic reward. This figure assesses the performance of MORL-FB in environments featuring stochastic reward functions. 30

Published as a conference paper at ICLR 2026

D.6

P ERFORMANCE C OMPARISON OF MORL-FB W ITH N ONLINEAR S CALARIZATION

While we focus on linear scalarization in this paper, MORL-FB can be readily extended to nonlinear scalarization schemes by replacing r⊤ λ with fλ (r) when sampling z for preference-guided exploration and when computing Q loss, where fλ (r) is the general scalarization function. This is feasible since the original FB is designed to handle any scalar reward function, and MORL-FB inherits this property from FB and can also handle nonlinear scalarization. To demonstrate this generalizability, we further evaluate MORL-FB onHalfcheetah2d P by training under smooth Tchebycheff scalarizaλi (r−rref ) m tion as fλ (r) = µ log , where λi = is the i-th entry of preference vector, i=1 exp µ µ is the smoothing parameter and set to 0.1, and rref is set to [2.0, 0.0] across training (Lin et al., 2024; Qiu et al., 2024). We see that MORL-FB still achieves comparably strong performance in HV and UT under nonlinear scalarization. Table 19: Performance of MORL-FB with Smooth Tchebycheff scalarization.

D.7

Environments

Metrics

MORL-FB (linear scalarization)

MORL-FB (Smooth Tchebycheff scalarization)

HalfCheetah2d

UT(× 103 ) HV(× 108 )

7.69 ± 0.08 1.24 ± 0.00

6.33 ± 0.02 1.00 ± 0.01

S AMPLE E FFICIENCY OF MORL-FB

We demonstrate the sample efficiency of MORL-FB by evaluating its performance at an intermediate stage of 1.5M training steps. Notably, as shown in Figure 16, MORL-FB achieves superior HV and UT scores across most tasks compared to baseline methods trained for a full 3M steps. This indicates that MORL-FB can attain high performance with significantly fewer environment interactions. Further evidence, presented in Figure 17 and Figure 18, corroborates that MORL-FB reaches proficient performance levels with reduced training data. D.8

C ROSS -O BJECTIVE T RANSFER C APABILITY OF MORL-FB

To investigate how well MORL-FB handles transfer across different numbers of objectives, we conducted an empirical study on Hopper across different objective dimensions. We analyze the following cases: • Hopper2d: Moving forward speed on the x-axis, control cost of the action • Hopper3d: Moving forward speed on the x-axis, jumping height on the z-axis, control cost of the action • Hopper4d: Moving forward speed on the x-axis, jumping height on the z-axis, jumping up speed on the z-axis, control cost of the action Table 20 summarizes the quantitative results presented visually in Figure 7. MORL-FB consistently outperforms FB across all configurations in terms of utility and hypervolume.

D.9

RFRL AS A SOURCE OF AUXILIARY TASKS

During training, the z vectors computed for each preference λ are diverse, covering both CCS and non-CCS policies. Learning from non-CCS policies serves as auxiliary tasks. From Figure 19, we find that the return vectors achieved by those z-induced polices at the 1.5 million training step span both non-CCS and CCS regions.

Published as a conference paper at ICLR 2026

Figure 16: Performance of MORL-FB on continuous control tasks. We evaluate MORL-FB (1.5M training steps) against several benchmark MORL algorithms (3M training steps) on diverse continuous control tasks from MO-Gymnasium. Utilizing key metrics, these results demonstrate that MORL-FB outperforms baselines, achieving superior HV and UT across most tasks despite significantly fewer training steps.

Figure 17: Learning curves for MORL-FB and benchmark algorithms on Ant3d. This figure presents the learning curves in terms of Hypervolume (HV) for MORL-FB and several benchmark MORL algorithms evaluated on Ant3d.

D.10

Figure 18: Learning curves for MORL-FB and benchmark algorithms on Ant3d. This figure presents the learning curves in terms of Utility (UT) for MORL-FB and several benchmark MORL algorithms evaluated on Ant3d.

C OMPARISON OF PARETO F RONTS

To evaluate the sample efficiency of MORL-FB, we conduct experiments on 2-objective MuJoCo tasks with a whole range of 21 preference vectors ([0.0, 1.0], [0.05, 0.95], [0.1, 0.9], ···, [1.0, 0.0]). As a comparison baseline, we consider SORL which trains a separate policy for each individual preference. Each single-object SAC (SOSAC) model requires 3M steps, resulting in a total training budget of 63M steps for all 21 preferences. By contrast, MORL-FB only uses 3M steps in total to learn policies that generalize across the entire preference set. Figure 20 shows the return vectors attained by MORL-FB and the collection of 21 SOSAC models on the Walker2d task. MORL-FB 32

Published as a conference paper at ICLR 2026

Table 20: Zero-shot cross-objective transfer from Hopper2d to Hopper3d and Hopper4d using vanilla FB and MORL-FB: This figure presents the results demonstrating effective transfer by MORL-FB, supporting the efficacy of its proposed enhancements. Environments

Metrics

MORL-FB

2D to 3D

UT(× 103 ) HV(× 1010 ) ED

0.02 ± 0.00 0.40 ± 0.00 0.07 ± 0.03

1.77 ± 0.00 7.81 ± 0.06 -

2D to 4D

UT(× 103 ) HV(× 1013 ) ED

0.06 ± 0.00 0.05 ± 0.00 0.05 ± 0.01

1.59 ± 0.00 7.78 ± 0.10 -

Figure 19: Return vectors (Moving Speed vs. Energy Cost) achieved at the initial, intermediate, and final training stages in Walker2d with preference [0.7, 0.3]: The scatter plot, particularly at 1.5M steps, highlights the diverse policies beyond the CCS policy, supporting that RFRL serves as auxiliary tasks beneficial for MORL.

achieves comparable or even superior return vectors with only 1/21 of the samples, demonstrating its strong sample efficiency and generalization ability across diverse preference vectors.

(a) Humanoid2d

(b) Walker2d

Figure 20: Return vectors (Moving Speed vs. Energy Cost) achieved under 21 different preference vectors [0, 1], [0.05, 0.95], . . . , [0.95, 0.05], [1, 0] across different methods. Each scatter cloud corresponds to the learned policies under a specific preference, illustrating how MORL methods adapt to diverse trade-offs between objectives. 33

Published as a conference paper at ICLR 2026

D.11

ROBUSTNESS UNDER W ORST-C ASE P REFERENCES

Beyond average performance metrics, it is crucial to assess the robustness of multi-objective reinforcement learning (MORL) algorithms under unfavorable preference settings. In practice, policies deployed in dynamic environments may encounter user preferences that significantly diverge from those seen during training. To capture this aspect, we evaluate each method under worst-case preferences using the Conditional Value-at-Risk (CVaR). Following the risk-sensitive reinforcement learning formulation of Tamar et al. (2015), we compute [email protected] as the mean of the lowest 10% scalarized returns. Specifically, for each algorithm we uniformly sample 500 preference vectors, apply linear scalarization to obtain utility values, sort the results, and average the worst 10%. This quantifies the expected return conditioned on being in the lowest α-quantile: ϕ(θ) = Eθ [R | R ≤ να (θ)],

(18)

Table 21 reports [email protected] across HalfCheetah2d and Walker2d. MORL-FB achieves the highest CVaR in both environments, significantly outperforming prior baselines. These results indicate that MORL-FB not only excels on mean metrics but also demonstrates superior robustness against adverse preferences, avoiding catastrophic failures more effectively than existing methods. Table 21: [email protected] performance across HalfCheetah2d and Walker2d. Higher is better. Algorithm PD-MORL Q-Pensieve CAPQL PGMORL MORL/D PCN SFOLS GPI-LS GPI-PD MORL-FB (Ours)

D.12

HalfCheetah2d

Walker2d

2425.91 6275.89 4874.64 2630.02 407.66 -0.11 1390.73 3884.67 3999.36 7123.87

284.89 1516.20 1297.40 1113.49 40.47 98.28 402.19 1546.96 276.74 2304.02

C OMPUTATIONAL C OST A NALYSIS

To provide a fair and comprehensive comparison of computational efficiency, we evaluated all methods under the same hardware environment. Each algorithm was trained for 100K environment steps on a workstation equipped with a single NVIDIA RTX 4090 GPU, an Intel Core i7-13700K CPU, and 64 GB of system memory. Table 22 reports the total wall-clock time required by each method, ensuring a fair comparison of time and resource usage across methods. Compared to strong baselines such as PD-MORL, Q-Pensieve, and GPI-LS, FB-MORL maintains a reasonable training time. While some methods require higher computational costs, our approach strikes a good balance between performance and efficiency. D.13

D ETAILED E XPERIMENTAL R ESULTS OF S ECTION 4

In this part, we will provide the table we use on plotting the bar chart on Section 4. The results in Tables 23 to 26 correspond to the visualizations shown in Figure 2, Figure 4, Figure 12 and Figure 13, respectively.

Published as a conference paper at ICLR 2026

Table 22: Wall-clock time comparison (100K steps). Algorithm PD-MORL CAPQL MORL/D SFOLS Q-Pensieve PGMORL PCN GPI-LS GPI-PD MORL-FB (Ours)

Wall-Clock Time (seconds) 1166 3369 793 550 12960 4237 4445 3611 5237 1874

Table 23: Performance of MORL-FB and benchmark algorithms in discrete MORL environments. Environments

Metrics

PD-MORL (w/i interpolator)

Envelope

PCN

MORL/D

SFOLS

MORL-FB (Ours)

Optimal

Deep Sea Treasure

UT HV(×102 ) ED

0.52 ± 0.00 9.33 ± 0.00 0.69 ± 0.00

6.53 ± 0.00 9.91 ± 0.00 0.77 ± 0.00

6.12 ± 0.95 8.71 ± 1.49 0.81 ± 0.01

3.88 ± 0.26 5.63 ± 0.41 0.15 ± 0.10

5.65 ± 0.95 8.46 ± 0.01 0.25 ± 0.00

6.43 ± 0.10 9.92 ± 0.00 -

6.89 9.92 -

Fruit Tree Navigation

UT HV(×104 ) ED

5.03 ± 0.00 1.25 ± 0.00 0.76 ± 0.01

5.07 ± 0.00 1.16 ± 0.00 0.95 ± 0.00

5.00 ± 0.02 1.01 ± 0.06 0.72 ± 0.03

4.19 ± 0.03 0.18 ± 0.05 0.01 ± 0.01

4.77 ± 0.01 0.87 ± 0.01 0.08 ± 0.01

5.01 ± 0.01 1.16 ± 0.01 -

5.08 1.25 -

Table 24: Performance of MORL-FB, PD-MORL, and Q-Pensieve under constrained preference training. Environments

Metrics

PD-MORL

Q-Pensieve

MORL-FB

Hopper3d

UT(× 10 ) HV(× 1011 ) ED

1.05 ± 0.02 0.61 ± 0.01 0.01 ± 0.00

1.72 ± 0.01 0.88 ± 0.01 0.11 ± 0.00

2.26 ± 0.01 1.16 ± 0.01 -

Ant3d

UT(× 103 ) HV(× 1011 ) ED

1.69 ± 0.04 2.18 ± 0.03 0.22 ± 0.02

0.49 ± 0.02 0.51 ± 0.00 0.26 ± 0.04

3.11 ± 0.24 3.17 ± 0.05 -

Humanoid2d

UT(× 102 ) HV(× 107 ) ED

-0.04 ± 0.00 0.06 ± 0.00 0.00 ± 0.00

4.51 ± 0.38 1.51 ± 0.04 0.40 ± 0.05

8.19 ± 0.03 1.83 ± 0.01 -

Table 25: Performance comparison of MORL-FB and its ablated versions across environments. This table evaluates MORL-FB against variants lacking preference-guided exploration or the Q-loss component, showing their performance across different environments.

Environments

Metrics

MORL-FB (w/o preference-guided exploration)

MORL-FB (w/o q loss)

MORL-FB

Hopper3d

UT(× 103 ) HV(× 1011 )

2.03 ± 0.42 0.91 ± 0.02

0.33 ± 0.07 0.44 ± 0.02

2.36 ± 0.00 1.15 ± 0.01

Ant3d

UT(× 103 ) HV(× 1011 )

1.23 ± 0.03 1.47 ± 0.03

-1.01 ± 0.01 0.06 ± 0.00

3.43 ± 0.22 4.18 ± 0.04

Humanoid2d

UT(× 102 ) HV(× 107 )

2.38 ± 0.30 1.78 ± 0.00

-1.43 ± 0.14 0.96 ± 0.00

8.13 ± 0.01 1.75 ± 0.02

1.29 ± 0.02 0.92 ± 0.00 0.05 ± 0.00

1.29 ± 0.11 2.42 ± 0.05 0.11 ± 0.05

-0.05 ± 0.00 0.06 ± 0.00 0.07 ± 0.03

0.38 ± 0.01 3.91 ± 0.04 0.03 ± 0.00

UT(×103 ) HV(×1011 ) ED

UT(×102 ) HV(×107 ) ED

UT(×103 ) HV(×1016 ) ED

Ant3d

Humanoid2d

Humanoid5d

1.70 ± 0.01 4.02 ± 0.01 0.27 ± 0.03

UT(×103 ) HV(×107 ) ED

Walker2d

UT(×103 ) HV(×1011 ) ED

3.17 ± 0.04 0.95 ± 0.00 0.06 ± 0.01

UT(×103 ) HV(×108 ) ED

Halfcheetah2d

Hopper3d

PD-MORL (w/o interpolator)

Metrics

Environments

36 0.12 ± 0.00 1.80 ± 0.02 0.00 ± 0.00

4.51 ± 0.03 1.51 ± 0.04 0.40 ± 0.04

1.81 ± 0.08 2.61 ± 0.02 0.14 ± 0.04

1.82 ± 0.02 1.25 ± 0.01 0.35 ± 0.01

1.66 ± 0.08 4.54 ± 0.02 0.36 ± 0.01

6.85 ± 0.01 1.13 ± 0.00 0.08 ± 0.01

Q-Pensieve

0.08 ± 0.00 0.00 ± 0.00 0.04 ± 0.00

-0.04 ± 0.00 0.01 ± 0.01 0.08 ± 0.00

1.01 ± 0.00 0.45 ± 0.00 0.08 ± 0.05

1.25 ± 0.01 0.57 ± 0.01 0.00 ± 0.00

1.49 ± 0.01 0.03 ± 0.01 0.00 ± 0.00

5.39 ± 0.01 0.89 ± 0.00 0.06 ± 0.01

CAPQL

0.54 ± 0.00 0.59 ± 0.00 0.10 ± 0.00

1.55 ± 0.04 1.01 ± 0.02 0.07 ± 0.01

0.80 ± 0.00 0.21 ± 0.03 0.08 ± 0.05

0.78 ± 0.00 0.01 ± 0.00 0.03 ± 0.01

1.15 ± 0.02 3.52 ± 0.09 0.18 ± 0.01

2.72 ± 0.03 0.73 ± 0.01 0.03 ± 0.01

PGMORL

0.58 ± 0.00 0.31 ± 0.00 0.08 ± 0.00

2.71 ± 0.02 0.62 ± 0.00 0.18 ± 0.01

0.04 ± 0.00 0.01 ± 0.00 0.05 ± 0.03

0.11 ± 0.00 0.01 ± 0.00 0.00 ± 0.00

0.05 ± 0.00 0.35 ± 0.00 0.03 ± 0.01

0.41 ± 0.01 0.08 ± 0.00 0.05 ± 0.01

MORL/D

-0.10 ± 0.01 0.00 ± 0.00 0.00 ± 0.00

-3.79 ± 0.72 0.25 ± 0.01 0.00 ± 0.00

0.05 ± 0.01 0.07 ± 0.01 0.06 ± 0.03

0.01 ± 0.01 0.00 ± 0.00 0.00 ± 0.00

0.10 ± 0.00 0.17 ± 0.00 0.03 ± 0.01

0.00 ± 0.00 0.00 ± 0.00 0.05 ± 0.01

PCN

0.54 ± 0.00 2.15 ± 0.08 0.05 ± 0.00

0.45 ± 0.03 0.75 ± 0.05 0.04 ± 0.00

0.28 ± 0.01 0.12 ± 0.00 0.06 ± 0.04

0.03 ± 0.01 0.03 ± 0.00 0.00 ± 0.00

0.41 ± 0.01 1.22 ± 0.00 0.15 ± 0.02

1.39 ± 0.03 0.59 ± 0.03 0.01 ± 0.01

SFOLS

0.72 ± 0.02 3.48 ± 0.03 0.09 ± 0.00

1.43 ± 0.12 0.77 ± 0.21 0.15 ± 0.01

1.45 ± 0.10 0.73 ± 0.15 0.17 ± 0.02

0.57 ± 0.23 0.29 ± 0.00 0.02 ± 0.10

1.92 ± 0.18 3.35 ± 0.33 0.32 ± 0.01

4.15 ± 0.84 0.67 ± 0.12 0.05 ± 0.02

GPI-LS

0.30 ± 0.00 0.20 ± 0.01 0.03 ± 0.00

1.35 ± 0.00 0.42 ± 0.03 0.14 ± 0.00

0.66 ± 0.00 0.35 ± 0.01 0.08 ± 0.05

1.32 ± 0.01 0.51 ± 0.00 0.10 ± 0.05

0.54 ± 0.00 1.13 ± 0.00 0.15 ± 0.02

4.06 ± 0.03 0.68 ± 0.00 0.06 ± 0.01

GPI-PD

0.55 ± 0.03 4.69 ± 0.20 0.03 ± 0.00

2.44 ± 0.00 1.19 ± 0.05 0.06 ± 0.00

0.01 ± 0.01 0.08 ± 0.00 0.03 ± 0.02

0.01 ± 0.00 0.00 ± 0.00 0.00 ± 0.00

0.29 ± 0.01 1.97 ± 0.12 0.00 ± 0.00

1.26 ± 0.00 0.26 ± 0.00 0.04 ± 0.02

Table 26: Comparative performance of MORL-FB and various benchmark algorithms across continuous control tasks in MuJoCo.

1.11 ± 0.00 6.99 ± 0.06 -

8.13 ± 0.01 1.75 ± 0.02 -

3.43 ± 0.22 4.18 ± 0.04 -

2.36 ± 0.01 1.15 ± 0.00 -

1.87 ± 0.05 4.56 ± 0.00 -

7.69 ± 0.08 1.24 ± 0.00 -

MORL-FB (Ours)

Published as a conference paper at ICLR 2026

We also evaluated the probability of improvement (POI) between MORL-FB and the benchmark algorithms suggested by Agarwal et al. (2021) in Figure 21. POI quantifies the likelihood that one algorithm will outperform another. The results demonstrate that MORL-FB consistently achieves superior performance compared to baselines. Additionally, we assessed the performance of training with constraint preferences using the method from Agarwal et al. (2021), presented in Figures 22 and 23.

$OJRULWKP; *3,3' *3,/6 )% 6)2/6 3&1 025/' 3*025/ &$34/ 43HQVLHYH 3'025/

3;<

$OJRULWKP< 025/)%2XUV 025/)%2XUV 025/)%2XUV 025/)%2XUV 025/)%2XUV 025/)%2XUV 025/)%2XUV 025/)%2XUV 025/)%2XUV 025/)%2XUV

Figure 21: Probability of Improvement (POI) of MORL-FB against benchmark algorithms. This figure illustrates the POI of MORL-FB relative to various benchmark MORL algorithms.

MORL-FB (Ours) Q-Pensieve PD-MORL

Median

IQM

0.4 0.6 0.8 1.0 0.4

0.6

Mean 0.8

0.4 0.6 0.8 1.0

Normalized Weighted Reward

Figure 22: Medium, IQM, and Mean performance of MORL-FB and benchmarks trained with small preference sets. This figure displays the Median, Interquartile Mean (IQM), and Mean performance for MORL-FB and other benchmark algorithms, trained with only a small set of preference vectors.

Algorithm X

Algorithm Y

Q-Pensieve

MORL-FB (Ours)

PD-MORL

MORL-FB (Ours)

0.5 0.6 0.7 0.8 0.9 1.0 P(X < Y)

Figure 23: Probability of Improvement (POI) of MORL-FB under constrained preference training. This figure illustrates the POI of MORL-FB relative to other benchmark algorithms, all trained with specific constraint sets of preference vectors.

Record · ID 138935 · SHA-256 069e20542febda1f
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.