To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization

To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization Taehyung Kim1 , Jongeun Choi1∗ 1

arXiv:2608.18770v1 [cs.LG] 19 Aug 2026

Yonsei University, Seoul, South Korea {kth8606, jongeunchoi}@yonsei.ac.kr

Abstract Learning a reward model from human feedback and optimizing a policy against it is one approach to aligning AI systems with individual users. From a fairness perspective, existing work improves such alignment by developing data-efficient and accurate reward models that capture minority preferences despite scarce data. We push this line of inquiry one step further and argue that data-efficient and accurate per-user reward models are not sufficient: users whose reward models are difficult to optimize at the policy level can become a new underserved group. We start from the observation that one user’s reward model can be easy to optimize from the initial policy while another’s is not. We argue that, given a sufficiently diverse user population, a curriculum naturally emerges between easy- and hard-to-optimize reward models. Building on this insight, we propose CurriPO, which grows a tree-structured curriculum to accommodate diverse user-specific objectives, covering the population in a single traversal. Specifically, CurriPO automatically constructs a curriculum over diverse user reward models, allowing it to branch from the existing curriculum and reuse reward models previously incorporated into the curriculum. To the best of our knowledge, this is the first work to explicitly exploit multi-user structure to address optimization in AI alignment. Extensive experiments on personalized continuous control in a simulated environment show that CurriPO achieves 1.2–2.1× the population satisfaction of the strongest baseline while substantially reducing training time. Additional analysis attributes much of this improvement to the users left underserved by conventional optimization.

1

Introduction

A widely adopted approach to aligning AI systems with human values is to elicit preference feedback (e.g., pairwise comparisons), fit a reward model, and optimize a policy against it with reinforcement learning (RL) (Christiano et al. 2017). This approach has since been applied and extended across a range of domains, achieving substantial gains (Ouyang et al. 2022; Cabi et al. 2020). As these systems have become widely deployed, there has been a growing demand for alignment at the level of individual users (Guan et al. 2025). Prior work has developed various reward model-based preference optimization methods to support personalized alignment. These include reward models that capture heterogeneous preferences by modeling distributions over latent utilities (Siththaranjan, Laidlaw, and Hadfield-Menell 2024), ∗

Corresponding Author.

Figure 1: CurriPO aims to construct an effective curriculum across users, enabling better alignment for those who would otherwise remain underserved by conventional methods.

shared preference-space methods with few-shot onboarding (Poddar et al. 2024), and methods that use limited reward models based on mixtures of preference groups or combinations of shared prototypes (Shenfeld et al. 2025; Kim et al. 2026). From the perspective of protecting minority preferences, these approaches leverage the structural advantage of multi-user settings to improve personalized reward model accuracy and data efficiency, helping preserve preferences that might otherwise be overlooked. Although these approaches are effective, we go one step further by highlighting that even when accurate personalized reward models are available for all users, including minorities, subsequent RL optimization can itself create new forms of underrepresentation. When modern methods drive a reward model to high held-out accuracy, it remains unclear whether a policy can be effectively optimized against it. Ng, Harada, and Russell (1999) characterize a family of reward transformations that preserve the optimal policy while substantially altering the difficulty of learning it. Extending this line of analysis to learned reward functions, Skalse et al. (2023) show that accuracy on held-out feedback alone cannot distinguish reward models that are easy to optimize from those that are not. Similarly, Razin et al. (2025b) show that a reward model can correctly rank every pair while still inducing a flat optimization landscape that makes policy optimization difficult. This optimization issue can be particularly pronounced in per-

user alignment, where a separate policy must be optimized for each user under a user-specific reward landscape. To address the reward optimization problem across a diverse user population, we begin with the observation that reward models learned from different users vary substantially in how easily a policy can be optimized against them. Our key intuition is that users whose reward models are easier to optimize can serve as stepping stones toward users with progressively harder reward models, enabling a curriculum-like optimization procedure. We instantiate this idea as CurriPO, a tree-structured traversal of policy space that treats the diverse reward model population as its curriculum. Starting from an initial policy π0 , CurriPO repeatedly optimizes a selected user’s reward model, stores the resulting policy checkpoint, and selects the next reward model to optimize. All checkpoints generated during traversal are retained in a growing set, which we call the wake. At each stage, CurriPO probes the entire wake by running short rollouts and evaluating them under users’ reward models, thereby identifying the next reward model–checkpoint pair to optimize. Once the curriculum is complete, each user is ultimately served the checkpoint in the wake that is most preferred by that user’s own reward model. CurriPO incorporates two key mechanisms. First, branching allows the next optimization to start from any checkpoint in the wake, rather than only from the most recent one. This prevents optimization from degenerating into a single chain and enables broader coverage of heterogeneous users. Second, reusing continues to probe previously optimized reward models, allowing them to re-enter the curriculum when probing identifies a later checkpoint as a more promising initialization. Thus, a reward model previously difficult to optimize can be reused from a better initialization. Through these mechanisms, CurriPO turns the multiplicity of users—often viewed as an obstacle to per-user alignment—into a sequence of learning stages. By explicitly leveraging this multi-user structure, CurriPO reduces the risk that users associated with harderto-optimize reward models are systematically underserved during the alignment process. We consider a setting in which CurriPO optimizes a policy using pairwise preferences collected from users. In our evaluation, we focus on personalizing relatively simple robots that are likely to be deployed around humans in the near term, instead of highly complex systems such as humanoid robots. We study personalization on continuous-control tasks in MuJoCo (Todorov, Erez, and Tassa 2012). We show that CurriPO substantially improves population satisfaction while reducing training time relative to both conventional reward model optimization and direct preference optimization baselines. Ablation studies further validate the contributions of its key components. Additional analysis demonstrates that CurriPO provides particularly large gains for users who are poorly served by existing optimization methods. Our key contributions are summarized as follows: • We propose CurriPO, a novel method for improving optimization in per-user alignment in multi-user settings, which treats the diverse user population as a curriculum to identify favorable optimization trajectories for users whose reward models are difficult to optimize.

• CurriPO admits branching and reusing when ordering the curriculum, yielding a tree-structured curriculum that serves all users in a single traversal and thus achieves more efficient and accurate user alignment. • Extensive experiments in continuous control show that CurriPO improves population satisfaction over conventional optimization baselines, with the largest gains observed for users whose reward models are difficult to optimize against using conventional methods.

2

Related Work

Preference-Based Policy Learning and Alignment. Preference-based RL replaces manually specified rewards with human preferences (Christiano et al. 2017). This paradigm has been steadily refined across diverse domains of continuous control (Lee, Smith, and Abbeel 2021; Park et al. 2022). As preference feedback increasingly comes from heterogeneous populations, recent work has sought to account for individual differences. Poddar et al. (2024) personalize robot control policies by inferring a per-user latent preference within a shared latent space, and, in a similar spirit, Kalra et al. (2026) decompose the preferences of multiple users into a common preference basis. Kim et al. (2026) group users with similar preferences and learn a reward model for each group, aiming to obtain robot policies that serve the population well. While these methods assume a multi-user setting and devote their efforts to constructing better reward models, whether multiple users can also be leveraged to improve the policy optimization stage itself has rarely been explored. To the best of our knowledge, CurriPO is the first to address this gap, using policies obtained by optimizing one user’s reward as stepping stones that make other users’ reward models easier to optimize. A parallel line of work bypasses explicit reward modeling and optimizes policies directly from preferences. IPL (Hejna and Sadigh 2023) learns a preference-consistent Q-function via an implicit reward, while CPL (Hejna et al. 2024) directly optimizes the policy with a contrastive objective. However, removing the explicit reward model does not eliminate optimization difficulties: direct preference optimization remains constrained by the coverage of the preference data and by the policy distribution from which optimization proceeds (Razin et al. 2025a; Guo et al. 2025). Discarding the reward model therefore does not resolve the problem CurriPO targets. Curriculum Learning. Curriculum learning organizes training examples or tasks to match the learner’s evolving capabilities (Bengio et al. 2009; Kumar, Packer, and Koller 2010), with automatic variants adapting the curriculum using signals such as learning progress (Graves et al. 2017; Matiisen et al. 2019). In RL, self-paced RL adapts task distributions to the agent’s competence (Klink et al. 2020), while PLR (Jiang, Grefenstette, and Rocktäschel 2021) prioritizes levels with high learning potential. Unsupervised Environment Design further generates or curates tasks near the learning frontier; examples include PAIRED (Dennis et al. 2020), replay-guided environment design (Jiang et al. 2021), and ACCEL (ParkerHolder et al. 2022). Recent methods improve task generation and selection through learned task representations (Azad et al.

2023) or cross-task learnability (Cho et al. 2026). Related open-ended methods such as POET transfer policies across challenges, using intermediate solutions as stepping stones toward otherwise difficult behaviors (Wang et al. 2019, 2020). CurriPO differs in two key respects. First, it requires no separate process for designing or generating curriculum tasks: the heterogeneous reward models already present in the user population form the task space, allowing population diversity to be directly exploited as a structural resource for optimization. Second, CurriPO is not a curriculum toward a single target. Through branching and reusing, it expands toward multiple user-specific objectives, effectively solving many user alignment problems within a shared traversal.

3 3.1

Method

Problem formulation

We aim to serve a diverse population with user-aligned policies, derived from π0 , using only pairwise user feedback collected over a behavior pool D. Let M = (S, A, P, ρ0 , H) be a reward-free episodic decision process, where a policy π(a | s) specifies the probability of choosing action a in state s, and executing the policy yields a trajectory τ = (s0 , a0 , . . . , sH−1 , aH−1 ) ∈ T . Let U = {1, . . . , N } be the population of users. We assume each user u carries a latent utility gu capturing how satisfied u actually is with a given behavior. From the feedback Fu that u provides on trajectory segments ξ = ((st , at ))L−1 t=0 , we fit a per-user reward model f : S × A → R, whose segment score u P f¯u (ξ) = L1 t fu (st , at ) is a learned proxy for gu . We call {fu }u∈U the reward model bank. Given the bank, we aim to provide each user u with a personalized P policy πu , improving overall population satisfaction N1 u Eξ∼πu [gu (ξ)] while better serving users who would otherwise remain underserved by conventional policy optimization.

3.2

Reward modeling from pairwise preferences

We instantiate the bank following the standard preferencebased pipeline (Christiano et al. 2017). Each element of Fu is a triple (ξ 0 , ξ 1 , y) with y ∈ {0, 1}, where y = 1 denotes ξ 1 ≻ ξ 0 and y = 0 denotes ξ 0 ≻ ξ 1 . To fit fu , most prior work (Christiano et al. 2017; Lee, Smith, and Abbeel 2021) defines a preference predictor following the Bradley–Terry model (Bradley and Terry  1952); we adopt the predictor per user,
expressed as Pfu ξ 1 ≻ ξ 0 = logistic f¯u (ξ 1 ) − f¯u (ξ 0 ) . Each fu is fit by minimizing the binary cross-entropy below, and repeating this for every u ∈ U yields the reward model bank: h   LCE (fu ) = − 0 1E (1 − y) log Pfu ξ 0 ≻ ξ 1 (ξ ,ξ ,y)∼Fu  1  i (1) 0 + y log Pfu ξ ≻ ξ .

3.3

CurriPO: Probe-Driven Reward Traversal

CurriPO exploits the reward model bank as an automatically discovered curriculum. Starting from π0 , CurriPO traverses the policy space by iteratively optimizing a selected user’s reward model, retaining the resulting policy checkpoint, and

selecting the reward model for the next stage. After k stages, these checkpoints form the wake Wk = {π0 , . . . , πk }. At stage k, CurriPO selects (wk , uk ), where πwk ∈ Wk is the checkpoint used to warm-start optimization of the selected reward model fuk . Probing the wake. CurriPO evaluates each checkpoint using short probe rollouts, scored by every user reward model. For each checkpoint–user pair, we estimate the competence 
 b ξ∼π Nu f¯u (ξ) , comp(π, u) = E (2) where Nu (x) = Pξ∼D [f¯u (ξ) ≤ x] is the empirical cumulative distribution function over the pool D, so competence is the pool percentile π reaches under u. Below we write compw (u) for the competence of checkpoint πw evaluated under user u’s reward model. Aggregating competence over the wake gives the coverage Ck (u) =

max

w : πw ∈Wk

compw (u),

(3)

so that Ck (u) measures how well user u is served by the best checkpoint currently in the wake. Selecting the next traversal step. Having probed the wake, CurriPO has an estimate of how well each user is currently served, summarized by the coverage Ck (u). It then uses this coverage information to decide where the traversal should expand next. CurriPO follows a curriculum principle of expanding progressively toward reward models that are sufficiently accessible from the current wake (Florensa et al. 2018). To this end, it first restricts attention to the admissible users, namely those some checkpoint in the wake already serves above a minimum threshold clo :  Uk = u : Ck (u) ≥ clo . (4) Among these, CurriPO targets the least-covered user and warm-starts from the checkpoint that currently serves that user best: uk = arg min Ck (u), u∈Uk

wk = arg

max

w : πw ∈Wk

compw (uk ),

(5) with ties broken toward the most recently added checkpoint. This design enables CurriPO to select the starting checkpoint afresh at every stage, without requiring every reward optimization stage to start from the immediately preceding checkpoint. As a result, the traversal can branch into a tree instead of being a single chain. This is a particularly effective design choice for our setting, where the goal is not to reach a single difficult objective, but to efficiently cover a diverse population of users with different hard-to-optimize objectives. Additionally, motivated by the observation that optimizing the same reward model from different initializations can induce substantially different optimization dynamics (Razin et al. 2025b), which we also empirically observe in Figure 4, we allow each reward model that has already been selected and used for optimization to be reused up to q times. CurriPO gives a previously selected reward model another opportunity for optimization whenever a newly added checkpoint in the wake achieves a better probe score than the checkpoint from

Figure 2: User satisfaction visualization across different optimization methods with N = 100 users under oracle feedback. User satisfaction landscapes are shown for HalfCheetah and Hopper under 1-D and 2-D preference settings, with target speed as the shared preference dimension and energy or height as the second dimension. Darker colors indicate higher user satisfaction. which that reward model was previously optimized. This mechanism is particularly well suited to our setting, where stepping-stone checkpoints remain relevant even after the curriculum is complete because they are retained as candidate policies for final user assignment. Reusing reward models can therefore improve the quality of these stepping stones while simultaneously inducing a finer-grained curriculum as the wake expands. Local stage optimization. Once (wk , uk ) is selected, CurriPO initializes the policy from πwk and performs a local optimization stage: X [ r̂uk − β DKL (π ∥ πwk ) ]. (6) πk+1 ≈ arg max Eτ ∼π π

t

Here, πwk is kept fixed as the reference policy throughout the stage. To make the shared regularization coefficient β comparable across users, we standardize each reward model over the pool D: fu (s, a) − µD u . (7) σuD The KL penalty preserves useful behaviors from the selected checkpoint while preventing each optimization stage from moving too far from its initialization. Re-anchoring the constraint at successive checkpoints allows CurriPO to progressively traverse policy space through local transitions. r̂u (s, a) =

Serving from the wake. The traversal terminates when minu Ck (u) ≥ cstop or when the interaction budget B is exhausted. The final output is the entire wake. Each user u is assigned the checkpoint preferred by that user’s own reward model,

πu = arg max comp(π, u). π∈W

4 4.1

(8)

Experiments

Experimental Settings

Environment. Our environment design follows the objective decomposition in MO-Gymnasium (Felten et al. 2023), which has been used to study trade-offs among multiple control objectives, and is conceptually aligned with Hwang et al. (2024), who model user preferences through combinations of such objectives. We evaluate the styles preferred by multiple scripted users across four environments: (i) HC-1D, with HalfCheetah as the agent, where user preferences are defined over its velocity; (ii) HC-2D, also using HalfCheetah, where preferences depend jointly on velocity and energy consumption; (iii) H-1D, with Hopper as the agent, where preferences are defined over its velocity; and (iv) H-2D, also using Hopper, where preferences depend jointly on velocity and vertical position (z-height). Datasets. We construct a custom dataset D covering a diverse range of locomotion styles, as existing public offline RL datasets do not provide such coverage. For each environment, we train 10 PPO (Schulman et al. 2017) policies with different random seeds for 600K steps using velocity-based reward functions with varying target values. We collect trajectories from nine checkpoints throughout training for each policy. The trajectories are then partitioned into non-overlapping 25-step segments and augmented with segments generated by a random policy. Finally, we construct a pairwise dataset by sampling segment pairs from the resulting pool.

Method

HC-1D RM 0.341±0.004 RM+RND 0.388±0.000 CPL 0.188±0.010 IPL 0.295±0.007 VPL 0.209±0.009 CurriPO 0.514±0.016

Oracle HC-2D H-1D 0.215±0.006 0.250±0.027 0.239±0.013 0.260±0.003 0.091±0.008 0.188±0.020 0.193±0.007 0.228±0.007 0.087±0.013 0.266±0.001 0.318±0.040 0.355±0.059

H-2D 0.175±0.001 0.157±0.001 0.076±0.008 0.125±0.008 0.099±0.005 0.328±0.006

HC-1D 0.311±0.012 0.320±0.003 0.194±0.003 0.304±0.001 0.192±0.021 0.386±0.034

B-Pref noise HC-2D H-1D 0.220±0.002 0.309±0.012 0.168±0.002 0.312±0.027 0.096±0.006 0.180±0.005 0.193±0.008 0.253±0.008 0.114±0.023 0.264±0.016 0.344±0.009 0.371±0.021

H-2D 0.163±0.011 0.156±0.009 0.085±0.005 0.128±0.009 0.098±0.006 0.340±0.038

Time (h) 2.6 3.4 5.6 5.7 3.5 1.1

Table 1: Satisfaction and training time for N = 12 users under oracle and B-Pref noisy feedback across three seeds. Method

HC-1D RM 0.352±0.000 RM+RND 0.344±0.005 CPL 0.173±0.015 IPL 0.299±0.008 VPL 0.208±0.020 CurriPO 0.512±0.030

Oracle HC-2D H-1D 0.227±0.001 0.309±0.002 0.221±0.000 0.318±0.008 0.089±0.011 0.191±0.017 0.195±0.005 0.240±0.006 0.107±0.031 0.275±0.002 0.319±0.019 0.415±0.051

H-2D 0.142±0.003 0.145±0.005 0.078±0.007 0.128±0.006 0.094±0.003 0.280±0.008

HC-1D 0.318±0.003 0.316±0.005 0.179±0.002 0.296±0.007 0.215±0.004 0.522±0.018

B-Pref noise HC-2D H-1D 0.183±0.001 0.244±0.007 0.171±0.004 0.267±0.003 0.093±0.001 0.194±0.006 0.191±0.004 0.261±0.018 0.125±0.030 0.275±0.006 0.253±0.026 0.350±0.018

H-2D 0.179±0.004 0.152±0.004 0.087±0.006 0.129±0.002 0.093±0.003 0.269±0.012

Time (h) 22 28 47 48 28 1.6

Table 2: Satisfaction and training time for N = 100 users under oracle and B-Pref noisy feedback across three seeds. User Sampling. Following common practice in prior work on personalized preference learning, we use scripted users to evaluate personalization across heterogeneous preferences (Poddar et al. 2024; Zollo et al. 2025; Bahlous-Boldi et al. 2024; Kim et al. 2026). In the 1-D settings (HC-1D and H-1D), users are sampled uniformly at random from a predefined speed range. In the 2-D settings (HC-2D and H-2D), users are sampled only from the feasible region. For example, in HC-2D, mismatched speed–energy combinations, such as high speed with low energy or low speed with high energy, are difficult to obtain under our data-collection procedure and are therefore excluded by construction. We conduct experiments with two population sizes, N = 12 and N = 100, for each environment. Figures in Appendix C.1 and Figure 2 visualize the preference ranges and user populations in the 1-D and 2-D settings for N = 12 and N = 100, respectively. To mitigate the limitations of scripted teachers, we further evaluate robustness to imperfect preference feedback by introducing noisy preference labels following the preference-noise setting of B-Pref (Lee et al. 2021), yielding reward model geometries that deviate from the oracle geometry (Liang et al. 2022; Park et al. 2022; Cheng et al. 2024). The preference-noise parameters are detailed in Appendix A.1. Evaluation Metrics. We evaluate each personalized policy using the user’s ground-truth utility, which is not directly observed during training. For user u, the per-step utility is defined as a band-shaped kernel over the behavioral feature vector dt (e.g., velocity and energy consumption) generated by the policy assigned to the user, 2! (dt − cu ) ⊘ ηu 2 , (9) gu (dt ) = exp − bu

we normalize deviations by user-specific tolerances to make heterogeneous preference dimensions comparable. The exponential kernel then maps this normalized distance to a bounded satisfaction score, smoothly capturing how utility decreases as behavior moves away from the user’s preferred region (Poddar et al. 2024). Our primary metric is population satisfaction, i.e., the utilitarian population satisfaction PN Wutil = N1 u=1 Gu (πu ), where πu is the policy served to user u, so that all users contribute equally. Each Gu is estimated from 10 fresh evaluation episodes. We also report the egalitarian population satisfaction minu Gu (πu ) and
1/N Q the Nash population satisfaction in the u Gu (πu ) Appendix C.2. Implementation Details. CurriPO is implemented in Python 3.11.15 with PyTorch 2.13.0 and MuJoCo 3.10.0 on Rocky Linux 9.4. All continuous-control training ran on Intel Xeon Gold 5320 CPUs, with each job allocated two CPU cores and 6 GB RAM. Reward model optimization methods share one PPO implementation: actor and critic MLPs of width 64, 8 parallel environments, 256-step rollouts, 10 epochs, Adam at 3×10−4 , γ=0.99, GAE λ=0.95 and clip 0.2. For CurriPO, we set clo =0.6, β=0.01, q=2, and cstop =0.9. Each traversal stage is allocated a PPO budget of 400K environment steps. We set a safety cap of B=9.6M environment steps, but this limit was rarely reached. For probing, we run six 1,000-step episodes from each checkpoint. π0 is randomly initialized and held fixed across all methods within a run. As assumed throughout the paper, each reward model is trained until it reaches at least 90% held-out preference accuracy against oracle labels. All baselines that rely on an explicit reward model use the same reward model bank as CurriPO.

where cu , ηu , and bu are the user’s preferred target, perdimension tolerance, and bandwidth, and ⊘ denotes elementwise division. The policy-level utility Gu (π) is the temporal average of gu (dt ) over fresh stochastic rollouts from π. Rather than using raw distance in the behavioral descriptor space,

4.2

Comparison with State-of-the-Art Methods

Baselines. We compare against two families of methods. Reward model optimization: RM (Christiano et al. 2017) optimizes each user’s reward model directly with PPO, i.e. the conventional per-user optimization pipeline; RM+RND adds

Figure 3: Comparison of learning curves with stronger PPO baselines in HC-1D with 12 users. Ground-truth goal satisfaction over three seeds is shown across different target preferences, comparing RM and RM+RND with increased environment steps.

Figure 4: Case studies of rescuing users from local optima. The x-axis denotes the wake number, which increases as the CurriPO tree progressively branches over training. The y-axis denotes the selected user’s ground-truth satisfaction. Circles indicate the checkpoint used to warm-start training for the selected user, with the preceding user’s target additionally provided for reference; v, e, and h denote target speed, target energy, and target height, respectively. Diamonds indicate the checkpoint obtained after training the selected user’s reward model starting from the corresponding warm-start policy.

a random network distillation (RND) bonus (Burda et al. 2019) to RM, to examine whether insufficient exploration accounts for its poor performance; and VPL (Poddar et al. 2024) infers a per-user latent preference and conditions a shared reward model on it. Direct preference optimization: CPL (Hejna et al. 2024) directly optimizes the policy using a contrastive preference objective, whereas IPL (Hejna and Sadigh 2023) learns a preference-consistent Q-function through an implicit reward induced by the inverse Bellman operator. Both methods learn policies from preferences without fitting an explicit reward model. Further implementation details for baselines are provided in Appendix A.2. Experiment Results. Table 1 reports population satisfaction for N =12. Under oracle feedback, CurriPO achieves the highest satisfaction in every environment, with particularly large margins in the more challenging 2-D settings, while no

single baseline consistently dominates. These gains persist under B-Pref noise: although noisy feedback affects absolute performance, CurriPO remains the best-performing method across all four environments, indicating that its advantage holds even under the altered reward model geometry induced by imperfect preference feedback. Table 2 shows that this advantage also persists as the population scales to N =100, with CurriPO again outperforming all baselines under both oracle and noisy feedback. CurriPO also tends to exhibit lower across-seed variability at N =100, consistent with more stable performance as the population grows. To understand where these gains arise, Figure 2 visualizes the satisfaction landscape and user distribution for the N =100 oracle setting (see Appendix C.1 for the satisfaction landscapes under the remaining population and feedback settings). For the baselines, satisfaction is concentrated near the low-velocity region and fades as the target moves outward; the additional exploration of RM+RND expands this region only slightly. In contrast, CurriPO maintains high satisfaction across a broad portion of the feasible range, showing that its gains arise primarily from users whose target bands lie in the regions that conventional optimization methods tend to underserve. Appendix C.2 further demonstrates that CurriPO achieves much strong gains under Nash and egalitarian population satisfaction. CurriPO also scales efficiently in terms of training cost: increasing the population from 12 to 100 raises its training time only from 1.1 h to 1.6 h, whereas the baseline training times increase from 2.6–5.7 h to 22–48 h. Across the eight environments, CurriPO significantly outperforms every baseline under a two-sided Wilcoxon signed-rank test (p=0.0078), with all comparisons remaining significant after Holm correction (padj =0.0391).

4.3

Analysis of Branching, Reusing and KL

We next examine the roles of CurriPO’s branching and reusing mechanisms. We compare CurriPO against two ablations: Chain, which removes both branching and reusing and traverses the curriculum along a single path, and No reuse, which retains branching but prevents a reward model from being selected more than once. As shown in Tables 3 and 4, Chain consistently performs substantially worse than CurriPO, while No reuse closes much of the gap but remains slightly worse in several settings. These results suggest that branching

Method Chain No reuse No KL CurriPO

HC-1D 0.389±0.037 0.514±0.016 0.244±0.139 0.514±0.016

Oracle HC-2D H-1D 0.249±0.030 0.324±0.013 0.318±0.040 0.352±0.190 0.253±0.132 0.362±0.138 0.318±0.040 0.355±0.059

H-2D 0.157±0.012 0.304±0.015 0.320±0.016 0.328±0.006

HC-1D 0.301±0.024 0.352±0.082 0.286±0.105 0.386±0.034

B-Pref noise HC-2D H-1D 0.223±0.008 0.307±0.006 0.344±0.009 0.370±0.081 0.293±0.059 0.358±0.102 0.344±0.009 0.371±0.021

H-2D 0.161±0.009 0.319±0.029 0.360±0.052 0.340±0.038

Table 3: Ablation on branching and reusing for N = 12 across three seeds. Method Chain No reuse No KL CurriPO

HC-1D 0.311±0.012 0.492±0.043 0.510±0.033 0.512±0.030

Oracle HC-2D H-1D 0.222±0.008 0.081±0.001 0.319±0.019 0.390±0.062 0.308±0.013 0.320±0.077 0.319±0.019 0.415±0.051

H-2D 0.169±0.018 0.276±0.015 0.211±0.114 0.280±0.008

HC-1D 0.280±0.010 0.522±0.018 0.424±0.010 0.522±0.018

B-Pref noise HC-2D H-1D 0.211±0.004 0.087±0.010 0.256±0.023 0.348±0.034 0.243±0.005 0.323±0.077 0.253±0.026 0.350±0.018

H-2D 0.161±0.015 0.267±0.014 0.207±0.119 0.269±0.012

Table 4: Ablation on branching and reusing for N = 100 across three seeds. is particularly important for reaching diverse, hard-to-reach users, while reusing provides additional flexibility to recover users that become more accessible from later checkpoints. To further examine the benefit of reusing, we conduct user-level case studies on how initialization affects reward model optimization. As shown in Figure 4, optimizing the same reward model can lead to markedly different outcomes depending on the initialization: in some cases, ground-truth satisfaction decreases from an earlier checkpoint, but increases substantially when the same reward model is reused from a policy learned for another user. This illustrates how reusing allows CurriPO to exploit newly discovered, more favorable initializations for previously difficult users. Finally, we ablate the KL penalty to the reference policy in Eq. (6) (No KL), where each stage optimizes from πwk without being constrained toward it. Tables 3 and 4 show that removing it degrades satisfaction in most settings and inflates the variance. We attribute this to the role of the penalty in keeping successive stages as local transitions, which prevents the traversal from abruptly leaving the curriculum it has built.

4.4

Additional Analysis of Hard-to-Reach Users

In the previous section, we showed that CurriPO’s gains arise primarily in regions that are not reachable by the existing baselines. A natural objection is that the baselines are merely undertrained on these users. Figure 3 examines this possibility by analyzing the training dynamics on the targets that RM and RM+RND serve most poorly. To this end, we scale the training budget for RM from 0.4M to 3.2M steps per user, well beyond the default 1.2M steps used for both RM and RM+RND. Increasing the training budget, however, leaves these targets essentially unserved: satisfaction at target velocity 5 remains at 0.001 throughout, while at target velocity 3.75, satisfaction does not improve monotonically with additional training. An exploration bonus helps but does not close the gap.

4.5

checkpoint that best matches their reward model. To isolate the effect of this serving rule, we apply the same rule to each baseline over its own policy pool. This improves baselines slightly, but explains only a small fraction of CurriPO’s overall gain, indicating that the benefit comes mainly from the traversal that expands the policy pool. We next examine whether CurriPO’s traversal order itself is important. For the targets in Figure 3 that are difficult to reach using conventional optimization, we randomize the curriculum followed by CurriPO. This tests whether warm-starting from other users’ policies alone is sufficient, or whether CurriPO’s specific traversal order is important. Randomizing the order sharply reduces satisfaction, and the performance gap widens as the target becomes more difficult, demonstrating the importance of CurriPO’s curriculum ordering. We further ablate clo ; removing this restriction substantially degrades performance, confirming the importance of restricting expansion to an appropriate frontier. Finally, performance remains stable around the default clo , suggesting that CurriPO does not require careful tuning of this threshold. Detailed setups and results are provided in Appendix B.

Additional Ablations and Sensitivity Analysis

We conduct additional experiments on checkpoint serving, the traversal order, and clo , along with a sensitivity analysis of clo . At serving time, CurriPO assigns each user the wake

5

Conclusion

This paper highlights the severity of the problem of unserved users that can arise when optimizing a reward model over a diverse user population, and introduces CurriPO, which addresses this issue by treating the diverse user population as a curriculum. To our knowledge, it is the first to exploit multi-user structure at the optimization stage. By growing a tree-structured traversal with branching and reusing, CurriPO serves an entire population within a single pass. Through extensive experiments on personalized continuous control, we demonstrate that CurriPO achieves higher population satisfaction than baselines at substantially lower training cost, with the gains concentrated on users that conventional optimization struggles to reach. Future work includes extending CurriPO to broader user alignment settings, such as large language models and vision-language-action models, and evaluating it with diverse real-user feedback.

References Azad, A. S.; Gur, I.; Emhoff, J.; Alexis, N.; Faust, A.; Abbeel, P.; and Stoica, I. 2023. Clutr: Curriculum learning via unsupervised task representation learning. In International Conference on Machine Learning, 1361–1395. PMLR. Bahlous-Boldi, R.; Ding, L.; Spector, L.; and Niekum, S. 2024. Pareto-Optimal Learning from Preferences with Hidden Context. Bengio, Y.; Louradour, J.; Collobert, R.; and Weston, J. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, 41–48. Bradley, R. A.; and Terry, M. E. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4): 324–345. Burda, Y.; Edwards, H.; Storkey, A.; and Klimov, O. 2019. Exploration by random network distillation. In International Conference on Learning Representations. Cabi, S.; Colmenarejo, S. G.; Novikov, A.; Konyushova, K.; Reed, S.; Jeong, R.; Zolna, K.; Aytar, Y.; Budden, D.; Vecerik, M.; Sushkov, O.; Barker, D.; Scholz, J.; Denil, M.; de Freitas, N.; and Wang, Z. 2020. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In Proceedings of Robotics: Science and Systems. Corvallis, Oregon, USA. Cheng, J.; Xiong, G.; Dai, X.; Miao, Q.; Lv, Y.; and Wang, F.-Y. 2024. RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences. In International Conference on Machine Learning, 8229–8247. PMLR. Cho, G.; Im, J.; Lee, J.; Yi, H.; Kim, S.; and Kim, S. 2026. TRACED: Transition-aware Regret Approximation with Colearnability for Environment Design. In International Conference on Learning Representations, volume 2026, 16051– 16086. Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. Dennis, M.; Jaques, N.; Vinitsky, E.; Bayen, A.; Russell, S.; Critch, A.; and Levine, S. 2020. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33: 13049– 13061. Felten, F.; Alegre, L. N.; Nowé, A.; Bazzan, A. L. C.; Talbi, E. G.; Danoy, G.; and Silva, B. C. da. 2023. A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023). Florensa, C.; Held, D.; Geng, X.; and Abbeel, P. 2018. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, 1515–1528. PMLR. Graves, A.; Bellemare, M. G.; Menick, J.; Munos, R.; and Kavukcuoglu, K. 2017. Automated curriculum learning for neural networks. In international conference on machine learning, 1311–1320. Pmlr.

Guan, J.; Wu, J.; Li, J.-N.; Cheng, C.; and Wu, W. 2025. A survey on personalized Alignment—The missing piece for large language models in real-world applications. In Findings of the Association for Computational Linguistics: ACL 2025, 5313–5333. Guo, J.; Li, Z.; Qiu, J.; Wu, Y.; and Wang, M. 2025. On the role of preference variance in preference optimization. Hejna, J.; Rafailov, R.; Sikchi, H.; Finn, C.; Niekum, S.; Knox, W. B.; and Sadigh, D. 2024. Contrastive preference learning: Learning from human feedback without reinforcement learning. In International Conference on Learning Representations, volume 2024, 18770–18798. Hejna, J.; and Sadigh, D. 2023. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36: 18806–18827. Hwang, M.; Weihs, L.; Park, C.; Lee, K.; Kembhavi, A.; and Ehsani, K. 2024. Promptable behaviors: Personalizing multi-objective rewards from human preferences. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16216–16226. IEEE. Jiang, M.; Dennis, M.; Parker-Holder, J.; Foerster, J.; Grefenstette, E.; and Rocktäschel, T. 2021. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34: 1884–1897. Jiang, M.; Grefenstette, E.; and Rocktäschel, T. 2021. Prioritized level replay. In International Conference on Machine Learning, 4940–4950. PMLR. Kalra, A.; Novoseller, E.; Lawhern, V.; and Brown, D. S. 2026. P3L: Learning Pluralistically Aligned Policies from Heterogeneous Crowd Feedback. Kim, T.; Lee, G.; Chang, M.; Lim, S.; and Choi, J. 2026. Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences. Klink, P.; D’Eramo, C.; Peters, J. R.; and Pajarinen, J. 2020. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems, 33: 9216–9227. Kumar, M.; Packer, B.; and Koller, D. 2010. Self-paced learning for latent variable models. Advances in neural information processing systems, 23. Lee, K.; Smith, L.; Dragan, A.; and Abbeel, P. 2021. B-Pref: Benchmarking Preference-Based Reinforcement Learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). Lee, K.; Smith, L. M.; and Abbeel, P. 2021. PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 6152–6163. PMLR. Liang, X.; Shu, K.; Lee, K.; and Abbeel, P. 2022. Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. In International Conference on Learning Representations. Matiisen, T.; Oliver, A.; Cohen, T.; and Schulman, J. 2019. Teacher–student curriculum learning. IEEE transactions on neural networks and learning systems, 31(9): 3732–3740.

Ng, A. Y.; Harada, D.; and Russell, S. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, 278–287. Citeseer. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744. Park, J.; Seo, Y.; Shin, J.; Lee, H.; Abbeel, P.; and Lee, K. 2022. SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning. In International Conference on Learning Representations. Parker-Holder, J.; Jiang, M.; Dennis, M.; Samvelyan, M.; Foerster, J.; Grefenstette, E.; and Rocktäschel, T. 2022. Evolving curricula with regret-based environment design. In International Conference on Machine Learning, 17473–17498. PMLR. Poddar, S.; Wan, Y.; Ivison, H.; Gupta, A.; and Jaques, N. 2024. Personalizing reinforcement learning from human feedback with variational preference learning. Advances in Neural Information Processing Systems, 37: 52516–52544. Razin, N.; Malladi, S.; Bhaskar, A.; Chen, D.; Arora, S.; and Hanin, B. 2025a. Unintentional unalignment: Likelihood displacement in direct preference optimization. In International Conference on Learning Representations, volume 2025, 24791–24834. Razin, N.; Wang, Z.; Strauss, H.; Wei, S.; Lee, J.; and Arora, S. 2025b. What Makes a Reward Model a Good Teacher? An Optimization Perspective. In Belgrave, D.; Zhang, C.; Lin, H.; Pascanu, R.; Koniusz, P.; Ghassemi, M.; and Chen, N., eds., Advances in Neural Information Processing Systems, volume 38, 59162–59222. Curran Associates, Inc. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. Shenfeld, I.; Faltings, F.; Agrawal, P.; and Pacchiano, A. 2025. Language Model Personalization via Reward Factorization. In Second Conference on Language Modeling. Siththaranjan, A.; Laidlaw, C.; and Hadfield-Menell, D. 2024. Distributional preference learning: Understanding and accounting for hidden context in rlhf. In International Conference on Learning Representations, volume 2024, 27448– 27471. Skalse, J. M. V.; Farrugia-Roberts, M.; Russell, S.; Abate, A.; and Gleave, A. 2023. Invariance in policy optimisation and partial identifiability in reward learning. In International Conference on Machine Learning, 32033–32058. PMLR. Todorov, E.; Erez, T.; and Tassa, Y. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, 5026– 5033. IEEE. Wang, R.; Lehman, J.; Clune, J.; and Stanley, K. O. 2019. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the genetic and evolutionary computation conference, 142–151.

Wang, R.; Lehman, J.; Rawal, A.; Zhi, J.; Li, Y.; Clune, J.; and Stanley, K. 2020. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning, 9940–9951. PMLR. Zollo, T.; Siah, A.; Ye, N.; Li, L.; and Namkoong, H. 2025. Personalllm: Tailoring llms to individual preferences. In International Conference on Learning Representations, volume 2025, 66949–66971.

A A.1

Appendix Additional Implementation Details

Preference Noise Configuration

Both the oracle and the noisy pairwise labels judge a segment ξ = ((st , at ))L t=1 through a weighted segment return Ru (ξ) =

L X

wt ut ,

ut = gu (dt ),

Component

Parameter

Myopic Skip Equal Stochastic Mistake

γ = 0.9 δskip = 0.1 δequal = 0.1 β = 12 pflip = 0.1

Table 5: Parameters of the B-Pref noise.

(10)

t=1

where gu is the ground-truth per-step utility of user u defined in Eq. (9) and dt is the behavioral feature vector at step t. The oracle pairwise label uses uniform  weights wt = 1 and returns the deterministic label y = 1 Ru (ξ 1 ) > Ru (ξ 0 ) . The noisy pairwise label follows the design of B-Pref (Lee et al. 2021) and applies all five perturbations—myopia, stochasticity, mistakes, skipping, and equal labels—to every query. Writing ∆R = Ru (ξ 1 ) − Ru (ξ 0 ) and letting R̄u denote the mean segment return of user u over the behavior pool D, each query (ξ 0 , ξ 1 ) is processed in the following order. • Myopic weighting. The return in Eq. (10) is computed with the discounted weights wt = γ L−t , so that later timesteps within a segment carry larger weight and the teacher judges primarily on the terminal behavior of the segment. • Skip. If neither segment is informative, i.e. max{Ru (ξ 0 ), Ru (ξ 1 )} < δskip R̄u , the query is discarded and no label is produced. • Equal. If the two segments are nearly indistinguishable, i.e. |∆R| < δequal R̄u , the query receives the soft label y = 0.5, which contributes to Eq. (1) as an evenly split target. • Stochastic labeling. Otherwise the label is drawn as
y ∼ Bernoulli σ(β ∆R) with rationality constant β, rather than taken deterministically from the sign of ∆R. • Mistake. Finally, every hard label is flipped independently with probability pflip . Table 5 lists the values used for the five components. This configuration perturbs the preference labels in five qualitatively different ways at once, and therefore yields reward models whose geometry deviates substantially from the oracle geometry.

A.2

Baseline Implementations

Common protocol. Every baseline receives exactly the same supervision as CurriPO: RM and RM+RND reuse the identical bank {fu }u∈U , while VPL, CPL, and IPL are fit from the same per-user preference dataset. All methods that interact with the environment share the PPO implementation reported in the main text, start from the same initial policy π0 , and consume the pool-standardized reward of Eq. (7). RM. The conventional per-user pipeline (Christiano et al. 2017) runs one PPO job per user on that user’s reward model fu for 1.2M environment steps. The policy is initialized at π0 at the start of each run.

RM+RND. RM+RND augments RM with a Random Network Distillation bonus (Burda et al. 2019) of coefficient 1.0. The target and predictor networks are width-256 MLPs producing 64-dimensional features, and the predictor is trained online on visited states. The per-user budget is again 1.2M steps. This baseline isolates whether insufficient exploration alone accounts for the users that RM fails to serve. VPL. VPL (Poddar et al. 2024) fits a single latentconditional reward model over all users jointly. An encoder q(z | A) maps an annotation context A of 8 preference pairs to a latent z ∈ R16 , and a decoder r(s, a, z) scores state– action pairs conditioned on it. Both are 3-layer width-256 MLPs, trained for 15k Adam steps with a Bradley–Terry reconstruction loss and an annealed KL term against the latent prior. Each user u is then served by the same 1.2M-step per-user PPO run used for RM, applied to the conditioned reward r(·, ·, zu ). CPL. CPL (Hejna et al. 2024) first behavior-clones a Gaussian policy with two hidden layers of width 256 on the pooled dataset D for 15k updates. For each user u, it then fine-tunes the policy for an additional 45k updates on the user’s preference pairs using the biased contrastive advantage objective, with temperature α = 0.1, conservative bias 0.5, and batch size 96. IPL. IPL (Hejna and Sadigh 2023) trains twin Q-networks, a V -network, and an actor with two hidden layers of width 256 for 60k gradient steps on the frozen pooled dataset. The value function is learned using IQL-style expectile regression with expectile 0.7, while the implicit reward r = Q − γV , induced by the inverse Bellman operator, is optimized using the Bradley–Terry preference loss with a χ2 regularizer. The actor is then extracted via advantage-weighted regression with temperature β = 1/3.

B

Additional Analysis on Serving from the Wake and Sensitivity Analysis

B.1

Analysis on Serving from the Wake

CurriPO serves each user the wake checkpoint their own reward model prefers, which raises the question of how much of its advantage comes from selecting over a pool rather than from the traversal that built the pool. Tables 6 and 7 give every baseline the same serving rule over its own policy pool. Nearly all baseline-setting pairs improve, confirming that cross-user serving is useful on its own, but the gains are small and leave a wide margin. The reason is that selection and traversal play different roles. Selection is a repair mechanism:

Oracle B-Pref noise HC-1D HC-2D H-1D H-2D HC-1D HC-2D H-1D H-2D RM 0.364 (+0.023) 0.238 (+0.023) 0.272 (+0.022) 0.185 (+0.010) 0.328 (+0.017) 0.226 (+0.006) 0.326 (+0.017) 0.200 (+0.037) +RND 0.419 (+0.031) 0.268 (+0.029) 0.275 (+0.015) 0.168 (+0.011) 0.331 (+0.011) 0.172 (+0.004) 0.317 (+0.005) 0.166 (+0.010) CPL 0.201 (+0.013) 0.121 (+0.030) 0.224 (+0.036) 0.102 (+0.026) 0.225 (+0.031) 0.141 (+0.045) 0.212 (+0.032) 0.111 (+0.026) IPL 0.305 (+0.010) 0.209 (+0.016) 0.243 (+0.015) 0.178 (+0.053) 0.323 (+0.019) 0.210 (+0.017) 0.262 (+0.009) 0.139 (+0.011) VPL 0.217 (+0.008) 0.101 (+0.014) 0.257 (−0.009) 0.108 (+0.009) 0.219 (+0.027) 0.130 (+0.016) 0.266 (+0.002) 0.116 (+0.018) CurriPO 0.514 0.318 0.355 0.328 0.386 0.344 0.371 0.340 Method

Table 6: Satisfaction for N = 12 users under CurriPO-style final serving using policy pools trained by different baseline methods. Each cell reports the served satisfaction, with the gain over the baseline’s own serving in parentheses. Oracle B-Pref noise HC-1D HC-2D H-1D H-2D HC-1D HC-2D H-1D H-2D RM 0.360 (+0.008) 0.233 (+0.006) 0.336 (+0.027) 0.156 (+0.014) 0.332 (+0.014) 0.197 (+0.014) 0.253 (+0.009) 0.193 (+0.014) +RND 0.360 (+0.016) 0.237 (+0.016) 0.342 (+0.024) 0.153 (+0.008) 0.338 (+0.022) 0.175 (+0.004) 0.275 (+0.008) 0.172 (+0.020) CPL 0.212 (+0.039) 0.102 (+0.013) 0.204 (+0.013) 0.105 (+0.027) 0.198 (+0.019) 0.103 (+0.010) 0.206 (+0.012) 0.099 (+0.012) IPL 0.311 (+0.012) 0.219 (+0.024) 0.248 (+0.008) 0.143 (+0.015) 0.312 (+0.016) 0.222 (+0.031) 0.275 (+0.014) 0.135 (+0.006) VPL 0.255 (+0.047) 0.116 (+0.009) 0.292 (+0.017) 0.112 (+0.018) 0.222 (+0.007) 0.132 (+0.007) 0.281 (+0.006) 0.107 (+0.014) CurriPO 0.512 0.319 0.415 0.280 0.522 0.253 0.350 0.269 Method

Table 7: Satisfaction for N = 100 users under CurriPO-style final serving using policy pools trained by different baseline methods. Each cell reports the served satisfaction, with the gain over the baseline’s own serving in parentheses.

Random Order CurriPO

to-optimize users reachable.

Target velocity

Method 2

3.75

4.5

4.75

5

0.428 0.613

0.212 0.559

0.159 0.433

0.147 0.441

0.128 0.445

Table 8: Random-ordering ablation on hard-to-reach targets.

when a user’s own optimization fails, it lets that user fall back on a neighbor’s policy that happens to score better under their reward model, recovering part of the loss within the behaviors the policy pool already contains. It cannot, however, extend the pool. Under per-user optimization, every policy in it was produced from π0 , so regions unreachable in that way remain unrepresented no matter how the serving rule redistributes them, and the users who lie there gain nothing. CurriPO’s traversal instead changes what is available to select from, because each stage starts from a new checkpoint that enables higher-quality exploration of previously unexplored regions.

B.2

Analysis on Ordering

CurriPO reaches a given user through a specific sequence of intermediate checkpoints. To test whether this ordering itself matters, we construct a Random Order baseline that takes the curriculum CurriPO traversed to reach each of the hard targets in Figure 3 and randomly permutes the order in which its intermediate reward models are optimized, running the same number of KL-anchored local stages in Eq. (6) with the same per-stage PPO budget and warm-starting each stage from the preceding checkpoint. Table 8 reports utilitarian satisfaction on N =12 under oracle feedback. Random Order is uniformly worse. Warm-starting from other users’ policies is thus not sufficient on its own—the progression discovered by CurriPO’s coverage-based selection is what makes hard-

B.3

Analysis on clo

Table 10 varies and ablate the key parameter clo . Setting clo =0 admits every user as a candidate at every checkpoint, causing Eq. (5) to reduce to always targeting the least-covered user. As a result, mean satisfaction drops from 0.365 to 0.208, demonstrating that removing the threshold substantially degrades performance. These results suggest that restricting expansion to an appropriate curriculum frontier is important for effective exploration. At the same time, performance is relatively insensitive around the default setting of clo =0.60: across clo ∈ [0.50, 0.70], mean satisfaction varies only from 0.356 to 0.365, indicating that CurriPO does not require careful tuning of clo .

C C.1

Additional Experimental Details

Additional User Satisfaction Visualization

Figure 2 in the main text reports the user satisfaction landscape for the N =100 population under oracle feedback. Here we provide the same visualization for the remaining three population–feedback configurations: N =12 under oracle feedback (Figure 5), N =12 under B-Pref noisy feedback (Figure 6), and N =100 under B-Pref noisy feedback (Figure 7).

C.2

Nash and Egalitarian Population Satisfaction

Metrics. The utilitarian satisfaction reported in the main P text, Wutil = N1 u Gu (πu ), weighs every user equally but is insensitive to how satisfaction is distributed: a method that serves part of the population well and abandons the rest can attain the same mean as a method that serves everyone moderately. Since the concern motivating this work is precisely that some users are left unserved, we additionally report two

Oracle (N =12) Method

Noisy (N =12)

Oracle (N =100)

Noisy (N =100)

HC-1D HC-2D H-1D H-2D HC-1D HC-2D H-1D H-2D HC-1D HC-2D H-1D H-2D HC-1D HC-2D H-1D H-2D

Egalitarian RM 0.018 RM+RND 0.002 CPL 0.006 IPL 0.089 VPL 0.002 CurriPO 0.348

0.010 0.018 0.013 0.088 0.005 0.212

0.012 0.014 0.008 0.016 0.050 0.014 0.043 0.062 0.008 0.012 0.102 0.037

0.005 0.004 0.006 0.075 0.001 0.139

0.002 0.002 0.022 0.085 0.007 0.201

0.013 0.009 0.010 0.020 0.034 0.027 0.035 0.060 0.013 0.011 0.075 0.091

0.001 0.001 0.006 0.075 0.002 0.235

0.018 0.013 0.004 0.074 0.003 0.155

0.008 0.011 0.012 0.011 0.012 0.007 0.014 0.019 0.008 0.009 0.055 0.077

0.000 0.000 0.003 0.048 0.002 0.206

0.001 0.001 0.002 0.057 0.002 0.119

0.005 0.006 0.010 0.008 0.001 0.006 0.011 0.017 0.008 0.009 0.052 0.058

Nash RM RM+RND CPL IPL VPL CurriPO

0.128 0.158 0.064 0.178 0.042 0.310

0.222 0.118 0.146 0.115 0.158 0.059 0.182 0.118 0.126 0.064 0.302 0.249

0.162 0.140 0.108 0.257 0.043 0.342

0.107 0.067 0.080 0.176 0.057 0.335

0.266 0.105 0.235 0.125 0.146 0.075 0.206 0.119 0.142 0.064 0.309 0.307

0.196 0.175 0.098 0.264 0.056 0.500

0.162 0.149 0.064 0.182 0.049 0.311

0.186 0.102 0.187 0.103 0.151 0.063 0.194 0.118 0.136 0.062 0.346 0.241

0.132 0.107 0.113 0.254 0.064 0.503

0.086 0.080 0.072 0.174 0.061 0.244

0.212 0.141 0.230 0.118 0.153 0.073 0.213 0.117 0.132 0.062 0.295 0.232

0.183 0.192 0.087 0.251 0.051 0.505

Table 9: Egalitarian and Nash population satisfaction on N =12 and N =100 over 3 seeds. Oracle (N = 100)

Noisy (N = 100)

clo

HC-1D

HC-2D

H-1D

H-2D

HC-1D

HC-2D

H-1D

H-2D

0.00 0.50 0.55 0.60 0.65 0.70

0.311 0.493 0.445 0.512 0.428 0.452

0.140 0.174 0.327 0.319 0.332 0.326

0.221 0.474 0.525 0.415 0.459 0.463

0.087 0.292 0.286 0.280 0.206 0.261

0.378 0.441 0.406 0.522 0.477 0.487

0.234 0.192 0.246 0.253 0.253 0.226

0.199 0.473 0.412 0.350 0.445 0.337

0.093 0.305 0.206 0.269 0.285 0.292

Table 10: Sensitivity and ablation analysis of clo on N =100. distribution-sensitive criteria, Wnash =

N Y u=1

1/N Gu (πu ) ,

Wegal = min Gu (πu ). u

(11) The egalitarian criterion is determined by the single worstserved user. The Nash criterion sits between the egalitarian and utilitarian: being multiplicative, it is driven toward zero by any user receiving near-zero satisfaction, while still rewarding broad gains. Tables 9 reports both criteria for N =12 and N =100 under oracle and B-Pref noisy feedback. CurriPO achieves the highest Nash welfare in all 16 settings and the highest egalitarian welfare in 15 of 16. The advantage is particularly pronounced under the egalitarian criterion, where most baselines leave at least one user with near-zero satisfaction, whereas CurriPO substantially raises the satisfaction of the worst-served users. These results indicate that CurriPO’s gains are not driven by concentrating improvements on already well-served users, but by extending effective coverage to users who would otherwise remain underserved.

Figure 5: User satisfaction visualization across different optimization methods with N = 12 users under oracle feedback. User satisfaction landscapes are shown for HalfCheetah and Hopper under 1-D and 2-D preference settings, with target speed as the shared preference dimension and energy or height as the second dimension. Darker colors indicate higher user satisfaction.

Figure 6: User satisfaction visualization across different optimization methods with N = 12 users under noisy feedback. User satisfaction landscapes are shown for HalfCheetah and Hopper under 1D and 2D preference settings, with target speed as the shared preference dimension and energy or height as the second dimension. Darker colors indicate higher user satisfaction.

Figure 7: User satisfaction visualization across different optimization methods with N = 100 users under noisy feedback. User satisfaction landscapes are shown for HalfCheetah and Hopper under 1D and 2D preference settings, with target speed as the shared preference dimension and energy or height as the second dimension. Darker colors indicate higher user satisfaction.

Related documents

• From Soft Targets to Reward Signals: How Assignment and Reward Objectives Interact#1129463
• PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning#798876
• PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning#797462
• PARM: Pipeline-Adapted Reward Model#120563
• Hybrid neural-cognitive models reveal how memory shapes human reward learning.#236302