Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning
arXiv:2606.14693v1 [cs.MA] 12 Jun 2026
Pengxin Wang1 , Lihao Guo1 , Yi Xie1 , Bo Liu1 , Siyang Cao1 , Jingdi Chen1 1 Department of Electrical and Computer Engineering, University of Arizona
Abstract Cooperative multi-objective multi-agent reinforcement learning (MOMARL) models team decision making under multiple, potentially conflicting objectives. In this setting, conflicts arise not only across objectives but also across agents with different observations, roles, and contributions. We propose Preference Coordinated Multi-agent Policy Optimization (PCMA), which learns coordinated agent-specific preferences to enable complementary trade-offs among agents. Theoretically, we formulate cooperative MOMARL as a team-optimal equilibrium problem, and show that, under suitable conditions, preference diversity yields a first-order improvement in the team objective. Experiments on multiple cooperative MOMA environments and a practical traffic-control scenario show that PCMA improves both performance and trade-off coordination.
1
Introduction
Multi-objective Multi-agent reinforcement learning (MOMARL) provides a natural framework for modeling complex decision making systems where multiple agents must coordinate under competing objectives. Existing applications on domains such as traffic signal control, dynamic traffic management, and natural resource allocation show that such systems often require agents to balance competing objectives such as efficiency, safety, energy consumption, and fairness [Li et al., 2010, Chung et al., 2018, Bone and Dragićević, 2009]. Therefore, modeling such problems with a single scalar reward is often inadequate Vamplew et al. [2022]. At the same time, conflict structure is extremely intricate in MOMA problem: conflicts may arise not only between objectives within each agent, but also between agents under the same objective and across agents with different objective priorities. Therefore, MOMARL is both practically important and theoretically challenging, calling for learning algorithms that model and coordinate such intertwined objectives in Multi-agent systems. Despite growing interest, research on MOMARL remains at an early stage. Several notable efforts have begun to structure the field, including the utility based analysis of Multi-objective Multi-agent decision making Rădulescu et al. [2020] and the introduction of MOMALand Felten et al. [2024], as the first benchmark environments in this field. A straight-forward approach adopted by existing works is to optimize scalarized objective using the same preference vector across all agents. However, enforcing identical objective emphasis for every agent can limit the effectiveness of coordination. When all agents prioritize objectives in the same way, they may compete along the same dimensions of the reward. For example, consider two autonomous vehicles passing through an unsignalized intersection. If both vehicles strongly prefer efficiency, they may aggressively enter the intersection and cause a collision. If both vehicles strongly prefer safety, they may both wait and waste time. A better joint behavior requires coordinated preferences: one vehicle may emphasize yielding and safety, while the other emphasizes passing efficiency. This motivates our central idea: rather than forcing all agents to follow the same trade-off, we allow agents to adapt their preferences through a team-level coordination mechanism. From a multiobjective perspective Roijers et al. [2013], Hayes et al. [2022], each agent’s policy can be viewed Preprint.
Preferences as points on the multi-objective Pareto front Shared preference
Coordinated preferences 3 3 R2
2
R2
2
1
w2 w1
Pareto front R1 Single agent (MO only) one w → one Pareto point
1
w3
w
Single objective (MA only) no front, no trade-off
Pareto front R1 MA + MO with coordination distinct points along the front
Figure 1: Illustration of preference coordination. Agents’ policies are projected on to the multiobjective return space (R1 , R2 ). A preference w from the origin selects one point on the front. Left: all agents use the same preference vector w, which could cause conflicts. Right: distinct w1 , w2 , w3 project to three different points along the front and the team jointly covers the front. A single agent can only occupy one point, a single objective produces no front at all, and only multi-agent + multi-objective with coordination spreads agents along the front.
as inducing a point in a shared objective space. By conditioning policies on different preferences, agents can approach different regions of the Pareto front. Coordinating these preferences enables the multi-agent system to assign diverse but complementary roles, thereby reducing behavioral conflicts and improving team-level performance, as illustrated in Figure. 1. To realize this idea, in this paper, we propose Preference Coordinated Multi-agent Policy Optimization (PCMA) under the centralized training with decentralized execution (CTDE) framework Lowe et al. [2017]. We model preference as a latent coordination variable and train a stochastic planner for each agent to adapt its preference based on team improvement and its local observation, and then each agent executes a policy conditioned on its sampled preference. To encourage meaningful specialization, we further regularize the learned preference distributions so that agents avoid collapsing to the same preference direction. Our contributions are as follows: • We formulate cooperative MOMARL as a team-optimal equilibrium problem, where the goal is to learn a coordinated preference profile whose induced preference-conditioned equilibrium maximizes the team objective. • We provide a theoretical analysis of preference coordination, including a first-order team improvement decomposition and a local equilibrium-tracking result for preference-conditioned games. • We propose PCMA, a practical policy-based method with stochastic preference planners and preference-conditioned actors, and provide empirical evaluations on selected cooperative tasks from MPE, SMAC, and MOMAland, as well as a traffic-control validation scenario.
2
Related Work
Current work on cooperative Multi-objective Multi-agent reinforcement learning (MOMARL) remains at an early stage of exploration. As a pioneer, work on Multi-objective games has put forward solution concepts such as Pareto Nash equilibrium, which extends Nash equilibrium to vector valued utilities by requiring Pareto optimality under unilateral deviations Borm et al. [1988], Voorneveld et al. [1999], Lozovanu et al. [2005], Dai and Duan [2024]. However, Pareto Nash equilibrium can be ill-posed as an optimization criterion for cooperative MOMARL. Since a unilateral deviation is excluded only when it improves at least one objective without decreasing any other objective, many policies that are suboptimal under meaningful trade offs may still satisfy the Pareto Nash condition. We provide an illustrative example in Appendix C.1, where any joint policies satisfy the Pareto Nash condition, while only a small subset corresponds to meaningful team level trade offs. More recently, MOMALand Felten et al. [2024] considers linear scalarization based approaches to convert the vector valued team reward into a scalar reward, and then train policies by standard MARL algorithms such as MAPPO Yu et al. [2022]. However, they need re-train a separate policy 2
for new preference vector, making them impractical for desired trade off during execution. Another work, MoMix Hu et al. [2023], extends the value decomposition principle of QMIX Rashid et al. [2018] to vector valued rewards. While this provides an important step toward Multi-objective credit assignment, it relies on strong individual global max (IGM) assumptions and can’t be applied to continous action space.
3
Preliminaries and Problem Formulation
This section introduces the background setting and problem formulation. We first define the cooperative Multi-objective Dec-POMDP in Subsection 3.1, and then formulate MOMARL as a team optimal equilibrium problem in Subsection 3.2. 3.1
Multi-objective Dec-POMDP
We consider a Multi-objective Decentralized Partially Observable Markov Decision Process (DecPOMDP) Rădulescu et al. [2020], Felten et al. [2024], Hu et al. [2023], where N agents interact in a shared environment defined by N G = N, S, {Ai }N i=1 , P, Ω, O, rteam , {ri }i=1 , γ .
Here, S is the state space, {Ai }N i=1 are the action spaces, a = (a1 , . . . , aN ) is the joint action, and γ is the discount factor. The environment evolves according to st+1 ∼ P (· | st , at ). Each agent receives a local observation oi , and the joint observation o = (o1 , . . . , oN ) ∈ Ω is generated by O(o | s). The reward consists of a sparse team reward rteam (s, a) ∈ R and agent-specific guiding vector K rewards {ri (s, a)}N i=1 , where ri (s, a) ∈ R . The team reward is scalar, since the team task should be the same to all agents and all objectives. It is shared by all agents and measures task completion, such as +1 for team winning. The guiding vector reward provides auxiliary Multi-objective feedback for each agent, such as efficiency, cost, or risk. 3.2
MOMARL as Team Optimal Equilibrium.
Let θi parameterize agent i’s policy, and θ = (θ1 , . . . , θN ). We define the shared team objective and the agent-specific vector objective as " T # " T # X X t t Jteam (θ) := Eπθ γ rteam (st , at ) , Ji (θ) := Eπθ γ ri (st , at ) . t=0
t=0
Given a preference profile p(p1 , . . . , pN ), agent i’s payoff is defined as Ui (θ; pi ) := Jteam (θ) + p⊤ i Ji (θ). For a fixed p, this induces a preference-conditioned stochastic game G(p). The solution can be defined as a Nash equilibrium Nash [1951], Fink [1964], denoted by θ∗ (p), where no agent can improve its payoff Ui by unilaterally changing its own policy while the other agents’ policies remain fixed. However, an equilibrium under a fixed preference profile does not necessarily maximize the final team objective. The key problem is therefore to coordinate the preference profile so that the induced equilibrium achieves high team performance. Motivated by the social-welfare view of evaluating collective outcomes Harsanyi [1955] and the optimal-equilibrium perspective in team Markov games Wang and Sandholm [2002], we formulate the goal as finding a preference profile p and an induced equilibrium θ that maximize the team objective Jteam .
4
Theoretical Analysis
We now provide a theoretical explanation for coordinated preference learning. First, we characterize how a policy update affects the team objective and show that preference diversity can contribute a positive first-order improvement. Second, we show that equilibria under different preference profiles 3
are connected through a locally smooth stationary path, which implies that preference optimization can proceed gradually from an initial profile instead of treating each preference-conditioned game as an isolated problem. 4.1
First-order team improvement and preference diversity
In this subsection, we ask how the choice of preference profile (p1 , · · · , pN ) affects the resulting team improvement. Starting from the current policy parameters θ = (θ1 , . . . , θN ), each agent performs one preference-conditioned gradient update θi,new = θi + η∇θi Ui (θ; pi ), i = 1, . . . , N, We can define the following team-improvement matrix B ∈ RN ×K by ⊤
Bi,k := (∇θi Jteam (θ)) ∇θi Ji,k (θ), i = 1, . . . , N, k = 1, . . . , K. Here, Bi,k measures the first-order contribution of agent i’s k-th individual objective to the team PN PN objective. Let p̄ = N −1 i=1 pi and b̄ = N −1 i=1 bi be the averages across agents, and let p̃i = pi − p̄ and b̃i = bi − b̄ be their centered counterparts. We assume that the centered first-order contribution of each agent has a lower-bounded projection on its preference direction. Assumption 4.1 (Preference-improvement alignment). There exists κ > 0 such that, for every agent i with p̃i ̸= 0, the projection of b̃i onto p̃i is lower bounded by κ, i.e., p̃⊤ i b̃i ≥ κ, ∥p̃i ∥22
i = 1, . . . , N.
By first-order Taylor expansion, we can derive the following decomposition Theorem 4.2 (Team Improvement Decomposition). For sufficiently small update step η, the firstorder team improvement satisfies Jteam (θnew ) − Jteam (θ) ≥ η
N X
2
∥∇θi Jteam (θ)∥2 + ηN p̄⊤ b̄ + κDp
i=1
where Dp :=
N N 1 XX ∥pi − pj ∥22 . 2N 2 i=1 j=1
is the pairwise preference distance across agents. This theorem decomposes the first-order team improvement into direct team-gradient improvement, average preference alignment, and diversity-induced improvement. The first term is non-negative but can be noisy under sparse team rewards. The term ηN p̄⊤ b̄ captures average alignment, while ηN κDp shows that aligned preference diversity yields additional positive team improvement. 4.2
Equilibrium Tracking
For each preference profile p, there is a corresponding preference-conditioned game G(p). A natural question is how the solutions of these games are related as p changes. In this subsection, we first show that, under regularity conditions, the local stationary equilibrium changes continuously with respect to p. Then, when the preference profile varies slowly, gradient updates can track the corresponding moving equilibrium path. By applying the implicit function theorem Jittorntrum [1978] to the local nash condition Ratliff et al. [2014], we obtain the following local continuity result. The full proof is provided in Appendix A.3. Lemma 4.3 (Continuity of preference-conditioned stationary solutions). Suppose that θ∗ is a local Nash equilibrium of G(p), i.e., ∇θi Ui (θ∗ ; pi ) = 0, i = 1, . . . , N. Assume that the joint gradient (∇θ1 U1 , . . . , ∇θN UN ) is continuously differentiable near (θ∗ , p) with non-singular Jacobian. Then there exists a neighborhood U of p and a unique continuously differentiable mapping θ(·) : U → Θ such that θ(p) = θ∗ and, for all p̃ ∈ U, ∇θi Ui (θ(p̃); p̃i ) = 0, i = 1, . . . , N. 4
Figure 2: Overview of PCMA. Each agent uses a stochastic preference planner and a preferenceconditioned actor. The planner samples preferences from a Dirichlet distribution, and the actor selects actions conditioned on the sampled preference. During training, the team critic provides coordination feedback, while individual vector critics provide dense preference-aligned learning signals. The previous lemma guarantees the existence of a locally continuous equilibrium region. For learning dynamic analysis, we further impose the following two regularity assumptions. Assumption 4.4 (Attraction property near equilibrium). Let θ(p) denote a local Nash stationary solution under the preference profile p. Assume that there exist ρ ∈ (0, 1) and a neighborhood of θ(p) such that ∥θnew − θ(p)∥ ≤ ρ∥θold − θ(p)∥, for all θold in this neighborhood. Assumption 4.5 (Local Lipschitz continuity). Let θ(·) : U → Θ be the local stationary solution mapping induced by Lemma 4.3. Assume that this mapping is Lipschitz continuous in U: there exists C > 0 such that for any two preference profiles p, p′ ∈ U, ∥θ(p′ ) − θ(p)∥ ≤ C∥p′ − p∥. The following theorem shows that, when the preference profile changes slowly, the policy update remains close to the equilibrium associated with the new preference profile. See proof in Appendix A.4. Theorem 4.6 (Equilibrium tracking). Consider the iterative policy update θit+1 = θit + η∇θi Ui (θt ; pt ),
i = 1, . . . , N.
Under Assumption 4.4 and Assumption 4.5, the tracking error et := ∥θt − θ(pt )∥ satisfies et+1 ≤ ρet + C∥pt+1 − pt ∥. In particular, if ∥pt+1 − pt ∥ ≤ δ, then lim sup et ≤ t→∞
5
C δ. 1−ρ
Multi-agent Policy Optimization with Preference Coordinated Learning
Motivated by the preceding analysis, we now implement the preference-coordination idea in a practical PPO-based algorithm, called Preference Coordinated Multi-agent Policy Optimization 5
Algorithm 1 Preference Coordinated Multi-agent Policy Optimization(PCMA) 1: Initialize preference planner ϕψ , actor πθ , team critic V team , and vector critics {Vi }N i=1 2: for each iteration do 3: for each environment step do 4: for each agent i = 1, . . . , N do 5: sample local preference pi ∼ Dir(ϕψ (oi )) and Sample action ai ∼ πθ (· | oi , pi ) 6: end for ′ 7: Execute joint action a = (a1 , . . . , aN ) and store (o, p, a, rteam , {ri }N i=1 , o ) 8: end for N 9: Estimate the team advantage Ateam and individual vector advantages {Aind i }i=1
10: Update V team and {Vi }N i=1 by minimizing Lcritic defined in Eq. (1) 11: Compute AUi and Update πθ by minimizing the Lactor defined in Eq. (2) 12: Compute preference diversity Dα , and update ϕψ by minimizing the Lplan defined in Eq. (3) 13: end for
(PCMA). The key design is to treat preference pi as a latent coordination variable: it determines the utility optimized by agent i, while the planner adjusts the preference profile to encourage teambeneficial updates. PCMA follows the CTDE Lowe et al. [2017] paradigm, using centralized critics for training and decentralized policies for execution. 5.1
Preference-conditioned Policy Optimization
For a fixed preference profile p = (p1 , . . . , pN ), each agent is updated with respect to its utility Ui (pi ). Following preference-conditioned policy learning in MORL Abels et al. [2019], Yang et al. [2019], Basaklar et al. [2023], we use a simple concatenation design, where the preference pi is provided as an additional input to the actor. We use a dual-critic design to estimate the corresponding utility advantage: a centralized team critic estimates the sparse team advantage Ateam , and individual vector critics estimate the multi-objective guiding advantages Ai . The critics are trained by " # N X 2 2 Lcritic = Eτ V team (o) − Rteam + Vi (oi ) − Ri 2 . (1) i=1
The actor selects actions by ai ∼ πθ (· | oi , pi ) and is optimized with a standard PPO surrogate: Lactor (θ) = LPPO (πθ (· | oi , pi ), AUi ) ,
ind AUi = Ateam + λp⊤ i Ai .
(2)
Intuitively, pi specifies the local trade-off direction along which agent i improves its policy. For a fixed preference profile p, the agents therefore play a preference-conditioned game. However, an arbitrary profile may lead to poorly coordinated directions, so the induced equilibrium may fail to cover team-beneficial regions of the Pareto front, which motivates us to optimize the preference profile itself. 5.2
Coordinated Preference Planning
We then equip each agent with a stochastic preference planner that adapts its own preference based on its local observation. Since preferences lie on the simplex, a natural choice is the Dirichlet distribution. Specifically, the planner outputs αi = ϕψ (oi ), and samples pi ∼ Dir(αi ), where pi ∈ ∆K−1 . Motivated by Theorem 4.2, we encourage the planner to maintain diverse preference distributions across agents. We regularize the expected pairwise diversity of the sampled preferences Dα := E[Dp ]. This regularizer promotes coordinated specialization: different agents can focus on different objective trade-offs, rather than collapsing to the same preference direction. The planner is then trained with the team advantage: Lplan (ψ) = LPPO ϕψ (· | oi ), Ateam − λ1 Dα . (3) The PPO term encourages preference distributions that yield higher team advantage. Therefore, our algorithm 1 not only let agent learn from local utilities, but are guide them towards team-level optimality. 6
6
Experiments
In this section, we present empirical results to validate the effectiveness of our method. We evaluate on a diverse set of cooperative multi-agent environments, including particle-world coordination Lowe et al. [2017], drone control Felten [2024], walker locomotion Terry et al. [2021], and StarCraft combat Samvelyan et al. [2019]. To better study coordinated preference learning, we modify these environments by separating the sparse team reward from vector-valued individual rewards. This design allows us to test whether our method can organize vectorized local learning signals toward improved team performance. Additional environment descriptions are shown in Appendix B.1. We first analyze how preference coordination improves multi-agent cooperation, and then report team success rate and mean reward across agents and objectives as the main evaluation metrics in Table 1. 6.1
How Does Preference Coordination Improve Cooperation?
In this subsection, we study two representative cases and show the effect of preference coordination. Preference specialization in particle-world tasks. We first analyze preference specialization in MOMPE tasks. In Cooperative Spread, we use objective-specific distance rewards, where each objective encourages agents to approach a different landmark. In Predator-Prey, the objectives correspond to approaching different prey while maintaining safety. With preference coordination, agents specialize toward different objectives rather than collapsing to the same preference vector, leading to a well-covered Pareto front as is shown in Figure 3.
(a) Preferences (Spread).
(c) Preferences (Predator).
(b) Front (Spread).
(d) Front (Predator).
Figure 3: Preference coordination analysis on MOMPE tasks. In Cooperative Spread, preferences converge toward different objectives, and rollout returns cover different regions of the approximate Pareto front. In Predator-Prey, two agents focus on two different preys and others keep a balanced preference. Role specification in SMAC. The same effect appears in SMAC. Figure 4 measures each agent by damage dealt and damage taken, which reflect different combat roles such as aggressive attackers or front-line units. With coordinated preferences, agents spread across the behavior (a) 8m. (b) 2s3z. space instead of forming a homogeneous cluster. This indicates Figure 4: Role differentiation in SMAC. Each point represents an that preference coordination in- agent’s damage dealt and damage taken (averaged over episodes). duces role differentiation: different agents emphasize different combat trade-offs while still contributing to the same team objective. Thus, coordinated preference diversity provides a simple mechanism for improving cooperation. 6.2
Quantitative Performance
We compare PCMA with MADDPG, IPPO, and MAPPO, covering deterministic actor-critic learning, independent policy optimization, and policy optimization shared critic. All baselines use the same fixed scalarization weight [0.5, 0.5] for the vector-valued guiding reward. More details are shown in 7
Appendix B.3. For most environments, we report success rate and average reward as our main metric in Table 1. For Escort and MOMAwalker, where success rate is not directly defined, we instead report task-specific metrics such as average reward and forward distance. Fig. 5 shows the learning curves across the evaluated tasks. PCMA achieves the best or tied-best performance on most metrics, showing consistent improvements across diverse cooperative multi-agent tasks.
(a) Cooperative Spread
(b) Safe Predator Prey
(c) Catch
(d) Escort
(e) Walker 1
(f) SMAC 3m
(g) SMAC 2s3z
(h) SMAC 8m
Figure 5: Learning curves of PCMA and baseline methods across the evaluated cooperative multiagent tasks. Curves are averaged over 3 random seeds, and shaded regions indicate standard deviation.
Table 1: Results on multi-agent tasks. Values in parentheses denote standard deviation. Environment
6.3
MADDPG
IPPO
Success rate Cooperative Spread Average reward
Metric
0.38(0.43) −0.63(1.09)
0.27(0.38) 0.20(0.35)
0.80(0.00) 0.69(0.00)
MAPPO PCMA (Ours) 1.00(0.00) 0.89(0.00)
Safe Predator Prey
Success rate Average reward
0.68(0.06) 2.30(0.10)
0.60(0.01) 2.21(0.02)
0.91(0.05) 2.39(0.04)
0.96(0.02) 2.45(0.05)
Catch
Success rate Average reward
0.87(0.12) 4.71(7.94)
0.24(0.15) 8.61(0.64)
0.53(0.37) 11.33(3.83)
0.94(0.03) 14.21(0.83)
Escort
Average reward
16.38(0.80)
14.39(0.74)
14.48(0.63)
17.29(0.99)
MOMAwalker
Forward Distance 75.04(21.28) 6.69(2.36) 70.52(21.08) Average reward 18.28(16.18) −0.80(0.37) 10.11(15.35)
93.64(31.47) 21.62(20.05)
SMAC-3m
Success rate Average reward
0.50(0.37) 0.75(0.31)
0.83(0.05) 1.02(0.03)
0.97(0.05) 0.29(0.02)
0.97(0.05) 1.11(0.03)
SMAC-2s3z
Success rate Average reward
0.63(0.45) 0.22(0.10)
0.93(0.05) 0.29(0.01)
0.97(0.05) 0.30(0.01)
1.00(0.00) 0.31(0.01)
SMAC-8m
Success rate Average reward
0.23(0.33) 0.08(0.12)
0.00(0.00) 0.00(0.00)
0.80(0.16) 0.27(0.04)
0.87(0.05) 0.28(0.01)
Ablation Study
As shown in Figure 6, we conduct ablation studies on Cooperative Spread to analyze three key factors in PCMA: the diversity regularization coefficient λ1 , the actor balancing coefficient λ2 , and the learned preference planner. Specifically, λ1 controls the strength of the pairwise diversity regularizer in the planner objective, λ2 balances the team advantage and the scalarized individual advantage in the actor update. Moreover, to validate the effectiveness of our preference coordination, we compare PCMA with RAND and SAME. RAND samples each agent’s preference randomly without a learned planner, while SAME forces all agents to use the same preference by averaging their sampled preferences. 8
(a) Study of λ1 .
(b) Study of λ2 .
(c) Study of preference planner.
Figure 6: Ablation studies on Cooperative Spread. (a) λ1 = 0.1 and λ1 = 0.2 improve early learning over λ1 = 0, suggesting that preference diversity helps avoid collapse, while λ1 = 0.5 slows learning due to excessive diversity. (b) λ2 = 0.1 and λ2 = 0.2 achieve stable convergence, whereas λ2 = 0.5 becomes unstable, showing that individual guidance should not dominate the team signal. (c) The planner learns coordinated, agent-specific preferences rather than merely injecting random diversity.
6.4
OpenCDA-MARL CARLA Validation
Additionally, we use OpenCDA-MARL Guo et al. [2026] as a high-fidelity validation of preferenceconditioned control in CARLA Dosovitskiy et al. [2017]. Each CAV follows a fixed route through a four-way intersection and learns a target-speed command; the competitive setting reflects crossing order and yielding, not route selection. Rewards are grouped into efficiency and safety/interaction objectives. Detailed setting are shown in Appendix B.2. The validation results in Table 2 shows that preference coordination mechanism improves SAC. These results suggest that off-policy version preference-conditioned coordination is feasible within the current speed-control setting, while broader behavioral validation would require richer action spaces and interaction mechanisms. Table 2: OpenCDA-MARL/CARLA best-checkpoint evaluation. Values are reported as mean ± standard deviation across seeds.
7
Setting
Backbone
Utility ↑
Success (%) ↑
Collision (%) ↓
Throughput ↑
Coop. Coop. Coop.
SAC MAPPO PCMA
-4776.9 ± 672.0 -16793.3 ± 2668.9 -2072.9 ± 414.8
68.6 ± 1.9 55.4 ± 4.7 69.6 ± 1.2
31.4 ± 1.9 42.5 ± 3.5 30.4 ± 1.2
1692 ± 46 1417 ± 100 1716 ± 29
Comp. Comp. Comp.
SAC MAPPO PCMA
-5084.5 ± 241.9 -13660.2 ± 4695.8 -2877.1 ± 151.6
67.9 ± 0.7 42.1 ± 30.3 68.0 ± 2.2
32.1 ± 0.7 23.3 ± 17.6 23.3 ± 8.3
1674 ± 17 1066 ± 759 1192 ± 291
Conclusion and Future Work
This work frames cooperative MOMARL as a team-optimal equilibrium problem, where the goal is to find a preference profile whose induced equilibrium improves team performance. Based on this view, we propose PCMA, which coordinates agent-specific trade-off directions through stochastic preference planners and preference-conditioned actors. Our analysis connects preference diversity with first-order team improvement and local equilibrium tracking. Experiments on selected cooperative tasks show that learned preference coordination can improve team performance and induce interpretable agent specialization. One limitation of this work is that our study focuses on controlled cooperative benchmarks, where the objectives and reward decomposition are explicitly specified. A promising direction for future work is to extend preference coordination to more complex real-world multi-agent systems, such as autonomous driving and open-ended agentic systems, where agents may have different capabilities, contexts, and objectives that naturally emerge from real task demands. 9
References Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning, 2019. Toygun Basaklar, Suat Gumussoy, and Umit Y. Ogras. Pd-morl: Preference-driven multi-objective reinforcement learning algorithm, 2023. Christopher Bone and Suzana Dragićević. Gis and intelligent agents for multiobjective natural resource allocation: A reinforcement learning approach. Transactions in GIS, 13(3):253–272, 2009. Peter Borm, Stef Tijs, and Jan Van Den Aarssen. Pareto equilibria in multiobjective games. Methods of Operations Research, 60:302–312, 1988. Eunshin Chung et al. Multi-agent reinforcement learning for traffic control. Transportation Research Part C: Emerging Technologies, 92:230–245, 2018. Pengcheng Dai and Lingjie Duan. Multi-objective multi-agent reinforcement learning with paretostationary convergence, 2024. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 1–16. PMLR, 13–15 Nov 2017. Florian Felten. Multi-Objective Reinforcement Learning. PhD Thesis, Unilu - Université du Luxembourg [FSTM], Luxembourg, jun 2024. Florian Felten, Umut Ucak, Hicham Azmani, Gao Peng, Willem Röpke, Hendrik Baier, Patrick Mannion, Diederik M. Roijers, Jordan K. Terry, El-Ghazali Talbi, Grégoire Danoy, Ann Nowé, and Roxana Rădulescu. Momaland: A set of benchmarks for multi-objective multi-agent reinforcement learning, 2024. Arlington M. Fink. Equilibrium in a stochastic n-person game. Journal of Science of the Hiroshima University, Series A-I, 28:89–93, 1964. Lihao Guo, Louis Liu, Jiahao Tang, Bo Liu, and Siyang Cao. Opencda-marl: A unified benchmarking framework for cooperative autonomous intersection management with multi-agent reinforcement learning. IEEE Robotics and Automation Letters, 11(4):4291–4298, 2026. John C. Harsanyi. Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility. Journal of Political Economy, 63(4):309–321, 1955. Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022. Tianmeng Hu, Biao Luo, Chunhua Yang, and Tingwen Huang. Mo-mix: Multi-objective multi-agent cooperative decision-making with deep reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12098–12112, 2023. Krisorn Jittorntrum. An implicit function theorem. Journal of Optimization Theory and Applications, 25(4):575–577, 1978. Houli Li et al. A multi-objective reinforcement learning approach for traffic signal control. In International Conference on Intelligent Transportation Systems, 2010. Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actorcritic for mixed cooperative-competitive environments. Neural Information Processing Systems (NIPS), 2017. 10
Doina Lozovanu, Dorel Solomon, and Alexander Zelikovsky. Multiobjective games and determining pareto-nash equilibria. Buletinul Academiei de Ştiinţe a Republicii Moldova. Matematica, (3): 115–122, 2005. John Nash. Non-cooperative games. Annals of Mathematics, 54(2):286–295, 1951. Roxana Rădulescu, Patrick Mannion, Diederik M. Roijers, Peter Vamplew, Sam Devlin, and Ann Nowé. Multi-objective multi-agent decision making: a utility-based analysis and survey. Autonomous Agents and Multi-Agent Systems, 34(1):10, 2020. Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning, 2018. Lillian J. Ratliff, Samuel A. Burden, and S. Shankar Sastry. On the characterization of local nash equilibria in continuous games, 2014. Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multiobjective sequential decision-making. Journal of Artificial Intelligence Research, 48:67–113, 2013. Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019. Justin K. Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S. Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al. Pettingzoo: Gym for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pages 15032–15043, 2021. Peter Vamplew, Ben J. Smith, Joakim Källström, Markus Müller, Jesse Rubin, and Reiner Scholz. Scalar reward is not enough: a response to silver, singh, precup and sutton (2021). Autonomous Agents and Multi-Agent Systems, 36(2):41, 2022. Mark Voorneveld, Dries Vermeulen, and Peter Borm. Axiomatizations of pareto equilibria in multicriteria games. Games and Economic Behavior, 28(1):146–154, 1999. Xiaofeng Wang and Tuomas Sandholm. Reinforcement learning to play an optimal nash equilibrium in team markov games. In Advances in Neural Information Processing Systems, volume 15, 2002. Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation, 2019. Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, 2022.
11
A
Detailed Proofs for Theoretical Results
A.1
Assumptions
We begin by restating the assumptions required for the theoretical analysis. Assumption A.1 (Preference-improvement alignment). There exists κ > 0 such that, for every agent i with p̃i ̸= 0, the projection of b̃i onto p̃i is lower bounded by κ, i.e., p̃⊤ i b̃i ≥ κ, ∥p̃i ∥22
i = 1, . . . , N.
where p̃i := pi − p̄ and b̃i := bi − b̄. Assumption A.2 (Attraction property near equilibrium). Let θ(p) denote a local Nash stationary solution under the preference profile p. Assume that there exist ρ ∈ (0, 1) and a neighborhood of θ(p) such that ∥θnew − θ(p)∥ ≤ ρ∥θold − θ(p)∥, for all θold in this neighborhood. Assumption A.3 (Local Lipschitz continuity). Let θ(·) : U → Θ be the local stationary solution mapping induced by Lemma 4.3 Assume that this mapping is Lipschitz continuous in U: there exists C > 0 such that for any two preference profiles p, p′ ∈ U, ∥θ(p′ ) − θ(p)∥ ≤ C∥p′ − p∥. A.2
Proof of Theorem 4.2 in the main text
Theorem A.4 (Team Improvement Decomposition). For sufficiently small update step η, the firstorder team improvement satisfies Jteam (θnew ) − Jteam (θ) ≥ η
N X
2
∥∇θi Jteam (θ)∥2 + ηN p̄⊤ b̄ + κDp
i=1
where Dp :=
N N 1 XX ∥pi − pj ∥22 . 2N 2 i=1 j=1
is the pairwise preference distance across agents. Proof. Given preferences p1 , . . . , pN , each agent takes one gradient update θi,new = θi + η∇θi Ui (θ; pi ),
i = 1, . . . , N.
where Ui (θ; pi ) = Jteam (θ) + ⟨pi , Ji (θi )⟩, We write θnew = (θ1,new , . . . , θN,new ). By the first-order Taylor expansion of Jteam at θ, Jteam (θnew ) − Jteam (θ) = η
N X
⟨∇θi Jteam (θ), ∇θi Ui (θ; pi )⟩ + o(η)
i=1
=η
N X
⟨∇θi Jteam (θ), ∇θi Jteam (θ) + ∇θi ⟨pi , Ji (θi )⟩⟩ + o(η)
i=1
=η
N X
2 ∥∇θi Jteam (θ)∥2 + η
i=1
=η
N X
N X
* ∇θi Jteam (θ),
i=1 2
∥∇θi Jteam (θ)∥2 + η
i=1
=η
N X
2
N X K X
i=1
N X i=1
12
+ pi,k ∇θi Ji,k (θi )
+ o(η)
k=1
pi,k ⟨∇θi Jteam (θ), ∇θi Ji,k (θi )⟩ + o(η)
i=1 k=1
∥∇θi Jteam (θ)∥2 + η
K X
⟨pi , bi ⟩ + o(η),
PN PN Let p̃i = pi − p̄ and b̃i = bi − b̄. Then i=1 p̃i = 0 and i=1 b̃i = 0. Let P = (p̃1 , . . . , p̃N )⊤ ∈ RN ×K be the centered preference matrix, and then crossing term can be decomposed as follows. N X
Jteam (θnew ) − Jteam (θ) = η
2
∥∇θi Jteam (θ)∥2 + η
i=1 N X
=η
N D E X p̄ + p̃i , b̄ + b̃i + o(η) i=1
2 ∥∇θi Jteam (θ)∥2 + η
N ⟨p̄, b̄⟩ +
i=1 N X
≥η
N X
N X
⟨p̃i , b̃i ⟩
+ o(η)
i=1 2 ∥∇θi Jteam (θ)∥2 + η
N ⟨p̄, b̄⟩ + κ
i=1
=η
!
N X
! ∥p̃i ∥22
+ o(η)
(apply Assumption A.1)
i=1 2
∥∇θi Jteam (θ)∥2 + ηN p̄⊤ b̄ + ηκ∥P ∥2F + o(η)
i=1
where ∥ · ∥2F is the matrix Frobenius norm. Last, we show that the Frobenius norm of centered preference similarity matrix is exactly pairwise preference distance. N 1 1 X 2 ∥pi − p̄∥22 ∥P ∥F = N N i=1 N
=
1 X ∥pi ∥22 − ∥p̄∥22 N i=1
=
N N N 1 X 1 XX ⊤ ∥pi ∥22 − 2 p pj N i=1 N i=1 j=1 i
=
N N 1 XX ∥pi ∥22 + ∥pj ∥22 − 2p⊤ i pj 2 2N i=1 j=1
=
N N 1 XX ∥pi − pj ∥22 = Dp . 2N 2 i=1 j=1
(by the variance identity)
(by bilinearity)
Ignoring the higher-order term under a sufficiently small step size η, we obtain the following lower bound N X 2 Jteam (θnew ) − Jteam (θ) ≥ η ∥∇θi Jteam (θ)∥2 + ηN p̄⊤ b̄ + ηκN Dp i=1
A.3
Proof of Lemma 4.3 in the main text
Lemma A.5 (Continuity of preference-conditioned stationary solutions). Suppose that θ∗ is a local Nash equilibrium of G(p), i.e., ∇θi Ui (θ∗ ; pi ) = 0,
i = 1, . . . , N.
Assume that the joint gradient (∇θ1 U1 , . . . , ∇θN UN ) is continuously differentiable near (θ∗ , p) with non-singular Jacobian. Then there exists a neighborhood U of p and a unique continuously differentiable mapping θ(·) : U → Θ such that θ(p) = θ∗ and, for all p̃ ∈ U, ∇θi Ui (θ(p̃); p̃i ) = 0,
i = 1, . . . , N.
Proof. Define the joint stationarity mapping F (θ, p) := ∇θ1 U1 (θ; p1 ), . . . , ∇θN UN (θ; pN ) . 13
Since θ∗ is a local Nash equilibrium of G(p), by definition F (θ∗ , p) = 0. By assumption, F is continuously differentiable near (θ∗ , p), and its Jacobian ∇θ F (θ∗ , p) is nonsingular. Therefore, by the Implicit Function Theorem, there exists a neighborhood U of p and a unique continuously differentiable mapping θ(·) : U → Θ such that θ(p) = θ∗ and ∀ p̃ ∈ U .
F (θ(p̃), p̃) = 0, Expanding the definition of F , this gives ∇θi Ui (θ(p̃); p̃i ) = 0,
i = 1, . . . , N,
which proves the result. A.4
Proof of Theorem 4.6 in the main text
Theorem A.6 (Equilibrium tracking). Consider the iterative policy update θit+1 = θit + η∇θi Ui (θt ; pt ),
i = 1, . . . , N.
Under Assumption 4.4 and Assumption 4.5 on page 5 of the main text, the tracking error et := ∥θt − θ(pt )∥ satisfies et+1 ≤ ρet + C∥pt+1 − pt ∥. In particular, if ∥pt+1 − pt ∥ ≤ δ, then lim sup et ≤ t→∞
C δ. 1−ρ
Proof. For any t ≥ 0, by adding and subtracting θ(pt ), we have et+1 = ∥θt+1 − θ(pt+1 )∥ ≤ ∥θt+1 − θ(pt )∥ + ∥θ(pt ) − θ(pt+1 )∥. First, by Assumption 4.4, the policy update is locally contractive around θ(pt ). Therefore, ∥θt+1 − θ(pt )∥ ≤ ρ∥θt − θ(pt )∥ = ρet . Second, by Assumption 4.5, the equilibrium path changes Lipschitz continuously with the preference profile, so ∥θ(pt ) − θ(pt+1 )∥ ≤ C∥pt+1 − pt ∥. Combining the two bounds gives et+1 ≤ ρet + C∥pt+1 − pt ∥. If ∥pt+1 − pt ∥ ≤ δ, then et+1 ≤ ρet + Cδ. Applying this inequality recursively gives et ≤ ρet−1 + Cδ ≤ ρ2 et−2 + Cδ(1 + ρ) ≤ ··· ≤ ρt e0 + Cδ
t−1 X
ρk
k=0
1 − ρt = ρt e0 + Cδ . 1−ρ Since ρ ∈ (0, 1), take lim supt→∞ , and we obtain lim sup et ≤ t→∞
14
C δ. 1−ρ
(a) MOMPE
(b) Catch/Escort
(c) MOMAWalker
(d) SMAC
Figure 7: Environments used in our experiments. Table 3: Reward and success details Environment
Dimension Team reward
Individual reward
Success criterion
Cooperative Spread 2A2O
Sparse task reward or [- distance to landmark Two agents occupy two punishment: +1 when 1, - distance to landmark distinct landmarks. both landmark reached, 2]. −1 on collision.
Safe Predator-Prey 4A3O
Sparse task reward or [progress to prey 1, All prey are captured. punishment: +1 when progress to prey 2, catch prey, −1 on colli- close distance to obstasion or bump into obsta- cle/other predator]. cle.
Catch
4A2O
Sparse task reward or [progress to target, - catch the target punishment: +1 when close distance to other catch target, −1 on col- drone] lision.
Escort
4A2O
only collision penalty
MOMAWalker
3A2O
Package progress dis- [Walker progress, stabil- Not Applicable tance, fall/drop penal- ity penalty] ties.
SMAC
xA2O
+1 on success.
OpenCDA-MARL
≤10A2O
Route success, progress, [efficiency, safety/ step penalty, and colli- interaction risk] sion penalty.
B
[- distance to target, dis- Not Applicable tance to other drones]
[damage dealt, -health Defeat all enemies within loss] episode time Route completion without collision.
Implementation Details
To support reproducibility, we provide detailed environment specifications, training hyperparameters, and pseudocode in the appendix. The source code, including environment modifications and training scripts, is available at https://github.com/PengxinWang/PrefMARL. B.1
Environment Setting
This subsection provides additional details of the environments used in our experiments. We describe the task setup, action space, reward design, and success criterion for each environment. The detailed reward design and success criterion are summarized in Table 3. Cooperative Spread and Safe Predator-Prey. We use two particle-world tasks with discrete actions. Spread is a two-agent, two-landmark variant of MPE simple spread, where agents must coordinate to cover distinct landmarks. Predator-Prey uses four predators, two moving prey, and 15
obstacles in a bounded 2D arena. Compared with Spread, Predator-Prey introduces moving targets, obstacle avoidance, and a three-objective reward vector that separates progress toward the two prey from safety. Catch and Escort. Catch and Escort are continuous-control drone tasks adapted from CrazyRL/MOMALand. In both environments, four drones move in a bounded 3D space with 3D velocity actions. Catch focuses on approaching and capturing a target while avoiding unsafe proximity to other drones. Escort instead requires drones to maintain an escort formation around a moving target as it travels toward a goal. Thus, Catch emphasizes interception, while Escort emphasizes sustained formation control. MOMAWalker. MOMAWalker is adapted from the Multiwalker domain. Three walkers jointly transport a package across terrain using continuous joint-torque actions. We use this environment to evaluate coordinated continuous control under a shared physical task, where agents must trade off forward progress with package and body stability. SMAC. SMAC comprises cooperative StarCraft Multi-Agent Challenge maps, including 3m, 2s3z, and 8m, and difficulty is set as 5. In these tasks, agents control decentralized units using discrete actions with action masks. We use a two-dimensional combat reward vector based on damage dealt and health loss, together with a sparse team-success signal indicating whether all enemy units are defeated within the episode limit. OpenCDA-MARL. OpenCDA-MARL extends OpenCDA with a MARL controller for CARLA intersection control. The environment is a four-way intersection on a local OpenDRIVE map, with traffic entering from north, south, east, and west, and each episode lasts at most 2400 simulation steps. During training, we use live traffic with at most 10 active MARL-controlled CAVs; for checkpoint evaluation, we replay a fixed traffic file to compare algorithms under the same arrival pattern. B.2
OpenCDA-MARL/CARLA Experimental Details
In Carla’s world, connected autonomous vehicles (CAVs) are spawned at the intersection and scheduled to reach their destinations. Each CAV follows a fixed route generated by the OpenCDA planner. The learned action is a one-dimensional target speed command in km · h−1 , clipped to [0, 45], and the low-level planner tracks the resulting speed along the assigned route. Thus the experiment tests learned yielding and crossing-order behavior under speed control, not route selection. The base observation includes relative position to the intersection, absolute position, lane position, heading angle, distance to the intersection, distance to the front vehicle, and waypoint-buffer information. Preference-conditioned variants append a two-dimensional preference vector to this state. We constructed 2 reward profiles, cooperative and competitive, for both cooperative and competitive runs. Both profiles use a sparse event structure with collision penalty −400, success reward 300, step penalty −0.8, and progress shaping scaled by 0.4. For cooperative preference experiments, reward components are grouped into two objectives: efficiency, containing progress, success, and step penalty; and safety, containing collision. For competitive preference experiments, the second objective is written as interaction risk and is also driven by collision, while time-to-collision, yielding, and clearance are tracked as metric-only signals. The reported metrics are scalarized utility, success rate, collision rate, and throughput in vehicles per hour. Utility is the accumulated scalarized objective value under the assigned preference, success rate is the percentage of vehicles that complete their route, collision rate is the percentage of vehicles that collide, and throughput is computed from successful completions per simulated hour. B.3
Training Details
Algorithm 2 gives the full training procedure of PCMA. Training setup. Unless otherwise stated, all on-policy methods use the same PPO-style training setup. We use Adam for both actor and critic optimization, and all MLPs use Tanh activations. For PCMA and MAPPO-style baselines, the actor is a shared preference-conditioned MLP with two hidden layers of size 64. The actor separately encodes the local observation and the preference, then 16
fuses the two embeddings before the action head. The preference embedding dimension is 32. The team critic is a centralized vector critic conditioned on the global state and preference, with two hidden layers of size 128. When individual critics are enabled, they use the same centralized critic architecture but output agent-wise vector values. Table 5 lists the additional hyperparameters used by PCMA. Training time. All PCMA experiments were run on an internal Linux workstation with NVIDIA RTX PRO 6000 Blackwell GPUs (98GB GPU memory each). Each training run used one GPU and one training process. Non-SMAC experiments used 4 parallel environment workers, while SMAC used 1 environment worker due to the single-environment action-mask path. The measured wall-clock training time for one PCMA seed was 11.6 minutes for Cooperative Spread, 21.2 minutes for Catch, 1.12 hours for Safe Predator-Prey, 4.49 hours for MultiWalker, 1.12 hours for SMAC-3m, 2.31 hours for SMAC-2s3z, and 1.54 hours for SMAC-8m. Main comparisons use three seeds, so the reported PCMA training compute is approximately three times these per-seed costs for each environment. Execution-time evaluation and rendering use fixed policy rollouts without gradient updates and were run on the same machine; these jobs are lightweight compared with training and typically finish within minutes per checkpoint. Table 4: Common training setup. Hyperparameter Value Optimizer Actor learning rate Critic learning rate Discount factor γ GAE parameter λ Rollout length PPO epochs Minibatches per update PPO clipping coefficient ϵ Value loss coefficient Entropy coefficient Max gradient norm Actor hidden layers Preference encoder dimension Critic hidden layers Actor parameter sharing Agent-ID embedding
Adam 3 × 10−4 3 × 10−4 0.99 0.95 256 4 4 0.2 0.5 0.01 0.5 [64, 64] 32 [128, 128] Fully shared across agents Dimension 8
Table 5: PCMA-specific hyperparameters. Environment λ1 λ2 Simple Spread Catch Escort Predator-Prey MultiWalker SMAC
17
0.10 0.05 0.05 0.05 0.02 0.02
0.20 0.10 0.10 0.20 0.20 0.10
Algorithm 2 Preference Coordinated Multi-Agent Policy Optimization (PCMA) Require: number of agents N , objective dimension d, rollout length T Require: actor πθ , preference planner ϕψ , team critic Vηteam , individual critics {Vξi }N i=1 1: for iteration = 1, 2, . . . do 2: for t = 0, . . . , T − 1 do 3: for agent i = 1, . . . , N do 4: Observe local observation oi,t 5: Generate inherent preference pi,t ∼ Dirichlet ϕψ (oi,t , p̄) 6: 7: 8: 9: 10: 11: 12: 13:
Sample action ai,t ∼ πθ (· | oi,t , pi,t ) end for Execute joint action at = (a1,t , . . . , aN,t ) Store team N (ot , p̄, {pi,t }N , {rind i=1 , at , log πθold , rt i,t }i=1 , ot+1 ) end for Estimate team vector returns R̂team and team advantages Ateam with GAE Estimate individual vector returns R̂ind and advantages Aind with GAE i i Scalarize the team advantage: Ateam = p̄⊤ Ateam
14:
Compute each agent’s individual utility advantage: ind AUi = p⊤ i Ai
15: 16: 17: 18:
Form the actor advantage:
Ai = Ateam + λind AUi
for PPO epoch = 1, . . . , K do for minibatch B do Update critics by minimizing Lcritic = Lteam + Lind .
19:
Compute PPO ratio
20:
πθ (ai | oi , pi ) . πθold (ai | oi , pi ) Update actor with clipped PPO objective: ρi =
Lactor = −EB [min (ρi Ai , clip(ρi , 1 − ϵ, 1 + ϵ)Ai )] − βent H(πθ ). 21: Compute preference diversity regularizer Dα ({pi }N i=1 ) 22: Update planner with Lplan . 23: end for 24: end for 25: end for
18
C
Illustrative Examples
C.1
Example for Pareto-Nash Equilibrium
For example, consider a two agent two objective game. Each agent follows a Gaussian policy with fixed variance, a1 ∼ N (µ1 , 1) and a2 ∼ N (µ2 , 1), where µ1 , µ2 ∈ [−1, 1]. Define team reward as i h i h . (r1 , r2 ) = exp − 12 (a1 − 1)2 + 12 (a2 − 1)2 , exp − 12 12 (a1 + 1)2 + 2(a2 + 1)2 In this example, every policy parametrized by (µ1 , µ2 ) ∈ [−1, 1]2 is a Pareto–Nash equilibrium, since any unilateral change that improves one objective necessarily decreases the other. However, most of these equilibria are not pareto optimal C.2
A Toy Example of Better Team Improvement with Agent-specific Preferences
We provide a simple two-agent Gaussian policy example to illustrate why diverse preferences can induce better team solutions than a single shared preference. Each agent uses a Gaussian policy with fixed variance: ai ∼ πi (·; µi ) = N (µi , σ), i ∈ {1, 2}, where µi is the only optimizable parameter. Let µi ∈ [−1.5, 1.5], σ = 1, and define the expected team return as R(µ) = Ea∼πµ [r(a)], where r(a) = (r1 (a), r2 (a)). Agent 1 and agent 2 use preferences w1 = (α, 1 − α),
w2 = (β, 1 − β).
The reward components are (a1 − 1)2 1 (a2 − 1)2 r1 (a1 , a2 ) = √ exp − − , 4 6 3 1 (a1 + 1)2 (a2 + 1)2 r2 (a1 , a2 ) = √ exp − − . 6 3 4.5 Starting from the same initial policy mean µ0 = (0, 0), we compare gradient ascent under a shared preference and under agent-specific preferences. With w1 = w2 = (0.5, 0.5), the trajectory converges to µshared = (−1.000, −1.000), with R(µshared ) = (0.905, 0.905) and Uteam = 0.905. In contrast, with agent-specific preferences w1 = (1.000, 0.000) and w2 = (0.000, 1.000), the trajectory converges to µagent = (1.000, 1.000), with R(µagent ) = (1.105, 1.105) and Uteam = 1.105, where Uteam (R) = 12 (R1 + R2 ). Thus, agent-specific preferences improve the final team utility by 0.200, or approximately 22.1%.
19