Conceptio › Archive › arXiv CS

arXiv CSopen access

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

arXiv:2604.08174v1 [cs.LG] 9 Apr 2026

Teng Pang1 , Zhiqiang Dong1 , Yan Zhang1 , Rongjian Xu1 , Guoqiang Wu *1 2 , and Yilong Yin *1 * 1

Corresponding Author

School of Software, Shandong University, Jinan, China 2 Luke EI, Jinan, China [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] April 10, 2026 Abstract

Offline multi-agent reinforcement learning (MARL) aims to learn the optimal joint policy from pre-collected datasets, requiring a trade-off between maximizing global returns and mitigating distribution shift from offline data. Recent studies use diffusion or flow generative models to capture complex joint policy behaviors among agents; however, they typically rely on multi-step iterative sampling, thereby reducing training and inference efficiency. Although further research improves sampling efficiency through methods like distillation, it remains sensitive to the behavior regularization coefficient. To address the above-mentioned issues, we propose Value Guidance Multi-agent MeanFlow Policy (VGM2 P), a simple yet effective flow-based policy learning framework that enables efficient action generation with coefficient-insensitive conditional behavior cloning. Specifically, VGM2 P uses global advantage values to guide agent collaboration, treating optimal policy learning as conditional behavior cloning. Additionally, to improve policy expressiveness and inference efficiency in multi-agent scenarios, it leverages classifier-free guidance MeanFlow for both policy training and execution. Experiments on tasks with both discrete and continuous action spaces demonstrate that, even when trained solely via conditional behavior cloning, VGM2 P efficiently achieves performance comparable to state-of-the-art methods.

Introduction

Multi-agent reinforcement learning (MARL) [1, 2, 3] is primarily applied to multiagent system tasks in real-world scenarios, such as multi-player strategy games [4], 1

multi-robot control [5], and traffic control [6]. The key challenge is how to effectively express powerful policies and facilitate communication among agents during interactions with the environment, thereby maximizing the overall reward of the system. However, due to the complexity of the real world, real-time interaction with the environment often involves risks and high costs, especially in large-scale tasks. Therefore, offline MARL [7, 8], which leverages pre-collected data for multi-agent policy learning, has gradually gained increasing attention. Similar to single-agent offline RL, offline MARL faces a series of distribution shift challenges. First, the limited and insufficient coverage of offline data makes agents more likely to access out-of-distribution (OOD) data during training. This issue becomes even more challenging as the number of agents grows. Additionally, the absence of real-time interaction with the environment hampers the proper exploration of the learned policies, thereby exacerbating extrapolation errors. Beyond these challenges, another key issue is how to effectively mine and utilize the communication between agents from the offline dataset. To address these challenges, existing research integrates the regularization methods from single-agent offline RL into the Centralized Training with Decentralized Execution (CTDE) framework [7, 8, 9, 10]. This approach ensures communication between agents while limiting OOD data access and mitigating extrapolation errors. Additionally, recent studies incorporate the impact of agents’ balance on policy learning, using sequential policy updates to further restrict OOD data access and extrapolation [11, 12, 13]. While these methods effectively mitigate distribution shifts and communication collaboration issues in multi-agent systems, the commonly used Gaussian policy fails to capture the multi-modal nature of the joint policy, thereby constraining the policy expressiveness of the agents and the scope of their applications. With the recent development of generative models, some studies apply models like diffusion [14] and flow matching [15] to offline MARL, particularly in policy modeling [16, 17] and trajectory generation [18]. Although these models are powerful, their complex sampling processes incur high generation costs, and the generated actions cannot be directly used for policy updates. Besides, some research studies one-step distillation [19] or one-step generative models [20], such as MeanFlow [21], as a behavioral regularization method to efficiently sample and generate optimal actions, but such approaches are highly sensitive to the exploration-exploitation trade-off and heavily dependent on the regularization coefficient. To address the aforementioned issues, we propose a simple offline multi-agent policy learning method, Value Guidance Multi-agent Meanflow Policy(VGM2 P), which uses the advantage value as guidance and treats training the optimal policy as conditional behavior cloning. In the training phase, to reduce sensitivity to the exploration–exploitation coefficient, VGM2 P calculates the global advantage value of offline data and integrates it into MeanFlow-based individual policies training with classifierfree guidance (CFG). For decentralized execution, to enhance exploration of the learned policies and the efficiency of action generation, VGM2 P generates actions for each agent through one-step sampling based on a preset condition. Experimentally, we apply VGM2 P to general offline MARL benchmarks, and a series of experiments show that VGM2 P, using only conditional behavior cloning, performs comparably to existing advanced methods. 2

Our contributions are summarized as follows: • We propose VGM2 P, a simple yet effective multi-agent training method that trains the optimal joint policy through conditional behavior cloning. • To enhance policy expressiveness and action generation efficiency, we leverage the classifier-free guidance MeanFlow for condition-based behavior cloning. • To enable agent collaboration, we incorporate the global advantage value as a guidance condition into conditional behavior cloning. • Experimental results demonstrate that, in both discrete and continuous action environments, our method efficiently achieves performance comparable to existing advanced algorithms.

Preliminaries

2.1

Problem setup

In this work, we model multi-agent reinforcement learning (MARL) as a decentralized partially observable Markov decision process (Dec-POMDP) represented by a tuple i N M = (I, S, {Oi }N i=1 , {A }i=1 , Ω, T , R, γ, ρ0 ). Here, I = {1, 2, · · ·, N } denotes a set of agents; S denotes the global state space; Oi and Ai denote the observation space and action space of the agent i ∈ I, A = A1 × ... × AN denotes the joint action space and a = (a1 , ..., aN ) ∈ A denotes the joint action, O and o similarly represent the corresponding joint observation space and joint observation; Ω(s, i) : S × I → Oi denotes observation function of the agent i that can observe oi ∈ Oi in current state s ∈ S and we set Ω(s) : S → O1 × ... × ON for simplicity; T (s′ |s, a) : S × A × S → [0, 1] denotes the state transition function; R(s, a) : S × A → R denotes the global reward model and rt = R(st , at ) denotes the global reward at time t; γ is the discounted factor and ρ0 is the initial state distribution. In Dec-POMDP, each agent i can only observe oit at each transition time t and execute the action ait according to its own policy π i (ait |oit ) : Oi × Ai → [0, 1]. The goal of MARL is to N learn the joint policy π tot = (π 1 , ..., πP ) that maximizes the discounted cumulative ∞ reward Jπtot = Es0 ∼ρ0 ,πtot (·|Ω(st )),T [ t=0 γ t rt ]. For the P joint policy π tot , we have ∞ tot a global Q-value function Qπtot (o, a) = Eπtot (·|Ω(st )),T [ t=0 γ t rt |o0 = o, a0 = tot a] and its corresponding value function Vπtot (o) = Ea∼πtot [Qtot π tot (o, a)]. In offline scenarios, we have a static dataset Doff = {Dβi }N collected by N agents following i=1 behavior joint policy πβtot = (πβ1 , ..., πβN ). Each agent i provides a sub-dataset Dβi = i {(oim , aim , o′ m , rm )}M m=1 consisting of M transition tuples. For the single-agent case, we drop the agent identifier and denote Vπ (o), Qπ (o, a), Dβ , and πβ for simplicity.

2.2

Centralized Training with Decentralized Execution

Centralized Training with Decentralized Execution (CTDE) [22] is a widely adopted training paradigm in MARL, where agents are trained jointly and execute independently at inference time. Under CTDE, value decomposition [23, 24], as a commonly 3

used training method, improves scalability by decomposing the joint observation-action space. This method typically relies on the Individual-Global-Max (IGM) principle [2], which requires that combining the individually optimal actions implied by each agent’s Q-value function Qiπi (oi , ai ) yields the optimal joint action: arg max Qtot π tot (o, a) a

N N = (arg max Q1π1 (o1 , a1 ), ..., arg max QN π N (o , a )). 1 aN

(1)

The IGM principle guarantees consistency between local optima and the global optimum.

2.3

Flow Matching and MeanFlow

Flow Matching [15] is a generative model that learns a velocity field to match the flow between a prior distribution and a target distribution. Formally, given data x ∼ ptarget and prior ϵ ∼ pprior (e.g., ϵ ∼ N (0, I)), we consider a linear schedule flow path xt = (1 − t)x + tϵ at time t ∈ [0, 1], which can lead to the sample-conditional velocity vc = ϵ − x by computing the time-derivative. In Flow Matching, the parameterized velocity network vθ is optimized by minimizing the following loss function: LFM (θ) = Ex,t,ϵ ||vθ (xt , t) − (ϵ − x)||2 ,

(2)

where t is sampled from the uniform distribution (i.e., t ∼ Unif([0, 1])) and ϵ is sampled from Gaussian distribution (i.e., ϵ ∼ N (0, I)). Since an intermediate sample xt can be formed as different (x, ϵ) pairs, Flow Matching essentially learns a marginal velocity field v(xt , t) ≜ Ep(xt |x,ϵ) [vc |xt ] over all possibilities p(xt |x, ϵ). The generative process in Flow Matching is described by the ordinary differential equation (ODE) d dt xt = vθ (xt , t) for xt . This ODE starts from x1 = ϵ to x0 = x. Unlike Flow Matching that models instantaneous velocity v(xt , t), MeanFlow [21] defines an average velocity between two time points t and r: Z t 1 uθ (xt , r, t) ≜ v(xτ , τ )dτ. (3) t−r r To learn the average velocity, Meanflow models it with a parameterized network uθ and trains with the following loss: LMF (θ) = Ex,t,r,ϵ ||uθ (xt , r, t) − sg(utgt )||2 ,

(4)

where (t, r) ∼ Unif([0, 1]), sg denotes a stop-gradient operation and utgt = v(xt , t) − d d u(xt , r, t) is the target velocity. To compute the dt u(xt , r, t), MeanFlow (t − r) dt further extends this partial derivative and finally implements the calculation using the Jacobian-vector product (JVP): d u(xt , r, t) = v(xt , t)∂x u(xt , r, t) + ∂t u(xt , r, t). dt

(5)

After training, MeanFlow can achieve one-step sampling, x0 = x1 − uθ (x1 , 0, 1), by simply setting (r, t) = (0, 1). 4

2.4

Behavior Regularization in Offline RL

In single-agent offline RL, policy training is typically achieved by the actor-critic framework under behavior regularization, resulting in a form of constrained policy optimization with behavior policy πβ or offline dataset Dβ : LQπ = E (o,a,o′ ,r)∼Dβ , [Qπ (o, a) − (r + γQπ (o′ , a′ ))]2 , a′ ∼π(·|o′ ) Lπ = −E (o,a)∼Dβ , Qπ (o, aπ ) − λDf (π(o|s)||πβ (o|s)) ,

(6) (7)

aπ ∼π(·|o)

where Df is used to measure the divergence between the policy π and the behavior policy πβ . To prevent excessive access to OOD actions during training, some works focus on weighted behavior cloning, such as AWAC [25, 26], which derives the representation of the optimal policy π ∗ of Eq. (7) when Df is the KL divergence: π ∗ (a|s) = R

exp( λ1 Qπ (s, a)) πβ (a|s). π (a′ |s) exp( λ1 Qπ (s, a′ ))da′ a′ β

(8)

Methodology

In this section, we present our method VGM2 P, a simple yet effective way to represent the optimal joint policy through conditional behavior cloning with MeanFlow for offline MARL. Based on the IGM principle, the optimal joint action can be derived from the optimal actions of individual agents. Therefore, to get the optimal joint action, we can first obtain the optimal policy for each agent and then use the IGM principle to derive the joint optimal policy. To achieve the above objective, VGM2 P consists of the following three aspects: 1) deriving the optimal policy for each agent through behavior policy conditioning on the advantage value; 2) modeling these policies with MeanFlow; and 3) obtaining the joint optimal policy based on the IGM principle.

3.1

Value Guidance Behavior Policy

In single-agent offline RL, policy improvement depends on the behavior policy [26, 27, 28]. When the policy is represented as a distribution, the optimal policy and the behavior policy are positively correlated, as shown in Eq.(8). According to Bayes’ theorem, there is a similar correlation between the conditional distribution and its corresponding prior distribution. Based on this insight, we can use a conditional behavior policy to approximate the optimal policy: Proposition 1 (Value-Guidance Behavior Policy). Given a behavior policy πβ (a|o) and the optimal policy π ∗ (a|o) derived from Eq.(8), for any variable c ∈ C and its related distribution p(c|o, a), when there exists c∗ ∈ C satisfying p(c = c∗ |o, a) ∝ exp( λ1 Qπ (o, a)), then we have the conditional behavior policy πβ (a|o, c = c∗ ) = π ∗ (a|o).

The proof is provided in Appendix B.1. Proposition 1 implies that, when we have a condition c∗ positively correlated with the value exp( λ1 Qπ (o, a)) to control behavior policy sampling, we can achieve the same distribution as the optimal policy π ∗ (a|o) derived from Eq.(8). To achieve p(c = c∗ |o, a) ∝ exp( λ1 Qπ (o, a)), we can define the advantage value function Aπ (o, a) = Qπ (o, a) − Vπ (o) and simply set c = 1 if Aπ (o, a) ≥ 0 else c = 0, which is similar to [29] in single-agent scenario. Then, we can define p(c = 1|o, a) :=

exp( λ1 Qπ (o, a)) . 1 exp( λ Qπ (o, a)) + exp( λ1 Vπ (o))

(9)

Intuitively, during training, when (o, a) has a non-negative advantage value, we set c = 1 to indicate that the action a comes from the optimal policy. Then, for the execution, we can fix c = 1 to sample from the optimal policy. In multi-agent settings, we can replace the local value function Qiπi with a global tot tot value function Qtot π tot and use the global advantage value Aπ tot (o, a) = Qπ tot (o, a) − tot Vπtot (o) as the guidance condition, enabling cooperative policy execution, which we will discuss in detail in Section 3.3.

3.2

Value Guidance MeanFlow Policy

To enhance the expressive ability of the policy, we present using MeanFlow to model it. As a specific implementation of Continuous Normalizing Flows (CNFs), MeanFlow has been widely adopted in image generation due to its ability to achieve both high efficiency and high-quality sample generation. Specifically, for a single-agent and its observation-action pairs (o, a) ∼ Dβ , we construct the action ak = (1 − k)a + kϵ at the timestep k ∈ [0, 1] of the flow process along with its sample-conditional velocity vc (ak , k|o) = ϵ − a and the parameterized average velocity uθ (ak , r, k|o) = Rk 1 k−r r vc (ak , k|o). During training, we formulate a MeanFlow-based behavior cloning loss as follows: LBC−MF (θ) = E(o,a)∼Dβ ,k,r,ϵ ||uθ (ak , r, k|o) − sg(utgt )||2 ,

(10)

d uθ (ak , r, k|o) where (k, r) ∼ Unif([0, 1]), ϵ ∼ N (0, I) and utgt = vc (ak , k|o)−(k−r) dk is the target velocity. Based on the Proposition 1, the optimal policy derived from Eq.(8) can be approximated by a behavior policy augmented with value-guidance conditioning. Based on it, we propose the Value-Guided MeanFlow Policy(VGMP), which is trained via a parameterized conditional average velocity ucθ (ak , r, k|o, c) and optimized through behavior cloning under the Classifier-free Guidance (CFG) MeanFlow: 2 LVGMP (θ) = E(o,a)∼Dβ ,k,r,ϵ ||ucθ (ak , r, k|o, c) − sg(ucfg tgt )|| ,

(11)

d c where c is the value-guidance condition, ucfg tgt = vcfg (ak , k|o, c)−(k−r) dk uθ (ak , r, k|o, c) is the target velocity and vcfg (ak , k|o, c) = ωvc (ak , k|o, c) + (1 − ω)ucθ (ak , k, k|o) is the ground-truth field that a weighted combination of the class-conditional field

vc (ak , k|o, c) and the class-unconditional field ucθ (ak , k, k|o) with a guidance weight ω under CFG. Following [21], we replace the class-conditional field vc (ak , k|o, c) with the sample-conditional velocity vc (ak , k|o) = ϵ − a and set the class-unconditional field ucθ (ak , k, k|o) = ucθ (ak , k, k|o, c = 1) to allow exploration of the offline dataset’s behavior even when using the optimal policy. For the execution, we sample a1 ∼ N (0, I) and set c = 1, generating each action with the conditional average velocity: ar = ak − (k − r)ucθ (ak , r, k|o, 1).

(12)

To improve generation efficiency, we can directly use one-step sampling, i.e., a = a0 = a1 − ucθ (a1 , 0, 1|o, 1).

3.3

Value Guidance Multi-agent MeanFlow Policy

In MARL, policy learning seeks to maximize the global value of joint actions, which requires explicitly obtaining and leveraging the global value during value guidance. However, achieving these goals suffers from two key limitations: first, the computational cost of the joint observation-action space typically grows exponentially with the number of agents; second, as a scalar, the global value cannot provide agent-specific guidance to all agents simultaneously. Under the IGM principle, the joint action that maximizes the global Q-value can be decomposed into actions that individually maximize each agent’s local Q-value. Therefore, to reduce computational cost and satisfy the IGM principle, we replace the global Q-value with individual agents’ Q-values. Specifically, we approximate the global Q-value function Qtot individual agents’ π tot (o, a) by summing parameterized PN i i i tot value functions Qϕi (o , a ), like VDN [23], i.e., Qπtot (o, a) = i=1 Qiϕi (oi , ai ), and then train them using global Temporal-Difference (TD) error: L({ϕi }N i=1 ) = E (o,a,o′ ,r)∼Doff , a′i ∼π tot (·|o′ )

N hX

Qiϕi (oi , ai )

i=1

− (r + γQiϕ̄i (o′i , a′i ))

i 2

(13)

where Qiϕ̄i denotes a slowly updated target Q-value network for stabilizing training. Since we approximate the global Q-value with individual agents’ Q-values, we can use these Q-values to guide the training of conditional behavior cloning. Proposition 2. Assuming that the behavior joint policy πβtot (a|o) and the global QQN i i i tot tot value Qtot i=1 πβ (a |o ) and Qπ tot (o, a) = π tot (o, a) are decomposable, i.e., πβ (a|o) = PN i i i tot,∗ (a|o), when the distribution pi (ci |oi , ai ) i=1 Qϕi (o , a ). For the optimal joint policy π 1 i i i i i i,∗ i i satisfies p (c = c |o , a ) ∝ exp( λ Qϕi (o , a )) for each agent i, then we have QN i i i i i,∗ tot,∗ (a|o). i=1 πβ (a |o , c = c ) = π The proof is provided in Appendix B.2. Proposition 2 indicates that under the value decomposition, the joint behavior policy with value guidance condition is the 7

Algorithm 1 Centralized Training of VGM2 P Input: Offline MARL dataset Doff ; individual conditional average velocity model i N {ucθi }N i=1 , individual Q-value model {Qϕi }i=1 ; guidance weight ω c N Output: {uθi }i=1 i N 1: Initialize {ucθi }N i=1 and {Qϕi }i=1

2: while not converged do 3: Sample tuple {(oi , ai , o′i , r)}N i=1 ∼ Doff 4: // Train individual Q-value model {Qiϕi }N i=1

′i ′i c ′i ′i Sample a′i 1 ∼ N (0, I), set a = a1 − uθi (a1 , 0, 1|o , 1) 6: Train Q-value model {Qiϕi }N i=1 with Eq. (13) 7: // Train individual conditional average velocity model {ucθi }N i=1 8: Sample âi1 ∼ N (0, I) and generate current action âi = âi1 − ucθi (âi1 , 0, 1|oi , 1) 9: Compute advantage value Ai (oi , ai ) = Qiϕi (oi , ai ) − Qiϕi (oi , âi ) and set the condition ci = 1 if Ai (oi , ai ) ≥ V (oi ) else ci = 0 10: Sample ai1 ∼ N (0, I), (k, r) ∼ Unif([0, 1]) 11: Train conditional average velocity model {ucθi }N i=1 with Eq. (11) 12: end while

Algorithm 2 Decentralized Execution of VGM2 P c N Input: local observation {oi }N i=1 , conditional average velocity model {uθi }i=1 Output: {ai }N i=1 1: Sample {ai1 }N i=1 ∼ N (0, I) 2: Compute ai = ai1 − ucθi (ai1 , 0, 1|oi , 1) for each agent i joint optimal policy π tot,∗ (a|o). Therefore, by considering the optimality of individual agents under the IGM principle, we can achieve the global optimum. In terms of implementation, we use the advantage value Ai (oi , ai ) = Qiϕi (oi , ai ) − i Qϕi (oi , âi ) where âi ∼ π i (·|oi ) to set the condition ci and fix the condition ci = 1 during execution for each agent. In addition, to further improve agent communication and collaboration, we share the parameters of both the policy network and the value function network across all agents. Combining the use of MeanFlow, we name the above approach Value Guidance Multi-agent MeanFlow Policy (VGM2 P), and present the complete training and execution processes in Algorithm 1 and 2, respectively.

Related Work

4.1

Offline Multi-Agent Reinforcement Learning

Offline multi-agent reinforcement learning (MARL) extends offline RL from singleagent to multi-agent settings, aiming to enable effective exploration while staying within the offline data distribution and preserving coordination among agents. Existing methods typically build on value and policy decomposition, reducing offline 8

SMACv1

Dataset

Extension of offline SARL BC(Diffusion) BC(FM) BC(MF)

MA-BCQ

MA-CQL

MADiff

Offline MARL DoF MAC-Flow

VGM2 P

3m-Good 3m-Medium 3m-Poor

16.0±1.0 8.2±0.8 4.4±0.1

19.5±0.5 13.3±0.7 4.2±0.2

20.0±0.0 14.7±1.5 4.5±0.1

19.8±0.4 15.0±2.8 4.2±0.3

3.7±1.1 4.0±1.0 3.4±1.0

19.1±0.1 13.7±0.3 4.2±0.1

19.3±0.5 16.4±2.6 10.3±6.1

19.8±0.2 18.6±1.2 10.9±1.1

19.8±0.2 18.0±3.2 10.6±2.2

19.5±0.7 16.9±1.1 14.9±1.5

8m-Good 8m-Medium 8m-Poor

16.7±0.4 10.7±0.5 5.3±0.1

19.4±0.5 18.6±0.6 4.8±0.2

19.5±0.2 18.2±0.8 4.9±0.1

19.5±0.6 18.7±0.8 4.8±0.1

4.8±0.6 5.6±0.6 3.6±0.8

18.9±0.9 15.5±1.5 7.5±1.0

18.9±1.1 16.8±1.6 9.8±0.9

19.6±0.3 18.6±0.8 12.0±1.2

19.7±0.3 19.4±0.6 11.5±0.8

19.7±0.4 18.2±1.6 4.9±0.1

2s3z-Good 2s3z-Medium 2s3z-Poor

18.2±0.4 12.3±0.7 6.7±0.3

18.0±1.0 13.4±1.4 6.2±1.2

19.5±0.1 15.1±2.0 6.9±0.8

19.1±0.9 14.3±1.8 7.0±1.0

7.7±0.9 7.6±0.7 6.6±0.2

17.4±0.3 15.6±0.4 8.4±0.8

15.9±1.2 15.6±0.3 8.5±1.3

18.5±0.8 18.1±0.9 10.0±1.1

19.5±0.5 17.6±0.6 8.5±0.6

19.9±0.1 16.5±0.6 7.9±0.7

5m vs 6m-Good 5m vs 6m-Medium 5m vs 6m-Poor

15.8±3.6 12.4±0.9 7.5±0.2

16.8±2.3 12.5±2.1 8.0±1.0

14.7±2.1 12.8±0.8 7.7±0.8

14.9±3.2 13.5±2.2 8.4±1.1

2.4±0.4 3.8±0.5 3.3±0.5

16.2±1.6 15.1±2.9 10.5±3.1

16.5±2.8 15.2±2.6 8.9±1.3

17.7±1.1 16.2±0.9 10.8±0.3

18.6±3.5 15.6±1.3 9.8±2.1

17.6±1.3 17.0±0.9 10.7±1.1

Average SMACv2

BC(Gaussian)

terran 5 vs 5-Replay zerg 5 vs 5-Replay terran 10 vs 10-Replay Average

11.2

12.9

13.2

4.7

13.5

14.3

15.9

15.7

15.3

7.3±1.0 6.8±0.6 7.4±0.5

9.3±0.9 8.1±1.7 5.5±1.5

8.3±1.9 4.6±0.5 5.8±1.7

9.3±2.0 6.2±0.4 5.6±0.6

13.8±4.4 10.3±1.2 12.7±2.0

11.8±0.9 10.3±3.4 11.8±2.0

13.3±1.8 10.2±1.1 13.8±1.3

15.4±1.3 12.0±1.1 14.6±1.1

16.6±4.3 9.8±1.5 13.0±4.7

12.2±1.8 9.6±4.1 7.7±0.8

7.2

7.6

6.2

7.0

12.3

11.3

12.4

14.0

13.1

9.8

MA-MuJoCo

Table 1: Comparative performance of VGM2 P with discrete actions environment. For the SMACv1 environment, we select 4 tasks, each with 3 datasets of varying quality. For the SMACv2 environment, we select three tasks with only 1 dataset. To distinguish different Behavior Cloning methods and simplify notation, we use FM and MF to represent Flow Matching and MeanFlow, respectively. We report the average performances and standard deviations of each algorithm across 6 seeds, with the best result in bold and the second-best result underlined. Dataset

Extension of offline SARL MA-TD3BC MA-CQL

MA-ICQ

OMAR

MAC-Flow

VGM2 P

6Halfcheetah-Expert 6Halfcheetah-Medium 6Halfcheetah-MR 6Halfcheetah-ME

4401.6±169.1 2620.8±69.9 3528.9±120.9 3518.1±381.0

4589.5±98.5 3189.4±306.9 3500.7±293.9 4738.2±181.1

2955.9±459.2 2549.3±96.3 1922.4±612.9 2834.0±420.3

-206.7±161.1 -265.7±147.0 -235.4±154.9 -253.8±63.9

3383.6±552.7 3608.1±237.4 2504.7±83.5 2948.5±518.9

4711.4±213.6 2650.0±365.4 2830.5±292.8 4410.9±836.8

4650.0±271.6 4358.5±369.2 3030.2±436.8 5139.9±84.1

4897.5±114.5 3684.8±130.4 4068.5±113.5 5159.2±156.3

3Hopper-Expert 3Hopper-Medium 3Hopper-MR 3Hopper-ME

3309.9±4.5 870.4±156.7 269.7±41.8 2904.3±477.4

3359.1±513.8 901.3±199.9 31.4±15.2 2751.8±123.3

754.7±806.3 501.8±14.0 195.4±103.6 355.4±373.9

2.4±1.5 21.3±24.9 3.3±3.2 1.4±0.9

859.6±709.5 1189.3±544.3 774.2±494.3 709.0±595.7

2853.3±593.8 1436.8±449.5 936.1±574.0 2810.4±723.2

3592.1±8.9 1023.5±253.0 1166.3±451.9 2988.3±480.2

2473.5±876.6 2008.6±1389.4 1426.6±665.5 3368.5±403.9

2Ant-Expert 2Ant-Medium 2Ant-MR 2Ant-ME

2046.9±17.1 1422.6±21.1 995.2±52.8 1636.1±96.0

2082.4±21.7 1033.9±66.4 434.6±108.3 1800.2±21.5

2050.0±11.9 1412.4±10.9 1016.7±53.5 1590.2±85.6

312.5±297.5 -1710.0±1589.0 -2014.2±844.7 -2992.8±7.0

2055.5±1.6 1418.4±5.4 1105.1±88.9 1720.3±110.6

2060.0±10.3 1428.4±14.7 1294.5±360.2 1740.2±158.9

2060.2±20.0 1432.4±17.8 1498.4±20.3 2053.3±20.4

2083.0±40.2 1429.4±15.8 1305.5±139.1 1974.7±116.1

2293.7

2367.7

1511.5

-611.5

1856.4

2430.2

2749.4

2823.3

Average

Offline MARL OMIGA MADiff

Table 2: Comparative performance of VGM2 P with continuous actions environment. For the MA-MuJoCo environment, we select 3 tasks, each with 4 datasets of varying quality. For simplicity, we use ME and MR to represent Medium-Expert and Medium-Replay, respectively.

MARL to independent offline RL for individual agents. ICQ [7] and CFCQL [9] leverage conservative Q-learning to improve exploration while maintaining coordination among agents. OMAR [8] and AlberDICE [11] study how multi-agent coordination affects policy improvement, while OMIGA [10] leverages value decomposition to further enhance policy learning. Additionally, graph-based multi-agent collaboration methods [30, 31, 32] use mechanisms such as graph attention to build the topological structure between agents for communication. Although these methods have made progress, the complex distributional nature of multi-agent scenarios often leads to improper credit assignment, which can hinder coordination among agents.

4.2

Diffusion-based and Flow-based RL

With diffusion and flow-based generative models achieving breakthroughs in image generation [14, 15], some studies begin applying them to offline RL. Diffuser [33] and

Expert

5000

Medium

Medium-Replay

4000

3000

Performance

4000 3000

2000

2000 Algorithms

1000

VGM2P MF FM

0 0

Steps (*5e4)

1000

Steps (*5e4)

1000

Steps (*5e4)

Performance

(a) Ma-MuJoCo: 6HalfCheetah (continuous action)

20.0 17.5 15.0 12.5 10.0 7.5 5.0 2.5 0.0

Good

Medium

20.0 17.5 15.0

12.5

10.0

7.5

VGM2P MF

Steps (*1e5)

5.0

Algorithms

2.5 0.0

Poor

Steps (*1e5)

(b) SMACv1: 5m vs 6m (discrete action)

Figure 1: The training curve between different BC and VGM2 P.

Decision Diffusion [34] use diffusion models to model trajectories, while methods such as DiffusionQL [27] model policy. Despite their effectiveness, multi-step sampling in the above models significantly raises computational costs, particularly for policy learning requiring multiple iterative rollouts. To accelerate policy learning under diffusion and flow models, EDP [35] uses a value-weighted diffusion training paradigm, while FQL [28] distills the policy into a one-step generator. Such techniques have also attracted attention in offline MARL. MADiff [18] extends Decision Diffusion to multiagent settings via an attention mechanism, generating trajectories that respect coordination constraints. DoF [17] generalizes value decomposition to distribution decomposition, naturally embedding multi-agent cooperation into diffusion-based generation. To improve inference efficiency, MAC-Flow [19] and OM2 P [20] extend FQL to multiagent scenarios and use flow models to represent individual policies. Additionally, MCGD [36] models multi-agent collaboration as a graph and enables communication using discrete and continuous diffusion models for dynamic scenarios.

Experiments

In this section, we evaluate the performance of VGM2 P by answering the following questions: • How does VGM2 P perform compared to flow-based multi-agent behavior cloning? • How does VGM2 P perform compared to existing offline MARL methods? • What factors affect the effectiveness of VGM2 P?

5.1

Setup

Benchmarks. We evaluate our method on three widely used MARL benchmarks, including two discrete action environments, StarCraft Multi-Agent Challenge (SMAC) v1 and v2 [37], and one continuous action one, Multi-Agent MuJoCo (MA-MuJoCo) [5]. • SMAC is a real-time combat environment with both homogeneous and heterogeneous unit settings, where agents must cooperate as a team to defeat opponents. There are two versions of datasets available [29]: SMACv1 includes three quality datasets for each map, such as Good, Medium, and Poor, while v2 consists of Replay datasets with more randomized initial positions and scenarios. • MA-MuJoCo treats the single robot as a collective of multiple agents, requiring collaboration among them to achieve a shared goal. There are four datasets of varying quality for each scenario [10]: Expert, Medium-Expert, Medium-Replay, and Medium. Baselines. We compare 10 representative offline MARL algorithms, covering 3 categories: extensions of single-agent methods, recent MARL solutions, as well as diffusionand flow-based methods. For single-agent methods, we mainly consider BCQ [38], CQL [39], and TD3BC [40]. In addition, we include behavior cloning (BC) methods with different modeling paradigms (i.e., Gaussian-based, Diffusion-based, Flow Matching-based, and MeanFlow-based ones) as additional baselines. For other methods, we consider the following: • ICQ [7] (MARL solutions) leverages implicit conservative Q-learning for training the joint multi-agent value. • OMAR [8] (MARL solutions) optimizes the value function using zero-order optimization. • OMIGA [10] (MARL solutions) introduces local implicit value regularization for policy optimization. • MADiff [18] (Diffusion-based MARL) uses the diffusion model to model trajectories and introduces an attention mechanism. • Dof [17] (Diffusion-based MARL) decomposes the centralized diffusion model into multiple independent diffusion models. 11

Expert

Medium

5000

4000

Performance

4000

3000

2000

1000

IGM w/o IGM

0 0

Steps (*5e4) Medium-Replay

Steps (*5e4) Medium-Expert

5000

4000

Performance

4000

3000

2000

1000

0 0

Steps (*5e4)

Figure 2: The training curve for different Q-value training methods of 6HalfCheetah scenarios in MA-MuJoCo.

• MAC-Flow [19] (Flow-based MARL) models policy with flow matching and adopts one-step generation through distillation. We evaluate 10 trajectories for each task and report the results based on experiments conducted with 6 seeds. We provide a detailed experimental introduction in Appendix A.

5.2

Comparison among Behavior Cloning

VGM2 P is a value-conditioned behavior cloning (BC) method that models the policy using MeanFlow. To provide a clear comparison with traditional BC, we perform unconditional BC using two generative models, Flow Matching (FM) and MeanFlow (MF), and present some comparison results in Figure 1. The results show that VGM2 P has more advantages than traditional BC in most cases. We attribute this to the fact that, unlike traditional BC, which merely replicates behavior policy, VGM2 P can dig more high-reward information with value guidance conditional generation.

Expert

Medium

5000

4000

3000

2000

3 1000 5 10 0 20

1000

Performance

4000 3000 2000 1000 0 0

Steps (*5e4)

Medium-Replay

0 0

Steps (*5e4)

Performance

(a) Ma-MuJoCo: 6HalfCheetah (continuous action)

20.0 17.5 15.0 12.5 10.0 7.5 5.0 2.5 0.0

Good

Medium

20.0

Poor

17.5

15.0

12.5

10.0

3 5 10 20 0

Steps (*1e5)

7.5

5.0

2.5 0.0

Steps (*1e5)

(b) SMACv1: 5m vs 6m (discrete action)

Figure 3: The training curve for different guidance weights.

5.3

Comparative Evaluation with Offline MARL

In this experiment, we evaluate VGM2 P’s performance in both discrete and continuous environments, comparing it with existing offline MARL methods. The results are shown in Table 1 and 2. In simpler discrete-action multi-agent tasks, such as those in SMACv1, VGM2 P performs well with conditional BC; however, in SMACv2, it only outperforms traditional BC. We guess this is due to the replay dataset quality in SMACv2 not supporting VGM2 P’s training with conditional BC. This will be a focus of our future work. To our surprise, VGM2 P performs comparably to existing stateof-the-art in continuous scenarios, which strongly validates the effectiveness of value guidance conditional generation.

5.4

Ablation Study

Effect of the Q-value training with IGM. To validate the effectiveness of joint Qvalue training based on the IGM principle, i.e., training with Eq.(13), we compare its performance with independent training of Q-value (i.e., each agent train Q-value

Average Running Time（min）

250 197

200

195

150 112

100

85 44

50 0

SMACv1 2

VGM P

MA-MuJoCo 2

VGM P(FM)

MAC-FLOW

Figure 4: Comparison of running time (minutes). These results are the averages across different tasks in each environment.

function with Eq. (6)), as shown in Figure 2. The results show that joint training based on the IGM outperforms training independently, especially under the Medium and Medium-Replay datasets. We believe that independently Q-value function training leads multi-agent systems to converge to each local optima, neglecting global optima. In contrast, training based on the IGM principle encourages agents to explore the global optima, especially when offline data quality is low. Effect of the guidance coefficient. To investigate the sensitivity of VGM2 P on the guidance coefficient, we conduct an ablation study to test its performance under different ω values. The result shown in Figure 3 reveals that VGM2 P is not sensitive to guidance coefficients within a certain range, and its performance does not degrade significantly with changes in the guidance weight. The runtime efficiency of VGM2 P. To evaluate the efficiency of VGM2 P, we compare it with the MAC-Flow, which improves efficiency through distillation and the Flow Matching version of VGM2 P, denoted VGM2 P(FM), with 10-step sampling for action generation. The results in Figure 4 show that our method achieves comparable efficiency to MAC-Flow in the Ma-MuJoCo environment and is more efficient in the SMACv1 environment. Additionally, the comparison with Flow Matching highlights that VGM2 P’s efficiency is due to MeanFlow’s 1-step generation.

Conclusion and Discussion

In this paper, we propose the value guidance multi-agent MeanFlow policy (VGM2 P), which leverages the advantage value as condition information and approximates the optimal joint policy through MeanFlow-based conditional behavior cloning. Experimental results show that relying solely on conditional behavior cloning, VGM2 P achieves 14

performance comparable to state-of-the-art offline MARL methods. In addition, ablation studies indicate that VGM2 P is both efficient and less sensitive to the guidance coefficient. While VGM2 P has yielded promising results, behavior cloning alone is insufficient for generalization in complex scenarios like SMACv2. Moreover, integrating more effective collaborative methods is expected to enhance VGM2 P’s performance further. These will be a primary direction for our future work.

References [1] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289–353, 2008. [2] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1–51, 2020. [3] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pages 321–384, 2021. [4] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019. [5] Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34:12208–12221, 2021. [6] Marco A Wiering et al. Multi-agent reinforcement learning for traffic light control. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML’2000), pages 1151–1158, 2000. [7] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299–10312, 2021. [8] Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17221–17237. PMLR, 17–23 Jul 2022. [9] Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conservative q learning for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36:77290–77312, 2023. 15

[10] Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multiagent reinforcement learning with implicit global-to-local value regularization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [11] Daiki E Matsunaga, Jongmin Lee, Jaeseok Yoon, Stefanos Leonardos, Pieter Abbeel, and Kee-Eung Kim. Alberdice: addressing out-of-distribution joint actions in offline multi-agent rl via alternating stationary distribution correction estimation. Advances in Neural Information Processing Systems, 36:72648–72678, 2023. [12] Zongkai Liu, Qian Lin, Chao Yu, Xiawei Wu, Yile Liang, Donghui Li, and Xuetao Ding. Offline multi-agent reinforcement learning via in-sample sequential policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19068–19076, 2025. [13] Dan Qiao, Wenhao Li, Shanchao Yang, Hongyuan Zha, and Baoxiang Wang. Offline multi-agent reinforcement learning via sequential score decomposition. In Submitted to The Fourteenth International Conference on Learning Representations, 2025. under review. [14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. [15] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In 11th International Conference on Learning Representations, ICLR 2023, 2023. [16] Zhuoran Li, Ling Pan, Jiatai Huang, and Longbo Huang. Beyond conservatism: Diffusion policies in offline multi-agent reinforcement learning, 2024. [17] Chao Li, Ziwei Deng, Chenxing Lin, Wenqi Chen, Yongquan Fu, Weiquan Liu, Chenglu Wen, Cheng Wang, and Siqi Shen. Dof: A diffusion factorization framework for offline multi-agent reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. [18] Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. MADiff: Offline multi-agent learning with diffusion models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [19] Dongsu Lee, Daehee Lee, and Amy Zhang. Multi-agent coordination via flow matching. arXiv preprint arXiv:2511.05005, 2025. [20] Zhuoran Li, Xun Wang, Hai Zhong, and Longbo Huang. Om2p: Offline multiagent mean-flow policy. arXiv preprint arXiv:2508.06269, 2025. [21] Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 16

[22] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 33(6):750–797, 2019. [23] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2085–2087, 2018. [24] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887–5896. PMLR, 2019. [25] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantageweighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. [26] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020. [27] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022. [28] Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In Proceedings of the 42nd International Conference on Machine Learning, pages 48104–48127. PMLR, 2025. [29] Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is a controllable policy improvement operator. arXiv preprint arXiv:2505.23458, 2025. [30] Shifei Ding, Wei Du, Ling Ding, Jian Zhang, Lili Guo, and Bo An. Multiagent reinforcement learning with graphical mutual information maximization. IEEE Transactions on neural networks and learning systems, 2023. [31] Ziheng Liu, Jiayi Zhang, Enyu Shi, Zhilong Liu, Dusit Niyato, Bo Ai, and Xuemin Shen. Graph neural network meets multi-agent reinforcement learning: Fundamentals, applications, and future directions. IEEE Wireless Communications, 31(6):39–47, 2024. [32] ZHAO Bocheng, HUO Mingying, LI Zheng, FENG Wenyu, YU Ze, QI Naiming, and WANG Shaohai. Graph-based multi-agent reinforcement learning for collaborative search and tracking of multiple uavs. Chinese Journal of Aeronautics, 38(3):103214, 2025.

[33] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902–9915. PMLR, 2022. [34] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2022. [35] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:67195–67212, 2023. [36] Xianghua Zeng, Hang Su, Zhengyi Wang, and Zhiyuan Lin. Graph diffusion for robust multi-agent coordination. In Forty-second International Conference on Machine Learning, 2025. [37] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019. [38] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052–2062. PMLR, 2019. [39] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179–1191, 2020. [40] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021. [41] Juan Claude Formanek, Asad Jeewa, Jonathan Phillip Shock, and Arnu Pretorius. Off-the-grid MARL: Datasets with baselines for offline multi-agent reinforcement learning, 2024.

Experimental Details

For the dataset, we primarily use the publicly available dataset library OG-MARL1 [41], which includes data from MARL scenarios collected through pretrained policies. Our experiments are implemented in Python with a JAX-based network architecture, and the experimental environment is Ubuntu 22.04. For computational resources, we use an RTX 3090 24GB GPU. Detailed hyperparameter settings are provided in Table 3. Hyperparameter Gradient steps Batch Size Optimizer Learning Rate Model Architecture Hidden Layer Hidden Dimension Discount factor The value of ω

Value 106 (SMACv1 and SMACv2), 5 × 105 (MA-MuJoCo) 64 Adam 3 × 10−4 MLP 4 512 0.995 [3, 5, 10, 20]

Table 3: Hyperparameter for Meanflow model

Proofs

B.1

Proof of Proposition 1

Proposition 1 (Value-Guidance Behavior Policy). Given a behavior policy πβ (a|o) and the optimal policy π ∗ (a|o) derived from Eq.(8), for any variable c ∈ C and its related distribution p(c|o, a), when there exists c∗ ∈ C satisfying p(c = c∗ |o, a) ∝ exp( λ1 Qπ (o, a)), then we have the conditional behavior policy πβ (a|o, c = c∗ ) = π ∗ (a|o). Proof. According to Bayes’ theorem, we have πβ (a|o, c) =

By comparing Eq. (8), we find that when there exists c∗ ∈ C satisfying p(c = c∗ |o, a) ∝ exp( λ1 Qπ (o, a)) (i.e., p(c = c∗ |o, a) = k∗exp( λ1 Qπ (o, a)), k is a constant), 1 https://huggingface.co/datasets/InstaDeepAI/og-marl

we have: πβ (a|o, c = c∗ ) = R

p(c = c∗ |o, a) πβ (a|o) π (a′ |o)p(c = c∗ |o, a′ )da′ a′ β

k ∗ exp( λ1 Qπ (o, a)) πβ (a|o) π (a′ |o)(k ∗ exp( λ1 Qπ (o, a′ )))da′ a′ β

exp( λ1 Qπ (o, a)) πβ (a|o) π (a′ |o) exp( λ1 Qπ (o, a′ ))da′ a′ β

= π ∗ (a|o).

B.2

(15)

Proof of Proposition 2

Proposition 2. Assuming that the behavior joint policy πβtot (a|o) and the global QQN i i i tot tot value Qtot tot (o, a) are decomposable, i.e., πβ (a|o) = i=1 πβ (a |o ) and Qπ tot (o, a) = PN πi i i tot,∗ (a|o), when the distribution pi (ci |oi , ai ) i=1 Qϕi (o , a ). For the optimal joint policy π satisfies pi (ci = ci,∗ |oi , ai ) ∝ exp( λ1 Qiϕi (oi , ai )) for each agent i, then we have QN i i i i i,∗ tot,∗ (a|o). i=1 πβ (a |o , c = c ) = π QN PN i i i i i Proof. When πβtot (a|o) = i=1 πβi (ai |oi ), Qtot i=1 Qϕi (o , a ) and p (c = π tot (o, a) = ci,∗ |oi , ai ) ∝ exp( λ1 Qiϕi (oi , ai )) (i.e., pi (ci = ci,∗ |oi , ai ) = k ∗ exp( λ1 Qiϕi (oi , ai )), k

is a constant), we have: N Y

πβi (ai |oi , ci = ci,∗ )

i=0

N Y p(ci = ci,∗ |oi , ai )

p(ci = ci,∗ |oi )

i=0

N Y

k ∗ exp( λ1 Qiϕi (oi , ai )) π i (ãi |oi )(k ∗ exp( λ1 Qiϕi (oi , ãi )))dãi ãi β

R i=0

N Y

exp( λ1 Qiϕi (oi , ai )) R

i=0

πβi (ai |oi )

ãi

πβi (ai |oi )

πβi (ãi |oi ) exp( λ1 Qiϕi (oi , ãi ))dãi

N Y

N Y 1 1 exp( Qiϕi (oi , ai )) · QN R πβi (ai |oi ) · i (ãi |oi ) exp( 1 Qi (oi , ãi ))dãi λ π i i=0 β ϕ i=0 i=0 λ

ã

= exp(

N 1X

λ i=0 N 1X

λ i=0

Qiϕi (oi , ai )) · R ã1 ×...×ãn

1 QN

1 i i i i 1 n i i i=0 πβ (ã |o ) exp( λ Qϕi (o , ã ))d(ã × ... × ã )

N Y

πβi (ai |oi )

i=0

1 Q

N i i i i=0 πβ (ã |o )

exp( λ1

i i i 1 n i=0 Qϕi (o , ã ))d(ã × ... × ã )

exp( λ1 Qtot π tot (o, a)) π tot (a|o) tot (ã|o) exp( 1 Qtot (o, ã))dã β π λ π tot ã β

= π tot,∗ (a|o)

(16)

N Y i=0

πβi (ai |oi )

Learning Curves of VGM2 P

Performance

3m-Good 20.0 17.5 15.0 12.5 10.0 7.5 5.0 2.5 0.0

Steps (*1e5)

3m-Medium

20.0

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

Steps (*1e5)

3m-Poor

20.0

17.5

8m-Good 15 10 5 0

Steps (*1e5)

2s3z-Medium

2s3z-Poor

Steps (*1e5)

terran_5_vs_5-Replay

Steps (*1e5)

15.0

12.5

10.0

7.5

5.0

2.5

0.0

Steps (*1e5)

zerg_5_vs_5-Replay

Steps (*1e5)

5m_vs_6m-Poor

6 4 2 0

Steps (*1e5)

terran_10_vs_10-Replay 8

8 6 4

Steps (*1e5)

5m_vs_6m-Medium

12.5

0 0

15.0

(d) SMACv1:5m vs 6m

17.5

5m_vs_6m-Good

20.0

2 0

8m-Poor 5

(b) SMACv1:8m

Performance

2s3z-Good

Performance

(a) SMACv1:3m 20 18 16 14 12 10 8 6 4

8m-Medium

20.0 17.5 15.0 12.5 10.0 7.5 5.0 2.5 0.0

Performance

Steps (*1e5)

(e) SMACv2

Figure 5: The training curve for SMAC.

Steps (*1e5)

6halfcheetah-Expert

5000

6halfcheetah-Medium

6halfcheetah-Medium-Replay

4000

3000

2000

1000

Performance

4000 3000 2000

Steps (*5e4)

6halfcheetah-Medium-Expert 5000 4000 3000 2000 1000 0

Steps (*5e4)

(a) 6HalfCheetah 3hopper-Expert

3000

Performance

3hopper-Medium

3500

2500

2000

2000 1500

1500

1000

500

Steps (*1e5)

2000 1750 1500 1250 1000 750 500 250 0

3hopper-Medium-Replay

3hopper-Medium-Expert 3500 3000 2500 2000 1500 1000 500

Steps (*1e5)

(b) 3Hopper 2ant-Expert

Performance

2ant-Medium

1500

3000 2000 1000 0 1000 2

Steps (*1e5)

2ant-Medium-Expert 2000

1000

1500

500

1000

500

500 0 500

1000

1000 0

2ant-Medium-Replay

1500

Steps (*1e5)

1000 0

Steps (*1e5)

Figure 6: The training curve for MA-MuJoCo.

Steps (*1e5)

Record · ID 2617 · SHA-256 76cb0763c65f0130
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.