Official Review · NEURIPS 2024

The paper proposed a scalable multi-agent constrained policy optimization for safe reinforcement learning. It is an extension of two previous work on safe reinforcement learning and scalable multi-agent reinforcement learning. The trust region policy updates and truncated policy/advantage function are combined to give theoritical performance bound. A practical algorithm based on PPO is also shown and the empirical results verifies the claimed performance. 1. The presentation is clear and easy to understand, the authors presented clearly the relation between this paper and the previous work. 2. It is a very solid combination of the ideas in previous works, and extends the scalable multi-agent reinforcement learning idea to multi-agent safe reinforcement learning. It will be beneficial to the safe RL community. 3. The theoretical results look correct to me. There are all minor weaknesses in this paper. 1. The author should discuss more clearly on how the assumptions are related to previous works, for example, the Dobrushin conditions in (1) and Assumption 2.1. They both appear in the previous scalable multi-agent reinforcement learning paper [1] but in slightly different formulations. 2. The experiments are only on a smaller number of agents (for 12 agents, we don't really need a scalable algorithm to handle it). [1] Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement 374 learning for networked systems with average reward. In NeurIPS, 2020. The theoritical results look okay to me, although I didn't check all the proofs. for the experimental results, 1. For figure 1 and 2, what is the constraint on the average episodic cost? It will be more clear to draw a horizontal line showing that. 2. The performance is still increasing at 1e7 steps, the algorithms might not converge. Can you explain a bit?