Official Review · ICLR 2025

This paper studies the decentralized, safe multi-agent reinforcement learning problem. The paper proposes a homogeneous, constrained Markov game framework to formulate the problem and an off-policy algorithm to solve it. The paper also proves the convergence of the on-policy version of the proposed algorithm under several assumptions. The proposed approach is validated in a safety-aware continuous swarm robot task in a simulation environment. 1. The study of the safety problem of multi-agent reinforcement learning is well-motivated. 2. The paper is well-structured. 1. Problem Formulation isn't sound to me: In 2.2 Problem formulation, the authors assume the availability of the global state and agents' joint actions, which violates the problem setting of the decentralized multi-agent. In decentralized multi-agent RL, the local agent should only have access to the information of its local neighborhood. 2. The novelty of the paper is limited. 1) problem formulation 2.2, policy gradient derivation, and the convergence proof seem like straightforward extensions of the work [1][4]. 2) The primal-dual method for multi-agent safe RL is also studied in [2-3]. 3. The proposed approach is only demonstrated in simple robotic swarming tasks in simulation, which is a bit insufficient for validating the effectiveness of the proposed algorithm. From the attached video, I also noticed that in the safe swapping scenario, there were two instances where collisions occurred between agents. It seems the learned policy is not safe enough. 4. I personally feel like the paper is a bit hard to follow due to the heavy notations. [1] Zhang, Kaiqing, et al. "Fully decentralized multi-agent reinforcement learning with networked agents." International conference on machine learning. PMLR, 2018. [2] Ying, Donghao, et al. "Scalable primal-dual actor-critic method for safe multi-agent rl with general utilities." Advances in Neural Information Processing Systems 36 (2024) [3] Paternain, Santiago, et al. "Safe policies for reinforcement learning via primal-dual methods." IEEE Transactions on Automatic Control 68.3 (2022): 1321-1336. [4] Zhang, Kaiqing, Zhuoran Yang, and Tamer Başar. "Multi-agent reinforcement learning: A selective overview of theories and algorithms." Handbook of reinforcement learning and control (2021): 321-384. How do you set the value of b in the loss function equation (15)?