CoGrid & the Multi-User Gymnasium: A Framework for Multi-Agent Experimentation
arXiv:2604.15044v1 [cs.HC] 16 Apr 2026
Chase McDonald Department of Social and Decision Sciences Carnegie Mellon University
Cleotilde Gonzalez Department of Social and Decision Sciences Carnegie Mellon University [email protected]
Abstract The increasing integration of artificial intelligence (AI) in everyday life brings with it new challenges and questions for regarding how humans interact with autonomous agents. Multi-agent experiments, where humans and AI act together, can offer important opportunities to study social decision making, but there is a lack of accessible tooling available to researchers to run such experiments. We introduce two tools designed to reduce these barriers. The first, C O G RID, is a multi-agent grid-based simulation library with dual NumPy and JAX backends. The second, Multi-User Gymnasium (MUG), translates such simulation environments directly into interactive web-based experiments. MUG supports interactions with arbitrary numbers of humans and AI, utilizing either server-authoritative or peer-to-peer networking with rollback netcode to account for latency. Together, these tools can enable researchers to deploy studies of human-AI interaction, facilitating inquiry into core questions of psychology, cognition, and decision making and their relationship to human-AI interaction. Both tools are open source and available to the broader research community. Documentation and source code is available at {cogrid, multi-user-gymnasium}.readthedocs.io. This paper details the functionality of these tools and presents several case studies to illustrate their utility in human-AI multi-agent experimentation.
Acknowledgements
This research was supported by the Defense Advanced Research Projects Agency
and was accomplished under Grant Number W911NF-20-1-0006 and by the NSF AI Institute for Societal Decision Making (AI-SDM), Award No. 2229881.
1
Introduction
Artificial intelligence (AI) research has made undeniable progress in producing highly capable systems, yet most benchmarks and evaluations emphasize isolated performance rather than interaction with humans. In real-world settings, AI must not only be competent but also interact effectively with humans— its main beneficiaries—and make decisions that align with and complement human decision-making processes (Russell, 2019; Wilder et al., 2020). Much of the literature has focused on agents that outperform humans in games or on established benchmarks (e.g., (FAIR)† et al. (2022), Perolat et al. (2022), and Silver et al. (2021)), with far less attention to how such agents interact with or impact human collaborators. Even when human interaction is considered, state-of-the-art AI systems are not always incorporated. For example, research on human-AI teaming often relies on surveys, rule-based expert systems, or Wizard-of-Oz paradigms rather than fully autonomous, learning-based agents (e.g., Duan et al., 2024; Salikutluk et al., 2024; Schelble et al., 2022; Zhang et al., 2021). Although early work highlights the importance of understanding human perceptions, preferences, and coordination dynamics (Carroll et al., 2020; Schmutz et al., 2024; Strouse et al., 2021), progress remains constrained by the lack of accessible platforms for conducting controlled human-AI experiments. Studying human-AI interaction poses unique challenges compared to pure simulation or human-only studies. In the context of reinforcement learning, designing custom settings for single agents in pure simulation can be cumbersome (Bamford et al., 2022). Adding human interaction further complicates such inquiries (Ouyang et al., 2022), requiring the implementation of necessary infrastructure, from networking to user interfaces and data collection pipelines that are often non-standard for complex tasks and rebuilt from scratch for each study. These are significant barriers to entry in empirical human-AI research. To address these challenges, we present a framework comprised of two complementary tools. The first, C O G RID, is a multi-agent grid-based simulation library originally inspired by Minigrid (Chevalier-Boisvert et al., 2023). The library is designed to facilitate extensibility and customization through modularized components. By adopting the PettingZoo API (Terry et al., 2021), C O G RID allows researchers to build cooperative or competitive environments that fit into standardized simulation infrastructure (e.g., reinforcement learning algorithm libraries). C O G RID is built to follow the PettingZoo API using a NumPy (Harris et al., 2020) backend; however, it is also built with an optional JAX (Bradbury et al., 2018) backend, enabling hardware acceleration for fast parallelized simulations. 2
The second tool, MUG, streamlines the deployment of simulation environments into interactive web-based experiments. MUG takes Python-based environments that follow the Gymnasium or PettingZoo APIs and makes them playable in the browser without the need for game engines or following the video game development cycle (Glazer & Madhav, 2015). Environments can run on the server or directly in participants’ browsers via Pyodide (Python compiled to WebAssembly; Team, 2021). For multiplayer experiments, it enables peer-to-peer networking with GGPO-style (Good Game Peace Out; Cannon, 2019) rollback netcode for latency correction. MUG provides common experiment infrastructure—including landing pages, waiting rooms, matchmaking, data collection, AI inference, and surveys—so that researchers do not need to rebuild these components for each study. In this paper, we detail the functionality of these tools, which we provide to the research community at {cogrid, multi-user-gymnasium}.readthedocs.io. We also present several case studies to demonstrate the use of both tools for designing simulation experiments and conducting both human-AI and human-human interaction experiments. In summary, our contributions are as follows: 1. We release C O G RID, a multi-agent grid-based simulation library with dual NumPy/JAX backends and the PettingZoo API, supporting both rapid prototyping and hardware-accelerated training. 2. We release MUG, a platform that deploys Gymnasium and PettingZoo environments as browserbased experiments, with support for client-side execution and peer-to-peer multiplayer. 3. We demonstrate the use of both C O G RID and MUG, illustrating how they can facilitate new lines of research in human-AI interaction.
2
Related Work
In recent years, there has been extensive development of simulation environments and platforms for behavioral research. In this section, we review the relevant prior work that motivates and complements both C O G RID and MUG. Minigrid has become a standard for reinforcement learning experiments. Despite the simplicity of gridbased environments, significant complexity can arise in task formulation and dynamics (Bamford et al., 2022; Chevalier-Boisvert et al., 2023), from social dilemmas (Agapiou et al., 2022) to difficult exploration and generalization tasks (Hafner, 2021). The current work on C O G RID is largely motivated by the accessibility and ease of use of Minigrid (Chevalier-Boisvert et al., 2023): C O G RID aims to provide an equally accessible 3
multi-agent library for developing simulation environments, with the additional requirement of supporting hardware-accelerated training through a dual backend. The primary goal is to provide a platform that can be easily manipulated, extended, and customized to ask novel questions, rather than serving as a platform for standardized benchmarks. MUG aims to bridge the gap between environments that are useful for training and evaluating AI in simulation, and tasks that can be used in human behavioral experiments. While there are several libraries that can be used to create environments, including C O G RID, few generalized platforms exist for translating a standard-API Python-based simulation environment into an interactive experiment (Aydın et al., 2025). MUG addresses this by allowing researchers to deploy simulation environments as browser-based experiments, with environments running either on the server or directly in participants’ browsers.
2.1
Multi-Agent Environments and Experiments
The standard interface for the agent-environment cycle in reinforcement learning is Gymnasium (Towers et al., 2023). It defines the API for how an agent can interact with the environment; however, it does not standardize or even allow for multi-agent interactions. The vast majority of multi-agent environments augment the Gymnasium API through minor extensions or alterations, maintaining most of the interface (i.e., Bamford et al., 2022; Liang et al., 2018; Terry et al., 2021). Whereas the development of single-agent environments has largely unified around the Gymnasium API, the adoption of a multi-agent API in the reinforcement learning research community has been less uniform, although several platforms have made strides toward a standard approach. In an extension of the Gymnasium API, the PettingZoo interface (Terry et al., 2021) has seen significant adoption. PettingZoo provides a standardized API for the agent-environment cycle in multi-agent settings, alongside a number of existing benchmark environments, such as the multi-agent Atari suite. An API for interacting with an environment is only one component of a multi-agent simulation framework. The other critical component is the implementation of the environment itself, including the state representation and transition dynamics. In the single agent setting, a wide range of mature frameworks and platforms support the construction and extension of environments (e.g., Chevalier-Boisvert et al., 2023; Samvelyan et al., 2021; Sukhbaatar et al., 2015). In contrast, the multi-agent domain offers substantially fewer such resources. Existing platforms typically define their own environment-creating formats, which vary considerably in complexity, abstraction level, and degree of customizability. 4
The framework most closely related to the present work is Griddly (Bamford et al., 2022). Griddly is an open-source gridworld game engine that uses its own description language, Griddly Description YAML (GDY), to configure environments and the interactions within them. It is highly extensible and supports a variety of agent interactions—including both single- and multi-agent. In the present work, we make similar concessions as those made by Chevalier-Boisvert et al. (2023): Griddly provides increased functionality relative to both Minigrid and C O G RID. However, this increases the barrier to entry for understanding the library and developing new environments. In addition to the GDY description language for configuration, Griddly relies on a C++ core game engine, trading off ease of use and readability for efficiency. In the approach we take here, we focus on lowering the barrier to entry to allow researchers to develop environments without needing significant outside knowledge. Melting Pot (Agapiou et al., 2022) provides an additional framework for grid-based multi-agent environments. Its goal is to provide a tool to help in the development and evaluation of agents in multi-agent environments and their ability to learn policies that generalize to novel partners. Melting Pot includes social dilemma environments utilized in previous work (Hughes et al., 2018; Jaques et al., 2019; Leibo et al., 2017), which were used to study learning dynamics and behavior with reinforcement learning agents in temporally extended variations of classic economic games (e.g., prisoner’s dilemma and tragedy of the commons). Melting Pot relies on a combination of the Lua programming language and Python for the development of environments, resulting in an increased barrier to entry and a higher level of complexity for developing custom environments. Aside from generalized platforms for building custom environments, there are a number of specialized frameworks and one-off implementations that provide examples of desired functionality. These range from relatively simple abstractions (e.g., Skrynnik et al., 2022) to complex 3D video games (e.g., Ellis et al., 2024; Norén, 2020).1 A particularly relevant example is the Overcooked-AI environment developed by Carroll et al. (2020). In their work, they adapted a popular collaborative video game, Overcooked, into a reinforcement learning environment to construct and evaluate agents for human-AI collaboration. Their setting has inspired a significant amount of follow-up work, with reimplementations and extensions of the Overcooked environment (for example, Agapiou et al., 2022). The ad hoc nature of these implementations also demonstrates the difficulty in designing general-use multi-agent environments, a difficulty that C O G RID is designed to alleviate. 1 A number of such examples can be found through the third-party environments linked in the PettingZoo documentation: https://pettingzoo.farama.org/environments/third party envs/.
5
In the same vein as C O G RID, there are existing efforts to add multi-agent functionality to Minigrid (Fickinger, 2020; Ndousse, 2020). Despite their initial efforts to provide multi-agent environments, these projects remain unmaintained and undocumented, preventing wider adoption, extension, and customization. Our work provides a complete and documented extension of Minigrid with increased customizability through modularization, as described below. A separate line of work has addressed the computational limitations of standard simulation environments through hardware acceleration. NAVIX (Pignatelli et al., 2024) reimplements Minigrid entirely in JAX, achieving substantial speedups, but retains Minigrid’s single-agent design. JaxMARL (Rutherford et al., 2024) extends this idea to the multi-agent setting, implementing multi-agent environments and algorithms entirely in JAX for hardware-accelerated training. However, both NAVIX and JaxMARL are JAX-only, which limits their applicability in settings that require web-compatible environments for browser-based execution. C O G RID takes a different approach through its dual-backend architecture: environments are written against a shared array namespace that dispatches to either NumPy or JAX at runtime, enabling the same environment code to be used for both rapid prototyping and high-throughput training. This allows C O G RID to utilize the JAX backend for accelerated training, then deploy the exact same environment to run natively in browsers using the NumPy backend in MUG. Table 1 summarizes the key dimensions along which the aforementioned environment libraries differ.
Minigrid NAVIX Griddly Melting Pot JaxMARL C O G RID
Language
Multi-agent
Hardware Acceleration
Standard API
Web Compatible
Python Python GDY + C++ Lua + Python Python Python
✗ ✗ ✓ ✓ ✓ ✓
✗ ✓ ✗ ✗ ✓ ✓
Gymnasium Gymnasium Custom Custom Custom PettingZoo
✗ ✗ ✓ ✗ ✗ ✓
Table 1: Comparison of RL environment libraries. Language indicates the language(s) required to implement new environments. Multi-agent indicates native support for multi-agent settings. Hardware Acceleration indicates support for hardware-accelerated (GPU/TPU) training. Standard API indicates compatibility with Gymnasium or PettingZoo. Web Compatible indicates the ability to run in a web browser (e.g., via WebAssembly compilation).
2.2
Human-AI Interaction and Interactive Platforms
In addition to training and testing autonomous agents in a pure simulation environment, there have been several efforts to provide platforms that enable humans to interact with these agents. Each platform has been developed largely to fill a specific need in the research community—from interactive versions of single 6
games (Carroll et al., 2020) to frameworks for decision-making research (Balietti, 2017)—and they highlight the growing amount of research into human-AI interaction. However, existing work has identified that there is a gap for generalized platforms that take complex tasks used in simulation and allow for human interaction (Aydın et al., 2025). We provide an overview of some of the existing approaches and the particular niche that they fill. A generalized framework for real-time games and experiments with human participants is nodeGame (Balietti, 2017). It provides researchers with a JavaScript framework to design games for individual human participants, multiple humans, and human-bot populations. It provides a framework that streamlines the creation of experiment flow, participant interaction, data collection, and much more. However, to implement more complex real-time games, as are commonly used in multi-agent reinforcement learning, the onus remains on the developer to integrate the game logic into the nodeGame experiment flow. Indeed, nodeGame is primarily used in settings where participants interact by selecting between alternatives (e.g., Prisoner’s Dilemma) or making allocations (e.g., Dictator Game), among other paradigms common in behavioral research. A particularly relevant development that inspired the setup and laid the groundwork for MUG is the interactive Overcooked-AI of Carroll et al., 2020. In their work, they provide an interactive demonstration of their Overcooked environment, which can be used to play the game with any combination of human and AI agents. Their framework is designed specifically for their implementation of Overcooked-AI, rather than being extensible to different environments. This lack of generality is exactly what motivated the development of MUG: we enhance their approach to provide a mapping from a generic class of simulation environments to interactive tasks, embedded within the full experiment pipeline—including surveys, condition randomization, instructions, and more. A recent and closely related effort is SHARPIE (Aydın et al., 2025), a framework for conducting experiments involving humans and reinforcement learning agents. SHARPIE provides a generic interface that wraps existing Gymnasium environments and supports several human-AI interaction paradigms. While SHARPIE and MUG share similar goals of reducing barriers to human-AI experimentation, they differ in emphasis and capabilities. SHARPIE provides breadth across interaction types (e.g., reward specification, preference elicitation) but does not address the challenges of real-time, low-latency multiplayer interaction, client-side execution, or a complete and configurable experiment flow. MUG is designed specifically to address these issues to provide an experience that is scalable and accessible.
7
Similarly, CrowdPlay (Gerstgrasser et al., 2021) is a web-based platform for crowdsourcing human demonstration trajectories in Gymnasium environments. CrowdPlay supports multi-agent games; however, it uses a server-authoritative architecture in which the environment renders frames as JPEG images that are streamed to the browser. No environment logic runs client-side. CrowdPlay provides no latency compensation beyond frame dropping, and its experiment lifecycle is limited (instructions and payment tracking, but no structured scene flow or surveys). Furthermore, the CrowdPlay project was deprecated in 2025 and is no longer maintained. HIPPO-Gym (Bewley et al., 2021) provides a web-based platform for human-in-the-loop RL research, allowing humans to interact with Gymnasium environments through a browser interface. HIPPO-Gym is limited to single-agent settings and supports only specific interaction paradigms (e.g., humans teaching RL agents), without support for multi-human or multi-agent experiments. A common limitation across these platforms is their reliance on server-authoritative architectures, where the environment runs on the server and each frame requires a network round trip to exchange actions and state. For single-player settings, this introduces latency proportional to the participant’s connection quality. For multiplayer, the problem compounds: all players’ inputs must reach the server before the environment can step, and the updated state must be sent back to all clients. At frame rates typical of real-time tasks (e.g., 30–60 frames per second), this can make interactions unusable for participants with moderate latency. MUG addresses this through client-side execution and, for multiplayer, peer-to-peer input exchange with rollback-based synchronization, as described in Section 4. Table 2 summarizes the capabilities of existing interactive experiment platforms for RL environments.
Overcooked-AI HIPPO-Gym CrowdPlay SHARPIE MUG
API Compat.
Multi-human
Client-side
Latency comp.
Exp. lifecycle
Custom only Gymnasium Gymnasium Gymnasium Gymnasium/PettingZoo
✓ ✗ ✓ ✓ ✓
✗ ✗ ✗ ✗ Pyodide
✗ ✗ ✗ ✗ GGPO
✗ Partial Partial Partial ✓
Table 2: Comparison of platforms for interactive experiments with simulation environments. API Compat. indicates which environment APIs are natively supported. Multi-human indicates simultaneous interaction between multiple human participants. Client-side indicates whether the environment can run in the participant’s browser. Latency Comp. indicates whether the platform provides a mechanism (e.g., rollback netcode) to mask network latency during real-time interaction. Exp. lifecycle indicates built-in support for the full experiment flow (instructions, matchmaking, surveys, data collection). As summarized in Tables 1 and 2, existing environment libraries require either non-Python languages for custom environments or lack hardware acceleration, and existing experiment platforms do not jointly sup8
port standard simulation environment APIs, real-time multi-human interaction, and client-side execution. C O G RID and MUG are designed to address these gaps.
3
C O G RID
All C O G RID environments are partially observable Markov decision processes (POMDPs), described by the tuple (X , A, O, T , R, Ω). Here, X is the state space, A the action space, O the observation space, T : X × A → X the transition function, R : X × A → R the reward function, and Ω : X → O the observation function. We first review the components of Minigrid that we have utilized in C O G RID, then detail the improvements made and the motivations behind each.
Minigrid Functionality. Minigrid (Chevalier-Boisvert et al., 2023) is a single-agent grid-world library built on the Gymnasium API (Towers et al., 2023). Environments are 2D n × m grids in which each cell is either empty or occupied by a WorldObj. The environment tracks the agent’s position, inventory, and direction separately from the grid, rather than representing the agent as a WorldObj—this is the primary reason, at the implementation level, that Minigrid does not support multiple agents. Minigrid provides fixed default observation, action, and reward interfaces: a partial grid view with agent direction and a mission string; a discrete action space for rotation, movement, and object interaction; and sparse rewards upon mission completion. Because Minigrid has no hardware-accelerated backend, simulation speed can become a bottleneck during large-scale training runs. NAVIX (Pignatelli et al., 2024) addresses this by reimplementing Minigrid in JAX for hardware-accelerated simulation, but it retains the single-agent design.
C O G RID’s Dual Backend.
All simulation code in C O G RID operates through a backend-agnostic ar-
ray namespace (cogrid.backend.xp) that dispatches to either NumPy or JAX. When the JAX backend is active, environment functions are automatically just-in-time (JIT) compiled, and jax.vmap can be used for batched execution across many environment instances in parallel. With the NumPy backend, the same code runs without a JAX dependency, which is critical for settings that require Pyodide-compatible libraries (e.g., for client-side execution via WebAssembly in MUG). This singleimplementation approach distinguishes C O G RID from libraries such as JaxMARL (Rutherford et al., 2024), which target GPU-accelerated training but cannot run in the browser. A concrete illustration of the benefits of GPU acceleration is provided in Section 5.1 where we demonstrate empirical results 9
in an example environment. In practice, users can swap backends simply by changing the backend argument in environment construction: cogrid.make("MyEnvironment", backend="numpy") or cogrid.make("MyEnvironment", backend="jax"). C O G RID Agents.
C O G RID defines agents as their own Agent object, inheriting from the same GridObj
class that all other environment objects do. This allows the environment to track an arbitrary number of agents within the Grid. C O G RID Observations.
Rather than setting defaults for agent observations, we have chosen to customize
the observation space individually for each environment. Environments specify features by name in a configuration file. Each Feature subclass provides a pure function that maps the current state to an observation vector. At initialization, the selected features are composed into a single observation function that concatenates outputs and composes them into a single observation for each agent in the environment. C O G RID Actions. The action space in C O G RID environments is, by default, similar to that of Minigrid. The default action space is discrete and consists of "turn left", "turn right", "move forward", "pickup or drop", "toggle", and "no-op". Picking up and dropping are consolidated into a single action, with the environment logic depending on which cell the agent is facing. The "no-op" action allows agents the option to do nothing. This has a number of use cases, such as an agent waiting for another. Beyond this default action set, we also provide direct movement actions that eliminate rotations. This allows the "turn left", "turn right", and "move forward" actions to be replaced with "move left", "move right", "move up" and "move down". The agent rotation is then changed to correspond with the direction moved. The motivation for this change, along with the unification of picking up and dropping, was to make the controls more intuitive for a human player if they were controlling an agent in the environment or interacting with other agents. Actions can be added or removed as long as the corresponding environment logic is added in the environment loop. The active action set is selected via the environment configuration. C O G RID Rewards. Similar to the observations in C O G RID, rewards have been modularized into their own class to allow increased flexibility and easy customization. Each Reward subclass implements a compute method that receives both the previous and current state together with the actions taken, and returns a per-agent reward array. C O G RID allows for abstractions that will automatically build rewards that are compatible with JAX’s JIT-compilation and parallel execution, alleviating complexity for researchers. 10
For example, we provide the InteractionReward base class that allows users to specify a small set of arguments to define custom rewards around basic interactions with objects. Further detail is provided in Appendix A and the online documentation.
C O G RID Visualization.
C O G RID retains an identical visualization scheme as Minigrid. In particular,
every object in the environment has an associated render function that adds to an RGB tile rendered at that object’s location. It is possible to make these tiles arbitrarily complex; however, most commonly, the images are based on graphical primitives defined through Minigrid’s rendering utilities (e.g., drawing circles, squares, and lines). An example visualization of a simple team-based search and rescue task2 is shown in Figure 1.
class GridAgent(GridObj): [...] def render(self, tile_img): tri_fn = point_in_triangle( (0.12, 0.19), (0.87, 0.50), (0.12, 0.81), ) # Rotate based on agent direction tri_fn = rotate_fn( tri_fn, cx=0.5, cy=0.5, theta=0.5 * math.pi * self.dir ) fill_coords( tile_img, tri_fn, self.color )
(a) A visualization of the full map from the C O G RID search_rescue environment. Agents are represented by triangles that point in their current direction. In this environment, two agents must work together to rescue “victims” in the environment (green, red, and yellow circles). There is rubble that must be cleared by picking up a pickaxe and doors that can only be opened by picking up a key. Yellow victims can only be rescued when an agent is holding the medical kit (red square with a white cross), and red victims require both agents to rescue them simultaneously.
(b) Every object has a render function, which can be arbitrarily complex. The rendering shown here uses the Minigrid primitives for drawing a triangle for an agent, rotated based on the agent’s direction.
Figure 1: Visualization of an example environment and sample code to draw the GridAgent object. The default visualization utilizes the rendering functions developed for Minigrid.
2
This task is included in C O G RID as an adaptation of the Minimap task developed by Nguyen and Gonzalez, 2023.
11
4
Multi-User Gymnasium (MUG)
While C O G RID provides a library for building grid-based multi-agent environments, MUG serves as a library for conducting interactive experiments with such multi-agent environments. Importantly, C O G RID is not a prerequisite for MUG: it is designed to be compatible with any environment that follows the same standardized API and can be customized to work with those that do not. We constructed the interface to allow experiments to be designed as a sequence of “scenes” (that is, Scene objects). All scenes represent configurations that a researcher can fill out to create a component of their experiment. Human interaction with simulation environments is achieved through a specific type of scene: a GymScene. Here, a researcher will specify all the information related to their simulation environment, how users interact with it, and how it is displayed to a participant in their study. This includes mapping keystrokes to actions, providing a mechanism to instantiate simulation environments, specifying AI policies, among much more. Behind the scenes, MUG facilitates client-server communication in order to send relevant data to the client, where a game engine—i.e., Phaser—is running to display the current state of the simulation to the participant. Any required information is then transmitted back from the client to the server to be processed and stored for analysis.3 An important specification that can be made is where the environment is executed. A typical approach to allow Python-based applications to interact with web-based clients would be to have a continuous communication loop where all Python code is executed on the server and updated information is passed to the client on each “tick.” This is the approach of both SHARPIE and HIPPO-Gym, and it works in cases where participants have very low latency connections to the server or the tick rate of the environment is low. In cases where this communication is frequent (e.g., a simulation environment updating many times a second), network latency can make this approach ineffective. The novel approach we have developed in MUG is to execute Python-based environments directly in client browsers using Pyodide, which eliminates the need for this communication and allows researchers to serve their experiments to participants who may have high-latency connections. The distinction and general construction of GymScenes are shown in Figure 2. Client-side execution removes the latency problem for single-player experiments. However, multiplayer introduces a new challenge. If each client runs its own copy of the environment, their states must remain synchronized. The common approach in existing work, where each client waits for all players’ inputs 3
Data storage may take any form defined by the user through our flexible API. Trajectories are currently automatically exported as sequences of states and actions on the server; however, researchers can define any data storage format or location (e.g., external database).
12
before stepping, reintroduces the same latency that client-side execution was designed to avoid. To address this, MUG implements a form of GGPO (Cannon, 2019) rollback netcode, a technique originally developed for latency-sensitive fighting games. Each client runs the environment locally with a shared random seed, applies its own input immediately, and predicts the other remote players’ input (by default, repeating their last known action). When another player’s input arrives that differs from the prediction, the client rolls back to the last confirmed state and replays the intervening frames with the correct inputs. This process is illustrated in Figure 3. Critically, to enable this functionality the environments must be deterministic given a random seed and must implement state retrieval and setting (get_state() and set_state() methods for MUG compatibility). Beyond the ability to define interaction with simulation environments in a GymScene, we have created a collection of alternative scenes that have distinct functionalities and customization. This enables researchers to modularize their experiments and integrate customized components or pages that do not require interaction with simulation environments. For example, static informational pages can be created alongside surveys or alternative forms of interaction. When constructing an experiment, all scenes are collected into a Stager that defines their order—and any manipulation (e.g., randomization)—and the flow of the experiment. This is illustrated in Figure 4. Data Collection.
In MUG, data collection is handled through hooks that are called throughout the envi-
ronment loop. Users can define GameCallbacks that provide hooks at each key environment event, such as episodes starting and ending, each environment step, players joining and leaving, among others. This provides functionality for data to be recorded (e.g., actions and environment state at each environment step) and saved either locally or to an external database. By default, all information returned in the infos dictionary from the environment is logged at every step and stored on the disk. If a use case requires logging customized data to an external database, for example, a user would write their explicit logic into a GameCallback that logs data to that source at a desired frequency. Additional Functionality. Beyond the core experiment loop, MUG provides several utilities common to online behavioral research. Completion codes can be generated automatically for integration with recruitment platforms such as Prolific or Amazon Mechanical Turk. Scene order can be randomized or counterbalanced at the participant level through the Stager. Participants can also be filtered on the basis of browser compatibility, device type, or measured network latency before they enter the experiment. Similarly, for multi-player experiments, we provide an interface for creating custom matchmaking logic. This 13
á Server
Ô
3
õ
Æ
s
Config.
Python Env.
Database
AI
Game Engine
Client
Client-Server Loop at
á
st
3
Ô
õ
s
Æ
Client Execution ⟨s0 , a0 , r0 , ..., sT , aT , rT ⟩
á
Ô
Ô
s
õ
3
Æ
Figure 2: The two ways in which we run simulation environments in GymScenes: either through a continuous client-server communication loop or by executing the Python environment code directly in the client’s browser. In the former, the server maintains the environment and interactions with it—as well as all AI agents—and simply accepts data from the user in the form of actions at , while providing updated states st at every step. In the second case, the server provides all data necessary—through the MUG configuration— for the client to execute the Python code in their browser.
14
a1t+1
a1t ŝt+1
st
a1t+2 st+2
T â2t
st
T (st , a1t , a2t )
st+1
a2t:t+1
T (st+1 , a1t+1 , a2t+1 ) Confirmed
st+3 T a2t+2
st+2
Predicted
Re-simulated
Figure 3: Illustration of GGPO rollback netcode in a two-player environment. The main timeline shows the simulation state at each tick; each transition T (st , a1t , a2t ) → st+1 requires both players’ actions. Client 1’s actions (a1 ) arrive on time at every tick. Client 2’s action at t+1 is delayed, so Client 1 predicts it (â2 ), producing a speculative state. When the delayed input arrives as a bundle a2t:t+1 at t+1 (green), the client rolls back to the last confirmed state st and re-simulates with the correct actions (green timeline), merging the corrected state back into the main timeline. Rollback occurs without rendering re-simulated frames and it takes place between rendered ticks, causing minimal visual disruption.
can be configured based on a number of attributes, from task performance to peer-to-peer latency. Full descriptions of these features and their usage are available in the online documentation.
5
Proof of Concept: Case Studies
To illustrate the functionality of C O G RID and MUG, we provide several demonstrations of experimental use cases: pure simulation, human-AI interaction, and human-human interaction. Any extension or variation is possible within these regimes, including multiple humans with any number of AI agents. These case studies are not intended as definitive psychological experiments, but as proof-of-concept demonstrations showing how C O G RID and MUG come together to offer (1) environment customization, (2) integration of trained agents into human-AI studies, and (3) end-to-end browser-based experiment deployment with Python environments and variable numbers of human participants. We provide methodological details to illustrate how researchers can adapt these pipelines for their own questions. For transparency, we also report the experimental design elements used in these demonstrations (e.g., recruitment, task structure, compensation). These details are provided not as empirical validation, but to illustrate the methodological templates that researchers can adapt for their own studies. All experiments conducted here use server-side execution for Python environments but are also compatible with browserbased execution.4 4 See the online documentation at multi-user-gymnasium.readthedocs.io for examples of browser-based versions of these experiments.
15
Stager
StartScene
GymScene
SurveyScene
EndScene
Figure 4: MUG experiment flow using a Stager and sequence of Scenes. The Stager defines the flow of the experiment and also has the capabilities to manipulate the order or assignment at the participant level for experiments with multiple conditions.
16
The first use case is solely in the C O G RID environment, demonstrating the ease of defining a new multiagent environment and training a reinforcement learning agent in that environment. We recreated the popular Overcooked environment originally developed by Carroll et al. (2020). In it, we trained a reinforcement learning policy with the Proximal Policy Optimization (PPO; Schulman et al., 2017) algorithm to complete the task. This process is detailed in Appendix A.2, and the efficacy of the agent and environment is demonstrated in the following experiments. Next, we demonstrate how MUG can be utilized for interactive human studies, both with and without integration with C O G RID. For the former, we demonstrate how agents trained in the C O G RID Overcooked environment can be integrated into a human-AI experiment. For the latter, we use an external environment that is not grid-based, Slime Volleyball (Ha, 2020). Lastly, we conducted studies with humans alone. In addition to human-AI experimentation, MUG enables researchers to run experiments where all actors are controlled by human players. Use cases can include data collection for imitation learning or simply collecting human data for the analysis of human-human interactions. All studies with human participants were conducted with approval from Carnegie Mellon’s Institutional Review Board, and participants were sourced from Amazon Mechanical Turk. For approximately 15 minutes of total experiment time, participants were compensated with a $1.50 base payment and an additional $1.50 possible bonus payment based on task performance: $0.05 per point scored in Slime Volleyball and $0.03 per dish delivered in Overcooked, capped at $1.50.
5.1
C O G RID Overcooked
The first case study illustrates how C O G RID supports flexible grid-based environment design and customization, and how MUG then enables us to use such environments for human-human and human-AI web-based experiments. We first demonstrate the benefits of the dual-backend approach to C O G RID. Hardware acceleration in JAX is a result of being able to parallelize operations, enabling running many instances of the environment at the same time. To illustrate the magnitude of the speedup, we show the efficiency gain when increasing the number of parallel instances from 1 to 1,024 in Figure 5. Hardware acceleration offers a drastic increase in throughput, and C O G RID demonstrates competitive performance with JaxMARL. 17
Figure 5: Environment throughput in the CoGrid Overcooked environment, comparing to the original Overcooked-AI (Carroll et al., 2020) and JaxMARL (Rutherford et al., 2024) implementations. The former has a constant rate of roughly 3,400 steps per second, while the latter scales from roughly 4,300 with a single instance to 2.9 million with 1,024 parallel instances. C O G RID’s JAX backend is competitive: scaling from 4,500 steps per second with a single instance to 5.6 million with 1,024 instances, leading to a 1.9x throughput improvement. C O G RID’s NumPy backend is by far the slowest at roughly 450 steps per second; however, it offers a a mode that enables browser-based execution via WebAssembly that is entirely absent with JaxMARL. Hardware accelerated execution was run on a single NVIDIA GeForce RTX 3090.
As an illustration of C O G RID, we implemented a replica of the Cramped Room layout in the Overcooked-AI environment originally developed by Carroll et al. (2020). In this cooperative task, two players coordinate to prepare and deliver onion soup, which requires coordinated sequential actions to combine ingredients, cook, and plate. Figure 6a shows the layout rendered using the basic rendering functions. This case study demonstrates how C O G RID enables researchers to reproduce complex cooperative environments with minimal effort. The full implementation and environment details are described in the online documentation. Training an Overcooked Agent. We then trained a reinforcement learning agent in the C O G RID Overcooked environment using RLlib (Liang et al., 2018) with the Proximal Policy Optimization (PPO; Schulman et al., 2017) algorithm. Training used self-play,5 and the trained agent achieved an average of 7.5 dishes delivered per episode. The training that produced the policy in the following experimentation was run using 5 Self-play is a training paradigm in multi-agent reinforcement learning where one policy controls all agents and interacts with itself.
18
the NumPy backend of C O G RID, rather than the JAX-accelerated variant. An illustration of training with the JAX backend, which takes a small fraction of the time, is provided in the documentation and source repository.
Human Experiments via MUG. Finally, we deployed the environment in a browser-based study using MUG. MUG allows for the specification of all required settings for experimentation, including mapping keyboard buttons to actions, creating landing pages and surveys, and specifying the environment display format. The experience for participants in the study can be seen in Figure 6b. These human experiments demonstrate how MUG can transform a C O G RID environment into an interactive online experiment. We conducted Human-Human and Human-AI studies through separate solicitations. We recruited 29 participants for the Human-AI study and 40 participants (20 pairs) for the Human-Human study through Amazon Mechanical Turk. Each participant completed 20 episodes of the Overcooked task with either the same human partner or a fixed reinforcement learning partner policy, as previously described. Participants first completed a consent form, then proceeded to the MUG landing page with task instructions and a start button. Participants in the Human-Human study were sent to a waiting room (minimum duration of 5 seconds) until they were paired with another participant. Human-AI study participants saw a simulated waiting room (randomized duration of 5-25 seconds). After completing the 20 episodes, all participants were redirected to a post-experiment questionnaire. The questionnaire asked a series of questions about relative contributions and behavior, including whether or not the participant thought their partner was a human or an AI. To ensure data quality, we applied exclusion criteria (participants with inactivity at or above 97.5% or those who left the webpage for 90% of an episode), leaving 23 participants in our Human-AI study and 36 (18 pairs) in the Human-Human study.
Results.
These results illustrate the types of analyses enabled by data collected through MUG. The per-
formance results for the Human-Human and Human-AI pairs for each episode are shown in Figure 7. The relative performance of AI-AI pairs and Human-AI pairs is consistent with the original results of Carroll et al. (2020), with AI-AI pairs (roughly 7.5 dishes per episode) substantially outperforming human-AI pairs. Human-AI pairs showed relatively stable performance across episodes, with no statistically significant change in performance over time. In contrast, human-human pairs improved steadily. The stability of human-AI performance is consistent with the findings of Carroll et al. (2020): the reinforcement learn19
(a) The C O G RID Overcooked Cramped Room implementation. The two chefs are represented by triangles, three yellow circles represent a stack of onions, three white circles a stack of plates, the grey circle a pot, and the green square the delivery zone.
(b) A view of the web page that participants saw after they started the task in their browser. The graphics used to render the game are taken from Carroll et al. (2020) and their original open-source implementation of Overcooked-AI. The instructions and icons are part of a customized HTML file that is passed to the MUG configuration. The score and time left display are part of a dynamic block, updated with a Python function that returns the values to populate the display.
Figure 6: The Overcooked visualization from C O G RID’s native rendering and visualized in the browser with MUG. The latter uses the assets originally used by Carroll et al. (2020).
20
Figure 7: Performance across episodes in both Overcooked studies. The reinforcement learning agent used, when playing with itself with a frame-skip of five, delivered an average of 7.5 dishes per episode (not displayed in the figure). Shaded regions represent standard error.
ing agent can complete most of the task independently, so although humans likely continue to learn, their improvement is masked by the agent’s high baseline capability. In contrast, human-human pairs must develop shared strategies from scratch, and the upward trend reflects this gradual coordination process. We also show the relative contributions of humans and AI in the Human-AI study, as shown in Figure 8. We measured the number of instances where either the human or AI delivered the dish to the delivery zone. Over time, humans adapt their behavior and begin contributing a larger share of deliveries. In interacting with AI, humans appear to be more inclined to complete the dish delivery, rather than taking on the role of putting onions in the pot, even though it results in no observable change in overall task performance. The results in Figure 8 also demonstrate the types of behavioral phenomena that can be captured through studies in human-AI interaction. This kind of behavioral trend—constant performance but varying contributions—exemplifies psychological or cognitive questions in human-AI interaction that can be pursued with MUG and C O G RID. 21
Figure 8: The relative contributions in the Human-AI study, the number of dishes delivered by the human and AI separately.
5.2
Slime Volleyball
Next, to complement the Overcooked case study, we demonstrate MUG’s compatibility with environments not based on C O G RID. We integrated Slime Volleyball (Ha, 2020) as a recreation of a classic 2D video game. In the game, players move laterally and jump to keep a ball off the ground on their side of the court. A game is won when the ball lands on the court of the other player, similar to volleyball. We again trained a reinforcement learning agent using the PPO algorithm. The environment was also fully ported through MUG. Full training and implementation details are provided in Appendix B. Figure 9 shows the original game visualization, as well as the MUG participant view. The experimental procedure mirrored that of the Overcooked study—the flow from the consent form, to instructions, waiting room, gameplay, and then the follow-up questionnaire remains unchanged. Participants completed 30 episodes of the task, each consisting of a single point. We recruited 21 participants for the Human-AI condition and 22 participants (11 pairs) for the Human-Human condition from Amazon Mechanical Turk. As before, we excluded participants for data quality. Participants were excluded in instances 22
(a) The original Slime Volleyball visualization from Ha (2020).
(b) Participants’ view of Slime Volleyball during the study, as implemented in MUG.
Figure 9: The original and MUG Slime Volleyball interfaces.
23
where they exceeded a maximum RTT,6 navigated away from the web page for 90% of the experiment, or were inactive (97.5% no-op actions). This resulted in dropping one participant from the Human-AI study.
Results.
Results are shown in Figure 10. Because Slime Volleyball is a zero-sum game, cumulative score
is not an informative measure; instead, we report the episode length as a proxy for player skill. Longer episodes indicate a greater ability to keep the ball in play. In these short sessions, we observed little-to-no improvement over time. Human–AI pairs maintained longer rallies than Human–Human pairs, reflecting the trained agent’s higher baseline skill. Indeed, AI–AI matches often reached the maximum length of 3,000 steps. Figure 11 further shows that the AI maintained higher “possession,” keeping the ball on its side for longer periods. The relatively flat episode lengths reflect the difficulty of the task: Slime Volleyball is a fast-paced motor task requiring precise timing with unintuitive physics, and 30 episodes provides limited practice for substantial skill improvement. Human-AI pairs sustain longer rallies because the trained agent reliably returns the ball, keeping it in the air for longer. Human-human pairs’ episode length does not benefit from the high skill of the AI. Although average performance remained relatively low, some participants substantially exceeded baseline, with the longest Human–AI match lasting 1,169 steps and the longest Human–Human match 538 steps. The Slime Volleyball study demonstrates that MUG is not limited to grid-based environments created with C O G RID. The support for fast-paced, physics-based tasks illustrates MUG’s flexibility for running studies with rich interaction dynamics and complexity, broadening the scope of potential investigation.
6
Discussion
Our contributions aim to expand the ecosystem of platforms for multi-agent research. Existing environments and libraries (e.g., Agapiou et al., 2022; Carroll et al., 2020; Terry et al., 2021) have provided standardized testbeds for AI agents, but they aren’t designed to prioritize customization or accessibility. Similarly, work in human-AI teaming has been carried out using non-standard or bespoke experimental platforms that have significant barriers to entry, preventing researchers from extending them or investigating research questions with modern AI. C O G RID and MUG help to solve these problems by enabling simple customization of multi-agent environments and uniquely providing a standardized method to translate simulation environments into web-based experiments, respectively. Our framework complements existing 6 The RTT is the “round trip time” of a message to go from the server to the participant and back. Given the fastpaced nature of the game, poor connections result in negative experiences.
24
Figure 10: The average episode length over time in the human-AI and human-human studies of Slime Volleyball. The average episode length is an effective proxy for skill because it measures how long the ball is kept in the air, which requires competency. Shaded regions represent standard error.
platforms by allowing researchers to move beyond isolated simulations and move to empirical evaluations with human participants. Our work on C O G RID extends the popular Minigrid library to the multi-agent domain, allowing researchers to easily define their own environments with minimal overhead and barriers to entry. It modularizes the components of Minigrid to enable arbitrary manipulation of environments and customizable execution logic in grid-based environments. We illustrated C O G RID’s functionality through a case study of implementing Overcooked (Carroll et al., 2020) in C O G RID, demonstrating the streamlined process by which new objects and interactions can be added to an environment. Multi-agent environments, like those that can be designed with C O G RID, provide opportunities to investigate interactions between agents—which is particularly important as AI is introduced into our social world. Where other libraries are primarily focused on benchmarks and standardized environments (e.g., PettingZoo (Terry et al., 2021), Melting Pot (Agapiou et al., 2022), Overcooked-AI (Carroll et al., 2020)), C O G RID emphasizes and prioritizes modularity, accessibility, and customizability. This focus situates C O G RID as a library that will support researchers in asking questions about novel interactions not represented by current 25
Figure 11: The average number of timesteps per episode in the Human-AI study where the ball is in possession of the human or AI. Possession is defined by the ball being on the corresponding side of the middle post. The AI agents have higher possession compared to the human counterparts. Shaded regions represent standard error.
26
benchmarks. The primary limitation of C O G RID is the grid-based nature of the environments. Although substantial complexity can arise in grid-based interactions, it excludes physics-based or continuous control settings that may be of interest in other domains. In addition to C O G RID, MUG is a unique contribution that establishes a streamlined process to translate simulation environments to interactive experiments in the browser. Where environments would typically have to be reimplemented to run in an interactive experiment, MUG provides a tool for researchers to produce a multiplayer game directly from a Gymnasium or PettingZoo environment. Through demonstrations with Overcooked and Slime Volleyball, we showed how MUG facilitates both Human-AI and Human-Human experiments without requiring bespoke reimplementations. These libraries lower technical barriers and allow researchers to study human-AI interaction in an efficient and scalable manner. A significant limitation is that MUG does not currently support explicit communication between participants or between humans and AI. All interaction is implicit, mediated through actions and observations in the shared environment. There is no built-in text chat or structured messaging system, which limits applicability to studies that require communication. Future work will address this by incorporating communication capabilities into MUG in the form of text-based chat. Furthermore, MUG is currently aimed at interactions within simulation environments, but could be extended to support complex interactions that include communication, demonstration, preference elicitation, and other advanced features. Despite these limitations, there are significant opportunities enabled by this framework. With C O G RID, researchers can prototype novel multi-agent scenarios or dilemmas to answer their particular questions rather than relying on pre-defined settings. With MUG, such environments can be brought online to study how humans interact with the AI that was trained in them to answer questions on perception, preferences, and behavior in human-AI interaction. Through the introduction of C O G RID and MUG, we have demonstrated a simple setting to develop multiagent simulation environments and a framework that streamlines the process of developing interactive experiments in simulation environments. The goal of this work is to support research in multi-agent systems, particularly in human-AI interaction and complementarity. By providing tools that support the rapid development and testing of ideas, we hope to contribute to progress in building autonomous agents that better serve the humans with whom they interact in the real world.
27
Open Practices Statement All source code, documentation, and tutorials for C O G RID and MUG are available at cogrid.readthedocs.io and multi-user-gymnasium.readthedocs.io, respectively. The scripts to recreate the experiments are available in the respective code repositories.
Declarations Funding This research was supported by the Defense Advanced Research Projects Agency and was accomplished under Grant Number W911NF-20-1-0006 and by the NSF AI Institute for Societal Decision Making (AI-SDM), Award No. 2229881. Conflicts of Interest
The authors have no competing interests to declare that are relevant to the content
of this article. Ethics Approval Ethics approval was received by the Institutional Review Board at Carnegie Mellon University under the project Building Human-Machine Shared Mental Models, study STUDY2021 00000324. Consent to Participate Informed consent was obtained from all individual participants. Consent for Publication The participants gave their consent for the publication of their behavioral data. Availability
of
Data
and
Materials Data
are
available
through
cogrid.readthedocs.io
and
multi-user-gymnasium.readthedocs.io. Code Availability
All source code and documentation are available through cogrid.readthedocs.io and
multi-user-gymnasium.readthedocs.io.
References Agapiou, J. P., Vezhnevets, A. S., Duéñez-Guzmán, E. A., Matyas, J., Mao, Y., Sunehag, P., Köster, R., Madhushani, U., Kopparapu, K., Comanescu, R., et al. (2022). Melting pot 2.0. arXiv preprint arXiv:2211.13746.
28
Aydın, H., Godin-Dubois, K., Braz, L. G., Hengst, F. d., Baraka, K., Çelikok, M. M., Sauter, A., Wang, S., & Oliehoek, F. A. (2025). Sharpie: A modular framework for reinforcement learning and human-ai interaction experiments. arXiv preprint arXiv:2501.19245. Balietti, S. (2017). Nodegame: Real-time, synchronous, online experiments in the browser. Behavior research methods, 49, 1696–1715. Bamford, C., Huang, S., & Lucas, S. (2022, July). Griddly: A platform for AI research in games [Number: arXiv:2011.06363 arXiv:2011.06363 [cs]]. https://doi.org/10.48550/arXiv.2011.06363 Bewley, T., Lawry, J., & Richards, A. (2021). Hippo gym: Facilitating human-in-the-loop reinforcement learning experiments. Workshop on Reinforcement Learning Tools and Environments, AAAI. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., & Zhang, Q. (2018). JAX: Composable transformations of Python+NumPy programs (Version 0.3.13). http://github.com/jax-ml/jax Cannon, T. “. (2019). GGPO: Good game, peace out rollback network sdk. Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., & Dragan, A. (2020). On the Utility of Learning about Humans for Human-AI Coordination [arXiv: 1910.05789]. arXiv:1910.05789 [cs, stat]. Retrieved July 6, 2021, from http://arxiv.org/abs/1910.05789 Chevalier-Boisvert, M., Dai, B., Towers, M., de Lazcano, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., & Terry, J. (2023, June). Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks [Number: arXiv:2306.13831 arXiv:2306.13831 [cs]]. https : //doi.org/10.48550/arXiv.2306.13831 Duan, W., Zhou, S., Scalia, M. J., Yin, X., Weng, N., Zhang, R., Freeman, G., McNeese, N., Gorman, J., & Tolston, M. (2024). Understanding the evolvement of trust over time within human-ai teams. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2), 1–31. Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J., & Whiteson, S. (2024). Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36. (FAIR)†, M. F. A. R. D. T., Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. (2022). Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067–1074. Fickinger, A. (2020). Multi-agent gridworld environment for openai gym.
29
Gerstgrasser, M., Trivedi, R., & Parkes, D. C. (2021). CrowdPlay: Crowdsourcing human demonstration data for offline learning in Atari games. Retrieved February 20, 2022, from https://openreview. net/forum?id=qyTBxTztIpQ Glazer, J., & Madhav, S. (2015). Multiplayer game programming: Architecting networked games. Addison-Wesley Professional. Ha, D. (2020). Slime volleyball gym environment. Hafner, D. (2021). Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780. Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Rı́o, J. F., Wiebe, M., Peterson, P., . . . Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585(7825), 357–362. https://doi.org/10.1038/s41586-020-2649-2 Hughes, E., Leibo, J. Z., Phillips, M., Tuyls, K., Dueñez-Guzman, E., Garcı́a Castañeda, A., Dunning, I., Zhu, T., McKee, K., Koster, R., et al. (2018). Inequity aversion improves cooperation in intertemporal social dilemmas. Advances in neural information processing systems, 31. Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J. Z., & De Freitas, N. (2019). Social influence as intrinsic motivation for multi-agent deep reinforcement learning. International conference on machine learning, 3040–3049. Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T. (2017). Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03037. Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., Gonzalez, J., Jordan, M., & Stoica, I. (2018). Rllib: Abstractions for distributed reinforcement learning. International conference on machine learning, 3053–3062. Ndousse, K. (2020). Marlgrid. Nguyen, T. N., & Gonzalez, C. (2023). Minimap: An interactive dynamic decision making game for search and rescue missions. Behavior Research Methods, 1–22. Norén, J. F. W. (2020). Derk gym environment. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 27730–27744.
30
Perolat, J., De Vylder, B., Hennes, D., Tarassov, E., Strub, F., de Boer, V., Muller, P., Connor, J. T., Burch, N., Anthony, T., et al. (2022). Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623), 990–996. Pignatelli, E., Liesen, J., Lange, R. T., Lu, C., Castro, P. S., & Toni, L. (2024). Navix: Scaling minigrid environments with jax. arXiv preprint arXiv:2407.19396. Russell, S. (2019). Human compatible: Ai and the problem of control. Penguin Uk. Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson, G., Willi, T., Hammond, R., Khan, A., de Witt, C. S., Souly, A., Bandyopadhyay, S., Samvelyan, M., Jiang, M., Lange, R. T., Whiteson, S., Lacerda, B., Hawes, N., Rocktäschel, T., . . . Foerster, J. N. (2024). Jaxmarl: Multi-agent rl environments and algorithms in jax. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Salikutluk, V., Schöpper, J., Herbert, F., Scheuermann, K., Frodl, E., Balfanz, D., Jäkel, F., & Koert, D. (2024). An evaluation of situational autonomy for human-ai collaboration in a shared workspace setting. Proceedings of the CHI Conference on Human Factors in Computing Systems, 1–17. Samvelyan, M., Kirk, R., Kurin, V., Parker-Holder, J., Jiang, M., Hambro, E., Petroni, F., Küttler, H., Grefenstette, E., & Rocktäschel, T. (2021). Minihack the planet: A sandbox for open-ended reinforcement learning research. arXiv preprint arXiv:2109.13202. Schelble, B. G., Flathmann, C., McNeese, N. J., Freeman, G., & Mallick, R. (2022). Let’s think together! assessing shared mental models, performance, and trust in human-agent teams. Proceedings of the ACM on Human-Computer Interaction, 6(GROUP), 1–29. Schmutz, J. B., Outland, N., Kerstan, S., Georganta, E., & Ulfert, A.-S. (2024). Ai-teaming: Redefining collaboration in the digital era. Current Opinion in Psychology, 101837. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms [arXiv: 1707.06347]. arXiv:1707.06347 [cs]. Retrieved August 24, 2021, from http://arxiv. org/abs/1707.06347 Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021). Reward Is Enough. Artificial Intelligence, 103535. https://doi.org/10.1016/j.artint.2021.103535 Skrynnik, A., Andreychuk, A., Yakovlev, K., & Panov, A. I. (2022). Pogema: Partially observable grid environment for multiple agents. https://doi.org/10.48550/ARXIV.2206.10944
31
Strouse, D. J., McKee, K. R., Botvinick, M., Hughes, E., & Everett, R. (2021). Collaborating with Humans without Human Data [arXiv: 2110.08176]. arXiv:2110.08176 [cs]. Retrieved November 29, 2021, from http://arxiv.org/abs/2110.08176 Sukhbaatar, S., Szlam, A., Synnaeve, G., Chintala, S., & Fergus, R. (2015). Mazebase: A sandbox for learning from games. arXiv preprint arXiv:1511.07401. Team, T. P. D. (2021, August). Pyodide/pyodide (Version 0.29.3). Zenodo. https://doi.org/10.5281/zenodo. 5156931 Terry, J., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sullivan, R., Santos, L. S., Dieffendahl, C., Horsch, C., Perez-Vicente, R., et al. (2021). Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 15032–15043. Towers, M., Terry, J. K., Kwiatkowski, A., Balis, J. U., Cola, G. d., Deleu, T., Goulão, M., Kallinteris, A., KG, A., Krimmel, M., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J. J., Shen, A. T. J., & Younis, O. G. (2023, March). Gymnasium. https://doi.org/10.5281/zenodo.8127026 Wilder, B., Horvitz, E., & Kamar, E. (2020). Learning to complement humans. arXiv preprint arXiv:2005.00582. Zhang, R., McNeese, N. J., Freeman, G., & Musick, G. (2021). ” an ideal human” expectations of ai teammates in human-ai teaming. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3), 1–25.
A
Overcooked Environment & Task
The full implementation of our adaptation of the Overcooked-AI environment (Carroll et al., 2020) is documented at cogrid.readthedocs.io. In this section, we provide an overview of the design of the environment and interactive task. All code to use the C O G RID Overcooked environment with MUG is available as an example in the MUG repository, linked at: multi-user-gymnasium.readthedocs.io.
A.1
C O G RID Overcooked
The Overcooked environment is implemented by subclassing CoGridEnv and defining the task-specific components through C O G RID’s registration system. Each component—objects, rewards, and features—is a Python class that is registered at initialization. The environment layout is specified via an ASCII grid encoding in the configuration file. Full source code and implementation details are available in the online documentation at cogrid.readthedocs.io. 32
Listing 1 shows the Onion object, illustrating how environment objects are defined.
The
@register_object_type decorator registers the class within the environment scope. Interaction capabilities are declared as class attributes using the when() descriptor—here, can_pickup = when() indicates that onions can always be picked up. Each object specifies a unique char attribute used for ASCII layout encoding.
1 2 3 4 5
@register_object_type("onion", scope="overcooked") class Onion(GridObj): color = Colors.Yellow char = "o" can_pickup = when()
Listing 1: The Onion object. The @register_object_type decorator registers the class and the when() descriptor declares that onions can be picked up. Additional objects (Plate, OnionSoup, DeliveryZone, etc.) follow the same pattern.
Rewards are defined using the InteractionReward base class. Instead of writing complex reward logic, subclasses declare the triggering conditions as class attributes: the action the agent must perform, the item it must hold, and the object it must face. The base class handles all condition checking and reward broadcasting. Listing 2 shows the delivery reward, which triggers when an agent performs a pickup/drop action while holding an OnionSoup and facing a DeliveryZone. For more complex conditions (e.g., checking pot capacity or cooking status) subclasses can override an extra_condition() method to specify additional conditions.
1 2 3 4
class OnionSoupDeliveryReward(InteractionReward): action = "pickup_drop" holds = "onion_soup" faces = "delivery_zone"
Listing 2: The soup delivery reward using the declarative InteractionReward base. Only class attributes are needed: the action, what the agent holds, and what it faces.
Features for the observations follow a similar pattern, where users can define custom features by subclassing the Feature base class, and generate a function that maps from the current state of the environment to specific view of the environment (e.g., a one-hot encoding of which item an agent is holding). The full details and implementations are shown in the online documentation. 33
1 2 3 4
class OnionInPotReward(InteractionReward): action = "pickup_drop" holds = "onion" faces = "pot"
5 6 7 8 9
def extra_condition(self, mask, prev_state, fwd_r, fwd_c, reward_config): # Check pot has capacity for another onion return condition_satisfied
Listing 3: A reward with an additional condition. The extra_condition() override narrows the trigger mask to check pot capacity and ingredient compatibility. For the Overcooked environment, the observation features roughly correspond to those used by Carroll et al., 2020. For each agent j, we calculate the following features. The observation is then the concatenation of all agents’ feature arrays: • Agent j’s direction as a one-hot encoding. • Agent j’s inventory as a one-hot encoding of the possible inventory objects. • A multi-hot indicator of whether agent j is adjacent to a counter. • Agent j’s distance to the closest of each Onion, Plate, PlateStack, OnionStack, OnionSoup, and DeliveryZone. • Agent j’s pot features, consisting of: 1. an indicator of whether the pot is reachable, 2. a one-hot representation of the pot status, which can be empty, cooking, or ready; 3. the number of onions in the pot, 4. the number of cooking timesteps remaining for the pot, 5. an array of the row and column distances to the pot, 6. the row and column location of the pot. • Agent j’s distance to the other chef. • Agent j’s row and column position in the grid. A.2
Training a Reinforcement Learning Agent
We train a reinforcement learning agent in the Overcooked environment using RLlib (Liang et al., 2018) and the PPO algorithm. A dish delivery reward of 1.0 is given when a dish is delivered, and agents also 34
receive 0.1 reward when an onion is placed in a pot and 0.3 when a dish is plated. The full training script, including all hyperparameters, is available in the online documentation and source code. We also provide a complete example for training with full hardware acceleration (using the JAX backend). The trained policy delivers an average of roughly 7.5 dishes per episode when controlling both agents. A.3
Overcooked in MUG
The full source code is available in the examples of the MUG repository, documented at: multi-user-gymnasium.readthedocs.io. The online documentation provides all implementation details and configurations to launch Overcooked in MUG.
B
Slime Volleyball Environment & Task
The Slime Volleyball environment we use was originally developed by Ha, 2020. We adapted their implementation to conform to the Gymnasium API, but did not make any major alterations to the environment dynamics. The code for the Gymnasium formatted environment is available at https://github.com/ chasemcd/slimevolleygym. The observation ot ∈ O is a size 12 vector with the (x, y)-position and (x, y)-velocity for each player and the ball. We use a modified training regime for Slime Volleyball with a self-play curriculum. The agent trains against a fixed version of itself, which is updated every time the average reward against that agent exceeds 0.5— agents receive a reward of -1 for losing, 0 for ties, and 1 for winning. The full training script, including all hyperparameters, for Slime Volleyball are provided in the MUG documentation and source code. Over the course of training, the policy is able to achieve an average episode length of nearly the maximum of 3,000 steps in the Slime Volleyball environment. B.1
Slime Volleyball in MUG
The full configuration and implementation details to launch Slime Volleyball in MUG are documented at multi-user-gymnasium.readthedocs.io.
35