arXiv CSopen access

Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning

Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning

arXiv:2606.09610v1 [cs.RO] 8 Jun 2026

Mohamed Sayed1 , Wolfram Burgard1 and Tanja Katharina Kaiser1 Abstract— Cooperative object transportation is essential in numerous domains, including industrial to domestic services. A popular transportation strategy is to carry objects on top of multi-robot systems. The corresponding task is typically solved by decomposing it into three interconnected subproblems: formation control, cooperative navigation, and collision avoidance. A particular challenge posed by real-world objects is their potentially arbitrary shape and non-uniform mass distribution, necessitating robot formations that securely support the object. In this work, we address the challenge of pattern formation control for transporting such real-world objects by proposing a novel multi-agent reinforcement learning approach. Our approach enables a multi-robot system to autonomously position itself underneath an object to support its weight while avoiding obstacles during the formation process. Our evaluations with diverse environments and varying numbers of robots show that our approach leads to policies that reliably produce balanced formations and generalize to cluttered scenes and objects with complex geometry and non-uniform mass distribution.

I. INTRODUCTION Cooperative object transport is a central problem in the space of multi-robot systems (MRSs) with relevant applications in various domains, including logistics and warehouse automation. Existing strategies include pushing, grasping, and caging [1]. A different strategy is transporting an object on top of robots. A prominent example of this strategy are systems like Amazon’s Kiva robots [2]. In such systems, each robot transports a single, manageable unit (e.g., a shelf) by positioning itself underneath and lifting it. While effective, this approach relies on a structured interface between the robot and the object: shelving units and robots are designed to match in terms of form factor, lifting mechanism, and physical capabilities. Another approach is placing the object on top of a flexible sheet or platform that is jointly supported by several robots, which requires explicit modeling of tension and compliance effects [3]. While effective in controlled settings, these methods typically impose strong assumptions on the object’s geometry, mass distribution, or interface constraints. Moreover, the existing strategies primarily focus on navigation during transport, rather than on the prior challenge of forming a stable load-bearing formation beforehand. However, everyday objects often have irregular shapes and non-uniform mass distributions. For the transportation of such objects, we cannot rely on specifically designed interfaces and require formation control approaches that generalize to arbitrary objects. 1 All authors are with the Department of Computer Science and Artificial Intelligence, University of Technology Nuremberg, Germany.

[email protected]

(a)

(b)

(c)

Fig. 1. Overview of formation task for object carrying with (a) the scene description, as well as examples of (b) a valid formation and (c) invalid formations.

In this work, we focus on the initial phase of transport: forming a stable load-bearing formation beneath a passive rigid object of arbitrary shape and mass distribution (see Fig. 1). The robots start from random initial positions and must self-organize within the object’s footprint to achieve an even load distribution. A balanced initial formation is essential for safe and reliable transport afterward. To the best of our knowledge, we are the first to explicitly address the pattern formation problem for transporting objects with nonuniform mass distribution. We propose a multi-agent reinforcement learning (MARL) approach based on the centralized training decentralized execution (CTDE) paradigm in the physics-based simulation environment VMAS [4]. Robots learn decentralized policies that enable them to form load-aware support structures within an object’s footprint while avoiding collisions with each other and obstacles during the pattern formation process. Our experiments demonstrate that the learned policies generalize across cluttered environments, different object geometries and mass distributions, and varying numbers of robots. In summary, we make the following contributions: 1) We introduce a cooperative object transport scenario focused on load-bearing pattern formation under rigid objects with arbitrary shape and mass distribution. 2) We propose a simplified abstraction of the load-sharing problem using local load estimates and limited object footprint information. 3) We train decentralized MARL policies that enable autonomous pattern formation, balanced load distribution, and collision avoidance. 4) We demonstrate scalability and generalization across environments and object complexities.

II. RELATED WORK A. Cooperative Object Transport The task of transporting objects on top of a multi-robot system has been intensively studied in the past. It requires formation control, cooperative navigation, including formation marching, and reliable obstacle avoidance. Several previous works studying this transportation task rely on optimization methods [5], finite state machines [6], or genetic programming and decentralized negotiation strategies [7]. The transported object may either be firmly attached to the robots [7], [5], or loosely placed on top of the multi-robot system [6]. Some works assume a uniform mass distribution [6], which simplifies the formation control task by requiring robots to be approximately uniformly distributed. Others predefine target formations via robotobject attachment points [7] or inter-robot distances [5], thereby enabling the transport of objects with varying mass distributions. Although the robot formation is crucial for transporting objects on top of several robots, these works do not address how the target formation itself is determined. To the best of our knowledge, we are the first to explicitly focus on formation control under different mass distributions in the context of cooperative object transport. Pattern formation studies investigate how robots can arrange themselves into specific shapes, either by outlining [8] or by filling them [9], [10], [11]. Existing approaches include algorithmic methods [9], [12], potential field techniques [11], Large Language Models [13], and reinforcement learning [8]. However, these methods focus primarily on geometric shapes, rather than on mass distribution. Liu et al. [10] incorporate the center of mass (CoM) through an image moments-based approach. Nevertheless, different shapes can have the same CoM, and robots may end up outside the target shape in their approach. In contrast to these previous methods, our approach aims at achieving a uniform load distribution within the object footprint with all robots of the multi-robot system to ensure safe and stable transport. B. Multi-Agent Reinforcement Learning Learning-based approaches to robot control enable generalization across environments and address hard-to-engineer problems. In reinforcement learning (RL), robots train policies through trial and error, aiming to maximize a cumulative reward. Multi-Agent Reinforcement Learning (MARL) [14] extends this approach to multi-agent systems, introducing additional challenges such as non-stationarity, scalability, and credit assignment. The centralized training decentralized execution (CTDE) paradigm (see Fig. 2) addresses these challenges by improving learning efficiency while maintaining scalability [15]. During training, agents have access to shared or global information, whereas during execution, they solely rely on local observations. MARL has been applied to multi-robot systems in tasks such as order picking in warehouses [16], autonomous driving [17], and navigation [18]. Several works have also

Fig. 2.

Centralized Training Decentralized Execution Paradigm

applied MARL to the pattern formation problem. Iiama et al. [19] train agents in a 2D grid world to form patterns as quickly as possible, while Wang et al. [8] propose a CTDEapproach to train general-purpose pattern formation policies that can adapt to new formations with only minimal additional training. However, these methods rely on predefined target poses and do not consider object mass distributions. In contrast, we do not specify fixed robot positions in advance. III. METHODS A. Task Description In an environment with up to nO obstacles, a decentralized multi-robot system of N robots must arrange itself into a formation in order to carry an object of arbitrary shape and mass distribution on top of it. The object and nO obstacles are at random poses in the environment. We assume the object to be hovering over the robots, similar to being on stilts. Thus, robots cannot collide with the object. The footprint of the object is given by a 2D polygon P ⊂ R2 . We use three types of objects: simple ones with uniform density, more complex ones with non-uniform density, featuring both heavy and light regions, and complex objects with cavities and non-uniform density. The object’s center of mass CoM = (xc , yc ) is computed geometrically, accounting for the varying densities within the object, and is relevant with respect to the load distribution among robots. B. Problem Formulation We model the formation control subtask as a Partially Observable Markov Game (POMG), which captures the decentralized nature of our multi-robot setting. A POMG is defined as G = ⟨N , S, {Ai }, P, {Ri }, {Ωi }, O, γ⟩,

(1)

where • N is the set of agents i = 1, . . . , N , • S is the global state space, • Ai is the action space of agent i, and a = (a1 , . . . , aN ) is the joint action, ′ • P (s | s, a) is the state transition function,

where ∆pij = pj − pi are the relative positions and lj the loads of all other robots. Environment observation oenv is given by i o oenv i = Di

(7)

with vector Dio of nr equidistant distance measurements to ◦ static obstacles and agents covering 360 around the robot o with a fixed sensing radius Dmax . The target object observation oobj i is defined as (a) Local Observation

P oobj i = Di

(b) Local Frame agents

Fig. 3. Setting description. Information about teammates oi , obstacles obj oenv and target object oi are green, gray, and orange lines, respectively. i Current velocity vi is red arrow. Vectors are represented under robots’ local frames.

with nr -dimensional vector DiP of distance measurements to the boundary of the target object’s footprint P. Each distance measurement dP r is given by dP r = min rayr ∩ ∂P − pi

Ri (s, a) is the individual reward function for agent i, Ωi is the observation space of agent i, ′ • O(o | s , a) is the joint observation probability function, where o = (o1 , . . . , oN ), and • γ ∈ (0, 1) is the discount factor. At each time step t, agent i receives a local observation oti ∈ Ωi and selects an action ati according to its policy πθi (ai |oi ) parametrized by θi . The joint action at = (at1 , . . . , atN ) induces a state transition via P , and each agent receives a reward rit = Ri (st , at ). The objective of agent i is to maximize its expected discounted return Ji (π), as defined by "∞ # X t t t Ji (π) = Eπ γ Ri (s , a ) , π = (πθ1 , . . . , πθN ). (2) •

•

t=0

C. Agents, Observations and Actions Agents are modeled as circular non-holonomic robots with radius ra . Each agent’s policy πθi outputs linear and angular velocities vi and ωi , which are used to update the agent’s position (xi , yi ) and heading ψi using unicycle kinematics. The motion model is defined by ẋi = vi cos ψi ,

ẏi = vi sin ψi ,

ψ̇i = ωi .

(3)

Each agent receives a structured observation vector oi , which we define as robots env obj Sload oi = [oself , oi , oi , oi ] i , oi

(4)

robots , with self-observation oself i , observation of other robots oi env environment observation oi , target object observation oobj i , and similarity score observation oss i . Self-observation oself is given by i

oself = { pi , vi , ℓi } i

(5)

with robot position pi , linear velocity vi , and current load ℓi . The computation of a robot’s current load is described in more detail in Sec. III-D. We define the observation of other robots orobots as i orobots = { ∆pij , ℓj } j̸=i , i

(6)

(8)

2

,

r = 1, . . . , nr

(9)

with rayr representing the synthetic LiDAR rays. In addition, we include similarity score observation oSi load derived from the observed load values. It indicates the similarity of the current robot loads to an even load distribution, as explained in detail in Sec. III-D. This aggregated observation provides a compact summary of the team progress towards a successful carrying formation. D. Load Computation and Sharing Only robots located inside the object’s footprint polygon (I = {i ∈ {1, ..., n}| pi ∈ P}) will participate in the object transport. For each of these robots, we compute the relative vector to the object’s CoM as pci = pi − CoM = (xci , yic ),

i ∈ I.

The static equilibrium conditions are given by X X X ℓi = w, xci ℓi = 0, yic ℓi = 0, i∈I

i∈I

(10)

(11)

i∈I

where w is the object weight. We express this system in matrix form: A LI = B     ℓI1   1 1 ··· 1 w ℓI2    xcI xcI · · · xcI   .  =  0  (12) 1 2 k  ..  0 yIc 1 yIc 2 · · · yIc k | {z } | {z } ℓIk | {z } B (3×1) A (3×k) LI (k×1)

with the total number k of robots inside the object footprint. The loads vector of the robots inside the footprint LI is solved using least-squares for k ≥ 3 and the Moore-Penrose pseudoinverse otherwise, and normalized by L̂I = P

LI . LI + ε

(13)

This partial normalized loads vector L̂I is then padded with n−k zeros to account for robots outside the footprint, which do not carry any load. This yields the full load vector   L̂I L̂ = . (14) 0n−k

We compare this loads vector to the ideal uniform load vector u = N1 1N . We calculate a loads similarity score using cosine similarity, as defined by !β T L̂ u Sload = α · , (15) ∥L̂∥2 · ∥u∥2 + ε which can be simplified as follows X1 T 1 L̂i = , ∵ L̂ u = n n si   2 1 1 =√ , & ∥u∥2 = n n n  β 1 . ∴ Sload = α √ n ∥L̂∥2

(16)

(17) (18)

Here, α > 0 is a scaling coefficient and exponent β ≥ 1 controls how sharp the reward becomes. Larger β values put more emphasis on reaching a perfectly balanced load distribution. The constant ε ensures numerical stability in cases where the vectors are close to zero. E. Reward Design We define the reward function Ri , that is calculated for each robot i, as Ri = Ripos + Ricollide + RiloadShare + Risuccess .

TABLE I: E NVIRONMENT PARAMETERS . S PATIAL QUANTITIES ARE EXPRESSED IN SIMULATOR UNITS . Parameter

Value Agents, Observations and Actions ra (N ∈ {3, 5}) 0.05 ra (N ∈ {10, 15, 20}) 0.02 o Dmax 0.35 nr 12 P Dmax 1.4 ro 0.05 Load Computation and Sharing α 2 β 10 Reward Design Rin 0.1 αp 0.1 Rcollide -0.5 Rs 5 Similarity Score Threshold τs3 , τs5 , τs10 , τs15 , τs20 [1.6049, 1.2999, 0.7954, 0.5086, 0.3374] Obstacles nO 10 rO 0.05 c rO Uniform(0.06, 0.16) c wO Uniform(0.12, 0.35) c hO Uniform(0.06, 0.20) Objects Ps 0.4 Pv Uniformint (5, 25) Pc 0.2 ncav Uniformint (1, 2) kcav Uniform(0.1, 0.25)

(19)

Ripos rewards robots for moving toward and getting inside the object footprint, and is defined by ( Rin if p ∈ P, pos (20) R = t−1 t αp · (dP − dP ) otherwise. where dP is the closest distance from the robot to the footprint polygon and αp is a reward shaping factor. Rcollide is a collision penalty. RloadShare rewards sharing the load, which is represented by the similarity score (Eq. 18). In addition, we give the MRS a constant reward Rs when all robots are inside the object footprint area and the similarity score exceeds the threshold τs . The success reward Risuccess is given by ( Pn Rs success i 1pi ∈P = n & Sload ≥ τs , R = (21) 0 otherwise.

TABLE II: MAPPO H YPERPARAMETERS Parameter η ϵ βent λ γ K M ∥∇∥max Lπ LV Hπ , HV

•

Search Space LogUniform(1e − 5, 3e − 4) Uniform(0.05, 0.3) LogUniform(1e − 4, 3e − 2) Uniform(0.85, 0.97) {0.95, 0.97, 0.98, 0.99} {3, 5, 8} {1000, 2000, 3000, 6000} {0.5, 1.0} {2, 3} {2, 3} {256, 512, 1024}

Chosen Value 9e − 5 0.2 4e − 3 0.935 0.98 8 1000 1.0 3 2 512 (N ∈ {3, 5, 10}), 1024 (N ∈ {15, 20})

d = 2: Here, the worst case will be having as many normalized −1 and 1 load values as possible so that the total load will be 1. Theoretically, a load of −1 would require a robot to pull down the object, which is outside of the robot’s capabilities. Hence, the maximum possible ∥L̂∥2 is:

F. Similarity Score Threshold Choice (√

We choose τs as the minimum acceptable similarity score for each team size N and desired maximum difference in carried load between agents defined as d = max(L̂) − min(L̂),

(0 ≤ d ≤ 2).

(22)

From (Eq. 18), minimizing Sload means maximizing ∥L̂∥2 . There are three scenarios: • d = 0: This is the optimal case where L̂ = u and ∥L̂∥2 = √1n .

∥L̂∥2,max =

•

√

N N −1

(N mod 2 = 1), (N mod 2 = 0).

(23)

0 < d < 2: The worst case will be having one agent carrying the maximum load a and all the other agents carrying the minimum load a − d. a + (n − 1)(a − d) = 1,

∴a=

1 + d(n − 1) . (24) n

(a) E1 Fig. 4.

(b) E2

(c) E3

(d) E4

(e) E5

Environments E1 - E5 (purple circles: robots, red cross: CoM, orange lines: object footprint, black circles and rectangles: obstacles)

Hence, the maximum possible ∥L̂∥2 is: p ∥L̂∥2,max = a2 + (n − 1)(a − d)2 (25) r (1 + (n − 1)d)2 + (n − 1)(1 − d)2 . (26) = n2 We substitute this value back in (Eq. 18) and set the result Sload as our threshold τs . G. Policy Optimization We use Multi-Agent Proximal Policy Optimization (MAPPO) [20] to optimize robot policies. MAPPO extends Proximal Policy Optimization (PPO) [21] with decentralized policies (actors) and a centralized value function (critic). Each robot runs an identical stochastic policy that conditions only on its own observations, while the critic conditions on the joint observations of all agents following the centralized training decentralized execution paradigm (see Fig. 2). We parameterize the actor with a shared-parameter multiagent multi-layer perceptron (MLP) that takes each robot’s observation vector as input and outputs the parameters of a diagonal Gaussian policy. The network has three hidden layers and ReLU activations. The Gaussian action is squashed through a tanh function to respect the action bounds of the environment. All robots share the same actor parameters, reflecting their homogeneous nature. The critic is implemented as a multi-agent MLP with two hidden layers and ReLU activations. It is centralized, that is, it receives the concatenated observations of all robots as input and outputs a separate state-value estimate for each robot. This centralized critic is used to compute generalized advantage estimates for PPO. IV. EXPERIMENTAL SETUP A. Environment We conduct our experiments in the 2D simulator VMAS [4] and evaluate our approach across five environment variants of increasing difficulty, see Fig. 4. The first environment (E1) is an obstacle-free scene with a simple object with uniform mass distribution (Fig. 4a). The object is generated as a random star-shaped polygon parameterized by its size (Ps ), number of vertices (Pv ) and concavity (Pc ∈ [0, 1]). The concavity parameter controls the amplitude of radial perturbations used to construct the polygon. When Pc = 0, the object reduces to a regular convex polygon. As

Pc increases, larger radial variations are introduced, increasing the likelihood of concave geometries. The radius ri of each vertex i is sampled as: ri = 1 + 0.9 Pc U(−1, 1)

(27)

The factor of 0.9 prevents zero radii while still allowing substantial geometric variation. The second environment (E2) extends this setup by adding nO circular obstacles of radius rO at random positions in the scene, requiring the robots to avoid basic clutter during the pattern formation task (Fig. 4b). In the third environment (E3), we further increase the complexity by introducing a denser mixture of circular and rectangular obstacles with varying sizes, creating tighter passages and more irregular free space, while still keeping the simple object (Fig. 4c). Circular obstacles c , whereas rectangular are parameterized by their radius rO c and height hcO . obstacles are defined by their width wO In environment four (E4), we simulate a more complex object with non-homogeneous mass distribution by randomly shifting the object’s CoM (Fig. 4d). This shift is sampled from a normal distribution with σ = 0.1 ∗ (Ps /2). The last environment (E5) uses the complex object from E4 and additionally adds ncav cavities to the object (Fig. 4e). The cavities are generated similarly to the main polygon with a scaling factor kcav and randomly placed within the main object polygon. Table I gives the environment parameters. B. Hyperparameters For each MRS size N ∈ {3, 5, 10, 15, 20}, we train separate policies to study how our approach scales with the number of robots. We train using 150 parallel random environments, each simulated for 200 steps per episode, resulting in batches of 30 000 environment steps per policy update. We perform 125 policy updates, for a total of 3.75 × 106 environment steps. For N ∈ {15, 20}, we use 300 steps per episode to give the robots enough time to find a successful formation. While training, we create random environments with similar properties as E1 - E5 by randomly placing up to no obstacles as in E3, randomly shifting the CoM as in E4, and introducing objects with cavities as in E5 with probability Pcav = 0.35. We run the training with a fixed random seed (seed = 0) for reproducibility. We conduct a randomized hyperparameter search using Ray Tune [22] for setting MAPPO’s hyperparameters. In

particular, we select clipping parameter ϵ, discount factor γ, and GAE parameter λ for the clipped surrogate objective function; entropy coefficient βent , epoch reuse K per collected batch, and mini-batch size M for sampling from the replay buffer. We normalize advantages per robot. We optimize actor and critic parameters jointly using Adam and select learning rate η and apply gradient clipping with a maximum ℓ2 norm, ||∇||max . For each N , we tested a total of 64 independent configurations trained for 50 iterations each and selected the final configuration based on the mean episode reward. The search spaces and chosen values for each hyperparameter are shown in Table II. C. Post-Evaluation We post-evaluate the trained policies across environments E1 - E5 in 150 runs each with maximum 400 steps per run. We report two metrics: The Success Rate (SR) gives the percentage of runs in which the maximum normalized load difference between robots remains below 0.15, that is, when the load similarity exceeds the predefined threshold and the MRS receives the success reward. The Mean Execution Steps (MES) metric gives the mean number of environment steps required to reach this successful formation. In addition, we compare the resulting robot formations of our approach against two baselines for pattern formation: Dynamic Uniform Distribution (DUD) + Expectation Maximization (EM) and a randomized formation. In Dynamic Uniform Distribution (DUD) + Expectation Maximization (EM) [11], robots are first driven into the target shape using an attraction field derived from the gradient of a distance transform image, while mutual repulsive forces promote an approximately uniform distribution inside the shape (DUD). This dynamic configuration serves as an initialization for an EM-like refinement stage, in which each agent iteratively moves to the centroid of its assigned region, improving uniformity. As a neutral baseline, we include a randomized formation generated within the footprint by sampling x and y coordinates independently from uniform distributions. To quantify performance in the baseline comparisons, we compute the L1 transportation cost required to move from the final formation P = {pi }N i=1 to the closest optimal formation P∗ = {p∗i }N , where all robots carry equal load i=1 and satisfy a minimum separation constraint. To compute this closest optimal formation, we restrict the footprint to randomly created simpler shapes represented as the union of 2–5 overlapping rectangles. This representation allows us to formulate a mixed integer linear programming (MILP) problem that determines the closest optimal target configuration P∗ . The MILP minimizes the total L1 norm (Manhattan distance) between the current robot positions and the optimal ones while enforcing equal load distribution and minimum inter-robot spacing. We do 100 runs in an obstacle-free environment and objects with randomly shifted CoM per method and MRS size N ∈ {3, 5, 10, 15, 20}. Please note that we scaled the object by a factor of 2 for MRS sizes N ∈ {15, 20}.

Fig. 5.

Learning Curves

V. RESULTS A. Policy Optimization Figure 5 illustrates the learning curves of the trained policies for different MRS sizes N . All configurations show stable performance improvement, indicating effective policy optimization. For N = 3, the mean episode reward remains lower compared to N ∈ {5, 10}. This behavior is primarily due to the load-sharing reward component (Eq. 19), which is computed as a high-order cosine similarity between the actual and ideal load distributions. In small MRS sizes, torque balance around the center of mass is geometrically constrained, and each robot’s contribution significantly affects stability. Consequently, achieving near-uniform load distribution is more restrictive, making successful episodes less frequent and limiting cumulative reward. In contrast, larger MRS sizes N ∈ {15, 20} benefit from redundancy in load distribution, which relaxes geometric constraints and facilitates faster convergence. It is important to note that policies for these two MRS sizes were trained with a maximum episode length of 300 steps, whereas smaller MRS sizes used 200 steps. The extended episode length enables greater accumulation of positive shaping rewards Rpos and Rsuccess , resulting in higher absolute returns. Therefore, raw episode rewards are not directly comparable across different MRS sizes. Instead, policy quality is evaluated through postevaluation metrics such as success rate, execution steps, and formation cost (see Tables III–IV). Overall, the learning curves demonstrate stable convergence across all MRS sizes and confirm that the proposed training formulation scales effectively with increasing number of robots. B. Post-Evaluation During post-evaluation (see Sec. IV-C), all MRS sizes achieve high success rates across the first four environments, indicating that the learned policies generalize well to variations in obstacle configurations and non-homogeneous mass distributions, see Tab. III. However, performance drops for E5, where the presence of cavities makes the object geometry more irregular and reduces the feasible regions for robot placement, which in turn increases the difficulty of achieving

TABLE III: S UCCESS R ATE (SR) AND M EAN E XECUTION S TEPS (MES) FOR DIFFERENT MRS SIZES No. of Robots Environment

3

5

10

15

20

SR

MES

SR

MES

SR

MES

SR

MES

SR

MES

E1

94.00%

62.6

100%

59.23

98.67%

62.69

99.33%

47.23

99.33%

49.54

E2

93.07%

62.8

96.67%

66.38

96.00%

63.62

99.8%

47.63

99.33%

50.09

E3

92.33%

66.17

98.68%

66.26

94.00%

63.43

96.67%

49.52

95.33%

58.03

E4

92.00%

85.78

98.67%

80.83

93.33%

83.11

94.67%

70.71

94.60%

70.37

E5

73.67%

82.48

84.67%

105.10

86.67%

94.62

93.33%

74.27

91.33%

77.31

TABLE IV: C OMPARISON WITH BASELINES ACROSS DIFFERENT MRS SIZES Metric

Avg. L1 Cost

Success Rate

Avg. Sload

Method

No. of Robots 3

5

10

15

20

Random

0.56±0.29

0.90±0.45

1.49±0.81

3.89±1.73

4.53±2.37

DuD+EM

0.25±0.25

0.40±0.23

0.75±0.41

2.48±1.11

3.23±1.66

Ours

0.03±0.03

0.07±0.08

0.42±0.29

0.84±0.42

1.11±0.78

Random

5.00%

9.00%

33.00%

49.00%

77.00%

DuD+EM

14.00%

49.00%

79.00%

91.00%

96.00%

Ours

90.00%

83.00%

93.00%

95.00%

97.00%

Random

0.32±0.55

0.39±0.54

0.63±0.62

0.72±0.59

0.93±0.58

DuD+EM

1.03±0.54

1.19±0.56

1.27±0.51

1.23±0.47

1.30±0.46

Ours

1.84±0.41

1.63±0.63

1.55±0.44

1.28±0.44

1.27±0.56

balanced load distribution. As before, this effect is more pronounced for N = 3 robots. The MES values provide additional insight into transport efficiency. For small MRS sizes N ∈ {3, 5, 10}, the required number of execution steps is relatively higher than for larger MRS sizes, particularly in E4 and E5, reflecting the increased effort needed to achieve stable load balance. In contrast, MES decreases substantially across all environments for N ∈ {15, 20}. This indicates that larger MRS sizes converge more quickly to stable transport formations, reducing corrective repositioning and oscillatory behavior. Table IV compares our method with the two baselines. While Random initialization performs poorly across all metrics, DuD+EM achieves moderate improvements due to its geometrically uniform distribution strategy. However, DuD+EM does not account for non-uniform mass distributions or torque balance constraints. As a result, the average L1 formation cost grows significantly with increasing MRS size for DuD+EM, whereas our method maintains substantially lower deviation from the MILP-optimal solution. We also find that success rates for our method remain consistently high across all MRS sizes, particularly in small MRS settings, where geometry-only strategies struggle. Our approach also achieves higher average load similarity score (Sload ), especially for small MRS sizes, indicating more effective load distribution. Finally, Figure 6 provides a qualitative comparison for N = 10 robots transporting an arbitrary shaped object with non-uniform mass distribution. The pattern by the Random method shows scattered placements and poor alignment with

the MILP-optimal solution. DuD+EM produces a geometrically uniform distribution. However, it fails to properly compensate for asymmetric mass distribution. In contrast, our learned policy closely matches the MILP-optimal formation. Overall, we find that our policies implicitly learn to approximate the combinatorial optimization solution through decentralized control. VI. CONCLUSION In this paper, we present a framework based on multiagent reinforcement learning to load-aware cooperative object transport. We specifically addressed the formation phase preceding transport and trained decentralized robot policies using multi-agent proximal policy optimization within a centralized training decentralized execution paradigm. The resulting policies enable robots to navigate cluttered environments, enter an object’s footprint, and self-organize into stable support configurations for objects with both regular and highly irregular geometries and non-uniform mass distributions. Post-evaluation across multiple environments and numbers of robots demonstrates that the proposed approach achieves high success rates, small execution steps, and scalability with increasing numbers of robots. Moreover, the learned formations closely approximate mixed integer linear programmingoptimal solutions, indicating that the policies implicitly capture the underlying load-balancing structures. Our results show that explicitly incorporating load-sharing objectives into the learning formulation enables coordinated multi-robot transport beyond purely geometric strategies. To the best of

(a) Random Fig. 6.

(b) DUD+EM

(c) Our method

Visualization of the transportation cost to the closest optimal formation for N = 5 robots carrying an object with non-uniform mass distribution.

our knowledge, our work is the first to explicitly address nonuniform object mass distribution during pattern formation in object-carrying tasks, while also ensuring the participation of all robots in transportation. Despite our encouraging results, there as several aspects that warrant future research. We aim at extending the framework toward full transport execution after lifting, integrating onboard estimation of object geometry and mass distribution, and investigating more scalable policy architectures such as graph-based representations to further improve coordination in large-scale multi-robot systems. ACKNOWLEDGEMENTS This work has been partially supported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the Robotics Institute Germany (RIG). The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the FriedrichAlexander-Universität Erlangen-Nürnberg (FAU) under the BayernKI project v106be BayernKI funding is provided by Bavarian state authorities. R EFERENCES [1] E. Tuci, M. H. M. Alkilabi, and O. Akanyeti, “Cooperative object transport in multi-robot systems: A review of the state-of-the-art,” Front. Robot. AI, vol. 5, 2018. [2] P. R. Wurman, R. D’Andrea, and M. Mountz, “Coordinating hundreds of cooperative, autonomous vehicles in warehouses,” AI Magazine, vol. 29, no. 1, 2008. [3] T. Hardy, A. Q.-C. Nguyen, and M. J. Powell, “Distributed cooperative manipulation of an elastic object by a team of mobile robots,” arXiv preprint arXiv:2111.09046, 2021. [4] M. Bettini, R. Kortvelesy, J. Blumenkamp, and A. Prorok, “Vmas: A vectorized multi-agent simulator for collective robot learning,” in Distributed Autonomous Robotic Systems, ser. Springer Proc. in Advanced Robotics. Springer, Cham, 2024, vol. 28. [5] D. Koung, O. Kermorgant, I. Fantoni, and L. Belouaer, “Cooperative multi-robot object transportation system based on hierarchical quadratic programming,” IEEE Robotics and Automation Letters, vol. 6, no. 4, 2021. [6] M. Jurt, E. Milner, M. Sooriyabandara, and S. Hauert, “Collective transport of arbitrarily shaped objects using robot swarms,” Artificial Life and Robotics, vol. 27, 2022.

[7] G. L. Herranz, S. Hauert, and S. Jones, “Decentralised negotiation for multi-object collective transport with robot swarms,” in IEEE Int. Conf. on Autonomous Robot Systems and Competitions (ICARSC), 2022. [8] J. Wang, J. Cao, M. Stojmenovic, M. Zhao, J. Chen, and S. Jiang, “Pattern-RL: Multi-robot cooperative pattern formation via deep reinforcement learning,” in 2019 18th IEEE Int. Conf. On Machine Learning And Applications (ICMLA), 2019. [9] M. Rubenstein, A. Cornejo, and R. Nagpal, “Programmable selfassembly in a thousand-robot swarm,” Science, vol. 345, no. 6198, 2014. [10] C. L. Liu, I. L. D. Ridgley, M. L. Elwin, M. Rubenstein, R. A. Freeman, and K. M. Lynch, “Self-healing distributed swarm formation control using image moments,” IEEE Robotics and Automation Letters, vol. 9, no. 7, 2024. [11] Q. Bi and Y. Huang, “A self-organized shape formation method for swarm controlling,” in 2018 37th Chinese Control Conf. (CCC), 2018. [12] H. Wang and M. Rubenstein, “Shape formation in homogeneous swarms using local task swapping,” IEEE Trans. on Robotics, vol. 36, no. 3, 2020. [13] V. L. N. Venkatesh and B.-C. Min, “Zerocap: Zero-shot multi-robot context aware pattern formation via large language models,” 2024. [14] S. V. Albrecht, F. Christianos, and L. Schäfer, Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press, 2024. [15] W. Du and S. Ding, “A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications,” Artificial Intelligence Review, vol. 54, no. 5, pp. 3215–3238, 2021. [16] A. Krnjaic, R. D. Steleac, J. D. Thomas, G. Papoudakis, L. Schäfer, et al., “Scalable multi-agent reinforcement learning for warehouse logistics with robotic and human co-workers,” arXiv preprint, 2023. [17] M. Zhou, J. Luo, J. Villella, Y. Yang, and et al., “Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving,” in Proc. of the Conf. on Robot Learning (CoRL), Cambridge, MA, USA, 2020. [18] J. Lin, X. Yang, P. Zheng, and H. Cheng, “End-to-end decentralized multi-robot navigation in unknown complex environments via deep reinforcement learning,” in IEEE Int. Conf. on Mechatronics and Automation (ICMA), 2019, pp. 2493–2500. [19] H. Iima and Y. Kuroe, “Swarm reinforcement learning methods improving certainty of learning for a multi-robot formation problem,” in 2015 IEEE Congress on Evolutionary Computation (CEC), 2015. [20] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative, multiagent games,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022, arXiv:2103.01955. [21] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347 [22] R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, and I. Stoica, “Tune: A research platform for distributed model selection and training,” arXiv preprint arXiv:1807.05118, 2018.