Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage

Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage Omar Rady , Mohamed Ayman , Ali Arafa , Mohamed Shalma Faculty of Information Engineering and Technology The German University in Cairo Cairo, Egypt

While recent literature has applied DRL and heuristic techniques to mmWave propagation and dynamic load balancing [4], [5], [6], [7], many existing models rely on simplified, unobstructed environments or predefined candidate coordinates [8]. For example, while DQN can jointly optimize mmWave coverage and localization [9], such frameworks often rely on discrete grid representations that restrict action spaces to predefined movements [9]. Conversely, although continuous actor-critic DRL frameworks enable flexible 3D placement of multiple UAV base stations to maximize user Quality of Experience (QoE) [10], they assume unrestricted aerial mobility [10]. The vast majority of the above-mentioned works are limited to single-BS scenarios and single agent modeling. As the environment becomes more complex, the curse of dimensionality dominates leading to a notable degradation in the DRL agents performance. There is a lack of unified optimization frameworks that simultaneously address mmWave characteristics, multi-agent modeling, and the strict topological constraints of non-convex topologies. The primary contributions of this work are as follows: I. I NTRODUCTION • We address the joint optimization of multiple base stations Next-generation (B5G/6G) wireless networks increasingly placement for a multi-user scenario to enhance the depend on the millimeter-wave (mmWave) spectrum to acminimum rate for maximum fairness using mmWaves commodate explosive demands for massive connectivity and in a practical urban environment. multi-gigabit throughput. However, the high-frequency nature • The considered BS placement vicinity is a multi-space of mmWave transmissions makes them highly vulnerable non-convex topology that leads to a non-convex and to physical blockages, severe penetration attenuation, and consequently, an NP-hard optimization problem. atmospheric absorption. Consequently, defining the exact • We propose the use of both continuous DRL approaches spatial coordinates for base station (BS) infrastructure is no like DDPG and discrete ones as DQN. Furthermore, the longer a standard planning exercise; it constitutes an NP-hard two approaches are evaluated using singe and multioptimization challenge that dictates network reliability, spectral agent architectures. By introducing a boundary projection efficiency, and equitable user fairness. mechanism and a rigorous max-min fairness objective, the In practical urban and campus environments, deployment multi-agent DDPG framework surpasses other models and terrains rarely adhere to symmetric or convex geometries, achieves superior optimality, guaranteeing 100% coverage, which limits the signal coverage [1]. Conventional mathematabsolute minimum SNRs above 19 dB, and a Jain’s index ical solvers, such as mixed-integer non-linear programming of 0.94. (MINLP) or exhaustive grid-search algorithms, struggle to II. S YSTEM AND C HANNEL M ODEL resolve these non-convex spatial constraints. They either incur prohibitive computational overhead or fail to capture the highly A. Model Description directional propagation dynamics of mmWave signals. To We analyze a downlink mmWave communication architecture overcome these limitations, Deep Reinforcement Learning where BSs are strategically deployed to serve K stationary user (DRL) gained increased attention as a robust, model-free equipments (UEs) distributed throughout an indoor/courtyard alternative, enabling autonomous agents to iteratively learn environment. The geometrical layout strictly models the Cand refine optimal solutions [2], [3]. Buildings complex at the Germa university in Cairo (GUC), as

arXiv:2608.19049v1 [cs.LG] 19 Aug 2026

Abstract—Deep reinforcement learning (DRL) has recently gained a great attention due to its real-time adaptation and effectiveness in complex optimization problems. This paper investigates the optimal deployment of millimeter-wave (mmWave) base stations (BSs) in a realistic, non-convex campus topology. The optimization problem is NP-hard, due to the non-convex, non-smooth nature of the max-min fairness objective. To overcome these constraints, we formulate the BS placement as a Markov Decision Process (MDP) and systematically benchmark four DRL schemes: a discrete single-agent Deep Q-Network (DQN), a spatially partitioned Multi-Agent DQN, a continuous single-agent Deep Deterministic Policy Gradient (DDPG), and a geographically partitioned multi-agent DDPG framework. Numerical evaluations reveal that the multi-agent DDPG approach substantially outperforms single-agent in dense scenarios. Additionally full coverage is achieved, and a fairness Jain’s index of 0.94 is obtained. Finally, the multi-agent demonstrates highly efficient computational convergence of dense scenarios with 400 users. Index Terms—Base station placement, deep reinforcement learning (DRL), Deep Deterministic Policy Gradient (DDPG), Deep Q-Network (DQN), max-min fairness, millimeter-wave (mmWave).

Fig. 2. System model showing the 4 BSs and multiple users

Fig. 1. Aerial view of the full campus environment at the German University in Cairo (GUC) showing all main buildings.

illustrated in Fig. 1, featuring four adjacent rectangular structures that enclose a central open courtyard. The optimization goal is to extract the ideal two-dimensional (2D) horizontal coordinates for placing one BS on each of the four building roofs, with the UEs distributed at ground level.

spaces, and objective-driven reward signals shared across the learning agents. The DRL environments and policy networks are implemented utilizing the Stable-Baselines3 framework to ensure robust and reproducible training. • Observations: The environmental state captures the 2D spatial distribution of all users, defined as obsv = [(x1 , y1 ), (x2 , y2 ), · · · , (xK , yK )]

B. Channel Model The Euclidean 3D distance separating a candidate base station and a user k is formulated as p dk = (xBS − xk )2 + (yBS − yk )2 + (hBS − hk )2 (1) where (xBS , yBS ) and (xk , yk ) represent the horizontal plane coordinates of the BS and user k, while hBS and hk denote their respective heights. The mmWave channel state incorporating large-scale path loss and small-scale fading is modeled as p hk = L(dk ) gk (2) with gk ∼ CN (0, 1) capturing Rayleigh fading. The large-scale path loss follows the standard power law: L(dk ) = C0 d−α k

(3)

where C0 specifies the path loss at a 1-meter reference, and α is the attenuation exponent. The received signal vector at user k is given by p yk = Ptx hk x + nk (4) where Ptx is the transmission power, x is the unit-power data symbol, and nk ∼ CN (0, N0 ) is the additive white Gaussian noise. In a noise-limited mmWave regime, the signal-to-noise ratio (SNR) is expressed as Ptx |hk |2 γk = (5) N0 Yielding an achievable data rate for user k of: Rk = log2 (1 + γk )

(6)

III. P ROPOSED DRL-BASED BS P LACEMENT A. MDP Formulation The proposed DRL architecture is defined by an MDP tuple encompassing state observations, continuous/discrete action

•

Action Space: The agents output the optimal 2D coordinates for the four deployed BSs. Therefore, the joint action vector is formalized as at = [xBS,,1 , yBS,1 , . . . , xBS,4 , yBS,4 ]

•

(7)

(8)

Reward Function: The reward mechanism directly reflects the max-min fairness target, defined strictly by the minimum user throughput rate to ensure equitable spatial coverage reward = min log2 (1 + γk ) (9) k

B. Multi-Agent DQN The proposed Multi-Agent Deep Q-Network (MADQN) decomposes the BS placement problem into four cooperative subproblems by partitioning the campus into four regions. A DQN agent is assigned to each region and selects the optimal BS location from a discrete set of candidate positions. Each agent maintains its own Q-network, target network, and replay buffer, enabling decentralized learning while reducing the search space. After all agents complete their selections, the minimum SNR among all campus users is evaluated and used as a shared reward, encouraging fair wireless coverage across the campus. The overall optimization procedure is summarized in Algorithm 1. C. Continuous Multi-Partitioning via DDPG To overcome the inherent resolution limits of grid discretization and avoid local spatial optima, the multi-partition DDPG approach explicitly subdivides the non-convex topography of each of the four building roofs into 5 distinct geometric partitions (North, South, East, West, and Center arms). Consequently, a dedicated continuous actor-critic agent is independently trained on each of these 20 total sub-sectors (Algorithm 2).

Algorithm 1 Multi-Agent DQN for BS Placement 1: Input: User locations {(xk , yk )}K k=1 and candidate BS locations for each region 2: for each agent i = 1, . . . , 4 do 3: Initialize Q-network Qθi and target network Qθ̄i 4: Initialize replay buffer Di 5: end for 6: for each episode do 7: Reset the environment 8: for each agent i do 9: Observe local state si 10: Select action ai using an ϵ-greedy policy 11: if ai is invalid then 12: Apply invalid action penalty 13: end if 14: Assign BS location corresponding to ai 15: end for 16: Compute the global reward (minimum user SNR) 17: for each agent i do 18: Store transition (si , ai , ri , s′i ) in Di 19: Sample a minibatch from Di 20: Update Qθi 21: Update target network Qθ̄i 22: end for 23: end for 24: Output: Optimal BS locations for the four campus regions Unlike the discrete DQN framework, if the DDPG network predicts a coordinate outside the bounds of its designated partition, the point is automatically projected to the nearest valid geometric boundary on the specific roof polygon. The framework then determines the absolute best deployment for the building by selecting the winning partition that yields the highest maximum reward.

Algorithm 2 Multi-Agent DDPG for BS Placement 1: Input: Users {(xk , yk )}K k=1 , buildings b ∈ {1..4}, partitions p ∈ {1..5} 2: for each building b do 3: for each partition p do 4: Initialize actor πθb,p and critic Qϕb,p 5: Set target networks πθ̄b,p , Qϕ̄b,p 6: Initialize partitioned replay buffer Db,p 7: for each episode do 8: Initialize environment 9: for each time step t do 10: Observe state st (b,p) 11: Generate action: at = πθb,p (st ) + Nt Enforce partition boundary projection: 12: (b,p) (b,p) at ← ProjAb,p (at ) (b,p)

b Apply BS placement (xbBS , yBS ) ← at (b,p) Compute step reward rt Observe next state st+1 (b,p) (b,p) Store tuple (st , at , rt , st+1 ) in Db,p Sample minibatch from Db,p Update critic network using Bellman equa-

13: 14: 15: 16: 17: 18:

tion Ascend policy gradient to update actor Perform soft target updates: θ̄b,p ← τ θb,p +

19: 20:

(1 − τ )θ̄b,p end for end for end for b∗ Select winning partition per building: (xb∗ BS , yBS ) = (b,p) arg maxp,t rt 25: end for 26: Output: Optimal continuous coordinates for all 4 buildings

21: 22: 23: 24:

IV. N UMERICAL R ESULTS This section benchmarks the proposed optimization configurations via simulation using the environmental parameters detailed in Table I. The primary performance criteria include achievable max-min rates (bps/Hz), spectral efficiency, signalto-noise ratio (SNR) distributions, and overall sector coverage percentages. TABLE I S IMULATION C ONFIGURATION PARAMETERS Fig. 3. Training reward convergence profile for the Single-Agent DQN. Parameter Description

Value

Total Number of Users (K) Transmit Power (Ptx ) Noise Floor Power (N0 ) Path Loss Exponent (α) BS Antenna Height User Equipment Height DDPG Learning Rate DQN Learning Rate Replay Buffer Capacity Temporal Discount Factor (γ)

400 (100 per zone) 1.0 W 1 × 10−10 W 4.0 12.0 m 1.7 m 3 × 10−4 3 × 10−4 50,000 0.99

As depicted in Fig. 3, the baseline Single-Agent DQN struggles with the scale of the centralized discrete grid. The moving average reward plateaus at approximately 3.95 bps/Hz. This constrained performance highlights the geometric limitations of quantizing a non-convex space when attempting to optimize highly sensitive mmWave spatial coordinates. To mitigate centralized grid congestion, Fig. 4 tracks the Multi-Agent DQN formulation. By assigning distinct discrete sub-grids to each of the four rooftops, the distributed agents

TABLE II C OVERAGE OF C AMPUS AND FAIRNESS E VALUATION VIA JAIN ’ S I NDEX Optimization Technique Single-Agent DQN Multi-Agent DQN Single-Agent DDPG Multi-Agent DDPG

Total Users (n)

Mean Rate (bps/Hz)

Min SNR (dB)

Jain’s Index (J)

400 400 400 400

3.17 9.05 9.64 9.65

0.01 14.01 19.19 19.18

0.71 0.82 0.94 0.9486

Fig. 6. Episodic training reward progression for the continuous Single-Agent DDPG.

Fig. 4. Training reward convergence profile for the Multi-Agent DQN.

successfully partition the computational complexity, yielding a noticeably higher and more stable max-min reward floor compared to the single-agent variant.

partitioned campus geometry. By eliminating discrete grid limitations, the continuous agent achieves a substantially higher minimum rate convergence of approximately 5.6 bps/Hz. Nevertheless, managing four concurrent coordinates via a single centralized policy still introduces variance, as the agent struggles to balance overlapping coverage zones.

Fig. 7. Time-based convergence comparison of average achievable rates for Single-Agent and Multi-Agent DDPG.

Fig. 5. Time-based convergence comparison of average rates between SingleAgent and Multi-Agent DQN frameworks.

Furthermore, to highlight the computational efficiency of the distributed approach, Fig. 5 contrasts the average rate convergence against actual training time in seconds. The MultiAgent DQN not only achieves a higher average rate but also demonstrates that partitioning the state space does not incur prohibitive time delays, rapidly establishing a superior performance floor compared to the single-agent baseline. Evaluating continuous action spaces, Fig. 6 demonstrates the Single-Agent DDPG operating across the unbounded, un-

Building on the episodic performance, Fig. 7 illustrates the average rate convergence with respect to continuous training time. The multi-agent partitioning scheme maintains a highly stable average rate. By dedicating independent actors to bounded partitions, the system circumvents the extensive search times required by a single centralized agent, proving highly efficient in real-time execution. To achieve global spatial optimality, the Multi-Agent DDPG framework deploys dedicated continuous agents to the strictly bounded rooftop partitions. Fig. 8 confirms the superiority of this localized exhaustive approach. By isolating the continuous search bounds into 5 arms per building, the framework

Fig. 8. Optimal spatial mapping and resulting SNR distribution generated by the Multi-Agent DDPG framework.

avoids spatial minima and rapidly converges on an optimal deployment topology yielding a network-wide minimum reward of 6.393 bps/Hz. The system achieves a flawless 100% quadrant coverage (> 0.0 dB). Furthermore, the distributed agents deliver exceptional spectral efficiencies: 9.54, 9.52, 9.74, and 9.80 bps/Hz across Buildings A, B, C, and D, respectively, while sustaining absolute minimum SNRs above 19 dB. V. C ONCLUSION This study addressed the complex challenge of optimizing mmWave base station deployments across the non-convex rooftop infrastructure of the GUC campus. By formulating the environment as an MDP, we evaluated four distinct DRL methodologies targeting sum-rate and max-min fairness. Analytical results confirm that the geographically partitioned Multi-Agent DDPG approach significantly outperforms discrete grid-based and centralized continuous baselines. By dividing the complex rooftops into 5 distinct partitions and dedicating independent actor-critic agents to exhaustively search each continuous bound, the proposed framework bypasses dimensional scalability limits and local spatial minima. Ultimately, this approach achieves a superior minimum data rate of 6.393 bps/Hz, robust SNRs exceeding 19 dB, and perfectly equitable quadrant coverage. R EFERENCES [1] M. M. H. Shalma, E. A. Maher, and A. El-Mahdy, “Optimal power allocation and passive beamforming of double active ris-aided communication with af relay,” in 2023 2nd International Conference on Smart Cities 4.0, 2023, pp. 74–78. [2] B. Shamseldin, A. Zedan, M. Shalma, and A. El-Mahdy, “Comparative study of gradient descent and deep reinforcement learning in multi-user rsma systems,” in 2025 7th Novel Intelligent and Leading Emerging Sciences Conference (NILES), 2025, pp. 376–379. [3] M. M. H. Shalma, E. A. Maher, and A. El-Mahdy, “Hybrid deep reinforcement learning for joint resource allocation in multi-active risaided uplink communications,” in 2025 12th International Conference on Wireless Networks and Mobile Communications (WINCOM), 2025, pp. 1–6.

[4] P. Yu, J. Guo, Y. Huo, X. Shi, J. Wu, and Y. Ding, “Three-dimensional aerial base station location for sudden traffic with deep reinforcement learning in 5g mmwave networks,” International Journal of Distributed Sensor Networks, vol. 16, no. 5, 2020. [5] M. Vdovichev and D. Bogdanov, “Neural network–based optimization of base station placement for local positioning systems,” in 2026 ElCon Conference of Young Researchers in Electrical Engineering, Automation & Control Systems (ElCon-EE), 2026, pp. 411–414. [6] Z. Deng, C. Sun, F. Jiang, and J. Wang, “Multi-armed bandit based base stations deployment in millimeter wave network.” New York, NY, USA: Association for Computing Machinery, 2023. [7] V. R. Farré-Guijarro, J. C. Estrada-Jiménez, J. D. Vega Sánchez, J. A. Vasquez-Peralvo, and S. Chatzinotas, “Enhanced 5g/b5g network planning/optimization deploying ris in urban/outdoor scenarios,” in 2024 IEEE 29th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), 2024, pp. 1–6. [8] T. Hayakawa, T. Kimura, and J. Cheng, “Robust optimization-based mobile base station placement with uav-assisted data collection under demand uncertainty,” in 2026 40th International Conference on Information Networking (ICOIN), 2026, pp. 900–905. [9] A. Al-Tahmeesschi, J. Talvitie, M. López-Benı́tez, H. Ahmadi, and L. Ruotsalainen, “Multi-objective deep reinforcement learning for 5g base station placement to support localisation for future sustainable traffic,” in 2024 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit), vol. Conference Proceedings, 2024, p. 493–498. [10] L. T. Hoang, C. T. Nguyen, H. D. Le, and A. T. Pham, “Adaptive 3d placement of multiple uav-mounted base stations in 6g airborne small cells with deep reinforcement learning,” Draft/Preprint, 2023.

Related documents

• Artificial intelligence driven multi agent framework for adaptive cyber attack simulation and automated incident response in cyber range environments.#1392
• Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents#310753
• Continual Policy Consolidation for Lifelong Robot Learning#973051
• How do World Models and Policies Compose in LLM Agents? A Joint Spectral and Behavioral Account#799304
• How do World Models and Policies Compose in LLM Agents? A Joint Spectral and Behavioral Account#797881