FedCritic: Serverless Federated Critic Learning-based Resource Allocation for Multi-Cell OFDMA in 6G
arXiv:2605.21418v1 [cs.LG] 20 May 2026
Amin Farajzadeh, Member, IEEE, Melike Erol-Kantarci, Fellow, IEEE School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada Emails:{amin.farajzadeh, melike.erolkantarci}@uottawa.ca Abstract—In sixth-generation (6G) ultra-dense networks, aggressive frequency reuse amplifies inter-cell interference (ICI), making multi-cell orthogonal frequency-division multiple access (OFDMA) scheduling and power control strongly coupled across neighboring cells. We study distributed downlink resource management—joint subcarrier scheduling and power allocation—under interference coupling and long-term per-user quality-of-service (QoS) minimum-rate constraints. By using virtual-queue deficit weights to enforce long-term QoS, we develop FedCritic, a serverless federated multi-agent actor-critic framework with decentralized execution. Unlike centralized training with decentralized execution (CTDE) approaches that require centralized critic learning and joint trajectory aggregation, FedCritic federates the critic through lightweight gossip-based parameter averaging over the interference graph, enabling stable value estimation without a central coordinator while keeping policies local. Simulations in an interference-rich reuse-1 setting show that FedCritic improves mean signalto-interference-plus-noise ratio (SINR) and cell-edge rate, increases network-wide average sum-rate and fairness relative to non-coordinated and CTDE baselines, and achieves more stable training with lower coordination overhead. Index Terms—6G, multi-cell OFDMA, resource allocation, multi-agent reinforcement learning (MARL), serverless federated reinforcement learning.
I. I NTRODUCTION Sixth-generation (6G) networks in the IMT-2030 timeframe target higher area capacity, enhanced uniform user experience, and stricter latency/reliability under heterogeneous spectrum and ultra-dense deployments [1]. These objectives are expected to be pursued via additional spectrum (including upper-midband/cmWave) and denser reuse-1 operation, which strengthens cross-cell coupling and makes inter-cell interference (ICI) a primary limiter of cell-edge throughput and service consistency [2]. Orthogonal frequency-division multiple access (OFDMA) is expected to remain central in wideband radio access networks (RANs) due to its fine time–frequency granularity and compatibility with link adaptation and multi-antenna transmission [3]. In reuse-1 multi-cell operation, however, per-cell signal-to-interference-plus-noise ratio (SINR) and rate depend on simultaneous neighbor transmissions, turning ICI mitigation into a network-coupled control problem in the ultra-dense regime [4]. Among the available interferencemanagement levers, subcarrier-level scheduling and power
allocation are particularly effective in reuse-1 OFDMA systems because they directly control which links share each subcarrier and at what transmit power. This enables selective protection of dominant victim–interferer pairs at fine time–frequency granularity while still preserving the spectral-efficiency benefits of aggressive frequency reuse [5]. Conventional coordination becomes difficult in interferencedominant multi-cell OFDMA [5]. The problem is highdimensional, mixes discrete scheduling with continuous power control, and often relies on timely global information and reliable backhaul. This motivates learning-based control. Multiagent reinforcement learning (MARL) is well aligned with multi-cell resource management because it can optimize longterm objectives under dynamic interference coupling [6]. Many recent works adopt centralized training and decentralized execution (CTDE) actor–critic learning, where a centralized critic leverages joint information to stabilize training while execution remains local at each BS [7], e.g., proximal policy optimization (PPO)/MAPPO-style cooperative training frameworks [8]. However, CTDE typically requires joint trajectory collection and broad information sharing, which limits scalability as the network size and action dimension grow [9]. Recent work has also explored federated RL and MARL for wireless resource management [10]. On the federated RL side, prior studies considered server-based aggregation [11] and peer-to-peer exchange for decentralized time–frequency control [12]. On the MARL side, related works investigated joint resource-block allocation and power control in multi-cell systems [13], dynamic resource management [14], cooperative user association [15], and scheduling policies [16]. However, scalable and coordination-efficient learning for interferencerich multi-cell OFDMA with joint subcarrier scheduling and power control remains largely unaddressed. We address this gap through a fully decentralized federated MARL framework tailored to reuse-1 OFDMA. Each BS learns from local experience, while only critic parameters are exchanged and averaged among neighboring BSs through gossip over the interference graph. This removes the centralized critic bottleneck of CTDE while preserving critic cooperation and local execution. To the best of our knowledge, this is among the first works to jointly study subcarrier scheduling, discrete power control, long-term per-UE minimum-rate
QoS via virtual queues, and serverless neighbor-only critic federation in interference-rich reuse-1 multi-cell OFDMA. Simulations in an interference-rich multi-cell reuse-1 setting show that the proposed approach improves SINR and sum-rate distributions while achieving higher network-wide average sum-rate and fairness than CTDE baselines and non-learning heuristics, with faster and more stable training under substantially lower coordination overhead. The remainder of this paper is organized as follows. Section II presents the system model and problem formulation. Section III describes the proposed learning framework. Section IV reports simulation results, and Section V concludes the paper.
where N0 denotes the one-sided noise power spectral density (PSD). For n′ 6= n, gn′ ,k,m (t) denotes the cross-link power gain from interfering BS n′ to UE m ∈ Mn on subcarrier k. The instantaneous rate served by BS n on subcarrier k is X cn,k (t) = xn,k,m (t)∆f log2 1+γn,k,m (t) , (4) m∈Mn
and the per-UE and per-BS transmission rates in slot t are, respectively, K X Rn,m (t) = xn,k,m (t)∆f log2 1+γn,k,m (t) , (5) cn (t) =
k=1 K X
cn,k (t).
(6)
k=1
II. S YSTEM M ODEL
B. Optimization Problem Formulation
A. Network Setting and Channel Modeling
In each slot t, the controller at each BS jointly selects We consider the downlink of a multi-cell OFDMA network the subcarrier–UE scheduling and the per-subcarrier transmit composed of N BSs sharing a total bandwidth B, partitioned powers to maximize the instantaneous network-wide downlink into K orthogonal subcarriers of equal width ∆f = B/K. sum-rate, while satisfying OFDMA orthogonality and per-BS Each BS n ∈ {1,...,N } serves a set of user equipments (UEs) power budgets. The per-UE QoS requirement is defined as Mn . Time is slotted with index t = 0,1,2,..., and in each slot a long-term average minimum rate constraint and therefore t, scheduling and power allocation decisions are made online. cannot be enforced directly within a single-slot optimization. Within each slot, BS n schedules at most one UE on each Instead, it is handled by an online virtual-queue control subcarrier k using binary variables xn,k,m (t) ∈ {0, 1}, mechanism that converts the long-term constraint into timewhich satisfy the intra-cell OFDMA constraint varying per-slot weights. This yields the following per-slot P x (t) ≤ 1. The transmit power allocated by BS surrogate optimization problem: m∈Mn n,k,m n on subcarrier k in slot t is denoted by pn,k (t) ≥ 0 and is N K N X X X X P min constrained by the per-BS power budget K Qn,m (t) Rn,m −Rn,m (t) max cn,k (t)− k=1 pn,k (t) ≤ Pn . Let hn,k,m (t) denote the small-scale fading coefficient X(t),P(t)n=1k=1 n=1m∈Mn (7a) from BS n to UE m ∈ Mn on subcarrier k at slot t, and X let αn,k,m > 0 denote the large-scale channel gain capturing s.t. xn,k,m (t) ≤ 1, ∀n,k, (7b) path loss and log-normal shadowing. We model αn,k,m = m∈Mn 2 exp(zn,k,m ), where zn,k,m ∼ N (µPL ,σPL ), and µPL and σPL K X denote the mean and standard deviation of the log-domain pn,k (t) ≤ Pn , pn,k (t) ≥ 0, ∀n, (7c) large-scale coefficient. The resulting channel power gain is k=1 0 ≤ pn,k (t) ≤ Pn an,k (t), ∀n,k, (7d) g (t) = α |h (t)|2 . (1) n,k,m
n,k,m
n,k,m
The small-scale fading is modeled as a first-order complex Gauss–Markov process [11], p (2) hn,k,m (t) = ρhn,k,m (t−1)+ 1−ρ2 wn,k,m (t),
where wn,k,m (t) ∼ CN (0, 1) is i.i.d. across (n, k, m, t) and 0 ≤ ρ < 1 is the temporal correlation coefficient, set according to the Doppler frequency and slot duration. The channel is assumed constant within each slot and evolves across slots according to (2). For notational convenience, we define the subcarrier P activity indicator an,k (t) , m∈Mn xn,k,m (t) ∈ {0,1}, which indicates whether BS n transmits on subcarrier k in slot t. The downlink SINR experienced by UE m on subcarrier k in slot t is γn,k,m (t) = X
pn,k (t)gn,k,m (t) an′ ,k (t)pn′ ,k (t)gn′ ,k,m (t)+N0 ∆f
n′ 6=n
,
(3)
xn,k,m (t) ∈ {0,1},
∀n,k, m ∈ Mn ,
(7e)
where X(t) = {xn,k,m (t)} denotes the binary scheduling variables, and P(t) = {pn,k (t)} denotes the per-subcarrier transmit powers. Constraint (7b) enforces intra-cell OFDMA orthogonality, (7c) imposes the per-BS power budget, and (7d) couples power allocation to scheduling decisions to ensure that power is used only on active (scheduled) subcarriers. The coefficients {Qn,m (t)} in (7a) are virtual queues that quantify nonnegative QoS deficit pressure and evolve over time based on past rate shortfalls. The update is h i+ min Qn,m (t+1) = Qn,m (t)+Rn,m −Rn,m (t) , ∀n, m ∈ Mn , (8) where [·]+ , max{·, 0}. This increases Qn,m (t) when UE min m falls below Rn,m and decreases it otherwise. Since P min Q (t)R is constant at slot t, (7a) equivalently n,m n,m n,m maximizes a weighted served-rate objective that prioritizes UEs with larger deficits. As a result, repeatedly solving (7)
online steers the system toward satisfying long-term average QoS while still prioritizing instantaneous sum-rate. Problem (7) is a mixed-integer nonconvex program due to binary scheduling decisions and interference-coupled SINR expressions under reuse-1, making centralized real-time optimization impractical at scale. This motivates an online decentralized learning formulation in which each BS learns from local observations and coordinates only with neighboring BSs to account for interference coupling, without relying on a central controller. III. R EFORMULATION AS AN I NTERFERENCE –C OUPLED D EC –POMDP
We model the online per-slot control problem in (7) as an interference-coupled Dec–POMDP with decentralized execution at the BSs. The model captures partial observability, since each BS has access only to local measurements and lightweight neighbor summaries, and interference coupling, since achieved rates depend on other BSs’ actions through (3). Following the cooperative Dec–POMDP, all BSs optimize a shared team reward that matches the network-wide surrogate objective in (7a). Based on this formulation, we develop an onpolicy actor–critic method and introduce FedCritic, a serverless federated critic mechanism that stabilizes value estimation via periodic gossip mixing over the interference graph. A. Interference–Coupled Dec–POMDP Model We model the network as the Dec–POMDP tuple N N D , S,{An }N n=1 ,P,{On }n=1 ,{Zn }n=1 ,r,β ,
(9)
where β ∈ (0,1] is the discount factor. The global state s(t) ∈ S collects the channel variables governing the network dynamics, including the fading coefficients {hn,k,m (t)} evolving via (2) and the large-scale gains {αn,k,m }. To preserve Markovian dynamics under long-term constraints and neighbor summaries, we include in s(t) the virtual queues {Qn,m (t)} and the exbn,k (t)} used by ponential moving average (EMA) statistics {O the controller. Each BS n selects a local action an (t) ∈ An , and the joint action a(t) = (a1 (t),...,aN (t)) induces the next state through the Markov transition kernel P s(t+1) | s(t),a(t) , which captures the channel evolution. BS n does not observe s(t) directly; instead it receives a local observation on (t) ∈ On via a deterministic observation function Zn : S → On , i.e., on (t) = Zn (s(t)), consisting of locally available features augmented with low-rate neighbor summaries exchanged over an interference graph. All BSs receive the same cooperative team reward r(t) = r s(t),a(t) , implemented by the networkwide shaped utility defined in Section III-C. 1) Interference graph and neighbor summaries: Let the interference graph be GI = ({1,...,N },EI ), where (n,j) ∈ EI indicates that BS j is a dominant interferer to cell n, and let N (n) denote the neighbor set (assume |N (n)| ≥ 1; otherwise max set O n,k (t) = On,k (t) = 0). To provide a low-overhead proxy of interference pressure, BSs exchange subcarrier-level bn,k (t) be an EMA of the activity activity statistics. Let O indicator an,k (t) as
bn,k (t)+(1−αO )an,k (t), bn,k (t+1) = αO O O
with αO ∈ (0,1). BS n forms neighbor aggregates X 1 bj,k (t), O O n,k (t) = |N (n)|
(10)
(11)
j∈N (n)
max bj,k (t), On,k (t) = max O j∈N (n)
(12)
which are included in on (t). 2) Actions and feasibility mapping: Each BS n selects actions that determine its local components of (X(t),P(t)). For each subcarrier k, the action is parameterized as ãn,k (t) = un,k (t),mn,k (t),ℓn,k (t) , (13) where un,k (t) ∈ {0, 1} indicates mute/active transmission, mn,k (t) ∈ Mn is the scheduled UE when un,k (t) = 1, and ℓn,k (t) ∈ {1,...,L} selects a discrete power level p(ℓ) . Note that un,k (t) coincides with the activity indicator an,k (t). The induced optimization variables are xn,k,m (t) = 1{un,k (t) = 1, mn,k (t) = m}, (14)
pn,k (t) = un,k (t)p(ℓn,k (t)) . (15) P By construction, m∈Mn xn,k,m (t) = un,k (t) and the intra-cell OFDMA constraint (7b) is satisfied. To enforce (7c)–(7d), we apply masking (forcing pn,k (t) = 0 when muted) and a per-slot power normalization (projection) to satisfy the sum-power constraint o n Pn , ∀k, (16) pn,k (t) ← pn,k (t)·min 1, PK k′ =1 pn,k′ (t)+ǫ
where ǫ > 0 is a small constant for numerical stability. This normalization P preserves relative per-subcarrier levels while ensuring k pn,k (t) ≤ Pn . B. Reward Design and Long-Term QoS Enforcement
To mirror the per-slot objective (7a), we define each BS’s base transmission rate reward as rnrate (t) = cn (t) =
K X
cn,k (t).
(17)
k=1
Using the per-UE rate definition in (5), we incorporate QoS pressure via the deficit weights {Qn,m (t)} through X rnqos (t) = Qn,m (t)Rn,m (t), (18) m∈Mn
where constant terms independent of the slot-t decision are omitted. Furthermore, to promote interference-aware operation, we include a leakage-style penalty based on long-term cross-link gain proxies. Let ḡn→j,k denote the long-term average cross-link gain from BS n to cell j on subcarrier k, obtained by time-averaging the corresponding measured cross-link gains. Define K X X ηnj pn,k (t)ḡn→j,k , (19) Ln (t) = k=1 j∈N (n)
where ηnj ≥ 0 are fixed leakage weights. The shaped per-slot reward is r̃n (t) = rnrate (t)+rnqos (t)−λint Ln (t), (20) with λint ≥ 0. In the cooperative Dec–POMDP, we use the N team reward X r(t) , r̃n (t), (21) n=1
and the environment provides the same scalar r(t) to all BSs. The deficit weights {Qn,m (t)} evolve via the virtual-queue recursion (8). Mean-rate stability of these queues implies satisfaction of the long-term QoS constraints in the Lyapunov sense; operationally, large Qn,m (t) increases the urgency to serve UE m, consistent with (7a). C. Federated Critic Learning With Gossip-Based Aggregation (FedCritic) Each BS n runs an on-policy actor–critic learner conditioned on its local observation on (t). The actor πθn outputs a distribution over a structured, feasibility-constrained action an (t), factorized across subcarriers. For each subcarrier k, the BS first decides an activity/muting variable un,k (t) ∈ {0,1} and, if active, selects a scheduled UE index mn,k (t) ∈ Mn and a transmit-power level index ℓn,k (t) ∈ {1, ... , L}. The resulting policy factorization is πθn an (t) | on (t) = K Y
πθn un,k (t) | on (t) πθn mn,k (t) | on (t),un,k (t)=1
k=1
×πθn ℓn,k (t) | on (t),un,k (t)=1 ,
(22)
where invalid choices are removed via action masking to satisfy the intra-cell OFDMA constraint, and the per-BS sum-power constraint is enforced via the normalization in (16). The local critic Vψn estimates the expected team return conditioned on the local information state, "t+H−1 # X τ −t Vψn (on (t)) ≈ E β r(τ ) on (t) , (23) τ =t
where r(t) is the team reward in (21) and H is the rollout length. Given local rollout tuples {(on (t), an (t), r(t), on (t+1))}, the critic is trained via temporal-difference (TD) regression h 2 i (24) LVn (ψn ) = E Vψn (on (t))− Vbn (t) , Vbn (t) , r(t)+βVψn on (t+1) . (25)
The actor parameters θn are updated locally using a PPO surrogate objective with advantage estimates derived from Vψn . We estimate advantages using generalized advantage estimation (GAE) with parameter λadv ∈ [0,1], which controls the bias– variance tradeoff of the advantage estimator [17]. All updates above are decentralized and use only local trajectories. Gossip-based federated critic mixing: In interferencecoupled multi-cell OFDMA, each BS’s reward and transition dynamics depend on the (unobserved) actions of neighboring
Algorithm 1: FedCritic-MAPPO at BS n Input: neighbors N (n); weights {wnj }; gossip period Kg ; rollout length H; PPO epochs E; mini-batch size Bmb ; (β,λadv ); power budget Pn ; power levels {p(ℓ) }L ℓ=1 ; EMA factor αO . Output: actor params θn , critic params ψn . old ← θ Init: actor πθn , critic Vψn ; set θn n bn,k } queues {Qn,m }; occupancy EMA {O for r = 1,2,... do bj,k }j∈N (n) ; build on (t) Neighbor info: receive {O Buffer: Dn ← ∅ for t = 0,...,H −1 do Observe: on (t) Act: an (t) ∼ πθn (·|on (t)) (per-subcarrier mute/UE/power) Project: map to (xn,k,m (t),pn,k (t)); enforce feasibility via masking/normalization Step: execute; get Rn,m (t); compute P local r̃n (t) and obtain team reward r(t) = N j=1 r̃j (t) min Update: Qn,m ← [Qn,m +Rn,m −Rn,m (t)]+ bn,k +(1−αO )an,k (t) bn,k ← αO O O Store: (on (t),an (t),r(t),on (t+1)) ∈ Dn bn (t) using Vψ and (β,λadv ) Advantage (GAE): compute A n old ← θ PPO: set θn n for e = 1,...,E do foreach mini-batch B ⊂ Dn , |B| = Bmb do Critic: TD regression update on B bn Actor: PPO clipped update on B using θ old ,A n
if r mod Kg = 0 then exchange P {ψj }j∈N (n) ; ψn ← j∈N (n)∪{n} wnj ψj
BSs, which can destabilize purely local critic learning. FedCritic addresses this without a centralized coordinator by federating only the critic over the interference graph GI via lightweight gossip. Let W = [wnj ] ∈ RN ×N be a doubly-stochastic mixing matrix consistent with GI , i.e., wnj > 0 only if j = n or PN PN (n, j) ∈ EI , and j=1 wnj = 1, n=1 wnj = 1. Every Kg learning rounds (i.e., after each policy/critic update), BSs perform a neighbor aggregation step X wnj ψj , ∀n, (26) ψn ← j∈N (n)∪{n}
implemented by exchanging critic parameters with one-hop neighbors. Over each connected component of GI , repeated application of (26) drives {ψn } toward consensus, improving the stability of value estimates under strong interference coupling. Theorem 1 formalizes this effect by establishing consensus (in mean-square) under periodic gossip mixing, and Proposition 1 shows that critic consensus implies vanishing disagreement in TD-based advantage estimates used by PPO. Algorithm 1 summarizes the proposed FedCritic-MAPPO procedure. We federate only the critic to stabilize learning under interference coupling while keeping actor execution local and signaling lightweight. Theorem 1 (Critic consensus under periodic gossip mixing). Let t denote the environment slot index and s the learning round index. Assume a fixed joint policy π during the critic-update window. Each BS n maintains critic parameters ψns ∈ Rd and performs a local stochastic-gradient step
E[gn (ψns ;ξns ) | Fs ] = ∇Fn (ψns ),
(27) (28)
= (Ws ⊗Id )ψ
W, s ≡ 0 (mod Kg ), Ws = IN , otherwise, (29)
,
where Id and IN are the d×d and N ×N identity matrices, s ⊤ ⊤ respectively, and ψ s = [(ψ1s )⊤ ,...,(ψN ) ] . Assume W is doubly stochastic and consistent with a connected interference graph, and define 1 (30) σ , W − 11⊤ < 1, N 2 where 1 ∈ RN is the all-ones vector and k·k2 is the spectral norm. Assume: (A1) each Fn is L-smooth P and boundedPbelow; (A2) E[kgn (ψns ;ξns )k2 | Fs ] ≤ G2 ; (A3) s ηs = ∞ and s ηs2 < P PN 1 s ∞. Let ψ̄ s , N1 N n=1 ψn" and F (ψ) , N # n=1 Fn (ψ). Then N X 1) Consensus: lim E kψns − ψ̄ s k2 = 0. s→∞
2.0
n=1
2) Stationarity of the average: liminf Ek∇F (ψ̄ s )k2 = 0. s→∞
B1: CTDE B2: CTDE+VQ B3: F dAc+o)
1.0 50
75
100
125 150 training updat s
175
P)o(os d: F dC)i+ic GREEDY (h u)is+ic) QoS (h u)is+ic) 200
225
250
Fig. 1: Evaluation average sum-rate per-slot versus training updates. Avg. networ sum-rate (Mbps)
ψ
s+ 1 2
2.5
1.5
where {Fs } is the natural filtration, ξns is the local sample/mini-batch at iteration s, Fn : Rd → R is the local critic loss, and ηs > 0. Every Kg learning rounds (i.e., for s ∈ {0,Kg ,2Kg ,...}), critics are mixed via gossip as ( s+1
Avg. n t.ork sum-rate (Mb(s)
s+ 1
ψn 2 = ψns −ηs gn (ψns ;ξns ),
2.4 2.2 2.0 1.8 1.6 1.4
E
B1: CTD
E+VQ
B2: CTD
ctor B3: FedA
: Proposed
FedCritic
Fig. 2: Distribution of the per-slot average network sum-rate for the final learned policies. and enforces (7b)–(7d) via action masking and power normalization. Finally, periodic neighbor-only critic mixing in (26) provides serverless coordination over GI , aiming to improve stability in interference-dominant regimes.
Proof. See Appendix A. Proposition 1 (Implication for TD advantage estimation in Algorithm 1). Under Theorem 1, assume Vψ (·) is Lipschitz in ψ on O, i.e., |Vψ (o)−Vψ′ (o)| ≤ LV kψ−ψ ′k,
∀o ∈ O,
(31)
where LV > 0 is a Lipschitz constant. For the rollout collected at learning round s and any slot t, define the one-step TD advantage at BS n as b(n) A (32) t,s , r(t)+βVψ s on (t+1) −Vψ s on (t) , n
n
b(avg) and define A analogously using Vψ̄s . Then t,s b(n) b(avg) ≤ (1+β)LV Ekψns − ψ̄ s k, EA t,s − At,s
Hence, the advantage Ekψns − ψ̄ s k → 0.
disagreement
∀n,s, t. (33)
vanishes
as
Theorem 1 characterizes the critic mixing dynamics and, via Proposition 1, its impact on advantage estimation. Global convergence of the full PPO-based MARL loop with nonlinear function approximation is beyond scope. D. Learning Objective Under the shared team reward (21), the cooperative Dec–POMDP objective is " # max J(θ) = E
{θn }N n=1
H−1 X
β t r(t) ,
(34)
t=0
where θ , (θ1 , Q ... , θN ) and the joint policy factorizes as N πθ (a(t) | o(t)) = n=1 πθn (an (t) | on (t)) with decentralized execution. The mapping in (13) induces (X(t), P(t))
IV. S IMULATION R ESULTS We evaluate a reuse-1 multi-cell OFDMA downlink with N = 7 BSs, K = 32 subcarriers, and M = 8 UEs per cell. Each BS allocates transmit power under a per-BS budget Pmax = 1.0 using discretized levels {0.05,0.15,0.35,0.60,1.0}, with noise PSD 10−3 . Large-scale gains follow a log-normal model with (µPL , σPL ) = (−2.3, 0.8) and cross-link scaling 1.2, while small-scale fading evolves via a Gauss–Markov process with correlation ρ = 0.85. Inter-cell coupling is defined by a line neighbor graph with radius 1. We compare four learning methods: B1: CTDE (MAPPO with centralized training and decentralized execution), B2: CTDE+VQ (CTDE augmented with virtual queues), B3: FedActor (federated training with periodic actor mixing across BSs), and Proposed: FedCritic (neighbor-gossiped critic with local execution), along with two non-learning heuristics: GREEDY and QoS. Training uses PPO for 250 updates with rollout horizon H = 128; since actions are selected per subcarrier, each BS contributes H × K samples per update, making Bmb = 256 feasible. We set β = 0.99, λadv = 0.95, clipping ǫ = 0.2, 6 epochs per update, mini-batch size 256, and max gradient norm 0.5, with entropy coefficient decayed from 0.010 to 0.001. We set the per-UE minimum-rate target to Rmin = 2.0 and evaluate every 10 updates over 6 seeds, 6 episodes/seed, and 24 steps/episode; we report mean ± 95% confidence intervals over the 6 per-seed averages. We set Kg = 1 (critic mixing every PPO update) to track strong intercell coupling in the interference-dominant reuse-1 regime. Learning dynamics and final performance: Fig. 1 shows the per-slot average network sum-rate at evaluation
1.0
0.8 0.6
CDF
0.4
−6
B1: CTDE
Q
+V B2: CTDE
c or : FedCr t Proposed
ct B3: FedA
0.0
0.6
0.4
0.2
−8
0.8
B1: CTDE
(a)
Q
+V B2: CTDE
ic or : FedCrit Proposed
ct B3: FedA
(b)
Fig. 3: (a) Mean SINR and (b) neighbor-collision rate, over all active links across all BSs and subcarriers. checkpoints. Proposed-FedCritic exhibits the strongest learning dynamics and reaches the highest evaluation sumrate among the learning-based methods. GREEDY and QoS appear as fixed references: GREEDY is strong but interference-unaware, whereas QoS is more conservative due to its constraint-driven allocation. Fig. 2 reports the distribution of the final per-slot average network sum-rate across evaluation episodes. Proposed-FedCritic achieves the highest median and a tighter spread than the CTDE baselines and B3-FedActor, indicating more reliable performance. Interference robustness: Fig. 3 summarizes interference outcomes over active links. In Fig. 3(a), the CTDE baselines (B1/B2) operate in a substantially lower-SINR regime, whereas B3-FedActor and especially Proposed-FedCritic shift the operating point to higher SINR, reflecting more interference-aware scheduling under reuse-1. Fig. 3(b) reports the neighbor-collision rate. B1/B2 remain near fully colliding reuse patterns, while B3-FedActor and Proposed-FedCritic drastically reduce neighbor collisions, demonstrating learned coordination in the interference-rich setting. Distributional behavior (CDFs): Fig. 4(a) presents the CDF of per-active subcarrier SINR. Proposed-FedCritic produces a clear right-shift, reducing the probability of low-SINR transmissions under reuse-1 interference. Fig. 4(b) shows the CDF of per-active subcarrier-link rate (Mbps), where Proposed-FedCritic improves the high-rate tail by selectively activating and powering links that remain favorable under neighbor interference. Reuse structure and interpretability: Fig. 5 visualizes the learned reuse intensity as BS-by-subcarrier activity. B2: CTDE+VQ tends toward near-uniform high activity across the grid, consistent with its high neighbor-collision behavior. In contrast, Proposed-FedCritic learns structured spatialfrequency reuse with selective activation, which explains its improved SINR distribution and superior network-wide average sum-rate in Figs. 1 and 2. Overall, Proposed-FedCritic achieves a better interference–throughput trade-off in an interference-dominant reuse-1 multi-cell OFDMA regime. V. C ONCLUSION This paper studied interference-coupled reuse-1 multi-cell OFDMA resource management under practical information and coordination constraints. We proposed FedCritic, a fully decentralized MARL framework that removes the centralized critic bottleneck through neighbor-only gossip
0.0
B1: CTDE B2: CTDE+VQ B3: FedActor Proposed: FedCritic
0.4
0.2
0.2 −80
−60
−40
−20
SINR (dB)
0
0.0
20
0.00
0.03
0.05
0.08
0.10
0.13
0.15
Per-active subcarrier-lin rate (Mbps)
(a)
(b)
Fig. 4: (a) CDF of per-active subcarrier SINR (dB) and (b) CDF of per-active subcarrier-link rate (Mbps). 1.0
0
0.9
Mean activity
−4
0.6
2
0.8
4 6
0.7 0
5
10
15
Subcarrier
20
25
30
(a) B2: CTDE + VQ. 1.0
0
0.8
Mean activity
−2
1.0
B1: CTDE B2: CTDE+VQ B3: FedAct r Pr p sed: FedCritic
0.8
Base station
0
Base station
2
CDF
1.0
Neighbor-collisio rate
Mean SINR (dB)
4
2
0.6
4
0.4
6
0.2 0
5
10
15
Subcarrier
20
25
30
0.0
(b) Proposed: FedCritic.
Fig. 5: Activity (reuse intensity) heatmaps over BSs and subcarriers. averaging while keeping execution local. Simulations showed that FedCritic improves SINR and sum-rate distributions, increases average sum-rate and fairness over CTDE baselines and greedy heuristics, and converges more stably with lower coordination overhead. Future work will extend the framework to larger and time-varying interference graphs, heterogeneous settings, richer PHY models, and asynchronous communication-efficient aggregation. A PPENDIX A P ROOF OF T HEOREM 1 PN s ⊤ ⊤ Proof. Let ψ s = [(ψ1s )⊤ ,...,(ψN ) ] and ψ̄ s = N1 n=1 ψns . Define the vector δ s , ψ s − (1 ⊗ ψ̄ s ), so that PNdisagreement s 2 s s 2 kδ k = n=1 kψn − ψ̄ k .
(1) Average is preserved by mixing: Since Ws is doubly stochastic for all s (it is either W or IN ), we have 1⊤ Ws = 1⊤ , hence the gossip step preserves the average: 1 as ψ̄ s+1 = ψ̄ s+ 2 . The averaged iterate evolves N X 1 ψ̄ s+1 = ψ̄ s −ηs ḡ s , ḡ s , gn (ψns ;ξns ). (A.1) N n=1 Taking conditional expectation and using (28) yields PN E[ḡ s | Fs ] = N1 n=1 ∇Fn (ψns ). (2) Disagreement contracts at gossip rounds: At gossip rounds, Ws = W and, with J = N1 11⊤ , 1 δ s+1 = (W −J)⊗Id δ s+ 2
1
1
⇒ kδ s+1 k ≤ kW −Jk2 kδ s+ 2 k = σkδ s+ 2 k,
(A.2)
where σ = kW − Jk2 < 1 [18]. Between gossip rounds 1 (Ws = IN ), δ s+1 = δ s+ 2 . The local SGD step adds bounded noise to disagreement. s+ 1 From ψn 2 = ψns −ηs gn (ψns ;ξns ) and Assumption (A2), there exists Cδ > 0 such that
[12] Z. Yin et al., “Decentralized federated reinforcement learning for user-centric dynamic TFDD control,” IEEE J. Sel. Topics Signal Process., vol. 17, no. 1, pp. 40–53, Jan. 2023. [13] Y. Yang et al., “MADRL-based uplink joint resource block allocation and power control in multi-cell systems,” in Proc. IEEE Wirel. Commun. Netw. Conf. (WCNC), Glasgow, United Kingdom, 2023, pp. 1–6. s+ 1 2 [14] X. Du et al., “Multi-agent reinforcement learning for dynamic resource s 2 2 2 E kδ (A.3) k | Fs ≤ kδ k +Cδ ηs . management in 6G in-X subnetworks,” IEEE Trans. Wirel. Commun., vol. 22, no. 3, pp. 1900–1914, Mar. 2023. Combining (A.2)–(A.3) over periodic mixing yields a super[15] Y. Zhang and D. Guo, “Traffic-aware cellular user association via 2 martingale recursion with contraction factor P σ < 1 at gossip multi-agent reinforcement learning,” in Proc. IEEE 102nd Veh. Technol. rounds and additive term O(ηs2 ). Since s ηs2 < ∞, we obtain Conf. (VTC2025-Fall), Chengdu, China, 2025, pp. 1–5. lim Ekδ s k2 = 0, (A.4) [16] E. Eldeeb and H. Alves, “An offline multi-agent reinforcement learning s→∞ framework for radio resource management,” IEEE Trans. Mobile which proves the consensus claim. Comput., vol. 25, no. 1, pp. 1137–1150, Jan. 2026. [17] N. Y. Mitsuishi, Y. Ma, and J. B. Coder, “Optimized power allocation (3) Stationarity of the averaged critic: By L-smoothness P in multi-cell 4G/5G systems using multi-agent deep reinforcement of F (ψ) = N1 N (A.1), n=1 Fn (ψ) and learning,” in Proc. IEEE 102nd Veh. Technol. Conf. (VTC-Fall), N E D 1X Chengdu, China, 2025, pp. 1–6. ∇Fn (ψns ) +O(ηs2 ). E F (ψ̄ s+1 ) ≤ E F (ψ̄ s ) −ηs E ∇F (ψ̄ s ), [18] H. Li et al., “S-DIGing: A stochastic gradient tracking algorithm for N n=1 distributed optimization,” IEEE Trans. Emerg. Top. Comput. Intell., vol. (A.5) 6, no. 1, pp. 53–65, Feb. 2022. 1 P s s s
Decompose N n ∇Fn (ψn ) = ∇F (ψ̄ )+e , where by Lipschitz gradients, kes k ≤ √LN kδ s k. Using Young’s inequality,
E F (ψ̄ s+1 ) ≤ E F (ψ̄ s ) − η2s Ek∇F (ψ̄ s )k2 +O(ηs )Ekδ s k2 +O(ηs2 ). (A.6)
Summing over s and using that F is bounded below (A1), P 2 and Ekδ s k2 → P 0 from Step (2), yields Ps ηs < ∞ (A3), s 2 η Ek∇F ( ψ̄ )k < ∞. Since s ηs = ∞, it follows s s that liminf s→∞ Ek∇F (ψ̄ s )k2 = 0, proving the stationarity claim.
R EFERENCES [1] E. Hossain and A. Vera-Rivera, “6G cellular networks: Mapping the landscape for the IMT-2030 framework,” IEEE Trans. Technol. Soc., vol. 6, no. 4, pp. 377–392, Dec. 2025. [2] M. Na et al., “Operator’s perspective on 6G: 6G services, vision, and spectrum,” IEEE Commun. Mag., vol. 62, no. 8, pp. 178–184, Aug. 2024. [3] F. Mazzenga and A. Vizzarri, “Time synchronous OFDMA for dense wireless access in open-RAN,” IEEE Commun. Lett., vol. 30, pp. 66–70, 2026. [4] Y. Wang et al., “Interference coordination for autonomous small cell networks based on distributed learning,” in Proc. IEEE Int. Conf. Commun. (ICC), Dublin, Ireland, 2020, pp. 1–6. [5] A. Tusha and H. Arslan, “Interference burden in wireless communications: A comprehensive survey from PHY layer perspective,” IEEE Commun. Surv. Tutor., vol. 27, no. 4, pp. 2204–2246, Aug. 2025. [6] M. A. Hady et al., “Multi-agent reinforcement learning for resources allocation optimization: a survey,” Artif. Intell. Rev., vol. 58, Art. no. 354, Nov. 2025. [7] A. Kopic, E. Perenda, and H. Gacanin, “A collaborative multi-agent deep reinforcement learning-based wireless power allocation with centralized training and decentralized execution,” IEEE Trans. Commun., vol. 72, no. 11, pp. 7006–7016, Nov. 2024. [8] Y. Zhou et al., “Knowledge distillation-based MAPPO approach of wireless power and spectrum resource joint allocation for 6G networks,” in Proc. IEEE Wirel. Commun. Netw. Conf. (WCNC), Milan, Italy, 2025, pp. 1–6. [9] S. Bai, Z. Gao, and X. Liao, “Multi-agent reinforcement learning based distributed multi-user scheduling and beamforming design in multi-cell systems,” IEEE Trans. Veh. Technol., vol. 74, no. 3, pp. 4432–4444, Mar. 2025. [10] S. K. Das et al., “Federated reinforcement learning for wireless networks: Fundamentals, challenges and future research trends,” IEEE Open J. Veh. Technol., vol. 5, pp. 1400–1440, 2024. [11] P. Tehrani, F. Restuccia, and M. Levorato, “Federated deep reinforcement learning for the distributed control of nextG wireless networks,” in Proc. IEEE Int. Symp. Dyn. Spectr. Access Netw. (DySPAN), Los Angeles, CA, USA, 2021, pp. 248–253.