Federated Multiagent Soft Actor Critic for IRS Assisted Cognitive Radio Networks under Partial Observability

Federated Multi-Agent Soft Actor–Critic for IRS-Assisted Cognitive Radio Networks under Partial Observability Satvika Pullisani∗ , Usama Mir∗ , Senior Member, IEEE, and Ubaid Abbasi† ∗ Department of Computer Science and Physics, Wilfrid Laurier University, Brantford, ON, Canada † Department of Science, Northwestern Polytechnic, Grande Prairie, AB, Canada

arXiv:2610.22884v1 [cs.NI] 16 Oct 2026

Corresponding author: Usama Mir (e-mail: umir@wlu.ca) Abstract—Intelligent reflecting surfaces (IRSs) can improve spectrum efficiency and interference control in cognitive radio networks (CRNs) by reconfiguring the propagation environment. This paper presents FMASAC, a federated multi-agent soft actor–critic framework for IRS-assisted CRNs under partial observability, in which distributed secondary users (SUs) jointly select their transmit power and the phase shifts of an assigned subset of IRS elements using only local observations. Primary user (PU) protection is incorporated through an interferencepenalized reward, and SUs coordinate by periodically averaging model parameters at a federated server, without exchanging raw observations, channel state information, or replay data. We formulate the problem as a partially observable Markov decision process, specify the local learning and aggregation procedure, and characterize its per-round communication cost. Illustrative performance trends contrast FMASAC with centralized SAC, independent SAC, IRS-fixed multi-agent reinforcement learning, and a heuristic CRN strategy. We also outline the simulationbased evaluation and the open challenges, namely constraint handling, agent heterogeneity, and scalability, that are needed to validate the framework. Index Terms—Intelligent reflecting surface, cognitive radio network, federated multi-agent soft actor–critic, spectrum sharing, distributed wireless optimization.

I. I NTRODUCTION Intelligent reflecting surfaces (IRSs) have emerged as a promising technology for improving wireless communication through programmable control of the propagation environment [1]. By tuning passive reflecting elements, an IRS can strengthen desired links and suppress harmful interference without active radio frequency (RF) transmission, making it a low-power enabler for spectrum-efficient wireless networks. However, IRS performance depends on selecting phase-shift configurations under dynamic channel conditions, which becomes difficult when channel state information (CSI) is incomplete or rapidly changing. This limitation motivates learningbased control methods such as deep reinforcement learning (DRL), which can adapt IRS decisions through interaction with the wireless environment. DRL has been widely investigated for IRS optimization because it can learn reflection strategies through environment interaction rather than explicit channel models [2]. DRLbased and limited-feedback IRS configurations have shown strong potential in smart radio environments, but they focus

on standalone IRS control and do not jointly model multiple secondary users (SUs), primary user (PU) protection, and decentralized learning [3], [4]. This motivates the use of IRS and DRL in cognitive radio networks (CRNs), where licensed PUs own the spectrum and unlicensed SUs opportunistically access it without causing harmful interference to the PUs. How efficiently licensed spectrum is used also depends on how operators price and share access to it [5]. In this setting, IRSassisted DRL can improve SU communication by adapting reflection decisions to changing wireless conditions. At the same time, the problem becomes more challenging because multiple SUs must make transmission decisions from limited local observations while still protecting the PUs. IRS-assisted CRNs have attracted attention because an IRS can enhance SU links while reducing interference toward the PUs. DRL-based resource allocation and adaptive spectrum access improve IRS-assisted spectrum utilization and adaptability to changing channel and spectrum conditions, but they rely on centralized learning and lack federated multi-agent coordination across distributed SUs [6], [7]. IRSassisted spectrum sensing and primary–secondary transmission improve sensing and transmission performance but do not address decentralized joint learning of SU policies and IRSaware control [8]. DRL-based IRS-assisted CRN optimization shows the value of dynamically adjusting IRS configurations to the wireless environment [9], and hierarchical DRL has been applied to secure resource allocation in multi-IRS spectrum-sharing networks [10]; however, these approaches remain single-agent or centralized and assume broader state availability than distributed CRNs provide. When multiple SUs operate simultaneously in shared spectrum, the action of one SU affects its own link quality, the interference experienced by other SUs, and PU protection. This interaction makes decentralized decision-making difficult under partial observability. Cooperative multi-agent architectures, in which SU agents negotiate spectrum use among themselves without a central controller, were an early answer to this problem [11]. Multi-agent reinforcement learning (MARL) addresses this problem by allowing multiple agents to learn transmission policies while interacting through a shared wireless environment, and multi-agent DRL has already been used to adapt IRS configurations [12]. Cooperative MARL

has improved spectrum sensing efficiency, dynamic spectrum access, and multi-user access decisions [13]–[16], but existing MARL methods often rely on centralized training, shared observations, or frequent information exchange, which increases communication overhead and limits practicality in distributed CRNs. This limitation motivates federated learning, where SUs coordinate by sharing model parameters instead of raw observations or local interaction data. Federated learning also supports privacy-preserving collaboration. Prior studies on IRS-assisted cognitive radio communication [17] mainly focus on communication-level analysis and performance modeling, and IRS-assisted federated learning studies improve communication efficiency and secure parameter aggregation [18], [19]. However, these works do not address decentralized multi-agent learning, spectrum-sharing dynamics, or IRS-assisted reinforcement learning under partial observability. Conventional federated learning alone also does not model the interaction among multiple agents competing in shared spectrum. Federated multi-agent actor–critic learning has been applied to dynamic spectrum sharing [20], and federated multi-agent DRL has been surveyed for wireless communications [21], but neither provides a unified IRSassisted CRN framework with soft actor–critic (SAC)-based continuous control. To the best of our knowledge, no existing study jointly addresses IRS-assisted CRN optimization, federated learning, multi-agent SAC (MASAC)-based continuous control, and partial observability in one framework. To address this gap, this paper proposes a federated MASAC (FMASAC) framework in which each SU trains locally while a federated server aggregates model parameters without raw data exchange. The main contributions are as follows: • We propose FMASAC, a federated MASAC framework for IRS-assisted CRNs that operates without raw data sharing or global CSI. • We give a joint formulation of SU transmit power and IRS phase control under a PU interference constraint and partial observability, including an explicit IRS elementpartitioning model that resolves how multiple agents act on a single IRS. • We model the problem as a partially observable Markov decision process (POMDP), specify the reward, the SAC updates, and the federated aggregation rule, and characterize the per-round communication cost. • We present illustrative performance trends against four benchmarks and outline the simulation-based evaluation and open challenges required to validate the framework. The remainder of this paper is organized as follows. Section II presents the system model and POMDP formulation, Section III describes FMASAC, Section IV presents illustrative performance trends, Section V discusses limitations and the evaluation plan, and Section VI concludes the paper. II. S YSTEM M ODEL AND P ROBLEM F ORMULATION We consider an IRS-assisted CRN with one primary base station (PBS) serving one PU, and M distributed SU

TABLE I M AIN N OTATION Network and IRS parameters M, N Number of SUs; number of IRS elements Θ(t), θn (t) IRS reflection matrix; phase of element n Ni Subset of IRS elements controlled by SU i pi (t), Pmax Transmit power of SU i; power budget SINRi (t), Ri (t) SINR and rate at the receiver of SU i Ii (t), Imax Interference from SU i at the PU; threshold POMDP and learning variables s(t), oi (t) Global state; local observation of SU i ai (t), a(t) Action of SU i; joint action ri (t), λ Reward of SU i; interference penalty weight γ, ξ(t) Discount factor; channel variation and noise SAC and federated learning parameters Actor policy of SU i and its parameters πφi , φi Q (k) , k ∈ {1, 2} Twin critics of SU i and their parameters ψi

Di α K, R wi

Local replay buffer of SU i Entropy coefficient Local steps per round; number of rounds Aggregation weight of SU i

Fig. 1. IRS-assisted CRN topology with direct, reflected, and interference paths.

transmitter–receiver pairs that opportunistically access the licensed spectrum while keeping the aggregate interference at the PU below a prescribed threshold. An IRS with N passive reflecting elements assists the secondary transmissions. The principal symbols are summarized in Table I. A. Channel and IRS Model As shown in Fig. 1, the IRS creates a controllable reflected path that can strengthen useful SU signals while reducing interference toward the PU. All channels follow Rayleigh smallscale fading with log-distance path loss. The IRS reflection matrix is   (1) Θ(t) = diag ejθ1 (t) , . . . , ejθN (t) , where θn (t) ∈ [0, 2π) is the phase shift of the nth element. The effective channel from the transmitter of SU j to the receiver of SU i is hji (t) = hdji (t) + giH (t)Θ(t)fj (t), where hdji is

the direct link, fj is the transmitter-to-IRS channel, and gi is the IRS-to-receiver channel. The channel hjP (t) to the PU is defined in the same way. The SINR at the receiver of SU i is SINRi (t) = P

pi |hii |2 , 2 2 2 j̸=i pj |hji | + p0 |h0i | + σ

(2)

where p0 and h0i are the PBS transmit power and its channel to SU receiver i, and σ 2 is the noise power. The achievable rate is Ri (t) = log2 (1 + SINRi (t)), and the interference caused by SU i at the PU is Ii (t) = pi (t)|hiP (t)|2 . Because a single physical IRS cannot apply conflicting phase requests from several agents, the N elements are partitioned into disjoint subsets N1 , . . . , NM , and SU i controls only the phases of the elements in Ni . This keeps each agent’s action space fixed and avoids a central IRS arbitration step; alternatives such as a dedicated IRS controller agent are discussed in Section V. B. POMDP Formulation Each SU has access only to local information, namely received signal strength, partial CSI of its own links, a PU interference indicator broadcast by the PBS, and limited IRS feedback. The global state and the other SUs’ actions and observations are unavailable. The problem is therefore modeled as a POMDP ⟨S, A, O, P, R, γ⟩, where S, A, O, P , and R denote the global state space, joint action space, observation space, transition probability, and reward function, and γ ∈ (0, 1) is the discount factor. At time t, SU i observes oi (t) ∈ O and selects ai (t) = [pi (t), {θn (t)}n∈Ni ] , with pi (t) ∈ [0, Pmax ]. The joint action is a(t) (a1 (t), . . . , aM (t)), and the global state evolves as s(t + 1) = f (s(t), a(t)) + ξ(t),

(3) = (4)

where f (·) captures the environment dynamics and ξ(t) captures channel variations and noise. The aggregate PU interference must satisfy M X

Ii (t) ≤ Imax .

(5)

i=1

Constraint (5) is enforced through the reward. Each SU receives  + M X ri (t) = Ri (t) − λ  Ij (t) − Imax  , (6) j=1

where [x]+ = max(0, x) and λ > 0 weights PU protection against SU rate. The violation term is available to every SU through the PBS interference indicator, so no SU needs the other SUs’ channels or actions. The penalty enforces (5) softly: violations are discouraged but not ruled out during learning. The objective is to learn policies {πi } that maximize the expected discounted reward "∞ # X t max E γ ri (t) . (7) {πi }

t=0

Fig. 2. POMDP decision loop with federated aggregation.

In the federated setting, each SU learns from its own interaction data and periodically uploads only model parameters, and the server returns an aggregated model. III. P ROPOSED F EDERATED MASAC F RAMEWORK Building on the POMDP formulation, FMASAC combines local continuous-control learning at each SU with periodic federated aggregation, enabling distributed policy improvement without exchanging raw data. As shown in Fig. 2, each SU interacts with the IRS-assisted environment using only local observations and updates its policy and value networks from a local replay buffer. A federated server periodically aggregates the local model parameters and broadcasts the result, preserving distributed operation while improving crossagent consistency. A. Local SAC Learning Each SU runs a local SAC agent with actor πφi and twin critics Qψ(1) and Qψ(2) , both conditioned on local observations i i and actions only. SAC suits this problem because transmit power and phase shifts are continuous decisions, and entropy regularization improves exploration and stability under partial observability. The critic target is   ′ ′ ′ ′ y = r + γ min Qψ̄(k) (o , ã ) − α log πφi (ã | o ) , (8) k

i

(k)

where ã′ ∼ πφi (· | o′ ) and ψ̄i are target-critic parameters. Each critic minimizes the soft Bellman loss  2  (k) L(ψi ) = E(o,a,r,o′ )∼Di Qψ(k) (o, a) − y , (9) i

and the actor maximizes   J(φi ) = E o∼Di min Qψ(k) (o, a) − α log πφi (a | o) , (10) a∼πφi

k

i

where α controls the exploration–exploitation trade-off.

Algorithm 1 FMASAC Training Input: M SUs, local steps K, rounds R, weights {wi } (1) (2) 1: Initialize φi , ψi , ψi , target critics, and Di for each SU i 2: for round r = 1, . . . , R do 3: for each SU i in parallel do 4: for k = 1, . . . , K do 5: Observe oi (t); sample ai (t) ∼ πφi (· | oi (t)) 6: Execute ai (t); receive ri (t) and oi (t + 1) 7: Store (oi (t), ai (t), ri (t), oi (t + 1)) in Di 8: Update critics via (9) and actor via (10) 9: Soft-update target critics (1) (2) 10: Upload {φi , ψi , ψi } to the server 11: Server aggregates parameters using (11) 12: Server broadcasts aggregated parameters to all SUs Output: Aggregated actor and critic parameters Fig. 3. Average SU throughput versus training episodes (illustrative).

B. Federated Aggregation After K local steps, each SU uploads its actor and critic parameters, and the server forms the weighted average φ̄ =

M X

wi φi ,

i=1

|Di | wi = PM , j=1 |Dj |

(11)

with the same rule applied to each critic. The aggregated parameters are broadcast to all SUs. Averaging assumes that SUs are sufficiently similar for a shared model to be useful; we normalize observations per SU to support this, and Section V discusses heterogeneous settings where personalized aggregation is more appropriate. Algorithm 1 summarizes the procedure. C. Communication Cost Let |φ| and |ψ| denote the numbers of actor and percritic parameters. In each round, every SU uploads and downloads |φ| + 2|ψ| parameters, so the total traffic per round is 2M (|φ|+2|ψ|), and R rounds cost 2M R(|φ|+2|ψ|). The cost is independent of the replay-buffer size and of the number of interaction steps K, and no CSI or observations leave the SU. Increasing K reduces the number of rounds needed for a given amount of local experience but lets local models drift further apart between aggregations, which is the trade-off examined in the planned evaluation. IV. I LLUSTRATIVE P ERFORMANCE T RENDS A. Scope of This Section The curves in Figs. 3–6 were produced in Python (NumPy, Matplotlib) from controlled synthetic performance traces that encode the qualitative behavior expected of each method. They are not the outcome of training agents in a simulated network and should be read as illustrations of the tradeoffs that FMASAC targets, not as measured performance. A simulation-based evaluation is in progress, and its design is described in Section V. The illustrated setting consists of one PBS, one PU, and distributed SUs over a 1 km2 area, with a fixed-position IRS and Rayleigh fading with log-distance path loss, over 300 training episodes. FMASAC is contrasted with four

Fig. 4. PU interference level, normalized to Imax , versus training episodes (illustrative).

benchmarks: centralized SAC (a single agent with global information access, which serves as a reference but violates the distributed assumption), independent SAC (local learning without aggregation), IRS-fixed MARL (learned power control with a fixed IRS configuration), and a heuristic CRN strategy (conservative transmission under PU protection). Four metrics are considered: average SU throughput (spectrum utilization), PU interference level normalized to Imax (PU protection), average cumulative reward (convergence), and action variance across SUs (policy consistency). B. Illustrated Trends Fig. 3 illustrates the expected throughput ordering. FMASAC is shown above independent SAC, IRS-fixed MARL, and the heuristic strategy, reflecting the intended benefit of combining IRS-aware control with federated aggregation, while centralized SAC acts as an upper reference because it uses information unavailable to distributed SUs. Fig. 4 illustrates PU interference. The learning-based methods start above the normalized threshold and approach it as training proceeds, with FMASAC shown fluctuating less than independent SAC and IRS-fixed MARL. Only the heuristic

Simulation-based validation. Implement the channel model of Section II with SAC agents and report, for stated values of M , N , K, Pmax , σ 2 , λ, node positions, and network hyperparameters, the mean and confidence intervals over independent random seeds. • Aggregation interval and overhead. Vary K and measure throughput and constraint satisfaction against the communication cost derived in Section III-C. • Scalability. Vary M and N to test whether the elementpartitioning model and parameter averaging remain effective as the network grows. • Constraint handling. Compare the penalty in (6) with primal–dual or Lagrangian SAC, and report the probability of violating (5) rather than only the average interference level. • Heterogeneity. Evaluate non-identically distributed SU locations and channels, and compare plain averaging in (11) with personalized federated aggregation. • IRS control architecture. Compare element partitioning with a dedicated IRS controller agent that sets all N phases from aggregated SU feedback.

•

Fig. 5. Average cumulative reward versus training episodes (illustrative).

VI. C ONCLUSION

Fig. 6. Action variance across SUs versus training episodes (illustrative).

strategy stays below the threshold throughout, at the cost of lower throughput. This pattern reflects the soft, penaltybased enforcement of (5) in (6): early in training, constraint violations are expected. Applications that require guaranteed protection would need the constrained formulations discussed in Section V. Fig. 5 illustrates convergence. Periodic aggregation is expected to smooth learning in the non-stationary multi-SU environment, so FMASAC is shown converging more smoothly than the decentralized baselines, while centralized SAC converges well but relies on global information. Fig. 6 illustrates policy consistency through action variance across SUs. Aggregation is expected to reduce this variance relative to uncoordinated learning, particularly independent SAC, which would translate into more consistent multi-user behavior and less interference variation. Whether these trends hold in practice, and by how much, is precisely what the simulation study must establish. V. L IMITATIONS AND E VALUATION P LAN The framework leaves several questions open, which define the planned evaluation.

This paper studied spectrum access and IRS-assisted control in CRNs under partial observability, where distributed SUs must learn from local observations while protecting the PU. We proposed FMASAC, in which each SU performs local soft actor–critic learning over its transmit power and an assigned subset of IRS elements, and a federated server periodically averages model parameters without raw data exchange. We formulated the problem as a POMDP with an interferencepenalized reward, specified the learning and aggregation procedure, and characterized its communication cost. Illustrative trends indicate the trade-offs the framework is designed to improve: throughput, PU protection, convergence smoothness, and multi-user consistency relative to decentralized baselines. Validating these trends through simulation, studying the tradeoff between aggregation frequency, communication overhead, and learning performance, and strengthening constraint guarantees are the subjects of ongoing work. R EFERENCES [1] A. Taha, Y. Zhang, F. B. Mismar, and A. Alkhateeb, “Deep reinforcement learning for intelligent reflecting surfaces: Towards standalone operation,” in Proc. IEEE SPAWC, Atlanta, GA, USA, 2020, pp. 1–5. [2] Y. Omid, Y. Deng, and A. Nallanathan, “Deep reinforcement learningbased secure standalone intelligent reflecting surface operation,” in Proc. IEEE GLOBECOM, Rio de Janeiro, Brazil, 2022, pp. 5329–5334. [3] W. Wang and W. Zhang, “Intelligent reflecting surface configurations for smart radio using deep reinforcement learning,” IEEE J. Sel. Areas Commun., vol. 40, no. 8, pp. 2335–2346, Aug. 2022. [4] X. Zhu, H. Cao, Z. Li, and L. Yang, “System information feedback assisted reconfigurable intelligent surface configuration: Leveraging deep reinforcement learning,” in Proc. IEEE WCSP, Hefei, China, 2024, pp. 1079–1083. [5] U. Mir and L. Nuaymi, “LTE pricing strategies,” in Proc. IEEE 77th Veh. Technol. Conf. (VTC Spring), Dresden, Germany, 2013, pp. 1–6, doi: 10.1109/VTCSpring.2013.6692625.

[6] J. Guo, Z. Wang, J. Li, and J. Zhang, “Deep reinforcement learning based resource allocation for intelligent reflecting surface assisted dynamic spectrum sharing,” in Proc. IEEE WCSP, Nanjing, China, 2022, pp. 1178–1183. [7] V. Srivastava and B. Prasad, “Enhancing adaptive spectrum access: An intelligent reflecting surface assisted CRN for future wireless communication,” IEEE Access, vol. 13, pp. 129348–129358, 2025. [8] X. Liang, L. Huang, Q. Deng, F. Shu, G. Yu, and J. Wang, “IRS-assisted spectrum sensing and primary-secondary transmission for cognitive radio networks,” IEEE Trans. Cogn. Commun. Netw., vol. 11, no. 3, pp. 1508–1521, Jun. 2025. [9] C. Zhong et al., “Deep reinforcement learning-based optimization for IRS-assisted cognitive radio systems,” IEEE Trans. Commun., vol. 70, no. 6, pp. 3849–3864, Jun. 2022. [10] L. Wang, W. Wu, F. Zhou, Q. Wu, O. A. Dobre, and T. Q. S. Quek, “Hybrid hierarchical DRL enabled resource allocation for secure transmission in multi-IRS-assisted sensing-enhanced spectrum sharing networks,” IEEE Trans. Wireless Commun., vol. 23, no. 6, pp. 6330– 6346, Jun. 2024. [11] U. Mir, L. Merghem-Boulahia, and D. Gaı̈ti, “COMAS: A cooperative multiagent architecture for spectrum sharing,” EURASIP J. Wireless Commun. Netw., vol. 2010, Art. no. 987691, 2010, doi: 10.1155/2010/987691. [12] S. Monga et al., “Adaptive intelligent reflecting surfaces for enhanced wireless communication via multi-agent deep reinforcement learning,” IEEE Access, vol. 13, pp. 62071–62087, 2025. [13] A. Gao, C. Du, S. X. Ng, and W. Liang, “A cooperative spectrum sensing with multi-agent reinforcement learning approach in cognitive radio networks,” IEEE Commun. Lett., vol. 25, no. 8, pp. 2604–2608, Aug. 2021. [14] X. Tan et al., “Cooperative multi-agent reinforcement-learning-based distributed dynamic spectrum access in cognitive radio networks,” IEEE Internet Things J., vol. 9, no. 19, pp. 19477–19488, Oct. 2022. [15] O. Naparstek and K. Cohen, “Deep multi-user reinforcement learning for distributed dynamic spectrum access,” IEEE Trans. Wireless Commun., vol. 18, no. 1, pp. 310–323, Jan. 2019. [16] S. M. Vangaru, A. K. Sharma, and R. Budhiraja, “Multi-user opportunistic spectrum access in cognitive radio networks using multi-agent reinforcement learning,” Sensors, vol. 25, 2025. [17] H. T. Nguyen, T. Q. Duong, and H. D. Tuan, “IRS-aided cognitive radio short-packet communications over Nakagami-m fading channels,” ETRI J., vol. 47, no. 2, pp. 1–14, 2025. [18] Y. Wu and B. Mao, “Accelerating and securing federated learning via intelligent reflecting surface,” in Proc. IEEE ICCT, Wuxi, China, 2023, pp. 1219–1224. [19] B. Mao, Y. Wu, J. Liu, H. Guo, J. Wang, and N. Kato, “Optimizing secrecy rate for federated learning model aggregation with intelligent reflecting surface toward 6G ubiquitous intelligence,” IEEE Trans. Cogn. Commun. Netw., vol. 11, no. 2, pp. 1258–1267, Apr. 2025. [20] T. Yang, W. Zhang, Y. Bo, J. Sun, and C.-X. Wang, “Dynamic spectrum sharing based on federated learning and multi-agent actor-critic reinforcement learning,” in Proc. IEEE IWCMC, Marrakesh, Morocco, 2023, pp. 947–952. [21] H. D. Oliveira, M. Kaneko, and L. Boukhatem, “Federated multiagent deep reinforcement learning for intelligent IoT wireless communications: Overview and challenges,” IEEE Veh. Technol. Mag., vol. 19, no. 4, pp. 73–82, Dec. 2024.

Related documents

• Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning#271868
• Uno-Orchestra: Parsimonious Agent Routing via Selective Delegation#158562
• AI-driven adaptive traffic light optimization using dueling deep Q-networks#432255
• Multi-agent Collaboration with State Management#216915
• Improving LLM Collaboration via Multi-Agent Preference Learning#1117158