Temporally Consistent Graph Q-Networks for Intelligent Network Control
arXiv:2606.13848v1 [cs.NI] 11 Jun 2026
Zacharias Veiksaar1 , Maxime Bouton2 [email protected], [email protected] are typically task-specific, and as operators are expected to change intents dynamically, agents must adapt their policies accordingly. Existing algorithms capture intents via a reward function and thus require retraining whenever the intent changes. This incurs a significant cost in computation and data collection. Conversely, recent progress in single-agent RL has introduced self-supervised learning components to improve sample efficiency by learning an efficient representation of the environment. These methods have been highly successful on benchmark problems, improving performance and reducing data required for training [5, 6]. However, integrating these methods with graph-based MARL to learn a transferable network representation that supports adaptation to new intents remains unexplored. To bridge this gap, we propose TC-GQN, a novel algorithm that extends graph-based RL with self-supervised temporal consistency learning. We pool graph-aware local observation embeddings from each agent into a global state using an attention mechanism, and encode this into a self-predictive latent state representation. By utilizing an auxiliary loss to I. I NTRODUCTION predict inherent environment variables, rather than reward, TCRadio access networks are increasing in density and 6G GQN decouples the prediction task from the reward function is expected to support an increase in traffic as well as new and yields a task-independent encoding of the environment dyservices related to robotics, healthcare, and cloud gaming with namics. This allows us to maintain the coordination benefits of strict throughput and latency constraints. To achieve this, the graph-based techniques while introducing the rapid adaptation industry is shifting towards a new paradigm of intent-driven capability of representation learning in a multi-agent setting. AI-native autonomous networks. Operators are expected to In this work, we model intelligent network control as a steer the network through high-level intents, and AI is seen as cooperative MARL problem where intents on energy-saving an enabling technology to realize this intent through real-time and QoS are captured in a shared reward function, and agents decision-making by changing network parameters [1, 2]. can enable or disable carrier-sleep. In simulation, we show that Optimizing the network based on high-level intent requires the algorithm outperforms a rule-based baseline and graphhaving coordinated policies for all network nodes as changing based MARL methods. We further test adaptation to new parameters in one node can greatly affect other parts of the intents by adding more QoS constraints and comparing a network, for example by causing congestion in neighboring pretrained agent with a randomly initialized agent, and find that sites. In addition, network performance is highly dependent on the agent requires substantially less data and outperforms the traffic demand as well as radio propagation conditions which baselines from the start when pretrained. The learned global are hard to model. Therefore, heuristic strategies to realize network representation also changes little during fine-tuning intent-driven automation are often suboptimal and instead AIwhich indicates that it captures network dynamics and remains based methods such as multi-agent reinforcement learning informative across different intents. (MARL) have been proposed [3]. Prior work has applied MARL to coverage and capacity II. R ELATED W ORK optimization using graph neural networks (GNNs) to coordinate Reinforcement learning techniques have shown promising antenna tilt control across network nodes [3] as well as to results when applied to various network control tasks. In the the domain of energy-saving [4]. However, these RL agents energy efficiency domain, deep RL techniques have successfully Abstract—Mobile networks continue to grow in complexity and next generation networks are expected to support both increasing traffic loads and more diverse services. As network complexity rises, optimizing antenna parameters under dynamic or changing objectives becomes increasingly challenging. We propose a novel multi-agent reinforcement learning (MARL) algorithm for high-level control and orchestration of mobile networks. The Temporally Consistent Graph Q-Network (TC-GQN) algorithm learns a self-predicting representation of the whole network that is task-independent and aggregates information from all base-stations. A graph neural network is trained using a global reward function to assign coordinated local actions based on the learned encoding of the global network state. We evaluate the algorithm in a simulated environment to orchestrate an energy-saving feature across multiple sectors and multiple carriers under different quality of service (QoS) constraints. The proposed algorithm outperforms state-of-the-art graph-based baselines and a competitive rule-based controller by improving hardware sleep time while maintaining QoS. Moreover, the learned representation enables rapid adaptation to changing intents. Index Terms—multi-agent reinforcement learning, autonomous networks, graph neural networks, representation learning
1 Work done while at Ericsson Research, Sweden. 2 Ericsson Research, Sweden.
been applied to reduce energy consumption in networks under QoS constraints, both in single-agent settings [7] and
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
in large MARL settings [4]. Similarly, graph-based neural networks have also been shown effective in coordinating multiagent antenna tilt control, both through neighbor-enhanced observations [8] and by decomposing the importance of a global reward signal across nodes through value decomposition [3]. However, these approaches are typically limited to optimizing a single predefined objective defined through a reward function, leaving them unable to adapt to changing network intents without retraining. To deploy such solutions, one must first address the fundamental scalability challenge of multi-agent environments. The dimensionality of the state and action spaces can grow quickly and make naı̈ve approaches such as concatenating agent observations and actions become impractical. Such solutions also suffer from a lack of adaptability, for example, when the number of agents in the environment changes. Graphbased methods address this issue by enhancing single-agent algorithms with neighboring observations [8] or through more general coordinated algorithms that are robust to network topologies [3]. Apart from scalability, another problem that arises in a multi-agent setting is the lack of individual reward signals that promote cooperation among agents. One method for proper reward assignment is to learn a decomposition of a global reward signal [9, 10], which can be combined with graph-based networks for cooperation. However, relying solely on a scalar reward signal to inform complex cooperation behavior can be insufficient for learning a desired policy. To address this, representation learning in RL provides informative latent states by training an encoder with an auxiliary task. These tasks can be predicting future states, creating representations that capture the transition dynamics of an environment, which improves sample efficiency and generalization. This latent space can be used either to augment the observations of a traditional RL algorithm [5, 11] or for model predictive control [6] and has been highly effective in such single-agent RL settings. Our work bridges the gap between these domains by introducing TC-GQN, an algorithm which extends graph-based MARL with a self-supervised component that learns a taskagnostic representation of the global network state. By training the network to predict inherent environment variables rather than rewards, we decouple the learned representation from the immediate task and instead focus on encoding the underlying environment dynamics. This allows for rapid adaptation to new intents while maintaining the coordination benefits of graph-based techniques. III. BACKGROUND A. Multi-Agent Reinforcement Learning Cooperative MARL can be defined as a decentralized partially observable Markov decision process (decPOMDP). A dec-POMDP is formally defined by the tuple (n, S, {Ai }ni=1 , {Oi }ni=1 , P, O, R, γ), where n is the number of agents, S is the global state space, Ai is the action space of agent i, and Oi is the observation space of agent i. P (st+1 | st , a1t , . . . , ant ) is the unknown transition probability
OFF
1
Sector - agent
0
Offered traffic intensity 1
OFF
0
1
1
ON
User equipment
0
Network site with 3 sectors
OFF
1 0
1
ON
ON
Carrier utilization in a sector
0
0
Fig. 1. Illustration of the RAN with 21 sectors and 4 carriers per sector and an associated graph. Traffic is distributed non-uniformly over the map.
of the global state given the previous state and the agents’ actions, and O(o1t+1 , . . . , ont+1 | st+1 , a1t , . . . , ant ) describes the unknown probability of each agent receiving a local observation given the global state and actions. The shared reward function R outputs a reward at each step rt = R(st+1 , a1t , . . . , ant ) ∈ R and γ ∈ [0, 1) is a discount factor for future rewards. The goal of our algorithm is to find a joint policy π : O1 × . . . × On → A1 × . . . × A Pn∞that maps local observations to actions and maximizes E[ t=0 γ t rt+1 | s0 ] where rt is the shared reward obtained by following the policy. Often, a value function Q can be learned to estimate this quantity, and a policy can then be derived by maximizing the Q-function. B. Deep Learning on Sets Agents can be represented by an undirected graph or a set, and an adaptable algorithm must be invariant to the ordering of agent observations. To represent the Q-function as a parametric function, deep learning architectures such as attention mechanisms and graph neural networks are particularly useful [12]. We use GNNs to transform the state of the environment into local embedding vectors which contain information about neighboring agents according to a graph topology provided by the environment. In particular, we adopt the graph attention network architecture [13], which uses selfattention to identify the most relevant neighbor observations and supports a variable number of neighbors. Since GNNs output graphs with vector embeddings, we compress them into a global latent network state using an attention-based pooling mechanism [12] detailed in Section V. IV. I NTELLIGENT RAN E NERGY C ONTROL AS MARL A. System Model To evaluate the benefit of MARL with global network representation learning for RAN control, we consider an energysaving problem in which certain carriers in the network can be shut down to reduce energy consumption during periods of low traffic demand. Disabling a carrier may offload users to neighboring cells, so decisions must account for the entire network rather than a single cell. Energy-savings must be balanced with maintaining stringent service requirements and this trade-off is expected to be changed dynamically based on operator intent, requiring the AI agent to adapt quickly.
The RAN consists of multiple geographical sectors equipped with multiple base-station antennas operating at different frequency bands, and user equipment connects through one or more of these frequencies. A given frequency in a specific sector, and its connected users, is referred to as a cell. We consider a geographical area with a hexagonal deployment consisting of 21 sectors as illustrated in Fig. 1. Each sector is equipped with four carriers (also known as layers): 800 MHz, 1800 MHz, 2100 MHz, and 3500 MHz corresponding to one low- and mid-band coverage frequency, and two mid-band capacity layers referred to as L08, L18, L21, and L35 respectively. They have bandwidths of 10 MHz, 20 MHz, 20 MHz, and 100 MHz respectively. The L21 carrier is assumed to be equipped with an energysaving feature running on the base-station that can be activated remotely. In a real deployment, when activated, the feature could intermittently turn the cell on or off as fast as every millisecond [4]. For simplicity, we model the feature as keeping the cell off all the time when it is activated. We assume that our proposed algorithm is implemented as a non-real time radio intelligent controller in an open-RAN architecture [1]. Its role is to orchestrate the energy-saving features to satisfy high-level intents such as maintaining a good quality of service. This centralized controller needs to receive performance counters from each network node and typically operate at a frequency ranging from minutes to days. We assume that this centralized controller takes a decision every hour to enable or disable a local energy-saving feature. A total of 1000 users are placed uniformly on the map and randomly labeled indoor or outdoor to yield 80% indoor traffic. Users generate downlink traffic according to a Weibull distribution creating hotspots in the map [14]. The intersite distance is sampled uniformly between 400 m and 1000 m at the start of each training or evaluation episode (see Section VI). To model temporal traffic variation, we sample an hourly total traffic demand over the area from a dataset collected across more than 100 live 5G base stations. Users connect to the carrier providing the highest achievable bitrate. Bitrate is estimated using 3GPP-compliant models from the received signal to noise ratio per carrier. Signal strength is computed using the site specific propagation model of Asplund et al. [15]. The number of connected users per cell, the offered traffic per user, and their estimated throughput are then used to compute key performance indicators (KPIs). Our system model relies on three main KPIs: carrier utilization, throughput, and power consumption. The utilization of a carrier c associated with agent i is Uci and corresponds P n Uic = NuPRBcu where nu is the number of PRBs associated with user u and depends on the user throughput ρu , and NPRBc is the total number of PRBs available at the carrier, determined by its bandwidth. Throughput is defined per carrier and corresponds to the average P throughput of the users connected to that carrier: τci = |U1i | u∈Uic ρu where Uic represents the set of users c connect to carrier c in sector i. Finally, to estimate the power consumption of each RAN sector, we adopt a model in which the total power is the sum of the idle power consumption of the
radio at each carrier and a load-dependent component scaling approximately linearly with the utilization, as recommended in ETSI TR 132 972 [16]. B. MARL Formulation To formulate the network control problem as MARL we need to define the agents, the observation spaces and action spaces of each agent, and the shared reward function. The objective is to turn antennas off to save energy while maintaining a minimum level of throughput in a given network area with multiple base-stations. This creates a complex MARL problem where agents must coordinate to minimize energy consumption without disrupting user connectivity. We use the energy-saving problem described in the previous section as an example, with the observations, actions, and rewards being specific design choices. However, we anticipate that the proposed algorithm will generalize to other variants of this network control problem, such as tilt optimization [3]. We model the problem as a discrete task where each agent selects an action at each time step. During a step t, our algorithm observes the network topology, and gets local observation vectors, o1t , ..., ont . It performs inference and returns actions for all sectors, a1t , ...ant and observes the next observations and the shared rewards which are added to a replay buffer as described in the next section. We define agents as sectors in the network. In our test hexagonal deployment there are thus 21 agents. Each agent is equipped with the four carriers described above. The observation space corresponds to network KPIs for each carrier. Each agent observes the utilization of each carrier (four values), the average user throughput in each carrier (four values), the state of the energy-saving feature taking values -1 and 1. Finally, it also observes an encoding of the hour of 2πh the day: [sin( 2πh 24 ), cos( 24 )] where h ∈ {0, 1, . . . , 23} is the hour of the day. The observation space of each agent is thus an 11-dimensional vector. In addition, our algorithm observes the graph topology Γt . The topology is determined via geographical proximity as well as orientation of each sector. Two sites are connected if they are less than 2 km from each other via the sectors facing each other, and each co-site sector is connected, as illustrated in Fig. 1. In a live network, the graph could be obtained through standard feature like automatic neighbor relations based on the hand-over between cells. Our algorithm is agnostic to the specific graph size and topology. The action space is discrete and has only two values: ON and OFF. It corresponds to activating the energy feature on the L21 carrier. When the OFF action is selected we say that the carrier is considered to be in a sleep state. The shared reward is defined as a sum of local reward components. We define Ri (st+1 , oit , ait ) as the local reward associated to agent i. Note that this local reward depends on the unobserved global state which can be affected by the actions of other agents through possible offloading of users between sectors. Ri is defined using indicator functions as follows: [1[τi,L18 >50Mbps] −λ1[τi,L18 ≤50Mbps] ]1[ai =off] − P̃i 1[ai =on] where 1 is the indicator function, τi,L18 represents the average user
Fig. 3. Diagram of the prediction task with a horizon of H = 2 timesteps. Fig. 2. Diagram of the proposed algorithm during inference.
from the observations of the agent’s neighbors using a GNN. These encoded observations hkt are then pooled over the throughput in the L18 carrier for agent i, and P̃i is the power agent dimension using attention based pooling to generate consumption of the whole sector (considering all 4 carriers) a global graph state representation gt . It is this global graph normalized to be between 0 and 1. In simple terms, the reward representation gt that then acts as an input to the representation gives a bonus of 1 when the agent turns on sleep and good learning where we generate the temporally consistent global throughput is achieved and gives a penalty of −λ when the graph encoding zt . To generate q-values for each agent we agent enables sleep and causes poor throughput. When the concatenate okt , hkt , and zt which together act as the input to agent does not turn on sleep, the reward is proportional to the a standard DQN architecture. We keep target networks for power consumption. The P total shared reward is the sum of the the GNN, pooling, encoder, and value components which are n individual rewards: R = i=1 Ri . denoted with a superscript minus sign, e.g. θ− . In an intent-based autonomous network, this reward function The proposed algorithm is an off-policy algorithm and experimay change based on high-level intents. For example, the ence is gathered through interaction with the environment using 50Mbps constraint on throughput may be changed by the an epsilon-greedy exploration scheme. Gathered experience is operator, it may be defined on a different carrier than the L18 stored in a replay buffer for access during the training pass one, or simply the λ parameter may be adjusted. We expect where gradient updates are computed from a Q-function loss these changes to lead to different behavior when we solve and an auxiliary loss, both of which are explained in more the MARL problem. With standard algorithms, however, such detail in the subsequent sections. After gradient updates are changes in intent require retraining the agent from scratch. completed target networks are updated using Polyak averaging. Introducing the self-supervised prediction objective increases V. P ROPOSED A LGORITHM training-time computational cost relative to the closest benchA. Components and Training mark algorithm GQN [3] due to the additional forward passes The proposed algorithm uses a graph-based Q-network required for prediction and auxiliary loss evaluation. However, architecture that is augmented to ingest a global state rep- the complexity during inference is only marginally increased as resentation learned through the prediction-based auxiliary task, the algorithm requires a single additional forward pass through as illustrated in Fig. 2. To generate this representation, we first the global encoder without any prediction taking place. This pool agent-wise information using an attention mechanism to makes the algorithm close to parity with a standard GQN output a global graph state. This state is then passed through architecture in terms of inference latency. an encoder to generate a temporally consistent global state B. Learning the Encoder representation. We separate the algorithm into two main parts, the encoder Although we validate the TC-GQN algorithm using network and the value function. The encoding process encompasses control tasks, the architecture generalizes to any graph-based the GNN, state and action pooling, encoder, and dynamics MARL problem where a graph topology is available. components with the corresponding parameters θ, ϕ, χ, ψ, and The main components of the algorithm are: η. To train the encoder we employ a combined state and target GNN : {hkt }nk=1 = Gθ {okt }nk=1 , Γt prediction which is described in (1) and denoted by Laux . We begin the training process by generating GNN embedState Pooling : gt = Astate {hkt }nk=1 ϕ dings using the agent observations {okt }nk=1 and the global action k n Action Pooling : ut = Aχ {at }k=1 graph Γt as {hkt }nk=1 = Gθ {okt }nk=1 , Γt , and pool these as Encoder : zt = Eψ (gt ) gt = Astate {hkt }nk=1 . The attention-based pooling mechaϕ nism is implemented by using a learnable query vector which Dynamics : (zt+1 , pt+1 ) = Dη (zt , ut ) effectively compresses the agent dimension to a single global k k k Value : qt (a) = Qω (ot , ht , zt ) state representation vector [12]. The GNN component first transforms agent observations using To generate predicted future states ẑt+h for h = 1, ..., H, a simple MLP and then enriches this embedding with context where H is the prediction horizon length, we begin by encoding
the initial state ẑt = Eψ (gt ). To inform the dynamics model which will generate predictions we use learnable embeddings to turn agent actions into continuous vectors and then pool k n these as ut = Aaction {a } , where the attention pooling χ t k=1 for actions works analogously to the state pooling. The predictions are then generated autoregressively, such that (ẑt+h , p̂t+h ) = Dη (ẑt+h−1 , ut+h−1 ) for h ≥ 1. Here ẑt+h and p̂t+h are generated using separate MLPs with the same input. To generate latent state targets z̃t for these predictions we generate target GNN embeddings by {h̃kt }nk=1 = Gθ− {okt }nk=1 , Γt which are then pooled as g̃t = Astate {h̃kt }nk=1 . Using these ϕ− target global graph embeddings we then compute the target embeddings z̃t:t+H for the interval t : t + H by z̃t = Eψ− (g̃t ). The prediction target reference values p̃t are simply the actual target values given by the environment. The learning of the encoder and dynamics components is illustrated in Fig. 3. Taken together we formulate the auxiliary loss Laux as H X
" h γaux
2 ∥p̂t+h − p̃t+h ∥2 −
h=1
ẑt+h ∥ẑt+h ∥2
⊤
z̃t+h ∥z̃t+h ∥2
#
(1) where γaux ∈ [0, 1) is the discount rate. We use the cosine similarity loss for the state prediction task as it has been shown to achieve better performance with more stability compared to an MSE loss [5].
VI. E XPERIMENTS A. Experiment Setup We implement the MARL network control environment in a proprietary simulator environment following the system model described in Section IV. The simulator is used to generate synthetic episodes corresponding to a specific intersite distance sampled uniformly, a realization of the Weibull distribution modeling the traffic distribution in space, a realization of indoor/outdoor traffic distribution, and a specific temporal traffic pattern. The episode consists of 96 steps where each step spans one hour of network utilization. We compare different intelligent network control algorithms according to three different metrics. The first is the constraint violation probability which is the number of steps where the throughput constraint is not satisfied divided by the total number of steps in an episode. The second metric is the average daily sleep time per sector which is the number of steps where the sleep action was taken without violating the constraint divided by the number of days and sector. The final metric is the return which is the shared reward summed over all episode steps. For TC-GQN, we set the prediction targets pt to be the mean throughput per agent for L08, L18, and L35, and set hyperparameters as listed in Table I. TABLE I H YPERPARAMETERS
Parameter
C. Learning the Value Function
Training & Optimization Total Timesteps Learning Rate (Adam) Batch Size Discount Factor (γ) Polyak Coeff. (τ ) Gradient Clip Exploration (ϵ) Parallel Workers Min. Sampling Steps Replay Buffer
Value 25 000 10−4 256 0.95 0.005 40 Linear(1.0, 0.01, 17 500) 20 1 000 Unlimited
For learning the value function parameters ω we adopt a DQN architecture using double-Q learning with dueling, where the input consists of the raw agent observations, the enriched graph aware agent observation embeddings hkt , and the temporally consistent global graph representation zt . We generate the Q-value function predictions individually for each agent by qtk (akt ) = Qω sg(okt ), sg(hkt ), sg(zt ) , where sg(·) is the stop-gradient operator. We then pool these Q-value Architecture Specifics GNN Type GATv2Conv (4 heads) functions by taking the sum over all agents to generate the Pn Embedding Dim. 10 k k group level reward prediction q̂t (a) = q (a ) as in k=1 t Prediction Horizon 5 value decomposition networks [9]. The target Q-values q̃t (a) Hidden Layer Sizes are generated by first finding the online networks prediction GNN / Enc. / Dyn. / Q-Head [64, 64] for the best action as a∗ = argmaxa∈A q̂t+1 (a), where a is the Pre-GNN / Dueling Heads [64] vector of agent actions. We find the target Q-values through Activations ReLU k k k k q̃t (at ) = Qω− sg(ot ), sg(ht ), sg(zt ) where we use the ht and zt generated by the online encoding networks. We then Pn define target Q-value function by q̂t− (a) = k=1 q̃tk (ak ) and B. Performance of TC-GQN for RAN Energy-Saving − let q̃t = rt + γq · q̂t+1 (a∗ ) where rt is the actual group level To evaluate our proposed TC-GQN algorithm, we conduct reward from the environment and γq ∈ [0, 1) is the discount a comparative study on the energy-saving task against three 2 rate. The Q-loss is then given by LQ = q̂t (at ) − q̃t . An other methods: (1) a heuristic baseline controller that takes important distinction here is that the training of the two sets of OFF actions at night time and if the capacity cell utilization parameters, for the encoding and the Q-function respectively, is below 10%, and takes ON actions if the utilization of the is separated and each trained by its own optimizer. We also mid-band coverage cell is above 25%, (2) the GAQ algorithm employ the stop-gradient operator before computing the Q- which incorporates neighboring agent observations but which values so as to completely separate the two training processes. does not have a shared reward signal and uses the local reward
10000 20000 Training steps
15 10 5 10000 20000 Training steps GQN GAQ
15
1250 1000
Auxiliary loss
10
Global episode return
Daily sleep time (h)
Constraint violation (%)
20
750 500 250 TCGQN
10000 20000 Training steps Baseline
10 5 10000 20000 Training steps
Fig. 4. Performance of MARL agents during training in terms of sleep time (the higher the better) and probability of constraint violation (the lower the better).
Ri instead [8], and (3) the GQN algorithm that incorporates all other agent observations and employs a shared reward [3]. Each algorithm is trained on 25 000 steps with three random seeds and we report the mean and 95% confidence interval. The baseline is evaluated on 20 episodes. The results of the experiment are presented in Fig. 4 and we note that the auxiliary loss is well behaved and steadily decreases during the experiment, indicating that adding the temporal consistency objective does not destabilize training. We also note that TC-GQN achieves high episode returns faster and ends up outperforming all of the competing methods. Looking at the two network performance metrics constraint violation probability and sleep time, the heuristic provides a conservative approach, while the RL algorithms converge to more aggressive strategies. Notably both GQN and TC-GQN converge to a strategy which achieves roughly eight hours of daily sleep time, but where TC-GQN violates the constraint only about a third as often as GQN. As these two algorithms are the most similar in the benchmark it highlights the benefit of introducing temporal consistency. While GAQ and GQN converge to drastically different strategies, the two algorithms achieve the same episode reward in the end. This suggests that the reward signal might not clearly favor a certain type of behavior, potentially making it more difficult for an agent to learn an optimal policy. Despite this apparent ambiguity, however, TC-GQN is still able to learn a policy that performs well across all metrics.
trainable, and (3) Frozen Encoder, a pretrained agent where only the value function weights are trainable while the encoder is frozen. We set the number of training steps to 2500 which is 10 times less data than required for full training. The environment is changed to add two new throughput constraints on the L08 and L35 carriers. It is achieved by adding the following terms in Ri : −λ1[τi,L08 ≤15Mbps] 1[ai =off] and −λ1[τi,L35 ≤100Mbps] 1[ai =off] which penalizes the agent for causing the throughput to be below 15 Mbps for the low frequency carrier and 100 Mbps for the high frequency high bandwidth carrier. These throughput threshold values are used to model operator intents for quality of service. They are expected to change when new services are being rolled out. The results of the experiment are presented in Fig. 5 where the pretrained agents clearly converge much faster than the agent training from scratch, suggesting that the learned global state representation is transferrable across certain changes of intent. We also note that the mean value of the global state representation z remains similar between the two pretrained agents compared to the one learning from scratch, further indicating that the pretrained global state representation already contained elements that generalize across tasks. VII. C ONCLUSIONS
We propose Temporally Consistent Graph Q-Networks (TCGQN), a graph-based MARL algorithm that learns a taskagnostic global network representation using self-supervised C. Adapting to Different Intents representation learning. On a simulated RAN energy-saving A key advantage of training the temporal consistency encoder task, TC-GQN outperforms other graph-based RL methods and independently of the reward signal is that the learned repre- a rule-based method. It creates the most cell sleep opportunity sentation can be reused to quickly adapt to new adjacent tasks. with less violation of the QoS constraint. In adaptation Since the encoder is trained to predict network dynamics via experiments where we change the intent by making QoS the carrier throughputs at the next step rather than task-specific constraints more stringent, a pretrained TC-GQN outperforms rewards, the learned representation should remain robust to the baseline from the start and requires less training to reach a changing reward functions, provided that the prediction target strategy fitting the new goal. We also find that the learned global still has relevance for the new task. network representation remains relatively unchanged during To highlight this capability we conduct an adaptation fine-tuning. These results suggest that the learned encoder experiment to compare how fast an agent pretrained on captures network dynamics and can transfer to adjacent tasks the energy-saving task (Section VI-B) can adapt to satisfy with different intents, paving the way towards AI-native intentmore stringent QoS constraints. We compare three different driven network automation. In today’s networks, this algorithm training strategies: (1) From scratch, where a TC-GQN agent could be adopted in the non real-time RIC of an O-RAN is initialized with random weights, (2) Fine-Tuning, where an network and use existing O1 and A1 interfaces to observe agent is initialized with pretrained weights and all weights are cell performance and send actions [1]. A standard MLOps
10 0 0
1000 2000 Training steps Fine-tuned
10 0
1000 z mean
20
20
Episode return
Daily sleep time (h)
Constraint violation (%)
30
500 0
0
1000 2000 Training steps Frozen encoder From scratch
4 3 2 1 0
0
1000 2000 0 1000 2000 Training steps Training steps Baseline From scratch asymptotic performance
Fig. 5. Performance of TCGQN fine tuned and from scratch when transferred on a new task with additional constraints on all the carriers.
framework could be responsible for triggering fine-tuning upon changes in service requirements intents. Several limitations and areas of future work need to be addressed to fully adopt this technology in 6G networks. The encoder learns to predict future states solely from the current one, with no access to recent observation history. Incorporating short-term temporal context may allow for more accurate predictions of future states and thus improve the performance of the algorithm, especially for network control tasks requiring good KPI forecasting capabilities. Finally, we observe an apparent robustness of TC-GQN to reward misalignment with the network optimization intent, however, more systematic reward design from intent should be further investigated. R EFERENCES [1] [2] [3]
[4]
[5] [6] [7] [8] [9] [10]
[11]
N. A. K. Khani and S. Schmid, “AI-RAN in 6g networks: State-of-theart and challenges,” IEEE Open J. Commun. Soc., vol. 5, pp. 294–311, 2024. P. Stjernholm, A.-C. Eriksson, J. Niemöller, M. Sintorn, M. Z. Mujale, and E. Westerberg, “Intent-driven networks,” Ericsson, White Paper, Feb. 2025. M. Bouton, J. Jeong, J. Outes, A. Mendo, and A. Nikou, “Multi-agent reinforcement learning with graph q-networks for antenna tuning,” in NOMS IEEE/IFIP Network Operations and Management Symposium, 2023. T. Cai, Q. Wang, S. Zhang, Ö. T. Demir, and C. Cavdar, “Multi-agent reinforcement learning for energy saving in multi-cell massive mimo systems,” in 2024 IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN), 2024. Y. Zhao, W. Zhao, R. Boney, J. Kannala, and J. Pajarinen, “Simplified temporal consistency reinforcement learning,” in International Conference on Machine Learning, ICML, 2023. N. Hansen, H. Su, and X. Wang, “TD-MPC2: scalable, robust world models for continuous control,” in International Conference on Learning Representations, ICLR, 2024. J. Ye and Y. A. Zhang, “DRAG: deep reinforcement learning based base station activation in heterogeneous networks,” IEEE Trans. Mob. Comput., vol. 19, no. 9, pp. 2076–2087, 2020. Y. Jin, F. Vannella, M. Bouton, J. Jeong, and E. A. Hakim, “A graph attention learning approach to antenna tilt optimization,” in International Conference on 6G Networking, 6GNet, 2022. P. Sunehag et al., “Value-decomposition networks for cooperative multiagent learning based on team reward,” in International Conference on Autonomous Agents and MultiAgent Systems, AAMAS, 2018. T. Rashid, G. Farquhar, B. Peng, and S. Whiteson, “Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning,” in Advances in Neural Information Processing Systems, NeurIPS, 2020. M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. C. Courville, and P. Bachman, “Data-efficient reinforcement learning with selfpredictive representations,” in International Conference on Learning Representations, ICLR, 2021.
[12]
[13] [14] [15] [16]
J. Lee, Y. Lee, J. Kim, A. R. Kosiorek, S. Choi, and Y. W. Teh, “Set transformer: A framework for attention-based permutation-invariant neural networks,” in International Conference on Machine Learning, ICML, 2019. P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, ICLR, 2018. D. Lee, S. Zhou, X. Zhong, Z. Niu, X. Zhou, and H. Zhang, “Spatial modeling of the traffic density in cellular networks,” IEEE Wireless Communications, vol. 21, no. 1, pp. 80–88, 2014. H. Asplund, M. Johansson, M. Lundevall, and N. Jaldén, “A set of propagation models for site-specific predictions,” in 12th European Conference on Antennas and Propagation (EuCAP 2018), 2018. ETSI, “5G; Telecommunication management; Study on system and functional aspects of energy efficiency in 5G networks (3GPP TR 32.972 version 18.0.0 Release 18),” ETSI, Tech. Rep. TR 132 972 V18.0.0, 2024, ETSI Technical Report.