Temporally Encoded Double DQN for Proactive PRB Allocation in O-RAN Enabled Industrial Networks

Temporally Encoded Double DQN for Proactive PRB Allocation in O-RAN Enabled Industrial Networks Elahe Delavari, Xingqi Wu, and Junaid Farooq † Department of Electrical and Computer Engineering, University of Michigan-Dearborn,

arXiv:2605.30630v1 [cs.NI] 28 May 2026

Dearborn, MI, 48128 USA, Emails: {elahed, xingqiwu, mjfarooq}@umich.edu. Abstract—Fifth-generation (5G) wireless systems are increasingly adopted in smart manufacturing to support heterogeneous industrial workloads through services such as enhanced Mobile Broadband (eMBB) and Ultra-Reliable Low-Latency Communication (URLLC). However, industrial traffic is inherently processdriven and temporally correlated. So, static or reactive schedulers in the Open Radio Access Network (O-RAN) are inadequate for such non-stationary conditions, leading to sub-optimal utilization and violation of latency–reliability guarantees. This paper proposes a temporal-aware deep reinforcement learning (DRL) xApp for proactive Physical Resource Block (PRB) allocation in O-RANenabled industrial networks. The proposed framework integrates a long short-term memory (LSTM) encoder within a Double Deep QNetwork (DQN) to model sequential dependencies among slice-level Key Performance Indicators (KPIs), enabling predictive and stable decision-making. A continuous-time Markov chain (CTMC) traffic model is incorporated to emulate machine concurrency and process burstiness. Experimental results show that the LSTM–Double DQN improves slice satisfaction, and buffer stability under moderate and heavy load, with the longest sequence window providing the strongest gains. Index Terms—5G, O-RAN, PRB allocation, model predictive control, real-time optimization.

I. I NTRODUCTION Smart manufacturing environments are increasingly composed of interconnected cyber-physical systems wherein machines, sensors, and robots coordinate over wireless links [1]. In such industrial Internet of Things (IIoT) settings, network traffic is highly heterogeneous, i.e., high-throughput services such as vision inspection and digital twin synchronization coexist with latency-sensitive control loops that demand ultra-reliable low-latency communication (URLLC) [2]. These requirements evolve dynamically with production stages as shown in Fig. 1, rendering static or wired configurations inadequate for adapting to time-varying workloads [3]. This motivates the use of learning-based schedulers, particularly reinforcement learning (RL), which can optimize decision-making over time horizons rather than rely on static allocation policies. Fifth-generation (5G) networks support heterogeneous services through network slicing (NS), which enables multiple logical networks to coexist on shared infrastructure. Typical slice types include enhanced mobile broadband (eMBB), URLLC, and massive machine-type communication (mMTC) [4]. Although this architecture facilitates service isolation and qualityof-service (QoS) differentiation, industrial workloads are often structured by machine-cycle–driven temporal dependencies. Fixed physical resource block (PRB) quotas and reactive slice

Fig. 1: O-RAN–enabled industrial network architecture with closed-loop PRB allocation by the Near-RT RIC. scheduling are therefore insufficient, frequently resulting in degraded latency performance and inefficient spectrum use. The Open Radio Access Network (O-RAN) architecture introduces openness and intelligence into the radio access infrastructure, enabling programmable control through the near-realtime RAN Intelligent Controller (Near-RT RIC) [5]–[7]. This architecture supports the deployment of artificial intelligence (AI)–driven applications (xApps) that can implement closedloop control strategies tailored to dynamic and time-correlated traffic conditions [8]. In the context of smart factories, where workload fluctuations follow predictable production cycles, ORAN provides a suitable platform for deploying temporallyaware control policies. Conventional resource management strategies, such as proportional-fair scheduling and convex optimization, rely on instantaneous observations and are thus limited in settings where traffic patterns exhibit temporal correlation. Deep reinforcement learning (DRL) has emerged as a promising alternative for autonomous resource orchestration, with approaches such as Q-learning [9], Deep Q Networks (DQN) [10], and actor–critic methods [11] outperforming traditional heuristics. However, most existing DRL-based xApps operate in a memoryless fashion, failing to exploit the short-term temporal regularities inherent in industrial traffic. To mitigate this limitation, recent studies have incorporated temporal modeling via long shortterm memory (LSTM) encoders [12] or predictive LSTM augmentations [13]–[15]. These methods improve performance in dynamic or partially observable scenarios but have not been extensively applied to proactive slice control in deterministic, process-driven industrial networks. In this paper, we propose a temporally encoded DRL

xApp for proactive PRB allocation in O-RAN–enabled smart manufacturing networks. The proposed method integrates an LSTM encoder within a Double DQN architecture to capture short-term temporal dependencies in slice-level KPIs such as PRB share, throughput, and buffer occupancy. The resulting temporal embedding enables predictive PRB allocation decisions that improve latency compliance and buffer stability under dynamic workloads. We evaluate the proposed LSTM–Double DQN framework using an extended AI-RAN simulator incorporating a continuous-time Markov chain (CTMC) traffic model that emulates process-level concurrency. Experimental results demonstrate that temporal encoding improves slice satisfaction and significantly reduces queuing delay under moderate and heavy loads. The findings highlight the importance of temporal modeling in DRL-based O-RAN control applications for industrial settings. II. S YSTEM M ODEL We consider an O-RAN–enabled industrial network that serves heterogeneous wireless workloads generated by concurrently active machines on a factory floor. As shown in Fig. 1, the Near-RT RIC hosts the proposed temporal-aware deep DRL xApp that monitors slice-level traffic metrics and dynamically allocates PRBs to minimize end-to-end latency across services. A. Network Architecture and Slicing The system consists of a single O-RU providing connectivity to M industrial user equipments (UEs), denoted M = {1, 2, . . . , M }. Each UE belongs to one of two service slices: eMBB or URLLC denoted as ME and MU . The eMBB slice supports high-throughput flows such as video streaming or digital-twin synchronization, while the URLLC slice carries short control and actuation packets requiring minimal delay. The total available bandwidth Btot is divided into K orthogonal PRBs, each of bandwidth B. At each time step t ∈ T = {1, 2, . . . , T }, the Near-RT RIC determines a PRB allocation matrix E(t) = [em,k (t)], where em,k (t) ∈ {0, P 1} indicates whether PRB k is assigned to UE m, where m∈M em,k (t) ≤ 1, ∀k ∈ {1, 2, ..., K}. We then define PE (t) and PU (t) as the sum of the PRB allocation at time slot t over the set P of eMBB PK and URLLC users, respectively (i.e., Ps (t) = m∈Ms k=1 em,k (t), where s ∈ {E, U }), and PE (t) + PU (t) ≤ K. The O-RU enforces this allocation within the subsequent transmission interval. B. Traffic Generation and Queueing Dynamics At time t, a subset A(t) ⊆ M of machines is active, generating uplink or downlink traffic. Each active UE m ∈ A(t) produces packets of instantaneous size ϕm (t), which depends on its operational state and(service type: FE , m ∈ ME , ϕm (t) ∼ (1) FU , m ∈ MU , where FE and FU denote the packet-length distributions of eMBB and URLLC traffic, respectively. Arriving packets are buffered at the O-RU in slice-specific queues Qs (t) for s ∈

{E, U }. The queues at the base station evolves Xas follows: Qs (t+1) = max[0, Qs (t) − Ts Rs (t)] + ϕm (t), (2) m∈Ms

where Ts is the scheduling interval and Rs (t) is the aggregate transmission rate achieved by slice s given the PRBs assigned at time t. C. Data Rate Computation and PRB Demand The achievable data rate for each UE is determined by its selected modulation and coding scheme (MCS) level, which depends on its Channel Quality Indicator (CQI). The contribution of PRB k to UE m’s rate is given by: O × C × NRE Rm,k (t) = em,k (t), (3) Dslot where O is the modulation order and C is the effective coding rate associated with the selected MCS entry, NRE is the number of resource elements per PRB per slot, and Dslot is thePslot duration. The total rate for UE m satisfies K Rm (t) =P k=1 Rm,k (t). The aggregate rate for slice s is Rs (t) = m∈Ms Rm (t). The slice throughput ms (t) denotes the total downlink data P rate achieved by slice s at time t, computed as ms (t) = m∈Ms Rm (t). The downlink PRB demand for each UE is derived from its MCS, Guaranteed Bit Rate (GBR) contract, and instantaneous buffer backlog. A UE with downlink GBR requirement Gmm l Gm (in bps) is assigned a minimum PRB floor equal to Rm,k . In parallel, the O-RU monitors the UE’s downlink buffer, m whose backlog Qm (bytes) induces a rate demand Im = 8Q Ts , This into a backlog-driven PRB requirement equal m l translates Im to Rm,k . To avoid allocating PRBs due to very small or transient bursts, we enforce the GBR floor only when the UE buffer holds at least one scheduler-step’s worth of GBR traffic. In practice, this means the GBR constraint when l applying m Im Qm ≥ Gm8Ts and Under this condition, the > 0. Rm,k demand is raised to the minimum number of PRBs needed to satisfy the GBR; otherwise, no GBR floor is applied. The final PRB requirement l m l reported m to the xApp is therefore m m Dm = max RIm,k , RGm,k which ensures that sustained GBR obligations are met while remaining responsive to realtime queue dynamics. D. Latency Evaluation and Performance Characterization For each slice s ∈ {E, U }, the instantaneous latency at time t is derived from the relationship between the slice backlog and the serving rate of the gNB. The resulting slice latency is Qs (t) Ls (t) = . (4) Rs (t) To characterize how effectively a slice uses its PRB allocation to maintain acceptable delay, we introduce a slice-level PRBefficiency measure. In our model, efficiency depends not only on the PRB budget but also on how well the slice meets its latency target. A general expression is !
ψs Ls (t) ηs (t) = f , (5) Ps (t) where Ps (t) is the PRB quota assigned to slice s, and ψs :

R → [0, 1] is a latency-alignment function that assigns higher values when the observed slice latency Ls (t) approaches the slice-specific latency target L⋆s (much stricter for URLLC than for eMBB). The monotone function f (·) models diminishing returns with respect to PRB consumption. Consequently, ηs (t) reflects how efficiently slice s maintains its latency objective given the PRB resources allocated to it. E. Optimization Objective The Near-RT RIC adjusts {PE (t), PU (t)} at every scheduling interval according to the xApp’s control policy. And both slices are evaluated through a latency-alignment measure that is normalized by their PRB usage, the PRB-control problem aims to allocate sufficient resources to keep slice latency near its target while avoiding unnecessary spectrum consumption. This trade-off is captured through the efficiency indicators ηs (t) introduced above, which increase when latency is well aligned and decrease when excessive PRBs are used. The high-level PRB allocation problem is therefore described as follows: max G(ηE (t), ηU (t)) , (6) {Ps (t)}

where G(·) is a non-decreasing aggregation function representing overall system performance. Direct optimization of G is generally intractable due to the stochastic and queuecoupled dynamics. Therefore, a surrogate reward formulation is introduced in Section III to enable tractable reinforcement learning while preserving the underlying trade-offs. III. M ETHODOLOGY To enable predictive and latency-aware resource allocation in O-RAN–enabled industrial networks, we propose a RL–based control framework that dynamically adjusts PRB allocations across network slices. This section describes the RL formulation, state and action design, reward function, temporal encoding architecture, and training procedure of the proposed LSTM–Double DQN xApp. A. Reinforcement Learning Formulation The PRB allocation task is modeled as a sequential decisionmaking problem, where each action influences future slice latencies and buffer states. Due to short-term temporal dependencies in industrial traffic, the learning agent must exploit past state information to perform stable control under dynamic workloads. To this end, we adopt a Double Deep Q-Network (Double DQN) framework augmented with a Long Short-Term Memory (LSTM) encoder. This temporally aware design enables the xApp to capture evolving traffic patterns without requiring explicit prediction. We additionally implement a memoryless baseline using a multilayer perceptron (MLP)–Double DQN, where the instantaneous six-dimensional network state is processed through two fully connected layers. This baseline is used to quantify the performance gain achieved by temporal modeling in the LSTM–Double DQN agent. The problem is formalized as a Markov Decision Process (MDP) defined by the tuple (S, A, P, R, γ), where S denotes the slice-level network state, A is the action space, P represents the environment dynamics,

R is the reward function, and γ is the discount factor. The xApp observes the current state, selects an action to adjust PRB allocations, and receives a reward based on slice-level QoS metrics. Based on this MDP formulation, the components of the state space, action space, and reward function are defined as follows. 1) State Space: At time t, the agent observes st = [P̃E , m̃E , Q̃E , P̃U , m̃U , Q̃U ], where P̃s is the normalized PRB share, m̃s the normalized throughput, and Q̃s the normalized buffer backlog for slice s ∈ {E, U }. All features lie in [0, 1] for numerical stability. 2) Action Space: Each action adjusts slice PRBs in small directional increments at = (∆pE , ∆pU ), ∆ps ∈ {−1, 0, +1}. The effective change is ∆Ps = λ∆ps , where λ is the step size. The resulting actions increase, decrease, or maintain PRBs for each slice; any over-allocation is automatically rescaled to satisfy the PRB budget. 3) Reward Function: The reward promotes low-latency operation and efficient PRB usage. For each slice s ∈ {E, U } slice latency Ls relative to its target L⋆s is mapped through a logistic scoring function, 1 ψs = , (7) 1 + exp[−as (Ls − L⋆s ) − bs ] where as and bs control the slope and offset of the latency penalty. Efficiency is incorporated as:   αs .ψs (Ls ) Rs = f , (8) σs + P s where Ps is the current PRB quota. Here, σs > 0 is a small constant that prevents the denominator from vanishing and αs balances the latency score with the PRB term. The global reward is a weighted sum, R = wE RE + wU RU This formulation emphasizes URLLC reliability while encouraging balanced and proactive PRB allocation. Although G(ηE , ηU ) is not optimized directly, the reward is constructed as a tractable surrogate of the same objective. Each slice-level term Rs reflects the efficiency structure of ηs through latency alignment and PRB normalization, and the combined reward R serves as an RL-friendly approximation of G(·). B. Double DQN Training Procedure The PRB allocator is trained using the Double DQN, which reduces the overestimation bias of standard DQN by separating action selection and action evaluation. Two networks are maintained: an online network Q(s, a; θ) and a target network Q(s, a; θ− ) periodically synced with the online parameters. At each update, a transition (s, a, r, s′ , d) is sampled from the replay buffer, where d ∈ {0, 1} is the terminal flag indicating whether s′ corresponds to the end of an episode. The next action is chosen using the online network and evaluated using the target network. The Double DQN target becomes y = R + (1 − d)γ Q(s′ , arg max Q(s′ , a; θ); θ− ). (9) a

The online network parameters θ are updated by minimizing a Huber loss on the temporal-difference (TD) error δ = y − Q(s, a; θ), using minibatch updates optimized with Adam optimizer. Exploration follows a linearly decaying ϵ-

Algorithm 1 Episodic LSTM–Double DQN for PRB Allocation

Fig. 2: Proposed LSTM–Double DQN xApp. greedy policy, and transitions are sampled from a replay buffer for stable training. The MLP–Double DQN processes the instantaneous sixdimensional state through two fully connected layers of width 256 with ReLU activations. A final linear layer outputs Q-values for the nine discrete PRB-adjustment actions. This feedforward design corresponds to the memoryless agent used as a baseline throughout our evaluation. C. LSTM-Double DQN for Temporally-Aware PRB Allocation To exploit short-term temporal patterns in traffic, we extend the feedforward agent with an LSTM encoder. Instead of using only the instantaneous six-dimensional state, the agent forms a sequence window St = (st−ℓ+1 , . . . , st ), where ℓ is the sequence length hyperparameter that specifies how many past slice-level states are included in the window. This sequence is passed to a single-layer LSTM: ht = LSTM(St ). The vector ht serves as a compact summary of the recent traffic evolution and is forwarded to a fully connected head that outputs Qvalues for the nine PRB adjustment actions. Only the final hidden state ht is used because it represents the LSTM’s learned encoding of the entire sequence and acts as a fixed-length temporal embedding suitable for Q-learning. To clarify the overall processing pipeline of the temporal encoder and the Double DQN, Fig. 2 illustrates how a window of the most recent l slice-level states is passed through an LSTM to produce the temporal feature ht , which is then fed into the fully connected Qnetwork. Both online and target networks share the same LSTMbased encoder; the Double DQN update, target construction, and Huber-loss minimization remain identical to the feedforward case. The complete training loop is shown in Algorithm 1. The LSTM–Double DQN replaces the feedforward encoder with a single-layer LSTM of hidden size 128 that ingests the sequence window St . All the other layers are the same as the MLP version. IV. P ERFORMANCE E VALUATION We build on the Python-based AI-RAN Simulator [16], which emulates a 5G-compliant O-RAN system with gNBs, UEs, and a RIC. Our extended version1 simulates a 2 km×2 km region with a single gNB deployed at the center, serving one n78 cell using standard NR numerology. UEs are placed uniformly, and 1 https://github.com/ElaheDlv/Proactive-PRB-Allocation

Require: sequence length ℓ, replay buffer B, networks Q(·; θ) and Q(·; θ− ), discount γ 1: Initialize θ, θ − 2: for each episode e do 3: Reset env, observe s0 , set window W ← {s0 } 4: for each step t do 5: Form sequence St from last ℓ states in W 6: Select at via ϵ-greedy on Q(St , a; θ) 7: Execute at , observe st+1 , Rt , dt 8: Append st+1 to W (keep last ℓ) 9: Form St+1 from updated W 10: Store (St , at , Rt , St+1 , dt ) in B 11: if ready to train then 12: Sample minibatch (S, a, R, S ′ , d) from B 13: Compute a′ = arg maxa Q(S ′ , a; θ) 14: Compute target y = R + (1 − d)γ Q(S ′ , a′ ; θ− ) 15: Update θ by minimizing Huber(Q(S, a; θ), y) 16: if target update then θ− ← θ 17: end if 18: end if 19: if dt = 1 then 20: break 21: end if 22: end for 23: end for initial slice-level PRB allocations reflect training and testing configurations. Transport block sizes follow 3GPP TS 38.214, Table 5.1.3.1–2, using modulation orders O ∈ {2, 4, 6, 8} and corresponding coding rates C. Simulator parameters are summarized in Table I. A. Simulation Setup Traffic is generated using an event-driven CTMC that models the manufacturing processes executed by two industrial machines connected to the O-RAN base station. Machine 1 performs a machining operation (M1), Machine 2 performs an assembly operation (M2), and both may operate concurrently depending on the production workflow. The CTMC state space is S = {S1 , S12 , S2 }, where S1 corresponds to machining (M1), S2 corresponds to assembly (M2), and S12 represents the concurrent machining–assembly phase (M1+M2). State transitions follow the  rate matrix  −0.75 0.40 0.35 −0.20 0.20  , Q= 0 0.20 0 −0.20 P where the exit rate from state i is λi = j̸=i qij and dwell times are sampled from Exp(1/λi ). Normalizing the rows of Q yields the transition probability matrix P, ensuring statistically correct and memoryless state evolution. As the CTMC evolves, each active interval is mapped to a traffic slice: machining generates URLLC traffic, while assembly generates eMBB traffic. URLLC arrivals follow an Erlang distribution Erlang(k=3, scale=0.5 ms) with a packet size distribution FU as N (60, 5). The eMBB traffic is generated

TABLE I: Simulator Parameters and Reward Hyperparameters Simulator Parameters Param.

Reward Parameters

Value

B fc Cell radius Tx power max , P max PDL UL NRE Dslot GE , GU

Param.

100 MHz 3.5 GHz 800 m 40 dBm 218, 55 168 ≈ 1 ms 10, 1 Mbps

eMBB , URLLC

L⋆s as bs αs σs K ws

0.1, 0.001 s −35, −9000 4.5, 4.5 100, 50 5, 2 5, 5 0.5, 0.5

Training Curves 3500

Reward

3000 2500 2000 1500 1000 500 0

100

200

300

400 500 Training Step

600

MLP LSTM seq4 LSTM seq8 LSTM seq16 700 800

Fig. 3: Training episodic return comparison for the MLP-Double DQN and LSTM-Double DQN agents. using Lognormal inter-arrival times Lognormal(µ = ln 7, σ = 0.55) and packet size distribution FE is Lognormal(6.0, 0.5), producing heavy-tailed bursts. Packets are time-stamped according to the sampled inter-arrival intervals and queued at the base station buffer. To generate diverse traffic traces, we vary two scaling parameters: α amplifies forward transitions (S1 → S12 , S2 ) and β scales return transitions (S2 → S1 ), applied to the corresponding entries of Q. Varying these parameters produces multiple trace sets with different concurrency levels. The resulting packet-level traces are aligned to a global timeline and imported into the O-RAN simulator for evaluating PRBallocation policies. For reproducibility, the specific numerical values used in our experiments are summarized in Table I. For the latencyalignment component, the function ψs (Ls ) is instantiated using the logistic form in equation 5, which assigns higher scores when the slice latency approaches its target L⋆s . The shaping function appearing in the PRB-efficiency term is realized as f (x) = K ln(1 + x), where K serves as a global scaling factor controlling the magnitude of the reward. B. Experimental Setup Training was conducted over 800 episodes using randomized CTMC traces and variable load levels. Each episode spans 20 seconds of simulation time. 1) Training Configurations: The agent is exposed to diverse conditions by varying the number of active UEs {1, 3, 5, 7, 9}, all combinations of initial PRB quotas for eMBB and URLLC slices {5, 7, 9, 11}2 , and CTMC scaling factors α, β ∈ {0.25, 0.5, 1, 1.5, 2}. For training, we use a discount factor of γ = 0.99, a learning rate of 5 × 10−4 , and a PRB adjustment step size of λ = 5. 2) Generalization Tests: Generalization is evaluated on UE densities {1, 2, 4, 6, 8} using a fixed initial allocation of

eMBB_PRBs = URLLC_PRBs = 5. These intermediate load levels test robustness without departing from the training regime. C. Evaluation Metrics To assess the performance of the proposed scheduler under mixed eMBB and URLLC traffic, several quantitative metrics are monitored during simulation. These metrics capture both instantaneous and long term characteristics of satisfaction, buffer, and resource utilization at the base station. The following indicators are computed at each decision interval and aggregated over time to evaluate steady state performance: 1) Slice Satisfaction Ratio: The satisfaction ratio Ps /Ds quantifies how effectively each slice’s PRB demand is met, where Ps and Ds denote the granted and demanded PRBs, respectively. High satisfaction values indicate that the scheduler is able to allocate sufficient resources to maintain service quality under varying loads. 2) PRB Allocation Efficiency: It reflects how well each slice converts its allocated PRBs into low-latency service. We compute efficiency using the same latency-based scoring function and PRB-normalized shaping terms defined in the reward formulation. This metric therefore captures both latency performance and the diminishing returns of allocating excessive PRBs to a slice. 3) Average Buffer Size: The average buffer size measures the aggregate downlink queue occupancy for the corresponding slice. Large buffer values indicate congestion and insufficient PRB allocation, whereas small values reflect smooth packet delivery with minimal queuing. D. Simulation Results The performance of the proposed temporally encoded LSTM Double DQN xApp is evaluated using the test traces described earlier, with particular emphasis on the model’s behavior under previously unseen load levels and traffic temporal patterns. The results provide a detailed view of how temporal awareness improves resource allocation in the presence of bursty and non stationary industrial traffic. Fig. 3 compares the episodic return during training for the MLP–Double DQN and the LSTM variants with sequence lengths ℓ ∈ {4, 8, 16}. All models converge stably, but the LSTM agents achieve higher returns, especially with longer sequences. The generalization results under unseen UE densities are presented in Figs. 4–6. Satisfaction trends (Fig. 4) show that all agents perform well under light load, but the MLP baseline deteriorates under congestion. The LSTM agent with ℓ=16 achieves the highest satisfaction at moderate and heavy load, highlighting the benefits of long-term temporal context. PRB efficiency (Fig. 5) remains comparable across agents at low to moderate load. However, under increased traffic intensity, the LSTM agents show improved efficiency consistency, suggesting that temporal awareness helps preserve latency without over-allocating PRBs. Buffer size (Fig. 6) reveals that MLP agents exhibit queue buildup under heavy load, whereas the LSTM variant with ℓ=16 maintains significantly smaller buffers, indicating more proactive and anticipatory scheduling.

Satisfaction (Avg: eMBB & URLLC) vs Number of UEs

MLP LSTM (seq=4) LSTM (seq=8) LSTM (seq=16)

Satisfaction

0.65 0.60 0.55

Buffer (Avg: eMBB & URLLC) vs Number of UEs

104 Buffer (log scale)

0.70

103 102

0.50 1

2

4 6 Number of UEs

8

Fig. 4: Satisfaction vs. number of UEs. Efficiency (Avg: eMBB & URLLC) vs Number of UEs

MLP LSTM (seq=4) LSTM (seq=8) LSTM (seq=16)

11

Efficiency

10 9 8 7 1

2

4 6 Number of UEs

8

Fig. 5: Efficiency vs. number of UEs. The results confirm that temporally encoded DRL agents provide improved stability and robustness compared to memoryless counterparts. In particular, the LSTM–Double DQN with longer sequence lengths (ℓ = 16) consistently outperforms both the MLP and shorter LSTM variants across all metrics. Short sequences (e.g., ℓ = 4) may fail to capture the CTMC structure, leading to noisy embeddings and suboptimal decisions. Overall, temporal encoding improves slice satisfaction, reduces congestion, and ensures balanced resource utilization in nonstationary industrial networks. V. C ONCLUSION This paper presented a temporally aware RL xApp for proactive PRB allocation in O-RAN–enabled industrial networks. By integrating an LSTM encoder into a Double DQN framework, the proposed scheduler captures short-term temporal patterns in slice-level KPIs and anticipates CTMC-driven traffic variations. Experimental results show that the LSTM–Double DQN can improve slice satisfaction and reduces buffer buildup compared to the feedforward MLP baseline, particularly under moderate and heavy load. Longer sequence windows provide the strongest gains, while short windows capture insufficient temporal structure. These findings highlight the importance of temporal modeling for stable, resource-efficient PRB allocation in non-stationary industrial environments. Future work includes extending the design to multiple slices, exploring adaptive sequence-length selection, and deploying the approach on nearreal-time RIC platforms for large-scale validation. R EFERENCES [1] M. Noor-A-Rahim, R. Zhang, M. Bennis, and H. V. Poor, “Wireless communications for smart manufacturing and industrial IoT: Existing technologies, 5G, and beyond,” IEEE Communications Surveys & Tutorials, vol. 24, no. 3, pp. 1621–1661, 2022.

101

1

2

4 Number of UEs

MLP LSTM (seq=4) LSTM (seq=8) LSTM (seq=16) 6 8

Fig. 6: Buffer vs. number of UEs. [2] Ericsson Research, “Boosting smart manufacturing with 5G wireless connectivity,” Ericsson Technology Review, 2019. [3] S. Ludwig et al., “Reference network and localization architecture for smart manufacturing based on 5G,” in Advances in System-Integrated Intelligence, (Cham), pp. 470–479, Springer International Publishing, 2023. [4] S. Wijethilaka and M. Liyanage, “Survey on Network Slicing for Internet of Things Realization in 5G Networks,” IEEE Communications Surveys & Tutorials, vol. 23, no. 2, pp. 957–994, 2021. [5] N. H. Stephenson, A. J. Chiejina, N. B. Kabigting, and V. K. Shah, “Demonstration of closed loop AI-Driven RAN controllers using O-RAN SDR testbed,” in MILCOM 2023 - 2023 IEEE Military Communications Conference (MILCOM), pp. 241–242, 2023. [6] C.-F. Hung, C.-H. Tseng, and S.-M. Cheng, “Anomaly detection for mitigating xApp and E2 interface threats in O-RAN near-RT RIC,” IEEE Open Journal of the Communications Society, vol. 6, pp. 1682–1694, 2025. [7] G. M. Almeida, G. Z. Bruno, A. Huff, M. Hiltunen, E. P. Duarte, C. B. Both, and K. V. Cardoso, “RIC-O: Efficient placement of a disaggregated and distributed RAN intelligent controller with dynamic clustering of radio nodes,” IEEE Journal on Selected Areas in Communications, vol. 42, no. 2, pp. 446–459, 2024. [8] M. Polese, L. Bonati, S. D’Oro, S. Basagni, and T. Melodia, “Understanding O-RAN: Architecture, Interfaces, Algorithms, Security, and Research Challenges,” IEEE Communications Surveys & Tutorials, vol. 25, no. 2, pp. 1376–1411, 2023. [9] Y. Shi, Y. E. Sagduyu, and T. Erpek, “Reinforcement learning for dynamic resource optimization in 5G radio access network slicing,” in 2020 IEEE 25th international workshop on computer aided modeling and design of communication links and networks (CAMAD), pp. 1–6, IEEE, 2020. [10] K. Suh, S. Kim, Y. Ahn, S. Kim, H. Ju, and B. Shim, “Deep reinforcement learning-based network slicing for beyond 5G,” IEEE Access, vol. 10, pp. 7384–7395, 2022. [11] M. Alsenwi, N. H. Tran, M. Bennis, S. R. Pandey, A. K. Bairagi, and C. S. Hong, “Intelligent resource slicing for eMBB and URLLC coexistence in 5G and beyond: A deep reinforcement learning based approach,” IEEE Transactions on Wireless Communications, vol. 20, no. 7, pp. 4585–4600, 2021. [12] R. Li, C. Wang, Z. Zhao, R. Guo, and H. Zhang, “The LSTM-Based Advantage Actor-Critic Learning for Resource Management in Network Slicing With User Mobility,” IEEE Communications Letters, vol. 24, pp. 2005–2009, Sept. 2020. [13] K. Li, W. Ni, and F. Dressler, “LSTM-Characterized Deep Reinforcement Learning for Continuous Flight Control and Resource Allocation in UAV-Assisted Sensor Network,” IEEE Internet of Things Journal, vol. 9, pp. 4179–4189, Mar. 2022. [14] F. Lotfi and F. Afghah, “Open RAN LSTM traffic prediction and slice management using deep reinforcement learning,” in 2023 57th Asilomar Conference on Signals, Systems, and Computers, pp. 646–650, IEEE, 2023. [15] Y. Cai, P. Cheng, Z. Chen, M. Ding, B. Vucetic, and Y. Li, “Deep Reinforcement Learning for Online Resource Allocation in Network Slicing,” IEEE Transactions on Mobile Computing, vol. 23, pp. 7099– 7116, June 2024. [16] “AI-RAN simulator.” https://github.com/ntutangyun/ai-ran-sim. Accessed: 2025-11-24.