1
A Persistence-Aware Framework for Age Violation Control in Wireless Status Update Systems
arXiv:2605.13002v1 [cs.NI] 13 May 2026
Haoyuan Pan, Member, IEEE, Chen Chen, Shiyong Zhou, Kun Chen, and Tse-Tin Chan, Member, IEEE
Abstract—Timely and reliable status updates are essential for emerging QoS-sensitive wireless applications. Common age of information (AoI)-based metrics, such as average AoI and age violation rate (AVR), characterize time-averaged freshness or violation frequency but do not explicitly capture the temporal persistence of consecutive age violations, which can be critical in safety-sensitive wireless applications. To address this limitation, we develop a persistence-aware reliability framework based on the consecutive age violation rate (C-AVR) vector, whose components quantify AoI threshold violations over consecutive time windows of different lengths. Through flexible weighting schemes, the proposed framework unifies reliability objectives ranging from average persistence to tail-sensitive performance. Optimizing weighted C-AVR objectives is challenging because consecutive violations are temporally correlated, leading to sparse learning signals. To address this issue, we develop a distributional reinforcement learning approach based on a quantile regression dueling double deep Q-network (QR-D3QN). By modeling a quantile-based return distribution rather than only a scalar expected return, QR-D3QN provides richer valueestimation signals for rare but prolonged violation sequences under stochastic packet arrivals, unreliable channels, and transmission cost constraints. Simulation results demonstrate that QR-D3QN consistently outperforms expectation-based baselines across a wide range of weighting schemes and system settings, with particularly significant gains under tail-sensitive persistence objectives. Component-wise analysis further shows that distributional value learning substantially improves reliability across multiple persistence scales, especially for long consecutive violation sequences. Overall, our results establish the proposed C-AVR framework as an effective foundation for persistenceaware reliability evaluation and demonstrate the advantages of distributional reinforcement learning for wireless status update systems. Index Terms—Age of information, distributional reinforcement learning, reliability, wireless scheduling.
I. I NTRODUCTION Ensuring timely and reliable status updates is a fundamental requirement in wireless communication systems supporting industrial automation, autonomous driving, intelligent transportation, and real-time monitoring. In such applications, outdated information may degrade control accuracy, compromise safety, or destabilize closed-loop operation. To characterize information freshness, the age of information (AoI) has emerged as a widely adopted metric, measuring the time elapsed since H. Pan, C. Chen, S. Zhou, and K. Chen are with the College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China (e-mails: [email protected]; [email protected]; [email protected]; [email protected]). T.-T. Chan is with the Department of Mathematics and Information Technology, The Education University of Hong Kong, Hong Kong, China (e-mail: [email protected]).
the generation of the most recently received update [1]– [3]. Extensive studies have investigated AoI-aware scheduling, sampling, queue management, and resource allocation in wireless networks [4]–[7]. Most existing works focus on average AoI or related mean-based freshness metrics due to their analytical tractability and intuitive interpretation of longterm system performance [8]–[11]. However, average AoI is often insufficient for quality-ofservice (QoS)-sensitive applications. In many practical systems, occasional but severe freshness degradation may dominate operational risk even when the average AoI remains acceptable. To address this limitation, reliability-oriented freshness metrics have been introduced. In particular, the age violation rate (AVR) measures the probability that AoI exceeds a predefined threshold and therefore provides a thresholdbased characterization of timeliness reliability [12]–[14]. Compared with average AoI, AVR is more directly connected to outage-like QoS guarantees and has been studied in wireless systems with stochastic arrivals, unreliable channels, and constrained scheduling resources [15], [16]. Nevertheless, existing violation-based metrics primarily quantify the frequency of violations, while largely ignoring their temporal persistence. This limitation is particularly important in safety-critical systems. Consider two systems with the same AVR: in one system, violations occur sporadically and are rapidly corrected; in the other, violations occur in long uninterrupted runs. Although both systems exhibit identical violation probabilities, their operational risks can differ substantially. In applications such as autonomous control and industrial monitoring, prolonged information staleness may lead to sustained control errors, persistent sensing failures, or safety hazards [17]. Therefore, reliability-aware freshness control should characterize not only whether AoI violations occur, but also how long they persist consecutively. Motivated by this observation, we develop a persistenceaware reliability framework based on a vectorized characterization of consecutive AoI violations. Specifically, we introduce the consecutive age violation rate (C-AVR) vector, whose k-th component measures the probability that AoI exceeds a threshold over k consecutive time slots. This vector representation provides a persistence profile of age violations across multiple time scales. Scalar reliability metrics arise as special cases under appropriate weightings on this vector; in particular, the scalar C-AVR for a fixed k corresponds to a one-hot weighting. More generally, flexible weighting schemes over the C-AVR vector enable a unified treatment of reliability objectives, ranging from average persistence (uniform weighting) to tailsensitive criteria that emphasize long violation runs (e.g.,
2
exponential weights). We refer to this scalar objective as the weighted C-AVR. To the best of our knowledge, this is the first AoI-based framework that models and optimizes consecutive violation persistence across multiple time scales. We study this framework in a multiuser wireless status update system, where a central scheduler selects users under transmission cost constraints, stochastic packet arrivals, and unreliable channels. Optimizing persistence-aware C-AVR objectives in this setting is challenging. Consecutive violation events are temporally coupled: once AoI exceeds the threshold, the violation tends to persist unless a successful update occurs. As a result, unlike average AoI or standard AVR, system performance depends not only on frequent and short violations, but also on rare and prolonged violation sequences. Moreover, system dynamics such as packet arrivals and channel reliability are often unknown or difficult to model accurately, limiting the applicability of classical model-based optimization methods [18]–[20]. These challenges motivate the use of deep reinforcement learning (DRL), which has recently emerged as an effective framework for wireless scheduling and resource allocation under uncertainty [21], [22]. Existing DRL-based wireless scheduling methods commonly employ expectation-based value-learning architectures, such as deep Q-networks (DQN) and their variants [23], [24]. Such methods learn scheduling policies by optimizing the expected long-term return and have demonstrated strong performance in many communication and networking problems. DRL has also been applied to AoI-aware scheduling, particularly in scenarios where explicit system models are unavailable [22]. However, most existing DRLbased AoI studies focus on average AoI, average cost, or single-threshold reliability objectives, all of which can be adequately represented through scalar expected returns. Persistence-aware reliability objectives fundamentally differ from these formulations. Weighted C-AVR objectives induce heterogeneous return distributions dominated by both frequent short violation runs and rare but prolonged persistence events. Under such objectives, expectation-based value learning may obscure critical tail behavior. Different scheduling actions may produce similar expected AoI performance while exhibiting substantially different risks of generating long consecutive violation sequences. Since expectation-based DRL represents the future return using only a scalar expectation, important persistence-related information in the return distribution may be obscured. This limitation becomes particularly pronounced under exponential weighting schemes, where long violation runs dominate system reliability. To address this issue, we adopt distributional reinforcement learning, which models the full return distribution rather than only its expectation [25]–[27]. By preserving information about return variability during value learning, distributional DRL is well-suited to persistence-aware reliability objectives whose rewards are induced by temporally correlated violation events. In particular, quantile-regression-based methods such as QR-DQN approximate the return distribution using learnable quantiles, enabling accurate modeling of asymmetric and high-variance return structures induced by consecutive violation events [26]. Recent studies have demonstrated
the effectiveness of distributional DRL in uncertainty-aware wireless control, risk-sensitive scheduling, and reliabilityconstrained resource allocation [28]–[35]. Nevertheless, its role in persistence-aware AoI control has remained largely unexplored. Building on these observations, we develop a quantile regression-based dueling double deep Q-network (QR-D3QN) tailored for persistence-aware wireless scheduling. The proposed architecture combines Double Q-learning to mitigate overestimation bias [23], a dueling network structure to improve state-value and advantage estimation [24], and quantile regression to model the full return distribution [26]. These components jointly enable more stable learning under sparse and high-variance persistence-aware feedback, thereby reducing prolonged AoI violations while satisfying transmission cost constraints. Extensive simulations suggest that multi-scale persistence objectives can benefit from distributional value representations, and that distributional DRL provides an effective learning approach. Specifically, the proposed QR-D3QN consistently outperforms expectation-based baselines (i.e., DQN and D3QN) across a wide range of system configurations and weighting schemes, including uniform, exponential, and one-hot weighted C-AVR objectives. The performance gains become increasingly significant as the optimization objective places greater emphasis on long violation windows, confirming the importance of modeling the return distribution for persistence-aware reliability control. Furthermore, componentwise analysis of the C-AVR vector reveals that QR-D3QN achieves particularly substantial improvements at large persistence scales, demonstrating its effectiveness in suppressing prolonged consecutive AoI violations that dominate tail reliability. We further compare the proposed approach with the model-based drift-plus-penalty (DPP) method [36]. The results show that DRL methods achieve performance comparable to DPP under short persistence objectives, while QR-D3QN exhibits clear advantages when the objective emphasizes long violation windows. Moreover, all performance gains are achieved under the same transmission cost constraint, indicating that the improvements arise from more effective scheduling decisions rather than increased resource consumption. Overall, the results validate both the proposed persistence-aware CAVR framework and the effectiveness of distributional DRL for reliability-aware wireless scheduling. The main contributions of this work are summarized as follows: • We introduce the C-AVR vector to characterize AoI violations across multiple persistence scales, thereby capturing the full temporal persistence profile of information staleness. Based on this representation, we further develop the weighted C-AVR framework, which unifies average persistence, tail-sensitive reliability, and single-scale persistence objectives under flexible weighting schemes. • We demonstrate empirically that persistence-aware CAVR objectives induce heterogeneous and high-variance learning signals for which distributional value representations can be more effective than expectation-based DRL methods. To address this issue, we develop a
3
distributional DRL framework based on QR-D3QN for persistence-aware wireless scheduling under stochastic arrivals, unreliable channels, and transmission cost constraints. • We conduct extensive simulations under multiple weighting schemes and heterogeneous system settings. The results demonstrate that distributional value learning consistently outperforms expectation-based baselines, with particularly significant gains for long persistence windows and tail-sensitive reliability objectives. Componentwise analysis further confirms the effectiveness of the proposed method in suppressing prolonged AoI violation sequences across multiple temporal scales. II. S YSTEM M ODEL AND P ROBLEM F ORMULATION A. System Model We consider a time-slotted wireless status update system with stringent information timeliness QoS requirements, consisting of M independent information sources, indexed by {1, 2, ..., M }, and their corresponding receivers coordinated by a central scheduler, as illustrated in Fig. 1. Time is indexed by discrete time slots t = 1, 2, . . . . At the beginning of each slot, every source independently generates at most one fresh status update packet according to a Bernoulli process, i.e., source m generates a new packet with probability pm g . Newly generated packets are instantaneously delivered to the central node through an ideal uplink and overwrite any previously stored packet from the same source. Hence, the central node maintains a buffer of size M , consisting of M per-source entries, with one entry per source storing only the most recent packet. This per-source replacement structure reflects QoS-driven designs where only the freshest information is relevant [3]. Due to shared downlink resources, at most one packet can be transmitted in each time slot. A scheduled transmission to receiver m succeeds with probability pm s , independently across slots.1 Upon a successful transmission, the receiver sends an acknowledgment (ACK) through an error-free feedback link; otherwise, a negative acknowledgment (NACK) is returned. The freshness of information is quantified using the age of information (AoI), defined as the time elapsed since the generation of the most recently successfully received status update. To track information timeliness across the system, AoI is maintained at both the transmitter and the receiver for each source m ∈ {1, 2, ..., M }: • At the central node, the transmitter-side AoI for source m at slot t is the number of slots since the most recent packet from source m arrived at the buffer. The transmitterside AoI for source m at slot t, denoted by ∆s (t, m), is ∆s (t, m) = t − Ts (t, m), where Ts (t, m) denotes the arrival time of the most recent packet from source m. 1 For tractability, this work adopts a fixed transmission success probability
model, which is commonly used in the AoI literature. This abstraction enables clear evaluation of scheduling and decision-making mechanisms without introducing additional physical-layer complexities. The proposed CMDP and DRL framework can be readily extended to more realistic time-varying wireless channels, such as Rayleigh fading or Markov fading channels, due to its model-free nature.
A buffer of size M
1
Source 1
2
�1�
�1�
A CK
�� � Source M
Packet generation from sources to central node
M
/NAC
�� �
A CK Central node Packet transmission form central node to receivers
/NAC
K
K
Receiver 1
Receiver M
ACK/NACK feedback form receivers to central node
Fig. 1. A time-slotted wireless status update system in which a central node schedules update packets to different receivers.
The transmitter-side AoI is reset to zero upon a new packet arrival and increases by one in each subsequent slot otherwise. • At the receiver, the receiver-side AoI ∆r (t, m) measures the time elapsed since the generation of the most recently successfully received packet from source m. If an update packet from source m is successfully delivered at the end of slot t, then ∆r (t + 1, m) = ∆s (t, m) + 1; otherwise, ∆r (t + 1, m) = ∆r (t, m) + 1. The transmitter-side and receiver-side AoI are initialized to zero and one, respectively. Their coupled evolution is illustrated via an example in Fig. 2. The transmitter-side AoI ∆s (t, m) and the receiver-side AoI ∆r (t, m) are depicted by the red curve and the blue curve, respectively. Red arrows indicate packet-generation events at the beginning of slots. Blue arrows represent packet transmission outcomes at the end of slots, where solid arrows correspond to successful transmissions and dashed arrows denote failures. As shown in Fig. 2, the transmitter-side AoI is determined solely by packet generation events. Whenever a new packet is generated, ∆s (t, m) is reset to zero, and otherwise it increases linearly by one in each subsequent slot. In contrast, the receiver-side AoI depends jointly on the transmitter-side AoI and the transmission outcome. When a packet transmission is successful at slot t, the receiver-side AoI at the next slot is ∆r (t + 1, m) = ∆s (t, m) + 1, reflecting that the receiver obtains the most recently generated packet with a one slot delay. When a transmission attempt fails or when source m is not scheduled, the receiver-side AoI simply increases by one. Fig. 2 demonstrates that consecutive transmission failures or scheduling delays can lead to sustained growth of receiverside AoI even when fresh packets are available, highlighting the importance of capturing temporal persistence in reliability metrics. B. The Consecutive Age Violation Rate (C-AVR) Metric Most existing AoI-aware scheduling studies focus on longterm average performance metrics, with system average AoI being the most widely adopted measure of information freshness [10]. While effective for characterizing time-averaged performance, average AoI does not capture system reliability during rare but severe freshness-degradation events. To better
4
AoI 6 5 4 3 2 1 0
1
2
3
4
5
7
6
8
9
10
11
12
t
Fig. 2. Sample AoI evolution for source m. The red curve shows the transmitter-side AoI ∆s (t, m), and the blue curve shows the receiver-side AoI ∆r (t, m). Red arrows indicate packet generation events at the beginning of slots, while blue arrows denote packet transmission outcomes at the end of slots: solid for successful transmissions and dashed for failures.
capture reliability, recent works have introduced the age violation rate (AVR) metric, which measures the fraction of time slots in which the AoI exceeds a predefined threshold ζ, T
Φr = lim sup T →∞
M
1 XX E [1[∆r (t, m) > ζ]] . M T t=1 m=1
(1)
where 1[·] denotes the indicator function that returns 1 when its argument is true and 0 otherwise. Unlike average AoI, AVR captures the frequency of threshold violations rather than their magnitude, thereby providing a threshold-based notion of timeliness reliability. However, AVR fundamentally treats violations as independent events and ignores their temporal structure. In particular, it does not distinguish between isolated threshold crossings and sustained violation periods, despite their markedly different implications for system performance. From a QoS and safety perspective, prolonged sequences of outdated information are significantly more harmful than sporadic and shortlived violations, as they may induce sustained decision errors or trigger system-level failures. This observation highlights a key limitation of AVR and motivates the need for a reliability metric that explicitly captures the temporal persistence of AoI violations. To address this limitation, we put forth the consecutive age violation rate (C-AVR), which quantifies the probability that the AoI exceeds the threshold across multiple consecutive time slots. Let a positive integer k ≥ 1 denote a prescribed violation window length. A window of length k ending at slot t is said to be fully AoI violated for source m if 1[Vtk (m)] = 1, where Vtk (m) =
k \
{∆r (t − k + i, m) > ζ}.
(2)
i=1
That is, ∆r (t − k + i, m) > ζ for all i = 1, · · · , k. From a QoS standpoint, C-AVR characterizes the probability of timeliness outage events whose duration exceeds k consecutive slots, thereby directly capturing persistence in reliability degradation. The system-level C-AVR is defined as Ψkr = lim sup T →∞
T X M X 1 E 1[Vtk (m)] . M (T − k + 1) m=1 t=k
(3)
Optimizing C-AVR therefore enforces temporal QoS guarantees on information freshness, rather than merely improving average performance. C-AVR naturally recovers AVR as a special case when k = 1, while larger values of k place increasing emphasis on severe reliability degradation caused by prolonged violation sequences. Furthermore, since C-AVR depends on the joint temporal distribution of AoI exceedances, it cannot be inferred from AVR alone. This highlights that persistence is an intrinsic dimension of reliability that is not captured by existing metrics. To the best of our knowledge, no such window-based formulation that explicitly characterizes violation persistence has been systematically developed in the AoI scheduling literature. C. C-AVR Vector and Weighted Reliability Objective While the C-AVR defined in (3) captures persistence over a fixed window length k, it remains inherently a single-scale characterization. In practice, however, violations of different durations correspond to different levels of operational risk. Short consecutive violations may be tolerable in certain applications, whereas long violation runs can lead to sustained performance degradation or even system instability. Therefore, a metric defined at a single persistence scale is insufficient to fully characterize system reliability. We introduce a vectorized characterization of persistence, called the C-AVR vector, in which each component corresponds to a distinct violation window length. Specifically, for a maximum window length kmax , the C-AVR vector is defined as Ψ = Ψ1r , Ψ2r , . . . , Ψkr max , (4) This representation provides a multi-scale persistence profile of AoI violations, capturing reliability behavior across a spectrum of temporal horizons rather than at a single scale. As such, it elevates C-AVR from an isolated scalar metric to a structured description of temporal reliability. To enable tractable optimization while preserving this multiscale structure, we introduce a weighted aggregation of the C-AVR vector. Let W = (w1 , w2 , ..., kmax ) denote the Pw kmax weighting vector satisfying wk ≥ 0 and k=1 wk = 1, where wk specifies the relative importance assigned to persistence at scale k, i.e., the C-AVR scalar Ψkr . The resulting scalar objective, referred to as the weighted C-AVR, is defined as Ψ̄ = WΨ⊤ =
kX max
wk Ψkr ,
(5)
k=1
Importantly, this formulation establishes a unified optimization framework over the persistence profile. By adjusting the weights, the scheduler can emphasize different persistence regimes, ranging from average persistence to tail-sensitive criteria that prioritize long violation runs. In this sense, conventional scalar formulations arise as special cases within the proposed framework. In particular, a one-hot weighting reduces the weighted C-AVR to a single-scale C-AVR metric, while more general weightings capture interactions across multiple persistence scales. In this work, we consider the following representative weighting schemes:
5
•
Uniform Weights assign equal importance across all persistence scales, i.e., wk =
1 , kmax
k = 1, 2, . . . , kmax ,
(6)
thus reflecting scenarios where both short and long violation sequences are considered equally relevant. • Exponential Weights assign progressively larger weights to longer violation windows, i.e., βk wk = Pkmax j=1
βj
,
β > 1,
cast problem (9) as a constrained Markov decision process (CMDP), which serves as the foundation for the proposed distributional reinforcement learning approach.
(7)
thereby emphasizing tail behavior. Consequently, the resulting weighted objective is particularly sensitive to long but rare violations, with β controlling the degree of tail emphasis. • One-hot Weights concentrate all weight on a single persistence scale ko , i.e., wko = 1 and wk = 0 for k ̸= ko . In this case, the weighted C-AVR reduces exactly to the scalar C-AVR at window length ko , demonstrating that the scalar formulation is a special case of the proposed vector framework.
III. CMDP F ORMULATION The scheduling problem formulated in Section II-D involves long-term persistence-aware reliability optimization under an average transmission cost constraint. Such a problem can be naturally modeled as a CMDP [37]. In this section, we present a complete CMDP formulation that is explicitly aligned with the weighted C-AVR objective introduced previously. The CMDP is defined as a sextuple (S, A, P, R, C, γ), where S is the state space, A is the action space, P : S × A × S → [0, 1] is the state transition probability, R : S ×A → R is the reward function, C : S × A → R is the cost function, and γ is the discount factor. Each component is specified in detail below. a) State Space: The system state captures both the instantaneous information freshness and its temporal persistence at a given time slot. Since both packet generation and packet delivery affect AoI evolution, the state jointly includes the transmitter-side AoI, the receiver-side AoI, and the consecutive violation status for each source m. At the beginning of slot t, the system state is represented by a 3M -dimensional vector
D. Problem Formulation Let at ∈ {0, 1, ..., M } denote the scheduling action at slot t, where at = m indicates that source m is scheduled for transmission and at = 0 indicates no transmission. The longterm average transmission cost is defined as η = lim sup T →∞
T 1X
T t=1
E[1[at ̸= 0]].
(8)
which must satisfy η ≤ ηmax , where ηmax ∈ (0, 1] reflects the resource constraint. Combining the persistence-aware reliability objective with the resource constraint, we formulate the weighted C-AVR scheduling problem as min Ψ̄, {at }
s.t.
(9) η ≤ ηmax .
By optimizing the weighted C-AVR, the scheduler is explicitly driven to control the temporal persistence of AoI violations, rather than merely their instantaneous occurrence. This distinction is critical because unlike AVR, which primarily reflects frequency-based characteristics, the weighted C-AVR objective is inherently sensitive to the temporal correlation structure of violations and is therefore dominated by rare but high-impact events. Furthermore, the weighting mechanism enables a unified treatment of reliability objectives across multiple persistence scales. In particular, uniform weighting captures average persistence behavior, exponential weighting emphasizes tail risk, and one-hot weighting isolates singlescale reliability requirements. This unified formulation highlights the conceptual contribution of the proposed framework: reliability is not characterized by a single metric, but by a structured persistence profile that can be flexibly aggregated based on system requirements. In the next section, we
st = ∆s (t, 1), ∆r (t, 1), v(t, 1), . . . , ∆s (t, m), ∆r (t, m), v(t, m), . . . , ∆s (t, M ), ∆r (t, M ), v(t, M ) .
(10)
Here, ∆s (t, m) and ∆r (t, m) denote the transmitter-side and receiver-side AoI of source m, respectively. The variable v(t, m) tracks the length of the current consecutive AoI violation segment relative to the threshold ζ, evolving as ( v(t − 1, m) + 1, if ∆r (t, m) > ζ, v(t, m) = (11) 0, if ∆r (t, m) ≤ ζ, with initial condition v(0, m) = 0. v(t, m) is reset whenever the AoI returns to a non-violating level. To ensure a finitestate MDP, the violation counter is truncated at kmax in this work, i.e., v(t, m) ∈ {0, 1, . . . , kmax }. Specifically, whenever the number of consecutive AoI violations exceeds kmax , the counter remains fixed at kmax . This state representation explicitly encodes both instantaneous AoI levels and their temporal persistence, which is essential for evaluating multi-scale violation events underlying the C-AVR vector. To ensure a finite state space, all AoI values are truncated by a predefined upper bound ∆max . Once an AoI value reaches ∆max , it remains at this value. This truncation preserves the relevant persistence structure while improving computational tractability. b) Action Space: At each time slot t, the scheduler selects an action at ∈ A, where A = {0, 1, . . . , M }. Action at = m schedules source m for transmission, while at = 0 indicates no transmission in the current slot. c) State Transition Probability: Given the current state st and action at , the system moves to the next state st+1 according to the transition probability P(st+1 | st , at ). The transition is determined by stochastic packet generation and
6
random transmission outcomes. Specifically, packet generation for source m follows a Bernoulli process with probability m pm g , and successful transmission occurs with probability ps . These system parameters are assumed unknown, making the transition model unavailable in closed form. Consequently, the scheduling problem naturally motivates a model-free reinforcement learning approach. d) Cost Function: To model resource constraints, each transmission incurs a unit cost that captures resource consumption, such as energy or bandwidth. The instantaneous cost at slot t is defined as ct = 1[at ̸= 0]. The long-term PTaverage 1 transmission cost is required to satisfy limT →∞ T t=1 ct ≤ ηmax , where ηmax ∈ (0, 1] specifies the allowable resource budget. e) Reward Function: The objective is to minimize the weighted C-AVR defined in (5), which aggregates violation events across multiple persistence scales. To incorporate this objective into the CMDP framework while satisfying the resource constraint, we adopt a Lagrangian relaxation approach. Specifically, after executing action at during slot t, the reward is defined as kmax M X 1 X k wk 1[Vt+1 (m)] − λ(ct − ηmax ). rt ≜ − M m=1
(12)
k=1
k Here, Vt+1 (m) denotes the event that the receiver-side AoI
of source m exceeds the threshold ζ for k consecutive slots ending at slot t + 1. The index t + 1 appears because the scheduling action taken in slot t affects the AoI state only after the transmission outcome is realized at the end of the slot. Consequently, the persistence-aware violation event induced by action at is evaluated using the updated AoI state at slot t + 1. The second term in (12) enforces the transmission cost constraint through the Lagrange multiplier λ ≥ 0. The first term represents a weighted aggregation of multi-scale violation events, and its time average corresponds to the empirical estimate of the weighted C-AVR defined in (5). It therefore penalizes violations across multiple persistence scales simultaneously, with their relative importance determined by the weight vector W. For example, uniform weights emphasize average persistence behavior, whereas exponentially increasing weights place greater emphasis on rare but prolonged violation runs associated with tail reliability degradation. The empirical average transmission cost is updated as 1 η ← η + ct − η , (13) t and the Lagrange multiplier λ is updated using the projected stochastic subgradient descent λ ← max (0, λ + ξ(η − ηmax )) ,
(14)
where ξ > 0 is the step size. This update dynamically adjusts the penalty to enforce the long-term constraint, i.e., increasing the penalty when the empirical transmission cost exceeds the allowable budget. f) Discount Factor: The discount factor γ ∈ (0, 1] controls the relative importance of immediate and future rewards. When the system performance P∞ is evaluated through the expected discounted return Eπ [ t=0 γ t rt ] under policy
π, a value of γ close to one emphasizes long-term reliability performance, which is critical for persistence-aware objectives where the impact of actions accumulates over consecutive time slots. While the CMDP formulation provides a principled framework for optimizing the weighted C-AVR objective under resource constraints, it also exposes a key challenge. The weighted C-AVR objective is affected by temporally correlated and potentially rare violation events, especially under tailsensitive weighting schemes. Consequently, the induced return distribution can be high-variance and asymmetric. When future performance is summarized solely by its expectation, some persistence-driven variability is compressed into a scalar value estimate. In particular, conventional DRL methods based on expectation-valued value functions may yield less informative learning signals for prolonged violation sequences under persistence-aware objectives. This observation motivates value representations that preserve distributional information during training. In the next section, we develop a distributional DRL framework tailored to weighted C-AVR optimization. IV. D ISTRIBUTIONAL DRL FOR W EIGHTED C-AVR O PTIMIZATION This section develops a distributional DRL approach explicitly aligned with the weighted C-AVR objective. By modeling the full distribution of future returns, rather than only their expectation, the proposed method captures the return variability and tail behavior induced by persistence-aware AoI violation penalties across multiple time scales. This richer representation enables the scheduler to distinguish policies with similar average performance but markedly different reliability profiles. Leveraging quantile-based distributional learning together with a reliability-aware state and reward design introduced in the CMDP formulation, we construct a QR-D3QN framework that is naturally suited to minimizing persistence-driven QoS violations under resource constraints. The overall learning framework is illustrated in Fig. 3. A. From Expected Value Learning to Distributional Value Learning In standard Deep Q-Networks (DQN), the agent learns a scalar action-value function Q(s, a), representing the expected discounted return obtained by taking action a in state s. The optimal action-value function satisfies the Bellman optimality equation h i ′ ′ Q(s, a) = E r + γ max Q(s , a ) | s, a , (15) ′ a
where r is the immediate reward, s′ is the next state, and γ ∈ (0, 1] is the discount factor. In practice, Q(s, a) is approximated by a neural network Q(s, a; w) with parameters w, trained by minimizing the temporal-difference (TD) loss h 2 i L(w) = E(s,a,r,s′ )∼B y − Q(s, a; w) , (16) where the TD target is y = r + γ max Q(s′ , a′ ; w− ), ′ a
(17)
7
Target network
Experience Replay Buffer
... ...
Mini-batch ... ...
Online network ...
K
AC
CK
...
/NA
...
ACK
C-AVR Environment
...
K/N
AC
Quantile Hube Loss
...
Fig. 3. QR-D3QN–based training framework for weighted C-AVR–aware scheduling, where distributional value estimation enables learning under rare but persistent AoI violation events.
and w− denotes the parameters of a periodically updated target network. Here, B represents mini-batches sampled uniformly from the replay buffer M, which mitigates temporal correlations and stabilizes learning. A known limitation of DQN is the overestimation bias introduced by the maximization operator. Dueling Double DQN (D3QN) mitigates this issue through two complementary mechanisms: • Double Q-learning: Action selection and evaluation are decoupled, yielding the TD target ′ ′ − y ∗ = r + γQ s′ , arg max Q(s , a ; w); w . (18) ′
To address this challenge, we adopt a distributional reinforcement learning framework. Instead of approximating the expected return, distributional methods model the entire return distribution. Let Z(s, a) denote the random return obtained by taking action a in state s. The expected action value is then Q(s, a) = E[Z(s, a)].
(20)
By explicitly representing the distribution of returns Z(s, a), distributional DRL preserves information about variability and low-probability, high-impact events, which are precisely the events governing weighted C-AVR performance.
a
Dueling architecture: The Q-function is decomposed B. QR-D3QN for C-AVR-based Scheduling into a state-value component and an advantage compoAmong distributional DRL approaches, Quantile Regression nent DQN (QR-DQN) offers a favorable balance between modeling 1 X Q(s, a; w) = V (s; w)+ A(s, a; w)− A(s, a′ ; w) , flexibility and computational efficiency. Rather than using |A| ′ a fixed categorical supports in C51, QR-DQN approximates the (19) return distribution using a set of learnable quantiles. where |A| is the number of actions. This separation For each state action pair (s, a), the cumulative distribution improves stability and emphasizes meaningful actions function (CDF) is partitioned into N uniform probability relative to the state. intervals {τ1 , τ2 , ..., τi , ..., τN }, where τi = i/N . The midpoint While D3QN improves learning stability and action discrimof the interval (τi−1 , τi ) is τ̂i = (τi−1 + τi )/2. The network ination, it still estimates only the expected return. This limilearns quantile values θi (s, a; w) representing the τ̂i -quantiles, tation is particularly relevant for weighted C-AVR optimizaand the return distribution of Z(s, a; w) is approximated by a tion. Specifically, the reward in (12) aggregates multi-scale uniform mixture of Dirac masses at these learned quantiles violation events and is influenced by temporally correlated and N potentially rare trajectories involving persistent AoI violations, 1 X δθ (s,a;w) , (21) Z(s, a; w) = especially under tail-sensitive weighting schemes. As a result, N i=1 i the induced return distribution can be heterogeneous and highvariance. When learning is based solely on expected returns, where δz denotes a Dirac distribution centered at z. This these persistence-driven variations are compressed into scalar representation captures not only the central tendency of the value targets. In particular, actions or policies with similar return but also its dispersion and tail behavior. In the conexpected returns may still be associated with return distri- text of weighted C-AVR optimization, the reward explicitly butions of different variability, which can make expectation- penalizes persistence of AoI violations across multiple time based value learning less informative under persistence-aware scales. Consequently, trajectories containing prolonged violation sequences accumulate substantially larger penalties and reliability objectives. •
8
therefore induce significantly lower cumulative returns. These rare but high-impact trajectories primarily influence the lower quantiles of the learned return distribution. Consequently, the distributional representation preserves richer training information about persistence-driven reliability degradation, whereas expectation-based methods compress such information into a single scalar target. To further improve learning stability, we integrate quantile regression with the D3QN architecture. For each quantile index i ∈ {1, . . . , N }, the corresponding quantile value θi (s, a; w) is decomposed into a state-value component and an advantage component as 1 X Ai (s, a′ ; w) , θi (s, a; w) = Vi (s; w) + Ai (s, a; w) − |A| ′ a (22) with 1 X θi (s, a′ ; w), (23) Vi (s; w) = |A| ′ a
Ai (s, a; w) = θi (s, a; w) − Vi (s; w). (24) P This decomposition enforces a Ai (s, a; w) = 0 for each quantile i, yielding a consistent separation between statevalue and action-dependent effects at every quantile level. It extends the classical dueling architecture to the distributional setting, where the components Vi (s) capture the system-wide reliability baseline at different return quantiles, while Ai (s, a) quantify the action-specific deviations relative to this baseline. The quantile-wise dueling architecture thus enables the model to distinguish how different actions influence not only the expected performance but also the distributional characteristics of returns, including variability and tail risk induced by persistent violation events. For action selection, the expected action value is computed by averaging the quantile estimates N
Q(s, a; w) =
1 X θi (s, a; w), N i=1
(25)
In QR-D3QN, the mean action value used for scheduling is obtained by averaging the learned quantile returns. The key distinction from expectation-based DRL lies in how this action value is learned. Instead of directly fitting a single scalar TD target, QR-D3QN learns a set of quantile targets through distributional Bellman updates, thereby preserving information about return variability, asymmetry, and low-probability persistence events during training. As a result, the resulting mean action-value estimates are informed by a richer distributional representation, which helps the scheduler identify actions that reduce prolonged AoI violation sequences under high-variance persistence-aware rewards. In addition, actions are selected according to an ϵ-greedy exploration policy, where the exploration rate ϵe at episode e follows the linear decay schedule e−1 ϵ (ϵstart − ϵend ), e ≤ Edecay , start − Edecay − 1 ϵe = ϵend , e > Edecay . (26)
Algorithm 1 QR-D3QN for Weighted C-AVR Scheduling Input: Number of training episodes Emax ; episode length Tmax ; minimum replay buffer size Mmin for training; discount factor γ; learning rate α; target update frequency G; linear decay exploration schedule parameters (ϵstart , ϵend , Edecay ); transmission cost constraint ηmax ; Lagrange multiplier λ; number of quantiles N ; step size ξ for λ update. 1: Initialize online network w and target network w − ← w, replay buffer M, Lagrange multiplier λ ≥ 0, empirical cost estimate η ← 0, and global time slot t ← 0 2: for episode e = 1 to Emax do 3: Reset the environment 4: Update exploration rate ϵe according to the linear decay schedule (26) 5: for n = 1 to Tmax do 6: Update global time slot index t ← t + 1 7: Observe the current state st consisting of AoI states and consecutive violation counters 8: Compute quantile values {θi (st , a; w)}N i=1 for all a ∈ A using the quantile-wise dueling architecture (22) 9: Compute mean action values Q(st , a; w) = PN 1 θ (s i=1 i t , a; w) N 10: Select action at using ϵe -greedy exploration 11: Execute at , observe cost ct , compute the Lagrangian reward rt using the current multiplier λ, and obtain the next state st+1 12: Store transition (st , at , rt , st+1 ) in M 13: Update empirical average transmission cost estimate η by (13) 14: if |M| ≥ Mmin then 15: Sample a mini-batch of transitions B = {(s, a, r, s′ )} uniformly from the replay buffer M 16: Select the greedy next action using the online network by (27) 17: Compute target quantiles by (28) 18: Compute quantile Huber loss L(w) by (29) 19: Gradient update: w ← w − α∇w L(w) 20: if t mod G = 0 then 21: Update target network: w− ← w 22: end if 23: end if 24: Update Lagrange multiplier λ by (14) 25: end for 26: end for 27: return Trained QR-D3QN parameters w Here, ϵstart and ϵend denote the initial and final exploration rates, respectively. Edecay specifies the number of training episodes over which the exploration rate decreases linearly from ϵstart to ϵend . The distributional Bellman update follows the double Qlearning principle. For a sampled transition (s, a, r, s′ ), the next action is selected as N
a∗ = arg max ′ a
1 X θi (s′ , a′ ; w), N i=1
(27)
9
and the target quantiles are computed as ′
∗
−
T θj = r + γθj (s , a ; w ),
TABLE I T RAINING PARAMETERS OF THE QR-D3QN A LGORITHMS
j = 1, . . . , N,
(28) Parameter AoI upper bound Discount factor DRL learning rate Step size for λ update Initial exploration rate Final exploration rate Number of exploration decay episodes Target network update period Replay buffer size Minimum replay buffer size for training Mini-batch size Hidden-layer size Number of quantiles Huber loss threshold Number of training episodes Time slots per episode
where w and w− denote the online and target network parameters, respectively. To train the network parameters, we minimize the quantile Huber loss L(w) =
N N 1 XX κ ρτ̂i T θj − θi (s, a; w) , N i=1 j=1
(29)
with ( ρκτ (u) =
1
1 2 2 u |τ − {u < 0}|, κ(|u| − 21 κ)|τ − {u < 0}|,
1
|u| ≤ κ, otherwise.
(30)
where κ > 0 controls smoothness near zero. This objective ensures accurate learning of the return distribution while maintaining training stability. As shown in Fig. 3, the proposed QR-D3QN framework integrates quantile-based distributional value learning with the D3QN learning architecture to optimize the weighted C-AVR objective in stochastic wireless systems. At each time slot, the agent observes the system state s, which includes transmitterside AoI, receiver-side AoI, and consecutive violation counters, and selects an action a via an ϵ-greedy policy based on the expected Q-values computed by averaging the learned quantiles. After executing the action, the resulting transition (s, a, r, s′ ) is stored in the replay buffer. Mini-batches sampled from this buffer are used to update the online network by minimizing the quantile Huber loss, where target quantiles are constructed through a distributional Bellman update combined with double Q-learning. A target network is periodically synchronized to ensure training stability. Crucially, by modeling a quantile-based return distribution during training, the learned value representation retains information related to both the frequency of AoI violations and their temporal persistence across different scales. This can improve the mean action-value estimates used by the mean-greedy scheduler when returns are high-variance and persistence-aware. Such capability is useful for weighted CAVR optimization, especially under weighting schemes that place larger penalties on prolonged violation events. The complete procedure is summarized in Algorithm 1. V. P ERFORMANCE E VALUATION This section evaluates the proposed persistence-aware scheduling framework from three perspectives: (i) the effectiveness of the weighted C-AVR objective in capturing multiscale reliability, (ii) the benefit of distributional reinforcement learning in reducing persistent AoI violations, and (iii) the robustness of the proposed method under varying system conditions. A. Simulation Setup All algorithms are implemented in Python 3.9 using PyTorch and evaluated under identical software and hardware configurations to ensure a fair and reproducible comparison.
Symbol ∆max γ α ξ ϵstart ϵend Edecay G |M| Mmin |B| – N κ Emax Tmax
Value 100 0.98 2 × 10−3 0.10 1.0 0.05 1000 3 106 1000 256 2 × 128 64 1 3000 100
Unless otherwise specified, we consider a system with M = 10 sources, and a homogeneous setting for different sources with packet generation probability pm g = 0.7, transmission success probability pm s = 0.7, AoI violation threshold ζ = 15, and transmission cost constraint ηmax = 0.75. Heterogeneous settings are considered in Section V-D. The AoI is truncated at ∆max = 100. The training parameters are summarized in Table I: discount factor γ = 0.98, learning rate α = 2 × 10−3 , replay buffer size |M| = 106 , minimum buffer size Mmin = 1000, mini-batch size |B| = 256, target update period G = 3, and number of quantiles N = 64. All results are averaged over 10 independent runs with different random seeds. For weighted C-AVR simulations, we set kmax = 9 unless otherwise stated. Three weighting schemes are considered: (i) Uniform weights; (ii) Exponentially increasing weights (default β = 2); and (iii) One-hot weights, which reduce the objective to a single-window C-AVR. We compare the proposed QR-D3QN method with the following baselines: DQN: Standard Deep Q Network that learns the expected return as a scalar value function. • D3QN: Builds on DQN with Double Q-learning and a dueling architecture to improve stability and action-value estimation, still based on expected returns. • QR-DQN: Distributional DRL algorithm using quantile regression to model the return distribution, without Double Q-learning or dueling structures. • QR-D3QN: The proposed approach combining quantilebased distributional learning with Double Q-learning and dueling architectures, designed to capture rare but persistent violations while maintaining stable learning. • Drift-Plus-Penalty (DPP) Policy: DPP is a wellestablished Lyapunov optimization framework for constrained stochastic optimization in networked systems and has been widely applied in AoI scheduling problems. Unlike DRL methods, DPP exploits system structure and does not rely on value function approximation, thereby providing a complementary benchmark from a classical control and operations research perspective. In this work, the resulting policy serves as a model-based heuristic benchmark derived from Lyapunov optimization prin•
10
D 8 Q L I R U P Z H L J K W V
' 4 1 4 5 ' 4 1 ' 4 1 4 5 ' 4 1 ' 3 3
: H L J K W H G & $ 9 5 Ψ̄
E ( [ S R Q H Q W L D O O \ L Q F U H D V L Q J Z H L J K W V
' 4 1 4 5 ' 4 1 ' 4 1 4 5 ' 4 1 ' 3 3
0 D [ L P X P Z L Q G R Z O H Q J W K kmax
0 D [ L P X P Z L Q G R Z O H Q J W K kmax
' 4 1 4 5 ' 4 1 ' 4 1 4 5 ' 4 1 ' 3 3
F 2 Q H K R W Z H L J K W V
9 L R O D W L R Q Z L Q G R Z O H Q J W K ko
Fig. 4. Weighted C-AVR performance Ψ̄ of DQN, D3QN, QR-DQN, QR-D3QN, and DPP under (a) uniform weights, (b) exponential weights, and (c) one-hot weights. The horizontal axes represent the maximum window length kmax in (a) and (b), and the violation window length ko in (c) (M = 10, pm g = 0.7, pm s = 0.7, and ζ = 15).
ciples. The details of the DPP policy are provided in Appendix A.
& $ 9 5 Ψ kr
B. C-AVR Performance under Different Weighting Schemes 1) Weighted C-AVR Performance: Fig. 4 shows the weighted C-AVR performance under different weighting schemes. Specifically, weighted C-AVR is plotted as a function of kmax under uniform and exponential weights in Fig. 4(a) and (b), respectively. For the one-hot setting, the objective reduces to a single-window C-AVR with window length ko . To facilitate comparison, we set ko = kmax and wko = 1, and plot the resulting performance as a function of ko in Fig. 4(c). Under different weighting schemes, the weighted C-AVR decreases monotonically as kmax increases for all methods. This trend follows directly from the definition of CAVR: longer violation windows occur less frequently and thus contribute smaller values to the aggregated metric. Considering the one-hot setting in Fig. 4(c), for example, increasing ko gradually shifts the objective from a frequency-oriented metric (ko = 1, equivalent to AVR) to a persistence-sensitive reliability metric (ko > 1). The observed monotonic decrease with ko is therefore primarily induced by the metric itself rather than by differences in control policies. Beyond this metric-induced effect, Fig. 4 shows a consistent performance ordering across different DRL algorithms. The two distributional methods (QR-DQN and QR-D3QN) achieve uniformly lower weighted C-AVR than the expectation-based baselines (DQN and D3QN). This indicates that weighted C-AVR optimization is inherently sensitive to the temporal persistence of AoI violations, and that learning the full return distribution provides a more informative learning signal than relying solely on expected returns. Moreover, the performance gap between QR-DQN and QR-D3QN is relatively modest, suggesting that the dominant improvement arises from distributional value representation, while Double Q-learning and the dueling architecture mainly contribute to stability and refinement of action evaluation. In addition, we include a Lyapunov-based DPP baseline in Fig. 4 to provide a structure-aware benchmark. When kmax (or
' 4 1 8 Q L I R U P kmax = 9 4 5 ' 4 1 8 Q L I R U P kmax = 9 ' 4 1 ( [ S R Q H Q W L D O kmax = 9 4 5 ' 4 1 ( [ S R Q H Q W L D O kmax = 9 ' 4 1 2 Q H K R W ko = 1 4 5 ' 4 1 2 Q H K R W ko = 1
9 L R O D W L R Q Z L Q G R Z O H Q J W K k
Fig. 5. Per-component C-AVR Ψkr versus violation window length k under different weighting schemes. Uniform and exponential weighting use kmax = m 9, while the one-hot setting uses ko = 1 (M = 10, pm g = 0.7, ps = 0.7, and ζ = 15).
ko in the one-hot case) is small, the weighted C-AVR objective is dominated by short-term violations and becomes closely aligned with instantaneous or near-term penalties. In this regime, by directly minimizing a one-step drift-plus-penalty surrogate, DPP captures the problem structure effectively, and DRL-based methods exhibit comparable performance. However, as kmax increases, the objective increasingly reflects long-horizon, persistence-driven penalties that depend on the evolution of violation sequences over multiple time slots. In this case, DPP’s myopic structure becomes suboptimal, as it cannot explicitly account for long-term temporal dependencies. In contrast, DRL methods, particularly distributional ones, propagate delayed penalties through the value function and can therefore better anticipate and suppress persistent violation patterns. This explains why QR-D3QN increasingly outperforms DPP as kmax grows. All these results highlight that C-AVR emphasizes controlling persistent AoI violations rather than isolated threshold crossings.
11
ηmax = 0.75
ηt
2) C-AVR Components Across Multiple Persistence Scales: While Fig. 4 provides an aggregated evaluation, it does not reveal how performance is distributed across persistence scales. To this end, Fig. 5 examines the full C-AVR vector by reporting Ψkr for k = {1, . . . , 9}. Here, kmax = 9 is used for uniform and exponential weights. At the same time, ko = 1 is adopted in the one-hot case, corresponding to AVR optimization, a special case of the proposed weighted C-AVR framework. It is important to emphasize that Fig. 5 evaluates the individual components of the C-AVR vector, whereas the optimization objective is the aggregated metric Ψ̄. Therefore, this figure serves as a diagnostic tool to assess how different objectives influence reliability across multiple persistence scales. Notably, even when training is performed under the one-hot setting (i.e., optimizing Ψ1r ), we can still evaluate Ψkr for all k by collecting the corresponding statistics under the learned policy. As expected, Ψkr decreases monotonically with k for all methods, reflecting the decreasing likelihood of long consecutive violations. More importantly, under uniform and onehot weighting, QR-D3QN consistently achieves lower Ψkr than D3QN across almost all k. This demonstrates that the performance gain of distributional value learning is not restricted to a particular persistence scale but instead extends uniformly across the full spectrum of violation durations. Under exponential weighting, an interesting trade-off is observed when examining the individual components of the C-AVR vector. Specifically, D3QN may achieve slightly lower Ψkr for small k (e.g., k < 5), whereas QR-D3QN significantly outperforms D3QN for larger k. This behavior is consistent with the structure of the objective, where exponential weights amplify the contribution of large-k violations, leading the reward signal to be dominated by rare but high-impact events. Distributional methods retain quantile-level information about low-return trajectories and can therefore provide richer training signals for these rare penalties. In contrast, expectation-based methods may average out such events and instead allocate more resources to controlling short-term violations, which explains their relative advantage at small k. Consequently, as k increases, the benefit of distributional value learning becomes more significant, and exponential weighting yields the lowest individual C-AVR Ψkr compared to other weighting schemes. Unlike exponential weighting, uniform weighting distributes emphasis evenly across all k, which is usually sufficient for short-to-moderate persistence scales. Note in Fig. 5 that the one-hot weighting scheme exhibits inferior performance for large k compared to both uniform and exponential weighting. This indicates that a policy optimized solely for AVR (ko = 1) does not generalize to persistencesensitive reliability metrics. More broadly, optimizing a single C-AVR component cannot guarantee good performance across other persistence scales. In contrast, the proposed weighted C-AVR objective provides a unified mechanism that not only optimizes an aggregate reliability measure but also improves individual components of the C-AVR vector in a coordinated manner. 3) Verification of Constraint Satisfaction: To verify constraint satisfaction, Fig. 6 depicts the evolution of the average
' 4 1 ' 4 1 4 5 ' 4 1 4 5 ' 4 1 ' 3 3
7 L P H V O R W V
Fig. 6. Average transmission cost over time under the cost constraint ηmax = m 0.75 with exponential weighting (M = 10, pm g = 0.7, ps = 0.7, and ζ = 15).
transmission cost for all algorithms under the exponential weighting scheme as an example. All methods satisfy the prescribed constraint ηmax = 0.75, indicating the feasibility of the learned policies. Importantly, the improved weighted CAVR performance of QR-D3QN is achieved without increasing resource consumption. This confirms that distributional value learning enhances persistence-aware reliability through improved decision quality rather than more aggressive resource usage. In the following subsection, we further quantify these advantages by comparing distributional and expectationbased methods across varying system parameters. C. Distributional DRL Performance under Different System Parameters Fig. 7 quantifies the benefit of distributional value modeling under the weighted C-AVR objective. Specifically, it reports the relative reduction in weighted C-AVR achieved by QRD3QN over D3QN for kmax ∈ {1, 5, 9} under exponential weighting, while varying key system parameters. Since QRD3QN and D3QN share the same backbone network structure and training framework, while differing mainly in their value representations (distributional versus expectation-based), this comparison isolates the impact of modeling the full return distribution. Across all subfigures, namely (a) packet generation rate, (b) AoI violation threshold, and (c) transmission success probability, QR-D3QN consistently outperforms D3QN. Notably, the performance gain increases with kmax , indicating that distributional learning becomes increasingly advantageous as the objective places greater emphasis on persistence-driven violations. In Fig. 7(a), as the packet generation rate increases, the performance gain of QR-D3QN becomes more significant, exceeding 20% for kmax = 9 while remaining moderate for kmax = 1. When the generation rate is high, packet availability is no longer the limiting factor; instead, timely delivery under the transmission cost constraint becomes the dominant factor. In this regime, sequences of non-transmission or un-
: H L J K W H G & $ 9 5 5 H G X F W L R Q 3 H U F H Q W D J H
12
(a)
4 5 ' 4 1 Y V ' 4 1 kmax = 1 4 5 ' 4 1 Y V ' 4 1 kmax = 5 4 5 ' 4 1 Y V ' 4 1 kmax = 9
(b)
4 5 ' 4 1 Y V ' 4 1 kmax = 1 4 5 ' 4 1 Y V ' 4 1 kmax = 5 4 5 ' 4 1 Y V ' 4 1 kmax = 9
3 D F N H W * H Q H U D W L R Q 5 D W H pgm
$ R , 9 L R O D W L R Q 7 K U H V K R O G ζ
(c)
4 5 ' 4 1 Y V ' 4 1 kmax = 1 4 5 ' 4 1 Y V ' 4 1 kmax = 5 4 5 ' 4 1 Y V ' 4 1 kmax = 9
3 D F N H W 7 U D Q V P L V V L R Q 6 X F F H V V 3 U R E D E L O L W \ psm
Fig. 7. Relative weighted C-AVR reduction (%) of QR-D3QN over D3QN for kmax ∈ {1, 5, 9} under exponential weighting and different (a) packet generation rates, (b) AoI violation thresholds, and (c) transmission successful probabilities.
successful transmissions lead to uninterrupted AoI growth, thereby increasing the likelihood of consecutive violations. Consequently, the performance bottleneck shifts from average behavior to the control of such persistence events, amplifying the role of the tail in the weighted C-AVR objective and favoring distributional value learning. Next, Fig. 7(b) shows that as the AoI violation threshold ζ decreases, the relative improvement of QR-D3QN over D3QN diminishes. With stricter thresholds, violations occur more frequently across all policies, reducing the relative contribution of rare and persistent violation events in the weighted objective. Hence, the advantage of modeling the return distribution is partially attenuated. Nevertheless, QR-D3QN maintains consistent gains across all threshold settings, indicating that distributional value modeling remains beneficial even when violations are more common. Finally, Fig. 7(c) shows that as channel reliability improves, the relative gain of QR-D3QN increases, particularly for larger kmax . A higher transmission success probability enhances the system’s ability to control AoI evolution, enabling proactive prevention of long violation sequences. In this regime, performance increasingly depends on anticipating and avoiding rare persistence events. Distributional value learning offers an advantage by retaining quantile-level information about low-return trajectories during training, thereby enabling more effective suppression of such events. D. Tail Persistence Index: σmin Analysis While the weighted C-AVR and the detailed C-AVR vector characterize persistence-aware reliability from aggregate and profile perspectives, they do not directly provide a guaranteeoriented interpretation. From a system design viewpoint, it is often desirable to determine the minimum persistence scale beyond which the reliability requirement is consistently satisfied. To this end, we introduce the tail persistence index σmin , defined as σmin = min k ∈ {1, . . . , kmax } : Ψkr ≤ ϵ̂ . (31) where ϵ̂ denotes a target violation level. If no persistence scale satisfies the target reliability level, we set σmin = kmax + 1, indicating that the violation probability remains above ϵ̂
throughout the considered persistence range. A smaller σmin implies that the policy can suppress consecutive AoI violations below the target level starting from shorter persistence windows, thereby providing a stronger persistence-level reliability guarantee. Fig. 8 evaluates σmin under heterogeneous wireless environments. Specifically, for each source m, the packet generation probability pm g and transmission success probability are independently sampled from the interval [0.6, 0.8]. We pm s set kmax = 9 and ϵ̂ = 0.05. For the one-hot weighting scheme in Fig. 8(b), we further set ko = 9. Fig. 8(a) illustrates the impact of the number of sources M on σmin . As the system size increases, QR-D3QN consistently achieves a smaller or equal σmin compared with D3QN. Under heterogeneous traffic and channel conditions, different sources contribute unevenly to persistence-level violations, leading to a more skewed and high-variance return distribution. By explicitly modeling this distribution, QR-D3QN can better distinguish actions with similar expected returns but different long-term reliability implications, thereby achieving stronger persistence-aware guarantees. This result highlights that the proposed tail persistence index σmin provides a direct and interpretable characterization of persistence-level reliability beyond aggregate weighted C-AVR metrics. Fig. 8(b) further examines the impact of the weighting scheme on σmin under a fixed system size M = 10. Since Fig. 8(a) already demonstrates the superiority of QR-D3QN over D3QN, only QR-D3QN is considered here. The results show that the weighting design significantly influences persistence-level reliability. In particular, the one-hot weighting scheme yields the largest σmin , indicating that optimizing a single persistence scale does not generalize well to broader persistence-aware reliability objectives. In contrast, both uniform and exponential weighting schemes achieve smaller σmin . In particular, exponential weighting places greater emphasis on long violation windows, thereby encouraging the learned policy to suppress persistent AoI violations more aggressively. This observation is consistent with the earlier C-AVR vector analysis, where exponential weighting achieved the most significant improvements for large persistence lengths. Overall, the proposed tail persistence index σmin offers a complementary guarantee-oriented perspective by explicitly
D
4 5 ' 4 1 ' 4 1
6 \ V W H P 6 R X U F H 1 X P E H U M
0 L Q L P X P 9 L R O D W L R Q : L Q G R Z σmin
0 L Q L P X P 9 L R O D W L R Q : L Q G R Z σmin
13
E
W 2 Q H K R
8 Q L I R U P Q H Q W L D O β Q H Q W L D O β Q H Q W L D O β R R R ( [ S ( [ S ( [ S
Fig. 8. Tail persistence index σmin under heterogeneous wireless environments: (a) comparison of σmin between QR-D3QN and D3QN as the number of sources M increases; (b) σmin of QR-D3QN under different weighting schemes with M = 10. For each source m, the packet generation probability pm g and transmission success probability pm s are independently sampled from [0.6, 0.8]. kmax = 9 and ϵ̂ = 0.05; for the one-hot weighting scheme in (b), ko = 9.
quantifying the minimum persistence scale at which a target reliability requirement is satisfied. The results further confirm that distributional reinforcement learning is particularly effective at improving tail-persistence guarantees in heterogeneous wireless status update systems. VI. C ONCLUSION We have investigated a reliability-aware scheduling for wireless status update systems, emphasizing the limitations of conventional metrics such as average AoI and AVR in capturing persistent information staleness. Specifically, we put forth the C-AVR vector, which characterizes AoI violation probabilities across multiple persistence scales, along with a weighted formulation that enables flexible prioritization of different violation durations, thereby providing a unified and fine-grained representation of timeliness reliability. Optimizing the weighted C-AVR objective is challenging due to the dominance of rare but high-impact long violation sequences and the resulting skewed return distributions. To tackle this problem, we formulated the scheduling task as a CMDP and developed a distributional reinforcement learning approach based on QR-D3QN. By modeling a quantile-based return distribution rather than only a scalar expectation during training, the proposed method provides richer value-estimation signals for trajectories associated with persistent violations, while satisfying transmission cost constraints. Extensive simulations validate both the proposed C-AVR vector framework and the distributional learning approach. The results show that distributional methods consistently outperform expectation-based baselines in optimizing weighted CAVR, with gains becoming more significant as the objective emphasizes long violation windows. Moreover, a detailed percomponent analysis further reveals that these improvements extend across persistence scales and are particularly significant for large k, confirming the effectiveness of distributional value modeling in suppressing prolonged violation sequences. The comparison across weighting schemes indicates that exponential weighting enhances tail suppression. Overall, these results
establish the C-AVR vector as a principled framework for persistence-aware reliability evaluation and highlight the benefit of distributional value learning when system performance is strongly affected by rare but persistent events. A PPENDIX A D RIFT-P LUS -P ENALTY (DPP) P OLICY FOR W EIGHTED C-AVR S CHEDULING This appendix presents the drift-plus-penalty (DPP) policy used in the simulations, which serves as a model-based heuristic benchmark derived from Lyapunov optimization principles. Unlike the proposed DRL approaches, the DPP policy exploits explicit system dynamics and performs online scheduling via a one-step optimization rule without value-function learning. The weighted C-AVR objective characterizes the long-term persistence behavior of consecutive AoI violations across multiple temporal scales. To apply Lyapunov optimization, this objective must first be rewritten as an equivalent timeaverage penalty process. Recall that the violation counter v(t, m) records the number of consecutive AoI violations of source m at slot t. Define H(n) =
kX max
wk 1[n ≥ k].
(32)
k=1
Using the equivalence 1[Vtk (m)] = 1[v(t, m) ≥ k], where Vtk (m) denotes the k-slot consecutive violation event defined in the main paper, the instantaneous weighted persistence penalty can be written as M
g(t) =
1 X H(v(t, m)). M m=1
(33)
This expression measures the aggregate persistence-aware reliability penalty incurred at slot t. Since H(v(t, m)) accumulates contributions from all persistence scales up to the current violation length, minimizing the long-term average of g(t) suppresses both frequent short violations and prolonged violation runs.
14
Since kmax is finite and all terms are bounded, the finite summation over k, expectation, and time-average (lim sup) can be interchanged without affecting the asymptotic value. In addition, the difference between denominators T and T −k +1 is a boundary term that vanishes as T → ∞. Therefore, minimizing the long-term average of g(t) is asymptotically equivalent to minimizing the weighted C-AVR objective, i.e., T
Ψ̄ = lim sup T →∞
1X E[g(t)]. T t=1
(34)
To enforce the average transmission cost constraint in (9), we introduce the virtual queue Z(t + 1) = max {Z(t) + c(t) − ηmax , 0} .
(35)
(43) into (44) and removing terms independent of the action, the per-slot DPP decision is a∗t = arg
min
a∈{0,1,...,M }
{Z(t)1[a ̸= 0] + VDPP E [g(t + 1) | st , a]} . (45)
Unlike exact dynamic programming approaches, the DPP method only minimizes a one-step conditional surrogate rather than the full long-term objective. The resulting policy should therefore be interpreted as a heuristic online control benchmark derived from Lyapunov optimization principles. We next derive the corresponding scheduling index. Suppose source m is considered for transmission at slot t. If no successful update occurs, the next-slot receiver-side AoI becomes
where c(t) = 1[at ̸= 0] and Z(1) = 0, following the time indexing t = 1, 2, . . . used in the main paper. If Z(t) is meanrate stable, then the average transmission-cost constraint is satisfied. Indeed, from (35),
If the transmission succeeds, the next-slot receiver-side AoI becomes
Z(t + 1) ≥ Z(t) + c(t) − ηmax .
(36)
drm,S = min{∆s (t, m) + 1, ∆max },
(37)
where ∆r (t, m) and ∆s (t, m) denote the receiver-side and transmitter-side AoI, respectively. The corresponding next-slot violation counters are ( min{v(t, m) + 1, kmax }, drm,F > ζ, (48) Fm = 0, drm,F ≤ ζ,
Summing over t = 1, . . . , T gives T
Z(T + 1) − Z(1) 1X c(t) ≤ ηmax + . T t=1 T
Therefore, if limT →∞ Z(T )/T = 0, then η ≤ ηmax . The virtual queue accumulates deviations from the desired average transmission budget. If the scheduler transmits excessively over time, Z(t) increases, thereby discouraging future transmissions. Stabilizing the virtual queue therefore approximately enforces the long-term transmission cost constraint. Let the Lyapunov function be 1 2 Z (t). 2 The one-slot conditional Lyapunov drift is defined as L(Z(t)) =
∆(t) = E [L(Z(t + 1)) − L(Z(t)) | st , Z(t)] .
(38)
(39)
From (35), we have 2
Z 2 (t + 1) ≤ (Z(t) + c(t) − ηmax ) 2
(40) 2
= Z (t) + (c(t) − ηmax ) + 2Z(t) (c(t) − ηmax ) . (41) Since c(t) ∈ {0, 1}, there exists a finite constant BZ such that 1 2 (c(t) − ηmax ) ≤ BZ . 2
(42)
∆(t) ≤ BZ + Z(t)E [c(t) − ηmax | st , Z(t)] .
(43)
Therefore,
Following the Lyapunov optimization framework, DPP minimizes an upper bound on the one-slot drift-plus-penalty function ∆(t) + VDPP E [g(t + 1) | st , Z(t)] ,
(44)
where VDPP > 0 controls the tradeoff between the weighted C-AVR penalty and the virtual-queue pressure. Substituting
drm,F = min{∆r (t, m) + 1, ∆max }.
(46)
(47)
and ( Sm =
min{v(t, m) + 1, kmax }, 0,
drm,S > ζ, drm,S ≤ ζ.
(49)
where ζ is the predefined AoI violation threshold. Equation (48) corresponds to the case where the violation process continues without successful refresh. In contrast, (49) captures the post-update evolution after successful transmission. Importantly, a successful transmission does not necessarily terminate the violation process because the delivered packet itself may already be stale, i.e., ∆s (t, m) + 1 > ζ. In such cases, the consecutive violation counter continues to increase even after successful delivery. A successful update therefore reduces the next-slot weighted persistence penalty from H(Fm ) to H(Sm ). Taking the transmission success probability pm s into account, the expected persistence reduction achieved by scheduling source m is Im (t) = pm s [H(Fm ) − H(Sm )] .
(50)
The quantity Im (t) can therefore be interpreted as a persistence-aware scheduling index that measures the expected reduction in future weighted violation severity achieved by transmitting source m. The source providing the largest expected reduction in weighted persistence penalty is selected as m∗ = arg max Im (t). (51) m∈{1,2,...,M }
Under the drift-plus-penalty rule in (45), scheduling source m∗ is preferred to remaining idle whenever VDPP Im∗ (t) > Z(t). M
(52)
15
Algorithm 2 DPP Policy for Weighted C-AVR Scheduling Input: Weight vector W = (w1 , . . . , wkmax ), threshold ζ, AoI bound ∆max , cost budget ηmax , success probabilities {pm s }m∈{1,2,...,M } , and DPP parameter VDPP after absorbing the constant factor 1/M . 1: Initialize Z(1) = 0, ∆s (0, m) = 0, ∆r (1, m) = 1, and v(1, m) = 0 for all m ∈ P {1, 2, ..., M } kmax = 2: Construct H(n) = k=1 wk 1 [n ≥ k] for n 0, 1, . . . , kmax 3: for each time slot t = 1, 2, . . . do 4: Update ∆s (t, m) according to packet arrivals for all m ∈ {1, 2, ..., M } 5: Observe st and the virtual queue Z(t) 6: for each source m ∈ {1, 2, ..., M } do 7: Compute Fm using (48) 8: Compute Sm using (49) 9: Compute Im (t) = pm s (H(Fm ) − H(Sm )) 10: end for 11: Select m∗ = arg maxm∈{1,2,...,M } Im (t) 12: if VDPP Im∗ (t) > Z(t) then 13: Schedule source m∗ , i.e., set at = m∗ 14: else 15: Stay idle, i.e., set at = 0 16: end if 17: Execute at and observe the transmission outcome if a source is scheduled 18: Update ∆r (t + 1, m) and v(t + 1, m) for all m ∈ {1, 2, ..., M } 19: Update Z(t + 1) by (35) 20: end for This condition admits an intuitive interpretation: transmission is performed only when the expected persistence reduction outweighs the current virtual congestion level induced by the transmission budget constraint. Since the factor 1/M is a positive constant independent of the action, it can be absorbed into the control parameter VDPP . The resulting DPP scheduling rule becomes ( m∗ , VDPP Im∗ > Z(t), at = (53) 0, otherwise. The resulting DPP policy is summarized in Algorithm 2. In our simulations, we set VDPP = 10. R EFERENCES [1] S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update?” in Proc. IEEE INFOCOM, Mar. 2012, pp. 2731–2735. [2] M. A. Abd-Elmagid, N. Pappas, and H. S. Dhillon, “On the role of age of information in the internet of things,” IEEE Commun. Mag., vol. 57, no. 12, pp. 72–77, Dec. 2019. [3] Y. Sun, I. Kadota, R. Talak, E. Modiano, and R. Srikant, Age of Information: A New Metric for Information Freshness. Morgan & Claypool, 2019. [4] A. Zakeri, M. Moltafet, M. Leinonen, and M. Codreanu, “Minimizing the aoi in resource-constrained multi-source relaying systems: Dynamic and learning-based scheduling,” IEEE Trans. Wireless Commun., vol. 23, no. 1, pp. 450–466, Jan. 2024. [5] C. Li, Q. Liu, Y. T. Hou, W. Lou, and S. Kompella, “Aequitas: A 5G scheduler for minimizing outdated information in IoT networks,” IEEE Internet Things J., vol. 11, no. 13, pp. 23 322–23 335, Jul. 2024.
[6] R. D. Yates, Y. Sun, D. R. Brown, S. K. Kaul, E. Modiano, and S. Ulukus, “Age of information: An introduction and survey,” IEEE J. Sel. Areas Commun., vol. 39, no. 5, pp. 1183–1210, Mar. 2021. [7] H. Pan, J. Feng, T.-T. Chan, V. C. Leung, and J. Li, “Timely status update in relay-assisted cooperative communications,” IEEE Trans. Veh. Tech., vol. 72, no. 12, pp. 15 745–15 761, Dec. 2023. [8] Y. Ji, Y. Lu, X. Xu, and X. Huang, “Age-optimal packet scheduling with resource constraint and feedback delay,” IEEE Trans. Commun., vol. 72, no. 7, pp. 4041–4054, Jul. 2024. [9] Q. Chen and et al., “Average aoi minimization with directional charging for wireless-powered network edge,” IEEE Trans. Mobile Comput., vol. 24, no. 6, pp. 4889–4906, Jun. 2025. [10] İ. Kahraman, A. Köse, M. Koca, and E. Anarim, “Age of information in internet of things: A survey,” IEEE Internet Things J., vol. 11, no. 6, pp. 9896–9914, Mar. 2024. [11] I. Kadota, A. Sinha, and E. Modiano, “Scheduling algorithms for optimizing age of information in wireless networks with throughput constraints,” IEEE/ACM Trans. Netw., vol. 27, no. 4, pp. 1359–1372, Aug. 2019. [12] J. P. Champati, H. Al-Zubaidy, and J. Gross, “Statistical guarantee optimization for age of information for the D/G/1 queue,” in Proc. IEEE INFOCOM Workshops, Apr. 2018, pp. 130–135. [13] R. Devassy, G. Durisi, G. C. Ferrante, O. Simeone, and E. UysalBiyikoglu, “Delay and peak-age violation probability in short-packet transmissions,” in Proc. IEEE ISIT, Jun. 2018, pp. 2471–2475. [14] H. Pan and S. C. Liew, “Information update: TDMA or FDMA?” IEEE Wirel Commun Lett., vol. 9, no. 6, pp. 856–860, Jun. 2020. [15] D. Zhang and et al., “Information freshness and timeliness analysis in the finite blocklength regime for mission-critical applications,” IEEE Trans. Commun., vol. 73, no. 12, pp. 14 458–14 468, Dec. 2025. [16] X. Zhou, I. Koprulu, and A. Eryilmaz, “Age-based multi-channelscheduling under constraints: Optimal and online designs,” IEEE Trans. Netw., vol. 33, no. 1, pp. 51–64, Feb. 2025. [17] L. Scheuvens, T. Hößler, P. Schulz, N. Franchi, A. N. Barreto, and G. P. Fettweis, “State-aware resource allocation for wireless closed-loop control systems,” IEEE Trans. Commun., vol. 69, no. 10, pp. 6604–6619, Oct. 2021. [18] Y.-P. Hsu, E. Modiano, and L. Duan, “Scheduling algorithms for minimizing age of information in wireless broadcast networks with random arrivals,” IEEE Trans. Mobile Comput., vol. 19, no. 12, pp. 2903–2915, Dec. 2020. [19] X. Zhou, I. Koprulu, and A. Eryilmaz, “Age-optimal multi-channelscheduling under energy and tolerance constraints,” in Proc. IEEE INFOCOM Workshops, Aug. 2023, pp. 1–8. [20] S. S. Vilni, M. Moltafet, M. Leinonen, and M. Codreanu, “Multisource aoi-constrained resource minimization under harq: Heterogeneous sampling processes,” IEEE Trans. Veh. Technol., vol. 73, no. 1, pp. 1084– 1099, Jan. 2024. [21] A. Alwarafy, M. Abdallah, B. S. Çiftler, A. Al-Fuqaha, and M. Hamdi, “The frontiers of deep reinforcement learning for resource management in future wireless HetNets: Techniques, challenges, and research directions,” IEEE Open J. Commun. Soc., vol. 3, pp. 322–365, 2022. [22] F. Fu, X. Wei, Z. Zhang, L. T. Yang, L. Cai, J. Luo, Z. Zhang, and C. Wang, “Age of information minimization for UAV-assisted internet of things networks: A safe actor-critic with policy distillation approach,” IEEE Trans. Netw. Sci. Eng., vol. 11, no. 1, pp. 1265–1276, Feb. 2024. [23] H. V. Hasselt, “Double Q-learning,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 23, Feb. 2010, pp. 2613–2621. [24] Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, “Dueling network architectures for deep reinforcement learning,” in Proc. Int. Conf. Mach. Learn. (ICML), vol. 4, Nov. 2016, p. 2939–2947. [25] M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” in Proc. Int. Conf. Mach. Learn. (ICML), 2017. [26] W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos, “Distributional reinforcement learning with quantile regression,” in Proc. AAAI Conf. Artif. Intell., 2018. [27] M. G. Bellemare, W. Dabney, and M. Rowland, Distributional Reinforcement Learning. MIT Press, 2023. [28] Y. Ma, D. Jayaraman, and O. Bastani, “Conservative offline distributional actor critic,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2021. [29] N. Urpı́, M. C. Campagnolo, S. Curi, and A. Krause, “Risk-averse offline reinforcement learning,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2021.
16
[30] J. Liu, D. Li, and Y. Xu, “Deep distributional reinforcement learningbased adaptive routing with guaranteed delay bounds,” IEEE/ACM Trans. Netw., vol. 32, no. 6, pp. 4692–4706, Dec. 2024. [31] J. Duan, W. Wang, L. Xiao, J. Gao, S. E. Li, C. Liu, Y.-Q. Zhang, B. Cheng, and K. Li, “Distributional soft actor-critic with three refinements,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 5, pp. 3935–3946, May. 2025. [32] Q. Zhang, W. Saad, and M. Bennis, “Distributional reinforcement learning for mmWave communications with intelligent reflectors on a UAV,” in Proc. IEEE Global Commun. Conf., 2020, pp. 1–6. [33] L. Huang, M. Fu, A. Rao, A. A. Irissappane, J. Zhang, and C. Xu, “A distributional perspective on multiagent cooperation with deep reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 3, pp. 4246–4259, Mar. 2024. [34] E. Eldeeb and H. Alves, “Offline and distributional reinforcement learning for wireless communications,” IEEE Commun. Mag., vol. 63, no. 8, pp. 71–76, Aug. 2025. [35] W. Zhang, M. Derakhshani, G. Zheng, and S. Lambotharan, “Constrained risk-sensitive deep reinforcement learning for eMBB-URLLC joint scheduling,” IEEE Trans. Wireless Commun., vol. 23, no. 9, pp. 10 608–10 624, Sept. 2024. [36] M. Neely, Stochastic Network Optimization with Application to Communication and Queueing Systems, ser. Synthesis lectures on communication networks. Morgan & Claypool Publishers, 2010. [37] E. Altman, Constrained Markov Decision Processes. CRC Press, 1999, vol. 7.