When and Which Sensor to Observe? Timely Tracking of a Joint Markov Source
arXiv:2606.30623v1 [cs.IT] 29 Jun 2026
Ismail Cosandal
Sennur Ulukus
Abstract—We investigate the problem of remote estimation (at a monitor) of a discrete-time joint Markov process with individual components which can be observed with dedicated sensors. At a given time slot, the monitor has the option of staying idle or sending a pull request to one of the sensors to obtain a partial state value, while the sensors are assumed to have heterogeneous sampling costs. Our goal is to develop a monitor pull policy, i.e., determining when and towards which sensor to send a pull request, in order to minimize a weighted sum of average age of incorrect information (AoII), or in short age, and sampling costs. As the communication model, we assume an erasure channel with a fixed one-slot delay from each sensor to the monitor. In this setting, the monitor does not perfectly know either the state of the process or the age, at any given time. We first obtain a sufficient statistic, namely belief, representing the joint distribution of the age and the current state of the observed process, by using the history of all pull requests and observations. Then, we formulate the optimization problem as a continuous state-space Markov decision process (MDP), namely belief-MDP, for the solution of which we propose two model predictive control (MPC) methods, namely MPC without terminal costs (MPC-WTC), and reinforcement learning MPC (RL-MPC). The effectiveness of the proposed methods is validated by numerical examples. Index Terms—Age of incorrect information (AoII), Markov decision process (MDP), belief MDP, model predictive control (MPC), reinforcement learning (RL).
I. I NTRODUCTION Age of incorrect information (AoII) is a joint mismatch and freshness metric that captures for how long there has been a mismatch between an observed random process and its estimate at a remote monitor [1]. AoII takes into account the dynamics of the source process, and is thus considered to be a semantic metric [1], [2]. Different from other mismatch metrics such as the mean squared error (MSE) or binary freshness (BF) [3], [4], AoII penalizes the duration of incorrect estimation by increasing linearly with time, and the AoII process is reset to zero as soon as the monitor correctly estimates the process. AoII is fundamentally different from other information freshness metrics derived from the age of information (AoI) [5], [6], since the AoI process can only drop upon the reception of a status update packet, while AoII can be reset not necessarily only with a status update, but with an update of the estimate of the monitor, or upon a state transition at the source to the estimation value at the monitor. I. Cosandal and S. Ulukus are with the University of Maryland, College Park, MD, USA. N. Akar is with Bilkent University, Ankara, Türkiye. Corresponding author: S. Ulukus (email: [email protected]). This work is done when N. Akar was on sabbatical leave as a visiting professor at the University of Maryland, MD, USA.
Nail Akar
action selection pull request channel
monitor
no sampling
sensor-1
1 2
4
sensor-2 erasure channel
3
Fig. 1. The illustration of the pull-based status update system. The two processes that sensors observe may not be individually Markov, but their joint process is Markov with its transition probabilities known to the monitor. Statespace of the joint process x(t) is ordered as {(a, α), (b, α), (a, β), (b, β)} with the index process xt taking the index of the joint state, as its value.
In this paper, we consider a unique scenario involving multiple sensors each of which observing a single component (i.e., partial state) of a joint Markov process, and a monitor which aims to minimize a weighted combination of the timeaveraged AoII (equivalently, the mean of the steady-state AoII, denoted by MAoII) and sampling costs, by either staying idle or selecting one of the sensors to observe a partial state, at each time slot. For the communication model, we assume that pull requests reach the sensors instantaneously on the reverse channel from the monitor towards the sensors. However, for the forward channel from the sensors towards the monitor, we assume that transmissions take place in one-slot with a given erasure probability. As a simple motivating example, consider an object performing a random walk on a 2D grid, where two sensors observe the object’s x and y components separately, and a monitor remotely track the location of the object. In this example, the monitor should make decisions on when and which sensor to observe, by considering the system dynamics and all previous observations, to track the object’s location, under energy consumption constraints. As another example, consider a joint fire/temperature process which is tracked by a remote monitor using two sensors; one sensor observing the temperature level (as low, moderate, high), and the other sensor observing the event of a fire (as fire, no fire). In this example, we assume that the fire event only occurs if the temperature is high, thus the individual events may not be Markov processes individually, but the joint process can be assumed to be Markov. As a result of the communication model with delays, partial observability, and errors in the forward channel, the monitor can never be exactly sure of the exact state of the process, and hence, the AoII value of its estimate. However, it is possible for the monitor to obtain the so-called belief
which corresponds to the joint distribution of the state of the Markov process and the AoII [7] based on all previous observations and actions. In addition, the monitor estimates the original process using the maximum a-posteriori (MAP) rule, which is also a function of the belief. This estimation method affects the distribution of the AoII, thus the transition probabilities among the beliefs and the cost of a belief. This relation hinders us from using the partially observable MDP (POMDP) formulation directly [8]. On the other hand, alternative formulations are proposed as extensions of the POMDP framework for problems where the cost or the state transition probabilities depend on the belief [9]–[11]. These extensions of POMDPs [12], and the POMDP itself [13], can be expressed with an equivalent continuous-state MDP where states are the distributions of unobserved states, which gives rise to a formulation called the belief-MDP. We will now overview the existing works on the solution of belief-MDPs which are most relevant to the current paper. The state-space of the belief-MDP, which is termed as belief space, contains uncountably many elements, and therefore it is not possible to obtain the value function for each belief state by using dynamic programming from the belief-MDP representation. One method to cope with the continuous belief space is the discretization of the belief space [14] which however suffers from low accuracy and high complexity for high and low discretization step sizes, respectively [15]. In some cases [16], [17], only a finite number of beliefs can be visited; hence, it is possible to convert the belief-MDP to an MDP with a finite state-space, which does not apply to the problem of interest here. Although an uncountable number of belief values can be visited in the current paper’s setting, the number of possible next beliefs from an individual belief in a finite horizon is countable due to the finite-dimensional observation and action spaces. In this paper, we utilize model predictive control (MPC) which evaluates all possible outcomes for a finite number of steps and adds terminal costs of states in the final step [18]. Subsequently, MPC obtains the action sequence that minimizes the expected cost from all calculated trajectories, and applies the first action from this sequence. This process is then repeated at each slot. MPC has successfully been used to solve belief-MDPs in several recent works owing to its ability to decide based on future costs in a finite horizon [19]–[21]. Regarding terminal costs, we consider two approaches. The first approach is MPC without terminal cost (MPCWTC) which considers all terminal costs to be equal (zero in particular), and the first action of the path which minimizes the expected cost on a finite horizon is taken [22], [23]. The main advantage of this method is ease of implementation since it does not require offline learning. However, a long time horizon may be required for improving performance at the expense of increased complexity of evaluating all trajectories. The second approach is based on reinforcement learning MPC (RL-MPC) where terminal costs are approximated with the aid of RL [24]. RL-MPC utilizes MPC-WTC to learn terminal costs in the first iteration, and increases its approximation horizon iteratively in
a similar fashion to fitted Q-learning [25]. This method allows us to find an action sequence that minimizes the cost for a larger horizon with a lower complexity compared to MPCWTC. On the other hand, approximation errors encountered at previous iterations may lead to performance degradation [26], and offline learning is required before run-time as opposed to MPC-WTC. The main contributions of our paper can be summarized as follows: • We investigate a timely tracking problem where a joint Markov process is observed partially only and with delays, at an observation instant, which to the best of our knowledge, has not been explored before, in the literature. • Considering heterogeneous sampling costs for the sensors, we aim to obtain a monitor pull policy which minimizes the weighted sum of MAoII and overall sampling costs. For this purpose, we adopt the method for calculating the joint distribution of age and state in [7] for the case of partial observations, and formulate the optimization problem as a belief-MDP. • The proposed belief-MDP has a continuous state-space for which the use of conventional dynamic programming techniques is infeasible. To solve the belief-MDP problem, we propose two MPC methods, namely MPCWTC and RL-MPC, and validate the effectiveness of the two proposed approaches, in terms of performance and computational complexity, through extensive numerical experimentation. The remainder of the paper is organized as follows. In Section II, related work from the literature is presented. The system model is given in Section III. The optimization problem is formulated as a belief-MDP in Section IV, and Section V is dedicated to the proposed MPC solutions. Numerical results are presented in Section VI, and the conclusions are given in Section VII. II. R ELATED W ORK POMDP is an extension of MDP [8] where the action taker cannot directly observe the state of the problem, but it can estimate the likelihood of the state from partial observations, which is called the belief, and the policy is defined as a function of the belief [27], [28]. In this formulation, the underlying process governing state transitions is a Markov process, and state transition probabilities and the cost function depend only on the state and the action. The study [9] further extends POMDP to address problems of belief-dependent cost, namely ρPOMDP, and demonstrates that if the cost function is convex, existing POMDP solutions can be adapted for ρPOMDP. In addition, the study [10] considers a scenario in which an agent aims to maintain an accurate belief alongside its main goal, and it defines this type of problem as POMDP with informational rewards (POMDP-IR). In this work, they propose expanding the action space with the estimation of the agent, and the cost function is modified to penalize/reward the incorrect/correct information. Additionally, in [11] it is shown that these two formulations can be converted to each other.
In the information freshness literature, correlated sources are studied mainly in two settings. In the first setting, each sensor observes a set of processes, with possible overlaps, and the sensors update a single monitor [29], [30]. Therefore, samples from different sensors may include asynchronous information about the same process, and received components have different freshness levels. In the second setting, there is a correlation between the processes observed by different sensors, thus getting an update from a sensor may contain partially fresh information about other processes [31]–[38], e.g., multiple cameras may surveil an overlapping area [31]. Additionally, in [39], the correlation is defined based on the spatial location of sources. Studies in [31]–[39] consider AoI of a correlated source as a weighted sum of the freshness of the corresponding sensors. In other words, when a source is sampled, AoI of other partially observed sources decreases proportionally with the correlation of the sampled source. In contrast, a metric called age of correlated information (AoCI) has been proposed in [31], and further studied in [36], [37], which considers a scenario where all sensors that include partial information about a process should be sampled in order to obtain fresh information about a process. Another related work [40] considers a single source observed by multiple sensors, where each sensor observes the source process with different state-dependent accuracy. In this work, a time-invariant mismatch metric is minimized. We note that the setting of this paper involving monitoring partial states of a joint Markov process has not been studied before, to the best of our knowledge. In a pull-based scheme for a sensor network, the monitor can only be aware of the freshness of the sensor that has just been sampled, thus, POMDP formulation is a useful tool to model these types of problems. In [16], [17], [41]–[44], the monitor estimates the freshness of sensors that were not sampled at that slot based on its observations and obtains a policy accordingly. In [41], [42], [43], and [45], sub-optimal policies are obtained via the index policy, the myopic policy, deep reinforcement learning, and the particle filter, respectively. The authors of [16] and [17] convert the POMDP into a fully-observable discrete-space MDP problem, and obtain an optimum policy. Furthermore, [46] and [47] consider a pushbased scheme where sensors estimate the freshness level of others to decide when to transmit. Reference [48] investigates a multi-agent optimization problem where agents should have fresh and accurate information about other agents’ status to take better actions. Consequently, [48] utilizes the AoII metric and proposes to use a decentralized POMDP (Dec-POMDP) formulation which aims to minimize the AoII of each agent maintained for other agents’ states. Similarly, in [49], uncrewed aerial vehicles (UAV) coordinate by estimating the states, e.g., speed, location, etc., of other UAVs, to collect fresh data from the ground sensors. In early studies, AoII is investigated for symmetric Markov chains, and a single threshold policy is proposed to minimize the average AoII [50]–[52]. On the other hand, in our previous works [53]–[55], we have shown that if the source process is
a general asymmetric Markov chain, a simple threshold policy would not be guaranteed to perform optimally. More specifically, in [54], [55] it is shown that the optimum transmission policy should take into account all the estimation, source, and age values. The most commonly used remote estimation rule in the AoII literature is the so-called martingale estimator [56], which estimates the process with the value encapsulated in the latest received status update from the information source [1], [50], [51], [57], [58]. However, this estimation rule is not applicable to our case, since we only get a partial observation with a status update. In a recent work [7], a MAP rule is proposed for the estimation that allows the monitor to update its estimation with the most likely state, and it is shown that the MAP rule is superior to the martingale estimator for AoII minimization for Markov sources. III. S YSTEM M ODEL We consider a time-slotted remote estimation system with K sensors, where sensor-k, k ∈ K = {1, 2, . . . , K}, observes a discrete-time process xt (k) ∈ Xk , |Xk | = Nk . Our main assumption is that these K individual partial processes are not necessarily Markov, but the joint collection of these processes denoted by xt = {xt (1), xt (2), . . . , xt (K)},
(1)
is a discrete-time joint Markov process (or Markov chain). We denote the state-space of the joint process by X . For convenience, we use an index process xt ∈ N = {1, 2, . . . , N } that takes the index (or order) of the state of the original process xt as its value, after a suitable enumeration of the joint states of the original process xt ; see Fig. 1 for an example scenario. The index process xt evolves according to a time-homogeneous transition matrix P = {pij }, which is a-priori known, where pij denotes the transition probability from index-state i to index-state j. The remote monitor’s goal is to track the joint process xt , equivalently its corresponding index process xt . At each time slot, the monitor decides whether to observe one of the sensors or stay idle. We denote the action taken by the monitor at time t with at = k, k ∈ K∪{0}, where the action 0 refers to staying idle, and action k > 0 refers to sending a pull request to sensor-k which is assumed to take place instantaneously. With the generate-at-will (GAW) principle, if at = k ̸= 0, then the sensor-k transmits the value of the sampled process xt (k) via the forward channel encapsulated in an information packet, with the sampling and transmission cost denoted by µk > 0. The forward channel is assumed to be an erasure channel with one-slot transmission time. In other words, the packet containing the value sampled from sensor-k at time t, xt (k), is subject to a time-homogeneous transmission error with erasure probability ρe , i.e., the information packet is received at time t + 1 by the monitor with successful transmission probability ρs = 1 − ρe .
We denote the observation at the monitor at time t with ot when action at−1 = k, k ∈ K, is taken at the previous time slot as, ( xt−1 (k), w.p. ρs , (2) ot = ∅, w.p. ρe , where ot = xt−1 (k) denotes the received partial state value from sensor-k and ∅ represents the channel error in the previous time slot. Additionally, the observation space for action at = k indicates all possible observations from sensor-k, thus it is defined as O(k) = Xk ∪ {∅}. On the other hand, if the monitor stays idle, i.e., at = 0, no pull request is sent to any sensor at time t. In this case, the observation space for at = 0 is O(0) = {∅}. Thus, at the next time slot, the observation will be ot = ∅ with probability (w.p.) one. Additionally, the sampling cost corresponding to staying idle is assumed to be zero, i.e., µ0 = 0. The described system model is illustrated in Fig. 1. From all previous actions and received observations, the monitor can calculate the conditional probability distribution of the index process as a row vector of size N , πt = πt (1) πt (2) · · · πN (t) , (3) where πt (i) corresponds to the conditional probability of the index process being in index-state i, πt (i) = P(xt = i|Ht ),
(4)
and Ht is defined as the history of all observations and actions until time t (excluding the action at time t), Ht = {o1 , a1 , . . . , at−1 , ot }.
(5)
The monitor estimates the index process using the MAP rule, x̂t = arg max πt ,
(6)
and the mismatch is quantified by AoII which progressively penalizes the incorrect estimation as the error stays. Particularly, the AoII process at time slot t, denoted by AoIIt , is a function of both the actual state xt and its MAP estimation at the monitor x̂t , and AoIIt evolves according to the following equation, ( AoIIt−1 + 1, x̂t ̸= xt , AoIIt = (7) 0, x̂t = xt .
Finally, we define the optimization problem that aims to minimize the weighted sum of MAoII and ASC as, min MAoIIϕ + λ · ASCϕ , ϕ∈Φ
(10)
where the coefficient λ ≥ 0 represents the relative weight of average sampling cost to average age cost, ϕ is any monitor pull policy, and Φ is the set of all such policies. It is worthwhile to note that one can use the solution of this unconstrained problem for the solution of the MAoII minimization problem under average sampling cost constraints, with a proper selection of the coefficient λ using an appropriate line search algorithm as in [7]. IV. P ROBLEM F ORMULATION The monitor aims to find the optimum policy described in (10) based on its previous actions and the observations it receives. It is important to note that the monitor will never have a perfect estimate of the joint process xt due to communication delays, erasures, and the partial observability of the state, and consequently, it will not perfectly know the instantaneous value of AoIIt for any value of t. Instead, the monitor keeps track of the joint distribution of index-state and AoII by using all observations it has received so far, which is termed as belief. In this section, we utilize the belief to formulate the optimization problem as a belief-MDP. For this purpose, we first elaborate on the estimation update method introduced in the previous section. Then, we define the belief for this problem, and explain how it evolves with the observations. Lastly, the belief-MDP formulation is developed using the transition probabilities among beliefs under an action, and the expected cost of a belief-action pair. A. Estimation Update In this subsection, we will describe how the estimation is updated as observations are made. For this purpose, in the first step, we assume that πt is available to the monitor, and the new observation ot+1 arrives. From (2), if the packet is not lost during transmission, the new observation includes information about x̂t , thus πt should be updated accordingly. Let us define the updated probability vector π̂t = π̂t (1) π̂t (2) · · · π̂N (t) , (11) where
We are interested in two types of costs in our setting. The first cost is MAoII, which is the time average of AoII due to the ergodicity of the corresponding AoII process, T
1X AoIIt . T →∞ T t=1
MAoII = lim
(8)
The second cost is the average sampling cost (ASC), which can be expressed as a function of actions, T
1X µat . T →∞ T t=1
ASC = lim
(9)
π̂t (i) = P(xt = i|Ht+1 ).
(12)
The following relation holds between πt and π̂t for a given observation ot+1 ,
1ot+1 ,i πt (i) , j=1 1ot+1 ,j πt (j)
π̂t (i) = PN
(13)
where 1ot+1 ,i = P(ot+1 |xt = i) is a binary function which indicates whether the observation ot+1 is possible from xt . From the example process in Fig. 1, 1a,i takes the value 1 when i = 1 and i = 3, because both states (a, α), (a, β) are the only states possessing partial information ‘a’ from the
observation, and their indices are 1 and 3, respectively. Notice that an empty observation ot = ∅ is possible for each state which makes π̂t (i) = πt (i) for all values of i when ot = ∅. In the second step, we obtain πt+1 from π̂t as follows, πt+1 = π̂t P .
1 2 0 1.0
3 4
1 2 0 0.6
3 4
1 2 0 0.85
3 4
1 2 0 0.54
1
0.15
3 4
1
1
1
0.09 .09 0.17
2
2
2
2
0.09 .01 0.01
3
3
3
3
0.1 0.1 0.2
(14)
Once we obtain πt+1 , the estimate x̂t can be obtained from (6).
Fig. 2. Evolution of the belief from an initial belief b0 for the process in Fig. 1 with the transition matrix in (19).
B. Belief Evolution In this subsection, we first define the belief bt as the following joint distribution bt (i, δ) = P(xt = i, AoIIt = δ|Ht ),
(15)
and then describe how the belief evolves as observations are made. Additionally, in this paper, we truncate the AoII values by ∆ for practical reasons. To describe the evolution of a belief, we first define the updated belief b̂t as b̂t (i, δ) = P(xt = i, AoIIt = δ|Ht+1 ), 1ot+1 ,i bt (i, δ) = P∆ PN . d=0 j=1 1ot+1 ,j bt (j, d)
(16) (17)
As (7) indicates, if a state transition is incurred to the estimated state, AoII resets to zero. Otherwise, the AoII process is incremented by one. In addition, we assume that the AoII value does not further increase upon reaching ∆ that comes from truncation. As a result of this, the belief at time t + 1 can be calculated using the updated belief, b̂t , and the source dynamics as follows, max πt+1 , i = x̂t+1 , δ = 0, N X b̂t (j, δ−1)pji , i ̸= x̂t+1 , 0 < δ < ∆, j=1 N X b̂t (j, δ−1)pji bt+1 (i, δ) = j=1 i ̸= x̂t+1 , δ = ∆, N X + b̂ (j, δ)p , t ji j=1 0, o.w. (18) Notice that when the estimation is correct at time t + 1, i.e., i = x̂t+1 , AoIIt+1 becomes zero regardless of its previous value, and its probability can be obtained from (14). Example: In this example, we consider again the process considered in Fig. 1, with the transition matrix 0.6 0.1 0.1 0.2 0.2 0.6 0.1 0.1 P = (19) 0.1 0.2 0.6 0.1 . 0.1 0.1 0.2 0.6 We assume that at time t = 0, the process is in state (a, α), i.e., the index process is in state 1, and this information is available to the monitor. Hence, π0 (1) = 1, π0 (i) = 0 for i ̸= 1. Also
notice that b0 (1, 0) = 1, b0 (i, δ) = 0 for (i, δ) ̸= (1, 0). We furtherassume that action 0 is taken at time t = 0. Therefore, π1 = 0.6 0.1 0.1 0.2 , and from (6), the estimation at the monitor at time t = 1 is x̂1 = 1. Meanwhile, the AoII process will remain at zero with probability 0.6, or it increases to 1 with probability 0.4, which makes E[AoII1 ] = 0.4, and the joint state-AoII belief b1 is obtained as in Fig. 2. Now, consider that the monitor decides to send a pull request to sensor-1 at time t = 1, and the observation o2 = a is successfully received at time t = 2, which means that the unobserved state at t = 1 was either (a, α) or (a, β). Thus, the updated belief b̂1 is obtained by (17) as b̂1 (1, 0) = 0.85, b̂1 (3, 1) = 0.15, and b̂1 (i, ∆) = 0, (i, ∆) ̸= (1, 0), (3, 1). From that observation, the monitor can construct b2 from (18), as illustrated in Fig. 2. A more detailed example with other possible observations is illustrated in Fig. 3. It is well-known that belief bt contains all necessary information about history Ht regarding decision making, and thus, it is a sufficient statistic [27]. Moreover, the new belief bt+1 conditioned on the entire history of actions and observations until time t + 1 can exactly be obtained from the previous belief bt along with the new observation ot using the equations (17) and (18) for which we say bt+1 = f (bt , ot+1 ), i.e., the current belief and the next observation (outcome of the current action) are mapped to the next belief, which is represented by the mapping (i.e., the function) f . In the previous example illustrated in Fig. 2, b2 is obtained from b1 and the observation o2 = a, thus b2 = f (b1 , a). C. Belief-MDP Formulation For the belief-MDP formulation, we first define T (bt+1 , at , bt ) as the transition probability from bt to bt+1 for a given action at . Although the belief space includes continuous-valued elements, the number of possible next beliefs is limited by the number of possible observations. In other words, for any belief bt and action at , there are |O(at )| possible next beliefs bt+1 . Additionally, transition probability between bt and bt+1 = f (bt , ot+1 ) for an action at is equivalent to the probability of the observation ot+1 for given bt and at , since f (bt , ot+1 ) is a deterministic mapping. This can be expressed as T (f (bt , ot+1 ), bt , at ) =P(bt+1 = f (bt+1 , ot+1 )|bt , at ) (20) =P(ot+1 |bt , at ).
(21)
The observation ot+1 = ∅ is included in the observation space of any action, including at = 0. If the action is at ̸= 0, the observation ot+1 = ∅ denotes the erasure on the channel, and it occurs with the probability of ρe . Additionally, if the monitor stays idle with action at = 0, it always observes ot+1 = ∅. Thus, we can express these probabilities as ( ρe , at ̸= 0, P(ot+1 = ∅|at , bt ) = (22) 1, at = 0. Furthermore, the observation ot+1 ̸= ∅ indicates the following: i) monitor sends the pull request to one of the sensors at time t, i.e. at = k ̸= 0, ii) by (2) the observation ot+1 is included the observation space of the action, i.e., ot+1 ∈ O(k), iii) transmission succeeds with probability ρs , iv) the observation at time t + 1 is the kth component of the process at time t, i.e., ot+1 = xt (k). We can collect all these conditions with the following equation: P(ot+1 ̸= ∅|at , bt ) = ρs
N X
Fig. 3. Illustration of all possible outcomes b̂1 , b2 and the probability of the corresponding observations when the action a1 = 1 is chosen for ρe = (1 − ρs ) = 0.1.
it with a function of bt as ∆ N X X Γ(bt ) = E[AoIIt ] = δ P(xt = i, AoIIt = δ|Ht ), (25) δ=0
1ot+1 ,i P(xt = i|Ht+1 ), at ̸= 0, ot+1 ∈ O(at ),
i=1
(23) Notice that statements (i) and (ii) are denoted with at ̸= 0, ot+1 ∈ O(at ), respectively. The successful transmission statement in (iii) adds the multiplication term ρs on the probability, and it is denotedPwith ot+1 ̸= ∅. The last statement in (iv) is N expressed with i=1 1ot+1 ,i P(xt = i|Ht+1 ) which is equivPN PN P∆ alent to i=1 1ot+1 ,i πt (i), or i=1 1ot+1 ,i δ=0 bt (i, δ). Finally, we can combine (21), (22) and (23) to obtain the function T as T (f (bt , ot+1 ), bt , at ) PN P∆ ρs i=1 δ=0 1ot+1 ,i bt (i, δ), ot+1 ∈ O(at ), ot+1 ̸= ∅, ρ , ot+1 = ∅, at ̸= 0 e = 1, at = 0, 0, o.w. (24) Notice that the conditions ot+1 ∈ O(at ), and ot+1 ̸= ∅ inherently indicate that the action is non-zero, thus the condition at ̸= 0 is simplified from the first case of the equation. In the example in Fig. 3, the action a1 = 1 is taken at time t = 1. This action has the observation space O(1) = {∅, a, b}. The observations ‘a’, ‘b’ include partial information for states x1 = 1, 3, and x1 = 2, 4, respectively, thus 1a,i = 1 for i = 1, 3, and 1b,i = 1 for i = 2, 4. From the given belief b1 , the transmission probabilities to all possible b2 values are calculated by (24) for the corresponding observation, and denoted under the corresponding arrows. The expected cost of the belief state bt and action at is defined as the sampling cost of the action plus the expected AoII of the next reached belief. First, we can calculate the expected AoII of a belief bt by its definition in (15), and denote
=
∆ X δ=0
δ
N X
i=1
bt (i, δ).
(26)
i=1
Then, the expected cost is defined as c(bt , at ) = X
T (f (bt , ot+1 ), bt , at )Γ(f (bt , ot+1 )) + λµat . (27)
ot+1 ∈O(at )
Finally, we summarize the belief-MDP by using formulations in [59], [60] as a tuple (B, A, T, c, γ): N (∆+1) • The belief space is B = [0, 1] , and the belief of the problem, bt ∈ B is defined in (15). • The action space of the problem is defined as A = K ∪ {0}. • After the reception of an observation, the state distribution is updated using (13) and (14), and the estimation is updated by (6). • Each observation ot+1 forces the belief bt according to bt+1 = f (bt , ot+1 ) with probability T (f (bt , ot+1 ), at , bt ). The derivation of f (bt , ot+1 ) is detailed in the steps (17) and (18), and the transition probability is derived in (24). • The cost of a belief-action pair (bt , at ) is defined through a cost function c(bt , at ) in (27). • It is a common practice to choose the discount factor as γ = 1 [19]–[22], [24], [61]. Hence, this term is omitted in the rest of the paper. V. M ODEL P REDICTIVE C ONTROL The belief-MDP formulation defined in the previous section has a continuous state-space and a discrete action space. Continuous state-space disallows us to find an optimum policy for the optimization problem in (10) via dynamic programming. On the other hand, the discrete action space indicates that for a single belief, there are finite number of next beliefs. Therefore, we utilize the MPC where the monitor searches
input layer
hidden layers
output layer
Fig. 5. Network architecture used for RL-MPC with parameter θ.
Fig. 4. An ℓ-step look-ahead table that starts a belief state and explores all possible belief states on ℓ steps.
the sequence of actions u∗t = [ut (1), . . . , ut (ℓ − 1)], which minimizes the expected cost by starting with a belief bt , and evaluating all possible reachable beliefs using an ℓ-step lookahead table. Then, the monitor applies the first action from the action sequence, i.e., a∗t = u∗t (1). An example of the look-ahead table is given in Fig. 4. The action sequence u∗t is expressed as " ℓ−1 X arg min E c(bt+m , at+m ) + V (bt+ℓ−1 ) at+m ∈A, m∈{0,...,ℓ−1}
m=0
# bt , at , . . . , at+ℓ−1 ,
WTC chooses the action as a∗t = arg min C0,ℓ (bt , a), and the value WTC of C0,ℓ (bt , a) is obtained recursively by Algorithm 1.
Algorithm 1 Cm,ℓ (bk ): The recursive algorithm to calculate the expected cost in the next ℓ − m steps. for ak ∈ A do for ok+1 ∈ O(ak ) do if m = ℓ − 1 then σbk ,ak ,ok+1 = V (bk ) else σbk ,ak ,ok+1 = Cm+1,ℓ (f (bk , ok+1 )) end if end for end for Return: minP ak ∈A c(f (bk , ok+1 ), ak )+ ok+1 ∈O(ak ) T (f (bk , ok+1 ), ak , bk )σbk ,ak ,ok+1
(28)
where V (bt+ℓ−1 ) is the terminal cost of the belief reached after ℓ actions. We propose two MPC approaches based on two terminal cost assumptions.
B. RL-MPC For RL-MPC, we adapt the iterative method in [24], which in each iteration d, aims to approximate the minimum expected cost of the first dℓ steps with a neural network as, Q(bt ; θd ) ≈
A. MPC-WTC In MPC-WTC, terminal costs of beliefs are fixed to 0. Therefore, the monitor chooses the action based on the expected cost from the first ℓ steps. This expected value is calculated recursively by defining Cm,ℓ (bk , ak ), the cost in next ℓ − m steps for the belief bk and the action ak , as WTC Cm,ℓ (bk , ak ) = c(bk , ak ) X WTC + T (f (bk , ok+1 ), ak , bk )Cm+1,ℓ (f (bk , ok+1 )), ok+1 ∈O(ak )
(29) WTC where Cm,ℓ (bk ) is the value of the belief by considering the actions that minimize cost for the remaining steps are applied, equivalently WTC WTC Cm,ℓ (bk ) = min Cm,ℓ (bk , ak ), ak ∈A
m < ℓ − 1.
(30)
WTC The value of Cℓ−1,ℓ (bk ) = V (bk ) = 0 stands for the terminal cost, which is fixed to 0. In each time slot t, the monitor
min
E
at+m ∈A, m∈{1,...,ℓ−1}
" dℓ X
# c(bt+m , at+m ) bt , at , . . . , at+dℓ−1 ,
m=1
(31) where θd , and Q(bt ; θd ) denote the network parameters and the output of the network at iteration d, respectively. The neural network design is illustrated with Fig. 5, and the trained network in each iteration is utilized to calculate the terminal costs in the next iteration. We implement RL-MPC as follows. We first initiate the network parameters to satisfy Q(bt ; θd ) = 0, bt ∈ B for d = 0, 1. Then, in each iteration, we train the network with the loss function RL J(θd ) = Q(bt ; θd ) − C0,ℓ,d (bt ),
t ∈ {0, 1, . . . , Tmax }, (32)
RL where Tmax is the training horizon, and Cm,ℓ,d (bt , at ) is defined similar to MPC-WTC as the cost in the next ℓ − m
minimizes the optimization problem in (10) for a given value of λ. The method we use for finding the optimum α∗ value for all benchmark policies is the use of simulations and brute force search. a) Benchmark I: In this benchmark policy, pull requests are generated with probability α, and a generated request is directed towards each of the sensors in a uniformly likely manner. The action at for Benchmark I taken at time t, can then be mathematically expressed as, ( α , n ∈ {1, . . . , N }, n, w.p. N (38) arand = t 0, w.p. 1 − α.
TABLE I PARAMETERS USED IN THE SIMULATION EXPERIMENTS .
dmax learning rate numbers of hidden layers nodes at hidden layers µk ∆ Tmax
4 10−3 2 60 1, for k ∈ K 15 106
steps for the belief bt and the action at in iteration d, and it has the recursive relation RL Cm,ℓ,d (bk , ak ) = c(bk , ak ) X RL + T (f (bk , ok+1 ), ak , bk )Cm+1,ℓ,d (f (bk , ok+1 )), ok+1 ∈O(ak )
(33) Different from MPC-WTC, the network from the previous RL iteration is used to calculate terminal costs as Cℓ−1,ℓ (bk ) = RL V (bk ) = Q(bk ; θd−1 ). Thus, Cm,ℓ,d (b) can be expressed for RL-MPC as ( RL min Cm,ℓ,d (bk , ak ), m < ℓ, RL Cm,ℓ,d (bk ) = ak ∈A (34) Q(bk ; θd−1 ), m = ℓ. RL (bt , at ) is applied, and Next, the action a∗t = arg min C0,ℓ,d RL the value of C0,ℓ,d (bt , at ) is obtained by Algorithm 1 by RL substituting Cm,ℓ (bk ) = C0,ℓ,d (bt ), and V (bk ) = Q(bk ; θd−1 ) at iteration d. Notice that, in the first iteration, the algorithm works identically to MPC-WTC with the same look-ahead step size WTC by learning Q(bt ; θ1 ) ≈ C0,ℓ (bt ), and in the remaining steps, the approximation horizon is extended. For instance, for d = 2 and ℓ = 1, the network output approximates WTC Q(bt ; θ2 ) ≈ min C0,1 (bt , at ) at ∈A X + T (f (bt , ot+1 ), at , bt )Q(f (bt , ot+1 ); θ1 ) (35) ot+1 ∈O(at )
WTC ≈ min C0,1 (bt , at ) at ∈A
+
X
WTC T (f (bt , ot+1 ), at , bt )C0,ℓ (f (bt , ot+1 ))
ot+1 ∈O(at )
(36) WTC ≈ C0,2 (bt , at ) .
(37)
VI. N UMERICAL R ESULTS We use the following open-loop policies as benchmark policies in order to compare their performance to the proposed MPC methods. For all benchmark policies, the sampling frequency is controlled with a single parameter α, and the optimum α∗ value is used for each benchmark policy, which
b) Benchmark II: In this benchmark policy, pull requests are generated almost uniformly with a period 1/α. In other words, the mth pull request is generated at round(m/α)th time slot, where the rounding operation, denoted by round(·), maps its argument to the nearest integer. In addition, the monitor chooses one of the sensors in a round-robin (i.e., circular) fashion. The action at for this policy can be expressed as, ( mod(arrη(t) , K) + 1, if t ∈ round m rr α : m∈N , at = 0, o.w. (39) Here, η(t) is the time of the last transmission before t. c) Benchmark III: This benchmark policy modifies the previous benchmark policy by incorporating erasureawareness, i.e., it repeats the previous action if the message is erased during transmission. The action for this case can be expressed mathematically as, ( m mod(arr,ea rr,ea ξ(t) , K) + 1, if t ∈ round α : m ∈ N , at = 0, o.w. (40) Here, ξ(t) is the time of the last successful transmission before t. In all simulations, we assume that the initial state is known by the monitor; hence, the initial AoII value is 0. For all policies, AoII values are truncated to ∆ = 15, and the MAoII values are obtained over at least 106 state transitions starting from the initial index-state x1 = 1. Unless otherwise stated, we assume a homogeneous sampling cost for all sensors which is fixed to 1. For RL-MPC, a neural network with two hidden layers is used with the learning rate of 10−3 . We train the network for at least dmax = 4 iterations before applying it. These parameters are summarized in Table I. Our numerical results pertaining to two different application scenarios are presented in the two subsections. A. Scenario I: Random Walk on a 2D Grid We consider an Lx × Ly grid space over which an object performs a random walk. We consider two sensors, sensor-1 and sensor-2, that are only able to observe the object’s x and y coordinates, respectively. For the random walk dynamics, we choose the probability of staying in the same state as 0.5, moving horizontally as 0.4 (with equal probability for right
x=1
y=1
y=2
y=3
11
12
13
0.5 0.05
x=2
21
0.2
22
0.2
23
0.05
x=3
31
32
33
Fig. 6. State diagram for a 3 × 3 grid. Only the transition probabilities from state (2, 2) are given. (a) λ = 0
and left), and moving vertically as 0.1 (with equal probability for down and up). For the states at the corners and boundaries, these probabilities are normalized. An example of the state diagram for N = 3 is illustrated in Fig. 6 with the x and y coordinates of all states. In Fig. 7, we compare the performance of various pull policies in terms of the average cost MAoII + λ · ASC for different grid sizes Lx × Ly while fixing ρs = 0.8. We first observe that round-robin policies (benchmarks II and III) provide performance gains in comparison to random sampling (benchmark I), and this performance gain is larger with the erasure-aware scheme (benchmark III) for λ = 0. However, as λ increases, the gain attained with erasure awareness appears to diminish. Furthermore, MPC methods outperform all benchmark methods in this scenario. We observe that MPCWTC performs substantially better with increasing the lookahead step size from ℓ = 1 to ℓ = 2. On the other hand, RL-MPC attains the same performance with the choice ℓ = 1. In Fig. 8, we fix the grid to Lx = Ly = 3 and compare the average cost of the MPC policies for different look-ahead step sizes as a function of ρs . We observe that increasing the lookahead step size further from ℓ = 1 for RL-MPC, and from ℓ = 2 for MPC-WTC, does not substantially improve their performance in terms of the average cost. In Fig. 9, when Lx = Ly = 3, we also fix the sampling cost for sensor-1 to µ1 = 1, and illustrate the frequency of actions in RL-MPC (ℓ = 2) when the sampling cost parameter of the second sensor is varied from µ2 = 0.5 to µ2 = 1.5. We denote the frequency of an action u ∈ A with fu , which measures how often an action is taken for a given policy, and it is written as,
Fig. 7. Comparison of methods for different grid sizes and λ values when ρs = 0.8.
Fig. 8. Comparison of MPC policies with different look-ahead step size and varying ρs for 3 × 3 grid size.
T
1X 1u,at , fu = T t=1
(b) λ = 0.5
(41)
where 1u,at is the indicator function which is 1 if at is the same with action u, and T is chosen as T = 106 for this experiment. We observe that even when the sampling costs of both sensors are equal, i.e., µ1 = µ2 = 1, the monitor tends to sample sensor-2 more frequently to track the location, as it is more likely that the object makes a horizontal move due to the dynamics of the joint process. On the other hand, as µ2 increases, the frequencies at which sensor-1 is pulled, and no
pull requests are made, also increase. B. Scenario II: Correlated Temperature/Fire/Freeze Events In this scenario, we consider a system model with three sensors. The first sensor observes the temperature of the environment. The temperature process has states ‘H’, ‘M ’, and ‘L’ corresponding to high, moderate, and low temperatures, respectively. The remaining two sensors observe events with
Fig. 9. Frequencies of the actions for applying RL-MPC (ℓ = 2) to 3 × 3 grid with varying sampling cost for sensor-2 (µ2 ) and fixed sampling cost for sensor-1 (µ1 = 1).
(a) λ = 0
(b) λ = 1 Fig. 11. Comparison of the studied policies for Scenario II in terms of the average cost for varying ρs when a) λ = 0, b) λ = 1. Fig. 10. State diagram for scenario II. Bidirectional arrows and self-loops indicate non-zero transition probabilities between two states in either direction, and non-zero self-transition probabilities, respectively.
state-space X2 = {e1 , ẽ1 }, and X3 = {e2 , ẽ2 }, and the event e1 (resp. e2 ) occurs only when the underlying process is in the state xt (1) = H (resp. xt (1) = L). These temperaturedependent events can be considered as “fire” and “freeze”, respectively. Therefore, the number of reachable joint states is reduced to 5 as illustrated in Fig. 10, and these states are sorted as X = {(H, e1 , ẽ2 ), (H, ẽ1 , ẽ2 ), (M, ẽ1 , ẽ2 ), (L, ẽ1 , e2 ), (L, ẽ1 , ẽ2 )}. Note that obtaining observations ot = e1 , ot = M , and ot = e2 indicates that the index process is xt−1 = 1, xt−1 = 3, and xt−1 = 5, respectively. The transition probability matrix for the index process, denoted by P , is chosen as, 0.1 0.7 0.1 0.1 0 0.4 0.4 0.1 0.05 0.05 P = 0.05 0.05 0.8 0.05 0.05 (42) . 0.05 0.05 0.1 0.1 0.7 0.1 0 0.1 0.4 0.4 Fig. 11 depicts the average cost as a function of the successful transmission probability ρs for two values of the relative weight λ. We first observe that for lower values
of ρs , benchmark III outperforms benchmark II due to its erasure awareness, whereas the performance gap between these two policies diminishes with increasing ρs , as expected. On the other hand, the performances of both these round-robin policies lag behind benchmark I. Furthermore, we observe that MPC methods outperform all open-loop benchmark policies by utilizing the observations through belief, i.e., beliefdependent policies. We observe that choosing ℓ = 1 is sufficient for both MPC methods when λ = 0. However, when λ = 1, looking only at the next step in MPC-WTC (ℓ = 1) falls short compared to the other MPC policies. In the second example, the average cost is depicted for varying values of λ while fixing ρs = 0.8, in Fig. 12. As λ increases, the monitor rarely sends pull requests, and the performance of MPC-WTC for ℓ = 1 degrades relative to the other belief-dependent MPC policies, which outperform the open-loop benchmark policies consistently. VII. C ONCLUSIONS We investigated a remote estimation problem for which a monitor obtains different components of a joint Markov process through pull requests and the monitor aims to minimize a weighted combination of the age and the average sampling
Fig. 12. Comparison of the studied policies for Scenario II in terms of the average cost for varying values of λ when ρs = 0.8.
cost by answering two fundamental questions: when and which sensor to observe? First, we obtained sufficient statistics for solving the problem, namely the belief, and formulated the problem as a belief-MDP. Then, we proposed the use of model predictive control (MPC) to solve the belief-MDP with the MPC-WTC and RL-MPC belief-based policies, the latter policy also making use of reinforcement learning (RL) for finding the terminal costs off-line, whereas the main advantage of MPC-WTC is that it does not require any offline learning. We observed that both MPC methods outperformed three benchmark open-loop policies for all the examples we studied. We have also observed that RL-MPC policy performed very well for the look-ahead size ℓ = 1 and gains with increasing this parameter further to ℓ = 2 were limited. On the other hand, there were several examples for which MPC-WTC with ℓ = 1 performed poorly, whereas increasing the look-ahead size ℓ = 2 improved the performance substantially. Further performance gains were also observed to be limited for MPCWTC, i.e., ℓ = 3 in the examples we studied. Consideration of other source dynamics other than the joint Markov structure, or the case when the source dynamics are not known, or only partially-known by the monitor, are left for future research. In addition, we assume that each time slot, the monitor can only send a pull request to at most a single sensor. However, the proposed methods are suitable for considering multiple pull requests simultaneously. R EFERENCES [1] A. Maatouk, M. Assaad, and A. Ephremides, “The age of incorrect information: An enabler of semantics-empowered communication,” IEEE Trans. Wireless Comm., vol. 22, no. 4, pp. 2621–2635, October 2022. [2] Z. Lu, R. Li, K. Lu, X. Chen, E. Hossain, Z. Zhao, and H. Zhang, “Semantics-empowered communication: A tutorial-cum-survey,” IEEE Commun. Surv. Tutor., November 2023. [3] M. Bastopcu and S. Ulukus, “Timely tracking of infection status of individuals in a population,” in IEEE Infocom, May 2021. [4] I. Cosandal and S. Ulukus, “Timely multi-goal transmissions with an intermittently failing sensor,” in IEEE MILCOM, October 2023. [5] M. Bastopcu and S. Ulukus, “Who should Google Scholar update more often?” in IEEE Infocom, July 2020. [6] O. Ayan, M. Vilgelm, M. Klügel, S. Hirche, and W. Kellerer, “Age-ofinformation vs. value-of-information scheduling for cellular networked control systems,” in ACM/IEEE ICCPS, April 2019.
[7] I. Cosandal, S. Ulukus, and N. Akar, “Joint age-state belief is all you need: Minimizing AoII via pull-based remote estimation,” in IEEE ICC, May 2025. [8] V. Krishnamurthy, Partially Observed Markov Decision Processes: From Filtering to Controlled Sensing. Cambridge University Press, 2016. [9] M. Araya-López, O. Buffet, V. Thomas, and F. Charpillet, “A POMDP extension with belief-dependent rewards,” Adv. Neural Inf. Process Syst., vol. 23, December 2010. [10] M. T. J. Spaan, T. S. Veiga, and P. U. Lima, “Decision-theoretic planning under uncertainty with information rewards for active cooperative perception,” Auton. Agents Multi-Agent Syst., vol. 29, pp. 1157–1185, December 2015. [11] Y. Satsangi, S. Whiteson, F. A. Oliehoek, and M. T. J. Spaan, “Exploiting submodular value functions for scaling up active perception,” Autonomous Robots, vol. 42, no. 2, pp. 209–233, August 2018. [12] M. Fehr, O. Buffet, V. Thomas, and J. Dibangoye, “ρ-POMDPs have Lipschitz-continuous ϵ-optimal value functions,” in NeurIPS, December 2018. [13] A. Eck, L. Soh, S. Devlin, and D. Kudenko, “Potential-based reward shaping for finite horizon online POMDP planning,” Agents Multi-Agent Syst., vol. 30, no. 3, pp. 403–445, May 2016. [14] L. Cregg, T. Linder, and S. Yüksel, “Reinforcement learning for nearoptimal design of zero-delay codes for Markov sources,” IEEE Trans. Inf. Theory, vol. 70, no. 11, pp. 8399–8413, June 2024. [15] M. Hauskrecht and B. Kveton, “Linear program approximations for factored continuous-state Markov decision processes,” Adv. Neural Inf. Process Syst., vol. 16, June 2003. [16] J. Liu, Q. Wang, and H. H. Chen, “Optimizing age of information in uplink multiuser MIMO networks with partial observations,” in IEEE WiOpt, August 2023. [17] J. Liu, R. Zhang, A. Gong, and H. Chen, “Optimizing age of information in wireless uplink networks with partial observations,” IEEE Trans. Comm., vol. 71, no. 7, pp. 4105–4118, July 2023. [18] D. Bertsekas, Dynamic programming and optimal control: Volume I. Athena scientific, 2012. [19] M. A. Sehr and R. R. Bitmead, “Performance of model predictive control of POMDPs,” in ECC, June 2018. [20] C. H. Ulfsjöö and D. Axehill, “On integrating POMDP and scenario MPC for planning under uncertainty–with applications to highway driving,” in IEEE IV, June 2022. [21] H. N. Esfahani, A. B. Kordabad, and S. Gros, “Reinforcement learning based on MPC/MHE for unmodeled and partially observable dynamics,” in ACC, May 2021. [22] A. Boccia, L. Grüne, and K. Worthmann, “Stability and feasibility of state constrained MPC without stabilizing terminal constraints,” Systems & Control Letters, vol. 72, pp. 14–21, October 2014. [23] L. Grüne, “NMPC without terminal constraints,” IFAC Proc. Vol., vol. 45, no. 17, pp. 1–13, September 2012. [24] M. Lin, Z. Sun, Y. Xia, and J. Zhang, “Reinforcement learning-based model predictive control for discrete-time systems,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 3, pp. 3312–3324, March 2023. [25] D. Ernst, P. Geurts, and L. Wehenkel, “Iteratively extending time horizon reinforcement learning,” in ECML, September 2003. [26] M. Lin, Y. Xia, Z. Sun, and L. Dai, “Learning-based model predictive control under value iteration with finite approximation errors,” Int. J. Robust Nonlinear Control, vol. 34, no. 4, pp. 2946–2971, December 2024. [27] R. D. Smallwood and E. J. Sondik, “The optimal control of partially observable Markov processes over a finite horizon,” Operations Research, vol. 21, no. 5, pp. 1071–1088, September 1973. [28] M. T. J. Spaan, “Partially observable Markov decision processes,” in Reinforcement Learning: State-of-the-Art. Springer, 2012, pp. 387– 414. [29] Z. Chen, T. Yang, N. Pappas, H. H. Yang, Z. Tian, M. Wang, and T. Q. S. Quek, “Improving information freshness via multi-sensor parallel status updating,” IEEE Trans. Commun., July 2024. [30] E. Erbayat, A. Maatouk, P. Zou, and S. Subramaniam, “Age of information optimization and state error analysis for correlated multi-process multi-sensor systems,” in MobiHoc, October 2024. [31] Q. He, G. Dan, and V. Fodor, “Minimizing age of correlated information for wireless camera networks,” in IEEE Infocom, April 2018. [32] V. Tripathi and E. Modiano, “Optimizing age of information with correlated sources,” in MobiHoc, October 2022.
[33] A. Zancanaro, G. Cisotto, and L. Badia, “Modeling value of information in remote sensing from correlated sources,” Computer Communications, vol. 203, pp. 289–297, March 2023. [34] J. Hribar, M. Costa, N. Kaminski, and L. A. DaSilva, “Updating strategies in the internet of things by taking advantage of correlated sources,” in IEEE GLOBECOM, December 2017. [35] L. Liang, S. Zhou, B. Tang, and G. Tan, “Optimizing age of information in random access networks with correlated sources,” in IEEE ICICSP, September 2024. [36] J. Tong, L. Fu, and Z. Han, “Age-of-information oriented scheduling for multichannel IoT systems with correlated sources,” IEEE Trans. Wireless Comm., vol. 21, no. 11, pp. 9775–9790, June 2022. [37] M. S. Kumar, A. Dadlani, O. Ardakanian, I. Nikolaidis, and J. J. Harms, “Age analysis of correlated information in multi-source updating systems with MAP arrivals,” IEEE Commun. Lett., July 2024. [38] Q. He, G. Dán, and V. Fodor, “Joint assignment and scheduling for minimizing age of correlated information,” IEEE/ACM Transactions on Networking, vol. 27, no. 5, pp. 1887–1900, September 2019. [39] M. Fidler, F. Gallistl, J. P. Champati, and J. Widmer, “2D-AoI: Age-ofinformation of distributed sensors for spatio-temporal processes,” 2024, available online at arXiv:2412.12789. [40] J. Tian, A. Zakeri, M. Codreanu, and D. Gundlegård, “Real-time remote tracking with state-dependent detection probability: A POMDP framework,” 2025, available online at arXiv:2509.09837. [41] Y. Shao, Q. Cao, S. C. Liew, and H. Chen, “Partially observable minimum-age scheduling: The greedy policy,” IEEE Trans. Comm., vol. 70, no. 1, pp. 404–418, October 2021. [42] A. Tahir, K. Cui, B. Alt, A. Rizk, and H. Koeppl, “Collaborative optimization of the age of information under partial observability,” in IFIP Networking, August 2024. [43] A. Gong, T. Zhang, H. Chen, and Y. Zhang, “Age-of-information-based scheduling in multiuser uplinks with stochastic arrivals: A POMDP approach,” in IEEE Globecom, December 2020. [44] G. Chen, S. C. Liew, and Y. Shao, “Uncertainty-of-information scheduling: A restless multiarmed bandit framework,” IEEE Trans. Inf. Theory, vol. 68, no. 9, pp. 6151–6173, August 2022. [45] G. Stamatakis, N. Pappas, A. Fragkiadakis, and A. Traganitis, “Autonomous maintenance in IoT networks via AoI-driven deep reinforcement learning,” in IEEE Infocom, May 2021. [46] Z. Zhao and I. Kadota, “Optimizing age of information without knowing the age of information,” 2025, available online at arXiv:2501.06688. [47] F. C., A. M., L. B., and P. P., “Goal-oriented medium access with distributed belief processing,” 2024, available online at arXiv:2412.07503. [48] G. He, S. Zhang, M. Feng, S. Li, and T. Jiang, “Age of incorrect information-aware data dissemination for distributed multi-agent systems,” IEEE Trans. Wireless Comm., vol. 23, no. 10, pp. 15 705–15 718, July 2024. [49] Y. Emami, H. Gao, K. Li, L. Almeida, E. Tovar, and Z. Han, “Age of information minimization using multi-agent UAVs based on AIenhanced mean field resource allocation,” IEEE Trans. Veh. Technol., April 2024. [50] A. Maatouk, S. Kriouile, M. Assaad, and A. Ephremides, “The age of incorrect information: A new performance metric for status updates,” IEEE/ACM Trans. Netw., vol. 28, no. 5, pp. 2215–2228, October 2020. [51] Y. Chen and A. Ephremides, “Minimizing age of incorrect information for unreliable channel with power constraint,” in IEEE Globecom, December 2021. [52] S. Kriouile and M. Assaad, “When to pull data from sensors for minimum distance-based age of incorrect information metric,” in IEEE WiOpt, February 2022. [53] I. Cosandal, N. Akar, and S. Ulukus, “Modeling AoII in push- and pullbased sampling of continuous time Markov chains,” in IEEE Infocom, May 2024. [54] ——, “AoII-optimum sampling of CTMC information sources under sampling rate constraints,” in IEEE ISIT, July 2024. [55] ——, “Multi-threshold AoII-optimum sampling policies for CTMC information sources,” IEEE Trans. Inf. Theory, vol. 71, no. 9, pp. 6968– 6988, July 2025. [56] N. Akar and S. Ulukus, “Query-based sampling of heterogeneous CTMCs: Modeling and optimization with binary freshness,” IEEE Trans. Comm., vol. 72, no. 12, pp. 7705–7714, June 2024. [57] S. Kriouile and M. Assaad, “Minimizing the age of incorrect information for real-time tracking of Markov remote sources,” in IEEE ISIT, July 2021.
[58] C. Kam, S. Kompella, and A. Ephremides, “Age of incorrect information for remote estimation of a binary Markov source,” in IEEE Infocom, July 2020. [59] O. Dagan, T. Becker, and Z. N. Sunberg, “Resolving multiple-dynamic model uncertainty in hypothesis-driven belief-MDPs,” in AAMAS, May 2025. [60] M. H. Lim, T. J. Becker, M. J. Kochenderfer, C. J. Tomlin, and Z. N. Sunberg, “Optimality guarantees for particle belief approximation of POMDPs,” Journal of Artificial Intelligence Research, vol. 77, pp. 1591– 1636, 2023. [61] D. P. Bertsekas, “Model predictive control and reinforcement learning: A unified framework based on dynamic programming,” in IFAC NMPC, August 2024.