arXiv CS open access

Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning

Sajjad Baghaeea et al. · arxiv_cs

arXiv CS · Papers

distributed-systems internet networking protocols

Deep-space communications operate under intermittent connectivity, long propagation delays, and relay storage constraints. In such networks, the Bundle Protocol enables store-carry-forward communication, requiring relay nodes to manage limited storage and scarce contact opportunities. Relay-side sch

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

77th International Astronautical Congress (IAC 2026), Antalya, Türkiye, 5–9 Oct 2026. Copyright © 2026 by the International Astronautical Federation (IAF). All rights reserved. This manuscript was presented at the 77th International Astronautical Congress (IAC 2026), held in Antalya, Türkiye, 5–9 October 2026.

IAC–26–B2,5,8,x113206 Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning Sajjad Baghaeea∗ , Aimin Lib∗ , Shanmugarajan Balasubramanianc , Elif Uysald

arXiv:2610.13923v1 [cs.NI] 9 Oct 2026

Communication Networks Research Group (CNG), Dept. of Electrical and Electronics Engineering, Middle East Technical University (METU), Ankara, Türkiye b c d a sajjad@baghaee.com sajjad@baghaee.com, aimin@metu.edu.tr, e279039@metu.edu.tr, sajjad@baghaee.com aimin@metu.edu.tr aimin@metu.edu.tr e279039@metu.edu.tr e279039@metu.edu.tr uelif@metu.edu.tr uelif@metu.edu.tr * Corresponding authors

Abstract Deep-space communications operate under intermittent connectivity, long propagation delays, and relay storage constraints. To span vast distances, data are often forwarded through relay nodes, where incoming traffic can vary over time. In such networks, the Bundle Protocol encapsulates data into store-carry-forward units called bundles, which are stored at intermediate nodes and forwarded only when contact opportunities become available. Therefore, relay-side buffer and flow management become critical challenges, since limited storage and scarce contact windows force the relay to decide which bundles to transmit or discard. This decision process is inherently goal-oriented: the relay should not merely forward data, but should allocate limited transmission opportunities to the bundles that are most valuable to the downstream mission objective. Existing schedulers, such as FIFO, are heuristic and do not explicitly optimize task utility; moreover, they often lead to unfair service across heterogeneous planetary sources. This paper presents a closed-loop modeling and learning framework for relay-side bundle and flow management in a contact-driven deep-space network, where updates generated by lunar and Martian sources are forwarded via a relay to Earth. These bundles are stored in a shared relay buffer and carry metadata including source and destination identities, generation time, and time-to-live. On + Earth, data utility depends on freshness, which we quantify via the excess age ∆ei (t) = [∆i (t) − di (t)] , where ∆i (t) denotes the Age of Information (AoI) of source i, i.e., the time elapsed since the freshest update currently available on Earth was generated, and di (t) is a source-dependent delay baseline. Based on this metric, we define a goal-oriented system loss that penalizes excess age in a source-dependent and nonlinear manner, allowing the relay to prioritize information according to mission relevance. We formulate relay control as a constrained semi-Markov decision process (CSMDP) and solve it using masked Lagrangian Proximal Policy Optimization (PPO), where feasibility-aware action masking restricts decisions to transmissions that can finish within the remaining contact window. Simulation results in DTNSim indicate that the proposed approach significantly improves age-based utility and achieves substantially better inter-flow fairness compared to baseline schedulers. By explicitly prioritizing information according to downstream task relevance and using this signal to guide transmission and dropping decisions, the proposed framework improves goal-oriented information delivery in deep-space networks and, to our knowledge, is among the first reinforcement learning frameworks to jointly optimize relay-side bundle scheduling and dropping, as well as flow control under contact-driven dynamics with explicit fairness considerations across heterogeneous sources. 1. Introduction Deep-space networks rely on intermittent contacts and store-carry-forward relaying to communicate across long propagation delays. Delay/Disruption-Tolerant Networking (DTN) and the Bundle Protocol (BP) enable storecarry-forward communication in such environments [1, 2], but leave a fundamental relay-side allocation problem unresolved: when multiple buffered bundles compete for a short contact window, which should be transmitted, and which should be dropped under buffer pressure. This decision becomes challenging in heterogeneous deep-space networks, where lunar and Martian traffic share a finite relay buffer but differ significantly in contact frequency and propagation delay. Simple queueing policies such as FIFO,

IAC–26–B2,5,8,x113206

LIFO, or fixed priority are therefore inadequate, as they may either waste scarce contact opportunities on outdated information or systematically disadvantage one source. To reason about this trade-off, we consider destination-side information freshness. Age of Information (AoI) captures the time elapsed since the most recently received update was generated [3, 4]. However, in deep-space settings, raw AoI conflates two effects: the relay’s scheduling decisions and the underlying contact structure of each source. To separate these effects, we measure freshness relative to a contact-aware baseline di(t) and define the excess age ∆ei (t) = [∆i (t) − di (t)]+ . This allows us to penalize only the staleness attributable to relay decisions, while avoiding bias toward inherently disadvantaged sources.

Page 1 of 15

To address this problem, we develop a relay-side scheduling and dropping model for deep-space DTNs, where bundle decisions are driven by downstream information utility under finite storage and intermittent contacts. We consider an Earth–Relay–Moon–Mars network with a shared relay buffer. The relay operates in an event-driven manner: at each decision epoch, it may transmit a feasible bundle, resolve buffer overflow through rejection or eviction, or remain idle. Since freshness at the destination is only updated upon Earth delivery, pending deliveries are retained in the system state so that the delayed reception dynamics remain Markovian. These dynamics lead to a constrained semi-Markov decision process (CSMDP), which we solve using masked Lagrangian Proximal Policy Optimization (PPO) with permutation-invariant encoding of the variable-size buffer state. Our main contribution is a goal-oriented framework for deep-space DTNs that jointly controls relay scheduling and dropping based on downstream information utility rather than queue order. It combines: (i) a closed-loop model with finite relay storage, scheduled contacts, stochastic source traffic, heterogeneous bundle sizes, and delayed destination updates; (ii) a contact-aware excess-age objective with explicit inter-source fairness constraints; (iii) an event-driven CSMDP formulation for joint scheduling and dropping; and (iv) masked Lagrangian PPO with permutation-invariant encoding of the variable-size relay buffer. The framework is implemented in DTNSim. 2. Background and Related Work 2.1 Relay-Local DTN Resource Management DTN and the Bundle Protocol (BP) enable persistent store-carry-forward communication across disrupted links while leaving relay-local bundle ordering and discard policies unspecified [1, 2, 5]. In scheduled space networks, Contact Graph Routing (CGR) and related methods exploit contact plans to construct time-dependent routes [6, 7], with extensions addressing heterogeneous traffic, contact capacity, and finite buffers [8, 9]. These mechanisms primarily determine where and when bundles can be forwarded across the network. Our focus is complementary: once multiple bundles coexist at a finite-buffer relay, the node must decide which bundle should use the next outgoing contact and which bundle should be discarded when storage becomes scarce. Such relay-local decisions can influence end-to-end performance under limited contact capacity [10]. Conventional rules such as FIFO, LIFO, fixed-priority, and TTL-based policies are simple but do not account for the freshness already available at the destination, motivating freshness-aware joint scheduling and dropping based on the downstream value of each candidate transmission.

IAC–26–B2,5,8,x113206

2.2 Freshness and Goal-Oriented Utility Age of Information (AoI) provides a natural measure of destination-side information freshness, and AoI-aware scheduling can substantially affect information timeliness [11, 12]. Related work has also examined fairness in freshness-oriented systems, since minimizing an aggregate age metric can produce highly uneven service across heterogeneous sources [13, 14]. Deep-space networks introduce an additional asymmetry: sources may experience substantially different propagation delays and contact opportunities even under the same relay policy. We therefore evaluate freshness relative to a policy-independent reference induced by the known contact plan, and combine this relative freshness objective with explicit source-specific service constraints. This is consistent with goal-oriented communication, where resources are allocated according to downstream utility rather than delivered data volume alone [15, 16]. 2.3 Learning-Based Control and Research Gap Learning-based methods have been studied for communication scheduling, routing, and resource management, including constrained satellite routing and PPO-based congestion control in deep-space DTNs [17, 18]. The central difficulty in our setting is temporal: a relay must decide which bundle to transmit or discard using its current buffer state, while the value of that decision is realized only when the transmitted information reaches Earth. During this propagation interval, new bundles may arrive, others may expire, contact conditions may change, and subsequent buffer-management decisions may have been made. Relay actions are therefore coupled across irregular decision epochs through delayed destination-side freshness. This structure calls for an event-driven control formulation that preserves the delayed effect of relay actions and the changing set of feasible decisions. We formulate the problem as a constrained SMDP, where event-driven transitions capture irregular decision times and delayed destination updates, feasibility-aware action masking removes transmissions that cannot be completed within the remaining contact window [19], and constrained policy optimization incorporates operational and source-specific service requirements [20, 21]. The formulation links relayside scheduling and dropping decisions to the freshness realized at Earth under stochastic, heterogeneous traffic. 3. Deep-Space DTN System Model 3.1 Network and Contact Model We consider a relay-side scheduling and dropping model for deep-space DTNs, where limited relay storage and intermittent contact opportunities require goal-oriented bundle management. As a representative scenario, we consider four nodes: Earth E, relay R, lunar source M , and

Page 2 of 15

Thus, only transmission completion must occur before the contact closes; relay reception may occur later due to propagation delay. 3.3 Bundle and Buffer Model Each bundle is represented as b = i(b), g(b), L(b), T ttl (b), tarr (b), ω(b), id(b) , [4]

Fig. 1. Earth–Relay–Moon–Mars network. Martian source S, with source set I ≜ {M, S}. Both sources generate time-stamped status-update bundles destined for Earth. Direct source–Earth delivery is excluded so that every bundle traverses the finite-buffer relay, the only node at which scheduling and dropping are controlled. Figure 1 shows this representative topology. Connectivity is specified by contact plans n oKxy (q) (q) (q) Cxy ≜ t(q) , [1] xy,s , txy,e , σxy , τxy q=1

where x and y denote link endpoints, q indexes contact segments, and Kxy is the number of segments on x → y; each segment specifies its start/end times, usable rate, and one-way propagation delay. We use the links M → R, S → R, and R → E. During active relay–Earth segment (q) (q) (q) q, σRE (t) = σRE , τRE (t) = τRE , and tend RE (t) = tRE,e ; end otherwise σRE (t) = τRE (t) = 0 and tRE (t) = t. Thus, the downlink is available iff σRE (t) > 0. 3.2 Source Generation and Source–Relay Transport Bundles are generated independently at the Moon and Mars according to Poisson processes with heterogeneous inter-generation times and randomly drawn sizes. Each source maintains a non-preemptive FIFO queue; bundles accumulate during contact gaps and are transmitted headof-line when a source–relay contact is active. A bundle is admitted to service only if its full serialization fits within the remaining contact interval. For source i ∈ I, if bundle b begins transmission at ttx,s (b) over contact segment q, its source-link completion i and relay arrival times are L(b) (q) ttx,e (b) = ttx,s (b)+ (q) , tarr (b) = ttx,e (b)+τiR , i i i σiR [2] with the feasibility condition (q) ttx,e (b) ≤ tiR,e . i

IAC–26–B2,5,8,x113206

[3]

where i(b) is the source, g(b) the update-generation time, L(b) the size, T ttl (b) the lifetime, tarr (b) the relay-arrival time, ω(b) an optional importance weight, and id(b) a unique identifier. Its expiry time is texp (b) ≜ g(b) + T ttl (b), and it is valid at time t only if t < texp (b). Let Qi (t) be the multiset of buffered source-i bundles. U The shared relay buffer is B(t) ≜ i∈I Qi (t), with occuP pancy B(t) ≜ b∈B(t) L(b) ≤ Bmax . An arriving bundle is admitted when capacity permits; otherwise the controller must reject it or evict a bundle so that the post-decision occupancy does not exceed Bmax . We assume each bundle fits within the relay buffer, i.e., L(b) ≤ Bmax . 3.4 Relay–Earth Service and Feasibility The relay–Earth downlink is a non-preemptive single server. Let busy(t) ∈ {0, 1} indicate whether a transmission is active, bsrv (t) the in-service bundle, and tcmp,srv (t) its completion time, fixed when transmission begins. When srv the server is busy, τRE (t) denotes the propagation delay of the relay–Earth contact segment used for the current transmission and remains fixed until completion. When the server is idle, bsrv (t) = ∅, tcmp,srv (t) = +∞, and srv τRE (t) = 0. For a bundle b selected at time t, its transmission, relay-completion, and Earth-delivery times are L(b) , σRE (t) tcmp (b, t) = t + ttx (b, t), ttx (b, t) =

[5]

tdel (b, t) = tcmp (b, t) + τRE (t), with ttx (b, t) = +∞ when σRE (t) = 0. When bundle b starts service at time t, tcmp,srv (t+ ) = tcmp (b, t). We use a non-preemptive, all-or-nothing service model: once transmission begins, the bundle must complete within the same active contact and cannot be resumed later. Accordingly, bundle b is schedulable at time t only if tcmp (b, t) ≤ tend tdel (b, t) < texp (b). [6] RE (t), The first condition ensures transmission completion before contact termination, while the second requires Earth reception before bundle expiry. 3.5 Pending Deliveries and Observability Propagation separates relay transmission completion from Earth reception. We retain the pending-delivery

Page 3 of 15

multiset G(t) ⊆ R≥0 × I × R≥0 , where (t̂, ı̂, ĝ) records an Earth-delivery time, source, and generation timestamp. Completion of bundle b inserts (tdel (b, t), i(b), g(b)) into G; at t̂, the record is removed and Earth-side freshness is updated as defined in Subsection 4.1. Retaining G(t) preserves the Markov state under propagation delay. Under the deterministic contact plan and transmission/propagation dynamics, the relay can reconstruct pending deliveries and Earth-side timestamps from its prior actions and the known contact plan; instantaneous Earth feedback is therefore unnecessary. The policy observes only relay-available information and the known contact plan. The SMDP state is defined in Section 5, while the policy feature representation is defined in Subsection 8.2. 4. Closed-Loop Freshness and Excess-Age Objective Relay-side transmission affects freshness only when the corresponding pending delivery reaches Earth. We therefore define destination-side Age of Information (AoI) and compare it with a source-dependent baseline derived from the known contact plan. 4.1 Earth-Side Freshness For each source i ∈ I, let ui (t) denote the generation timestamp of the freshest source-i update delivered to Earth by time t. When a pending delivery (t̂, ı̂, ĝ) ∈ G(t− ) reaches Earth at t = t̂, uı̂ (t) ← max{uı̂ (t− ), ĝ}. The AoI is ∆i (t) ≜ t − ui (t). The maximum prevents an out-of-order delivery from replacing fresher information at Earth. Relay transmission completion does not update ui (t); only Earth reception can advance this timestamp, while ∆i (t) increases linearly between useful deliveries. 4.2 Contact-Plan Baseline Raw AoI reflects both relay-controlled effects and source-dependent contact and propagation conditions. To account for this heterogeneity, we define a policyindependent baseline based on idealized, contention-free forwarding over the known contact plan. For an L-bit bundle available at node x at time s, its earliest feasible arrival at node y is Axy (s; L) ≜ " min

1≤q≤Kxy : (q) (q) ηxy (s)+L/σxy ≤t(q) xy,e (q)

(q) ηxy (s) +

(q)

L (q)

σxy

# (q) + τxy ,

[7]

where ηxy (s) ≜ max{s, txy,s } and Axy (s; L) = +∞ if no contact can carry the complete bundle. Let Lref denote a fixed reference bundle size, common to all sources and policies. For a hypothetical reference update generated at time g, define

IAC–26–B2,5,8,x113206

ai (g) ≜ ARE AiR (g; Lref ); Lref , ui (t) ≜ sup{g ≤ t : ai (g) ≤ t},

[8]

di (t) ≜ t − ui (t). Here, di (t) represents the freshness achievable by the reference update under idealized forwarding. Because it depends only on the contact plan and Lref , it is independent of relay scheduling, queueing, contention, dropping, and the realized traffic process. 4.3 Excess-Age Objective The excess age and instantaneous goal-oriented freshness cost are ∆ei (t) ≜ [∆i (t) − di (t)]+ , X [9] G(t) ≜ wi ϕi (∆ei (t)) , i∈I

where [x]+ ≜ max{x, 0}, wi > 0 denotes the relative mission importance of source i, and ϕi : R≥0 → R≥0 is nonnegative and nondecreasing. Thus, ∆ei (t) measures Earth-side staleness relative to the corresponding sourcedependent contact-plan baseline. Over an episode [t0 , T ], the relay minimizes the iexpected cumulative freshness hR T cost JG ≜ E t0 G(t) dt , subject to the operational and source-specific constraints introduced later. Its event-wise decomposition defines the SMDP reward in Section 5.3. 5. Event-Driven SMDP Formulation Relay-side scheduling and dropping occur at irregular times determined by contact changes, bundle arrivals, transmission completions, Earth deliveries, and expirations. We therefore model relay control as a semi-Markov decision process (SMDP) [22]. Let {tk }k≥0 denote the strictly increasing decision epochs and δk ≜ tk+1 − tk the holding time. Co-timed events are processed as one batch before control. If an arrival creates buffer overflow, the controller may take a finite sequence of rejection or eviction decisions at the same physical time until the overflow is resolved. A scheduling decision is then made when transmission is possible. These same-time control stages do not constitute separate SMDP transitions; physical time advances only after all applicable stages have been completed. 5.1 Events, State, and Action Sets The physical event types are contact boundary, transmission completion, Earth delivery, relay expiry, and relay arrival. Co-timed events are processed in this fixed order. This convention first establishes the active contact state, then resolves transmission and destination updates, removes expired bundles, and finally processes new arrivals, yielding a unique post-event state. An additional controller wake-up may be scheduled after an elective wait, as described below; it changes no physical state by itself. After processing events at tk , the relay-side control state is

Page 4 of 15

sk ≜ B(tk ), B(tk ), busy(tk ), bsrv (tk ), tcmp,srv (tk ), srv tk , σRE (tk ), τRE (tk ), τRE (tk ), tend k ), [10] RE (t new . {ui (tk )}i∈I , G(tk ), {Di (k)}i∈I , bk

is reconsidered. If the server is idle, σRE (tk ) > 0, and F(sdk ) ̸= ∅, then qk ∈ Asched (sdk ); otherwise qk = ∅. The complete control applied at epoch tk is ak ≜ (dk , qk ), where dk = (dk,0 , . . . , dk,Jk −1 ) is the drop sequence, empty when no overflow occurs.

The physical, service, and freshness variables are defined in Sections 3–4. The variables Di (k) are the fairness deficits defined in Section 6, while bnew is nonempty k only when a newly arrived bundle requires an overflow decision. Equation [10] collects the relay-side control state from which the policy observation is constructed. Let hk denote the upstream simulator state, including source FIFO queues, active source–relay transmissions, and in-flight source–relay bundles. The environment state is xk ≜ (sk , hk ). The environment evolves Markovianly on xk , whereas the relay policy acts only on information derived from sk . The hidden upstream state hk , together with source-generation and bundle-size draws, determines future relay arrivals but is not exposed to the controller. For scheduling, define the currently feasible bundle set n F(sk ) ≜ b ∈ B(tk ) :tcmp (b, tk ) ≤ tend RE (tk ), o [11] tdel (b, tk ) < texp (b) .

5.2 Post-Decision State and Next Event After the drop sequence has been completed, the scheduling component is applied: s̄k ≜ Ψsched (sdk , qk ), where Ψsched (s, ∅) = Ψsched (s, wait) = s. If qk = b ∈ F (sdk ), bundle b is removed from the relay buffer and the server state is set to busy = 1, bsrv = b, srv tcmp,srv = tk + ttx (b, tk ), and τRE = τRE (tk ). No holding time, reward interval, or discount factor is introduced between same-time drop stages or between overflow resolution and scheduling. Let Texp , Tct , Ttx , Tdel , and Tarr,k denote, respectively, the next buffered-bundle expiry, relay–Earth contact boundary, transmission completion, pending Earth delivery, and relay arrival. The candidate times are Texp (s̄k ) ≜ min texp (b) : b ∈ B̄k , texp (b) > tk , ( ) K[ RE (q) (q) Tct (s̄k ) ≜ inf t > tk : t ∈ {tRE,s , tRE,e } ,

When the relay–Earth server is idle, the contact is active, and F(sk ) ̸= ∅, the scheduling action set is Asched (sk ) ≜ F (sk ) ∪ {wait}. [12] Here, wait is an elective action. If F(sk ) = ∅, no scheduling decision is required. Overflow is resolved through one or more same-time drop stages. Let sd,0 k ≜ sk . Whenever a pending arrival bnew = ̸ ∅ would violate the shared relay capacity, the k available drop actions at stage j are new Adrop (sd,j } ∪ B(sd,j [13] k ) ≜ {bk k ). The drop stage is invoked only while admission of bnew k would violate the shared relay-buffer capacity. Selecting bnew rejects the arriving bundle and terminates overflow k resolution. Selecting b ∈ B(sd,j k ) evicts that buffered bundle. If the remaining capacity is still insufficient to admit bnew , another drop stage is performed at the same physical k time using the updated buffer state. Otherwise, bnew is k admitted and the overflow sequence terminates. For selected drop action dk,j , define sd,j+1 ≜ Ψdrop (sd,j k k , dk,j ). Let Jk denote the total number of rejection or eviction decisions executed during overflow resolution at physical epoch tk . Each such decision discards one bundle, and Jk = 0 when no overflow action is required. The k resulting post-overflow state is sdk ≜ sd,J , with sdk = sk k when Jk = 0. The controller therefore executes statedependent drop sequence dk,0 , . . . , dk,Jk −1 , where each action is selected from the intermediate state produced by the preceding stage. After overflow resolution, scheduling

IAC–26–B2,5,8,x113206

q=1

 tcmp,srv (s̄ ), busy(s̄k ) = 1, k tcmp,srv (s̄k ) > tk , Ttx (s̄k ) ≜  +∞, otherwise, n o Tdel (s̄k ) ≜ min t̂ : (t̂, î, ĝ) ∈ Ḡk , t̂ > tk .

[14]

Here, B̄k and Ḡk denote the buffered-bundle and pending-delivery multisets in the post-decision state s̄k . A minimum or infimum over an empty candidate set is defined as +∞. Let x̄k ≜ (s̄k , hk ) denote the complete post-decision state. The next relay-arrival time is a simulator-side random variable whose conditional law is generated by upstream evolution from x̄k , including subsequent source-generation and bundle-size draws. Neither hidden upstream information nor future random outcomes are exposed to the policy. Thus, Tarr,k > tk denotes the first unprocessed relay arrival strictly after the co-timed event batch, with Tarr,k = +∞ if no further arrival occurs. An elective scheduling wait introduces an additional decision timer ( tk + 30 s, qk = wait, Twake (s̄k , qk ) ≜ [15] +∞, otherwise. A physical event occurring before this timer cancels the need to wait for the full 30 s because the earlier event itself creates the next decision opportunity. Forced idle periods, for which no feasible transmission exists, do not create this timer and remain purely event-driven. For every nonterminal epoch tk < T , the next decision epoch is

Page 5 of 15

n tk+1 ≜ min Tarr,k , Texp (s̄k ), Tct (s̄k ), Ttx (s̄k ), o Tdel (s̄k ), Twake (s̄k , qk ),T .

[16]

Because all events and control operations occurring at tk are exhausted before this minimization, every nonterminal finite event candidate is strictly larger than tk . Hence, tk+1 > tk , δk ≜ tk+1 − tk > 0. [17] If several future physical events attain the same minimum, they are processed as one batch according to Subsection 5.1. At the terminal time T , physical events occurring exactly at T are processed before the episode stops. The simulator then terminates without draining the system; unexpired bundles still queued, in service, or in transit are recorded as unresolved backlog rather than as drops or expirations. 5.3 Event Updates, Transition Kernel, and Reward At tk+1 , only state components affected by the occurring physical events are updated. A relay arrival is admitted if capacity permits; otherwise the existing buffer is retained and the arrival becomes bnew k+1 , so that the first control stage at the resulting epoch is overflow resolution. If one eviction does not release sufficient capacity, additional drop stages are taken at the same timestamp until the arrival is either admitted or rejected. Scheduling is then reconsidered without advancing physical time. When a relay-buffered bundle reaches its absolute expiry time, it is removed from the relay buffer. Expiry accounting is maintained according to the bundle’s physical location in the simulator. A queued bundle that can no longer be delivered before its expiry is excluded from the scheduling set through Equation [11], but it is not proactively removed; it remains in the relay buffer until it is explicitly dropped or reaches its absolute expiry time. At transmission completion, the completed bundle is not yet treated as delivered because it must still propagate to Earth. Before clearing the server, the record srv − srv − tk+1 + τRE (tk+1 ), i(bsrv (t− (tk+1 )) is ink+1 )), g(b srv − serted into G. Here, τRE (tk+1 ) is the propagation delay of the contact segment on which the completed transmission was initiated. The server is then cleared by setting srv busy = 0, bsrv = ∅, tcmp,srv = +∞, and τRE = 0. An Earth-delivery event removes the corresponding pending record and updates the Earth-side timestamp according to Subsection 4.1. Thus, transmission completion and Earth-side freshness update remain distinct physical events. After all events at tk+1 are processed, the fairness deficits are updated using the holding time δk and the delivered freshness gains yi,k according to Section 6. The post-decision map, event scheduler, sourcegeneration processes, and transmission dynamics induce

IAC–26–B2,5,8,x113206

the controlled transition kernel on the complete environment state P (dx′ |xk , ak ) ≜ Pr{xk+1 ∈ dx′ | xk , ak }. The relay policy selects actions using only the observation constructed from sk and does not observe hk or future random outcomes. Conditional on a fixed physical traffic trace, the environment transition is deterministic. Under stochastic traffic, randomness originates from the independent source-generation processes and bundle-size draws; source FIFO service, source–relay contacts, and propagation then determine the resulting relay-arrival sequence. Because physical time is included in xk , the transition kernel also induces the holding time δk = tk+1 − tk . The reward accumulated during transition R tk+1k is the negative integrated freshness cost rk ≜ − tk G(t) dt. Integration over physical time correctly weights intervals of different duration. Drop and expiry counts are retained as constraint costs rather than being penalized in the reward. Because multiple overflow actions may occur at one physical epoch, define Nkdrop ≜ Jk , where each rejected arriving bundle or evicted buffered bundle counts as one drop. Let Nkexp denote the total number of bundles whose absolute lifetime expires during transition k; the simulator additionally records the physical location at which each expiry occurs. Their constraints are defined in Section 7. 6. Fairness Modeling Minimizing aggregate excess age does not guarantee balanced service across heterogeneous sources. We therefore measure fairness through useful delivered freshness: service is credited only when an Earth delivery advances the latest generation timestamp of that source. 6.1 Freshness-Service Deficit Because ui (t) changes only when a useful source-i update reaches Earth, we define the delivered freshness gain during transition k as yi,k ≜ ui (tk+1 ) − ui (tk ) ≥ 0. [18] The nonnegativity follows directly from Subsection 4.1, since the max-update rule makes ui (t) nondecreasing. This quantity equals the sum of the positive generationtimestamp advances produced by all source-i Earth deliveries during (tk , tk+1 ]; obsolete or out-of-order deliveries contribute zero through Subsection 4.1. Hence, yi,k measures useful freshness progress rather than the number or temporal regularity of delivered bundles. Let ρi > 0 denote the required long-run freshnessservice rate of source i, measured as delivered generationtime advance per unit physical time. Thus, ρi specifies a target rate of cumulative freshness progress, not a required bundle-delivery frequency or a bound on inter-delivery time. Because transition k lasts δk , its fairness cost is (fair,i) ck ≜ ρi δk − yi,k , [19]

Page 6 of 15

where positive values indicate service shortfall. Accumulated under-service is tracked by the nonnegative deficit h i+ (fair,i) Di (k + 1) ≜ Di (k) + ck . [20] By including Di (k) in the relay-side control state sk , the policy can condition scheduling and dropping decisions on accumulated source-specific freshness-service shortfall. The targets ρi should be compatible with the achievable long-run freshness-service rates under the contact and traffic dynamics. Excess service can reduce an existing deficit, while the projection in Equation [20] prevents surplus service from accumulating as negative credit. Hence, Di (k) tracks uncompensated freshness-service shortfall rather than imposing a bound on inter-delivery gaps. 6.2 Long-Run Fairness Interpretation Equation [20] is a virtual-queue recursion with demand ρi δk and service yi,k . Lemma 6.1 (Deficit rate stability implies freshness fairPK−1 ness). Let t̃K ≜ k=0 δk = tK − t0 . If t̃K → ∞ and Di (K)/t̃K → 0, then K−1 1 X lim inf yi,k ≥ ρi . K→∞ t̃K k=0 Proof. Since [x]+ ≥ x, summing Equation [20] gives PK−1 k=0 yi,k ≥ ρi t̃K − Di (K) + Di (0); division by t̃K and deficit rate stability yield the result. □ For finite-horizon learning, Di (k) serves as a state variable that exposes accumulated freshness-service shortfall (fair,i) to the policy. Separately, ck is used as the signed episodic constraint cost in Section 7. The lemma gives a conditional long-run guarantee on the average delivered freshness-progress rate, provided the deficit is rate stable. It does not guarantee bounded inter-delivery times, bounded AoI, or service within every finite time window. The mechanism targets long-run freshness-service balance rather than fairness over every finite time window. 7. Constrained SMDP Problem Formulation Using the event-driven SMDP defined in Section 5, we optimize destination-side freshness subject to operational and source-specific fairness constraints. 7.1 Returns and Constraint Costs Let πθ denote the stochastic relay policy applied at each applicable control stage using the policy observation constructed from the relay-side control state. For physical transition k, the complete same-time control sequence is ak ≜ (dk , qk ), with dk = (dk,0 , . . . , dk,Jk −1 ), as defined in Section 5. The drop-stage decisions are taken sequentially on their intermediate states, followed by the optional scheduling decision qk , while all such same-time stages belong to one physical SMDP transition.

IAC–26–B2,5,8,x113206

The expected finite-horizon reward and constraint re" # turns are X JR (θ) ≜ Eµ0 ,πθ ,P rk , k:tk <T

" JCj (θ) ≜ Eµ0 ,πθ ,P

[21]

# X

(j) ck

j ∈ J,

k:tk <T

where µ0 denotes the distribution of the initial complete environment state, and the reward rk and transition kernel P are defined in Subsection 5.3. For relay management, the constraint index set and operational costs are J ≜ {drop, exp} ∪ {(fair, i) : i ∈ I}, [22] (drop) (exp) ck ≜ Nkdrop , ck ≜ Nkexp , (fair,i)

The fairness cost ck is defined in Equation [19], while Nkdrop and Nkexp are defined in Subsection 5.3. Each constraint j ∈ J has an episodic budget dj . Let KT denote the number of SMDP transitions in an episode ending at T , so tKT = T . The undiscounted source-i fairness cost then satisfies KX T −1 (fair,i) ck = ρi (T − t0 ) − [ui (T ) − ui (t0 )] , [23] k=0

PKT −1 where the equality follows from k=0 δk = T − t0 and the telescoping definition of yi,k in Equation [18]. Thus, the episodic source-i constraint regulates the total delivered freshness progress over the episode. Because (fair,i) ck is signed, excess freshness service during some transitions can compensate for shortfall during others in the episodic constraint return. This differs from the projected deficit recursion in Equation [20], which does not retain surplus service as negative credit. This finitehorizon constraint is distinct from the conditional longrun freshness-service guarantee in Lemma 6.1, which additionally requires deficit rate stability. 7.2 Constrained Objective and Lagrangian The relay-control problem is max JR (θ) θ

s.t.

[24] JCj (θ) ≤ dj ,

j ∈ J.

Introducing a nonnegative multiplier λj ≥ 0 for each constraint, we form the Lagrangian X L(θ, λ) ≜ JR (θ) − λj JCj (θ) − dj , [25] j∈J

where λ ≜ (λj )j∈J and the primal–dual problem is max min L(θ, λ). [26] θ

λ⪰0

Section 8 implements this formulation using masked Lagrangian PPO with projected multiplier updates.

Page 7 of 15

8. Method: Masked Lagrangian PPO with VariableCardinality Set Pooling We develop a masked Lagrangian PPO controller [23] for the constrained event-driven relay-control problem. The method combines stage-specific physical-feasibility masking, a causal observation adapter for variable-cardinality bundle candidates, permutation-invariant set pooling, holding-time-aware advantage estimation, and Lagrangian constraint updates. Overflow resolution may require multiple same-time drop-stage decisions followed by an optional scheduling decision, while PPO optimization is performed at the level of physical SMDP transitions. Fig. 2 summarizes the controller. 8.1 Stage-Specific Feasibility Masking Let s denote the relay-side control state presented to an individual control stage. The nominal candidate sets are Aesched (s) ≜ B(s) ∪ {wait}, [27] Aedrop (s) ≜ B(s) ∪ {bnew (s)}. The scheduling slate contains the currently buffered bundles together with the elective wait action, whereas the drop slate contains the currently buffered bundles together with the pending arrival. The drop stage is active only when the pending arrival requires overflow resolution. For stage r ∈ {drop, sched}, define χr (u; s) ≜ 1{u ∈ Ar (s)} , u ∈ Aer (s), [28] where Ar (s) is the corresponding feasible action set defined in Section 5. During overflow resolution, every represented bundle in the current drop slate is eligible for rejection or eviction, while the special wait action is not available. After each selected rejection or eviction, the relay state, candidate set, observation, and mask are recomputed on the resulting intermediate state sd,j k . If the incoming bundle remains pending and the capacity violation persists, another masked drop decision is taken at the same physical time. Once the arrival is admitted or rejected, scheduling is reconsidered on the resulting state sdk . No physical time elapses between these same-time control stages. At a scheduling stage, the wait action is available and a bundle is enabled only if relay–Earth service is available, its complete serialization fits within the active contact, and its Earth reception occurs strictly before its absolute expiry, per Equation [11]. If no scheduling stage is enabled, the relay is forced idle and no scheduling action is sampled. The same physical masks are used during training and execution; deterministic evaluation selects the masked argmax. 8.2 Causal Observation & Set Pooling The policy does not observe the hidden upstream state hk introduced in Section 5. Instead, a deterministic adapter

IAC–26–B2,5,8,x113206

constructs a causal numerical observation from the relayside control state s. For a control stage with n(s) represented bundle candidates, o(s) ≜ (X(s), xg (s)) , X(s) ∈ Rn(s)×18 , xg (s) ∈ R50 , [29] where each row of X(s) is the 18-dimensional representation of one represented bundle and xg (s) is the 50-dimensional raw global context. Stable bundle identifiers are retained for action decoding but are not part of these numerical feature tensors. The same feature definitions and fixed normalization rules are used during training, validation, and testing. Each bundle is represented by 18 causal features grouped into four categories: source and size attributes, freshness relevance, expiry/contact feasibility, and buffermanagement context. These features include bundle age and remaining lifetime, potential freshness gain, transmission and deadline slack, overflow pressure, and contactopportunity indicators. The 50-dimensional global vector is naturally partitioned as src xg (s) = [xcore (s), xsrv (s), xpend (s), xsrc M (s), xS (s)] , [30] with dimensions 17 + 6 + 5 + 11 + 11 = 50. The 50-dimensional global context summarizes five information groups: the current relay/contact state, active service, pending Earth delivery, and source-specific recent ingress/contact history for the Moon and Mars. All components are constructed from information available at the relay and the known contact plan. Thus, realized future source arrivals, hidden source queues, future random bundle-size realizations, RNG state, traffic seeds, and trace identifiers are not exposed to the policy. To obtain a fixed-dimensional representation from the variable-cardinality bundle set, the network forms masked mean and elementwise-maximum summaries of the active bundle rows.n Let o I r (s) ≜ u ∈ Aer (s) \ {wait} : χr (u; s) = 1 [31] denote the active bundle support for control stage r. Define x̄r (s) = MeanPool ({xu (s) : u ∈ I r (s)}) , [32] r xmax (s) = MaxPool ({xu (s) : u ∈ I r (s)}) . The raw 50-dimensional global context is then concatenated with the two 18-dimensional pooled summaries: T z r (s) ≜ xg (s)T , x̄r (s)T , xrmax (s)T ∈ R86 . [33] Hence, the 50-dimensional raw global observation xg (s) and the 86-dimensional pooled network context z r (s) are distinct quantities. For value estimation, we use the same mean–max pooling operation over all bundles represented in the relay-side control state, irrespective of their immediate action feasibility. Define

Page 8 of 15

Fig. 2. Masked Lagrangian PPO architecture for event-driven relay control.

and

I V (s) ≜ B(s) ∪ {bnew (s) : bnew (s) ̸= ∅}, h z V (s) ≜ xg (s)T ,

[34] [35]

MeanPool{xu (s) : u ∈ I V (s)}T , iT MaxPool{xu (s) : u ∈ I V (s)}T ∈ R86 . Empty bundle sets use zero pooled summaries. For a represented bundle action u, the corresponding bundle row is concatenated with the pooled context, T x eru (s) ≜ xu (s)T , z r (s)T ∈ R104 . [36] Separate candidate-scoring networks are used for drop and scheduling stages. Thus, for a represented bundle, ℓrθ (u; o(s)) = fθr (e xru (s)) , r ∈ {drop, sched}, [37] whereas the elective wait score at a scheduling stage is obtained directly from the pooled context, ℓsched (wait; o(s)) = fθwait z sched (s) . [38] θ Feasibility masking is applied to the logits before normalization: exp(ℓrθ (u; o(s))) χr (u; s) πθr (u|o(s)) = X , exp(ℓrθ (u′ ; o(s))) χr (u′ ; s) er (s) u′ ∈A [39] for r ∈ {drop, sched}. Consequently, infeasible nominal actions receive zero probability and contribute neither to action selection nor to the stage entropy.

IAC–26–B2,5,8,x113206

8.3 Compound Transition Policy and Value Estimates d,j Let od,j k = o(sk ) denote the observation at drop stage q j, and let ok = o(sdk ) denote the post-overflow scheduling observation. Define Iqk = 1 when a scheduling stage is active and Iqk = 0 otherwise. The stage-wise policies induce the following probability for the complete physical-transition action ak = (dk , qk ) conditional on the initial relay-side state sk : JY k −1 πθ (ak |sk ) ≜ πθdrop dk,j | od,j k [40] j=0 q sched I × πθ (qk | oqk ) k . An empty drop product equals one. Accordingly, JX k −1 log πθ (ak |sk ) = log πθdrop dk,j | od,j k j=0

[41]

+ Iqk log πθsched (qk | oqk ) . Equation [40] subsumes all control cases. If no overflow resolution is required, the empty drop product equals one. If scheduling is enabled, the scheduling probability is multiplied by the probabilities of all preceding same-time drop decisions; otherwise only the drop-stage probabilities contribute. If neither a drop nor a scheduling decision is required, the complete action probability equals one. Thus, all same-time control stages belong to one physicaltransition policy probability.

Page 9 of 15

d,Jk The intermediate drop-stage states sd,0 and k , . . . , sk their policy observations are used only for same-time conditional action selection; they do not constitute separate physical SMDP transitions, holding-time intervals, rewards, constraint costs, or critic time steps. At physical transition boundary k, define zkV ≜ z V (sk ). The critic therefore remains well defined whether the current epoch contains a drop stage, a scheduling stage, or no active control stage. It produces one reward value and one value for each constraint: VR (zkV ), VCj (zkV ), j ∈ J . [42]

Critic evaluation occurs only at physical SMDP transition boundaries, not at intermediate same-time drop stages. 8.4 Holding-Time-Aware Lagrangian PPO To account separately for irregular holding times in temporal credit assignment, define δk time λk ≜ exp − , [43] τGAE where τGAE > 0 is the physical-time scale used by the advantage estimator. For numerical conditioning, rewards and constraint costs are rescaled as (j) c rk (j) rek = , e ck = k , [44] sR sj where sR > 0 and sj > 0 are fixed optimization scales. These rescalings affect only numerical conditioning and not the physical objective or constraint definitions. The reward and constraint TD residuals are V ζkR ≜ rek + VR (zk+1 ) − VR (zkV ), [45] C (j) V ζk j ≜ e ck + VCj (zk+1 ) − VCj (zkV ).

j Âk j = ζk j + λtime Âk+1 . k

where ηH ≥ 0 is the entropy coefficient. For physical transition k, the entropy contribution is the sum of the entropies of all active same-time control stages: JX k −1 q sched q Hdrop (od,j (ok ), [50] Hk = k ) + Ik H j=0

with X Hr (o(s)) = − πθr (u|o(s)) log πθr (u|o(s)). [51] u∈Ar (s)

Thus, infeasible nominal actions contribute neither probability mass nor entropy. Using critic targets [52]

[46]

Terminal transitions use zero bootstrap values. If a rollout segment is truncated before termination, its final critic estimates may instead be used for bootstrap. The episodic reward and constraint objectives remain undiscounted; λtime affects only temporal credit assignk ment in GAE. Let λ denote the current Lagrange multipliers. The Lagrangian advantage is X C R ÂL λj Âk j . [47] k ≜ Âk − j∈J

Before the PPO actor update, ÂL k is standardized over transitions with an active control stage; the individual reward and constraint advantages are not separately standardized.

IAC–26–B2,5,8,x113206

+ ηH E[Hk ],

old V R̂k = ÂR k + VR (zk ),

The holding-time-aware GAE recursions [24] are R time R ÂR Âk+1 , k = ζk + λk

For notational simplicity, we retain ÂL k for the standardized quantity thereafter. Let θold denote the behavior policy that generated the rollout. The PPO importance ratio for physical transition k is πθ (ak |sk ) ξk (θ) ≜ . [48] πθold (ak |sk ) In implementation, the behavior log-probability is stored when the transition is collected, and the numerator is reconstructed during optimization by summing the current log-probabilities of the same active control stages in Equation [41]. Stage observations, stable candidate identities, nominal action slates, and feasibility masks are stored with the rollout so that the behavior and current policies are evaluated on identical action supports. The clipped PPO actor objective is h Jactor (θ) = E min ξk (θ)ÂL k, i [49] clip ξk (θ), 1 − ϵppo , 1 + ϵppo ÂL k

Ĉj,k = Âk j + VCold (zkV ), j the joint loss for the five h critic outputsis 2 Lcritic = E VR (zkV ) − R̂k X 2 i + VCj (zkV ) − Ĉj,k .

[53]

j∈J

Actor and critic parameters are updated from the same physical-transition minibatches using their respective optimization losses and learning-rate schedules. Numerical optimizer settings are specified with the experimental training configuration rather than being part of the constrained SMDP definition. 8.5 Lagrange-Multiplier Update Multiplier updates use completed episodic constraint returns. Let Ecomp denote the set of episodes completed since the previous multiplier update. The empirical return

Page 10 of 15

Algorithm 1 Masked Lagrangian PPO for Event-Driven Relay Control

of constraint j is JbCj ≜

1 |Ecomp |

K e −1 X

ce,k ,

(j)

[54]

j ∈ J,

[55]

e∈Ecomp k=0

The projected dual update is " #+ JbCj − dj λj ← λj + ηλ , sj

where ηλ > 0 is the dual step size. Thus, an episodic return above its budget increases the corresponding multiplier, while a return below its budget permits the multiplier to decrease toward zero. The normalization by sj affects only the numerical dual update and does not change the physical constraint JCj ≤ dj . 8.6 Training Procedure Each rollout record corresponds to one physical SMDP transition and stores (j) ok , ak , rk , {ck }j∈J , δk , ok+1 , Ck , ιterm , [56] k where Ck contains the same-time control-stage observations, stable candidate identities, nominal action slates, feasibility masks, selected actions, and behavior logprobabilities required to reconstruct the compound action probability. The indicator ιterm ∈ {0, 1} identifies termik V nal physical transitions. The pooled contexts zkV and zk+1 are deterministic functions of the stored observations and masks and are reconstructed by the network when value estimates are evaluated. After rollout collection, the holding-time factors, reward and constraint advantages, critic targets, and Lagrangian advantages are computed from the physicaltransition sequence. The quantities used as PPO targets, together with the stored behavior log-probabilities, remain fixed during the PPO epochs for that rollout. Current policy probabilities, masked entropies, and value estimates are re-evaluated on each optimization minibatch. The actor and critics are trained from random initialization using PPO. Stage-specific action masks are applied during both training and execution. Multiplier updates use completed episodic returns and are performed after the PPO updates for the corresponding rollout, so the updated multipliers affect subsequent data collection rather than changing advantages already computed for the current rollout. Algorithm 1 summarizes the procedure. 9. DTNSim Implementation and Simulator Interface We extend DTNSim [25] to implement the event-driven relay environment of Section 5 and the masked control interface of Section 8. The implementation preserves the shared-buffer, non-preemptive service, delayed-reception, and co-timed event semantics defined by the model.

IAC–26–B2,5,8,x113206

Require: T, τGAE , ϵppo , ηH , ηλ , {dj }j∈J , KPPO 1: Initialize actor, critics, and λj ≥ 0 2: while training criterion not satisfied do 3: Initialize rollout buffer R and completed-episode set Ecomp 4: while rollout collection is incomplete do 5: Observe sk and construct ok ; initialize dk ← () 6: while overflow resolution is required do 7: Construct masked drop policy at sd,j k and sample dk,j 8: Apply dk,j and append it to dk 9: end while 10: if scheduling is enabled then 11: Construct masked scheduling policy at sdk and sample qk 12: else 13: qk ← ∅ 14: end if 15: Set ak ← (dk , qk ) and advance DTNSim to tk+1 (j) Observe (rk , {ck }j∈J , δk , ok+1 ) and store the transition 16: and active stage records in R 17: Accumulate completed episodic constraint returns 18: end while Compute λtime = exp(−δk /τGAE ), reward/constraint GAE, 19: k critic targets, and ÂL k 20: for p = 1, . . . , KPPO do 21: for each shuffled minibatch M ⊂ R do Re-evaluate masked stage probabilities and form PPO ratios 22: 23: Update actor and critic parameters 24: end for 25: end for 26: if Ecomp ̸= ∅ then 27: Estimate JbCj and update λj 28: end if 29: end while

9.1 Runtime Architecture and Environment Contract DTNSim owns the simulation clock, source generation and source queues, source–relay transmission and propagation, contact plan, bundle movement, and physical event calendar. A relay-control adapter exposes the relay-side state, constructs the causal observation and stagespecific masks, applies relay actions, and returns decisionto-decision SMDP transitions. The training driver updates the actor, critic parameters, and Lagrange multipliers without directly modifying simulator state. Each decision epoch begins after all physical events at time tk have been processed. If relay capacity is exceeded, the controller repeatedly selects same-time rejection or eviction actions dk = (dk,0 , . . . , dk,Jk −1 ) until the overflow is resolved, recomputing the observation and mask after each decision. Scheduling is then reconsidered on the resulting state. The complete relay action is therefore ak = (dk , qk ), with no physical time elapsing between its internal drop and scheduling stages. After the compound action is completed, DTNSim advances to the next decision epoch. Following an elective

Page 11 of 15

wait while a feasible transmission exists, the controller is awakened after at most 30 s or earlier if a physical event occurs. The interface returns the next observation, reward rk , constraint-cost vector, holding time δk , termination status, and the stage information required to reconstruct the compound policy probability during PPO optimization. At the terminal time T , physical events scheduled exactly at T are processed before termination. The simulator stops without draining the network state; unexpired backlog is unresolved rather than counted as loss. Learned and heuristic controllers use the same physical simulator, contact plan, traffic realization, and relay-control interface. 9.2 DTNSim Extensions and Validation Shared relay storage: Moon and Mars bundles share the finite relay-buffer budget Bmax . Bundles already in relay–Earth service or propagation are excluded from buffer occupancy. DTNSim applies no independent overflow rule; all capacity violations are resolved through the controllerdefined rejection/eviction sequence. Contact-aware service: Relay–Earth transmission is non-preemptive and single-server. A bundle may enter service only if its complete serialization fits within the active contact and its Earth reception occurs strictly before absolute expiry, i.e., tk + L(b)/RRE ≤ tend and tk + L(b)/RRE + τRE < texp b . Thus, service admission requires timely Earth reception, not merely transmission completion before contact closure. A buffered bundle that can no longer satisfy the delivery deadline remains stored until overflow removal or absolute expiry, but is masked from relay–Earth scheduling. Delayed reception and event ordering: Relay transmission completion releases the server but does not update destination freshness; the corresponding update occurs only at Earth reception. Co-timed events follow the deterministic ordering in Subsection 5.1, after which all applicable same-time control stages are completed before physical time advances. The frozen DTNSim integration was validated against the Python reference implementation before policy evaluation. Source-ingress equivalence was verified over 67 traces and 8,085 bundles with zero timing deviation, and 134 automated tests covered observations, masks, event handling, and metric computation. Additional regression tests verified shared-buffer accounting, action feasibility, strict Earth-before-expiry delivery, delayed reception, co-timed event ordering, and reproducibility under fixed traces and frozen checkpoints. After validation, the heldout evaluation campaign compared five controllers on 64 common traffic traces, yielding 320 evaluation episodes under identical simulator conditions.

IAC–26–B2,5,8,x113206

Table 1. Frozen evaluation and PPO training configuration. Parameter

Setting

Episode interval Traffic generation

t0 = 1800 s, T = 16200 s; H = 14400 s Independent Poisson; mean interarrival: Moon 180 s, Mars 360 s 20/100/300 Mbit with probabilities 0.5/0.3/0.2 5400 s from original generation 1.2 × 109 bits 1800 s M → R : [0, 1200); S → R : [900, 1500); R → E : [1200, 1800) 2 Mbit/s on all links τM R = 1.3 s; τSR = 600 s; τRE = 0.12 s Lref = 20 Mbit wM = wS = 1; ϕM (x) = ϕS (x) = x ∆M (t0 ) = 1190.0 s; ∆S (t0 ) = 1179.88 s ρM = 0.8441; ρS = 0.8364 τGAE = 900 s ϵppo = 0.2 10−4 → 10−5 (linear decay) 10−3 → 10−4 (linear decay) 0.01 → 0.001 (linear decay) 16 episodes/update; 4 PPO epochs; 512 physical transitions/minibatch 2048 episodes; random initialization ηλ = 0.05; λ0 = (0.1, 0.1, 0, 0) sR = HTc ; (sdrop , sexp , sfair,M , sfair,S ) = (17, 1, H, H) max. gradient norm 0.5; KL threshold 0.03 (ddrop , dexp , dfair,M , dfair,S ) = (17, 1, 0, 0) 5 controllers × 64 common traffic traces (320 evaluation episodes)

Bundle sizes Bundle TTL Relay capacity Bmax Contact cycle Tc Contact windows Link rates Propagation delays Reference bundle size Freshness objective Initial Earth AoI Fairness rates GAE time scale PPO clipping Actor learning rate Critic learning rate Entropy coefficient PPO batching Training duration Dual update Reward/cost scales

Optimization safeguards Constraint budgets Held-out evaluation

10. Evaluation Setup All policies are evaluated through the common DTNSim execution path of Section 9. They use identical traffic realizations, contact opportunities, relay capacity, expiry semantics, and physical feasibility rules, and differ only in relay-side action selection. Table 1 summarizes the frozen simulation, training, and evaluation configuration. 10.1 Scenario and Training Configuration The experiments use the Moon–Mars–relay–Earth topology of Section 3. Source generation, source-side FIFO service, propagation, shared-buffer admission, relay– Earth transmission, and delayed Earth reception follow the system model and are not repeated. The same contact-plan freshness reference is used for all policies. 10.2 Compared Policies We compare the proposed Masked Lagrangian PPO controller with four deterministic baselines. FIFO schedules the feasible bundle with the earliest relay arrival,

Page 12 of 15

Table 2. Excess-AoI performance over the 64 held-out traces. Values are means in seconds; lower is better. Policy Masked Lag. PPO FIFO LIFO EDF Freshest-first

Aggregate ↓

Moon ↓

Mars ↓

887.63 1747.87 993.57 1692.38 1009.56

216.64 686.89 232.83 806.15 232.82

670.98 1060.97 760.74 886.23 776.74

(a) Contact availability Moon → Relay Mars → Relay Relay → Earth

(b) Bundle arrivals at the relay 20 Mbit 100 Mbit

Moon

300 Mbit Moon

Mars

10.3 Evaluation Protocol and Metrics Each policy is evaluated on the same 64 held-out stochastic traffic traces, ensuring identical exogenous conditions across policy comparisons. Reported performance values are arithmetic means over these common traces. All training hyperparameters and evaluation rules were fixed before the held-out evaluation. The primary performance metric is the time-average RT 1 aggregate excess-age cost Ḡ ≜ H G(t) dt, where lower t0 values indicate less excess staleness relative to the contactplan reference. With unit source weights and linear penalties, this equals the time-average aggregate excess AoI used for policy comparison. Secondary diagnostics include source-specific excess, drop and expiry counts, delivered traffic, relay–Earth contact PKT −1utilization, cumulative useful freshness progress k=0 yi,k , terminal deficits Di (KT ), and episodic constraint returns. Relay–Earth contact utilization is defined as transmitted bits divided by the total available relay–Earth contact capacity. Constraint compliance is assessed against the budgets in Table 1. 11. Results and Discussion We compare Masked Lagrangian PPO with FIFO, LIFO, EDF, and Freshest-first under the common evaluation protocol of Section 10. All values are computed over the same 64 held-out stochastic traffic traces. 11.1 Freshness Performance Table 2 reports the time-average aggregate and sourcespecific excess AoI. Masked Lagrangian PPO achieves the lowest aggregate excess AoI among the evaluated policies. Relative to FIFO and EDF, the controller reduces mean aggregate excess AoI by 49.2% and 47.6%, respectively.

IAC–26–B2,5,8,x113206

Transmit

Tx Mars

Drop

Drop Moon marker size ∝ bundle size

Drop Mars

Occupancy [Gbit]

LIFO the latest relay arrival, EDF the earliest absolute expiry, and Freshest-first the largest generation timestamp. During overflow, each baseline discards its least-preferred bundle under the corresponding ordering: FIFO discards the latest arrival, LIFO the earliest arrival, EDF the latest expiry, and Freshest-first the smallest generation timestamp. Remaining ties are resolved using the stable bundle identifier. All policies use the same candidate sets and physical feasibility rules.

(d) Relay buffer occupancy after eviction

Bmax = 1.2 Gbit

offered (before eviction)

drop event

0 0.0

0.5

1.0

1.5

2.0

2.5

3.0

Time since episode start [h]

Fig. 3. Relay operation under the PPO policy. Relative to the LIFO and Freshest-first baselines, the reductions are 10.7% and 12.1%. The improvement is present for both Moon and Mars, showing that the aggregate result is not obtained by sacrificing one source. Because all policies use common traffic traces, performance can be compared trace by trace. Masked Lagrangian PPO has lower aggregate excess AoI than FIFO and EDF on all 64 traces and lower excess AoI than both LIFO and Freshestfirst on 59 of the 64 traces. Thus, the mean advantage is broadly distributed across the evaluation set rather than driven by a small number of favorable traces. 11.2 Operational Performance Table 3 summarizes relay operation under the heterogeneous bundle-size distribution. The bundle-based and bit-based delivery metrics are both reported because they need not rank policies identically when bundle sizes differ. The proposed controller produces substantially fewer overflow drops and delivers the largest number of bundles, giving the highest bundle delivery ratio. However, it delivers fewer total bits and uses less relay–Earth contact capacity than the baselines. Thus, its freshness improvement is not obtained by transmitting more data or occupying more downlink capacity. Instead, the results indicate more selective use of the available transmission opportunities according to information freshness. The distinction between bundle and bit delivery is important in this setting: Masked Lagrangian PPO achieves a bundle delivery ratio of 0.860 while its bit delivery ratio is 0.686. This difference reflects the heterogeneous bundle sizes and prevents bundle-count delivery alone from being interpreted as a throughput metric. Figure 3 provides an event-level view of the relay be-

Page 13 of 15

Table 3. Mean operational performance over 64 traces. Expiries count relay-side events used in the constraint. Policy

Drops ↓

Exp. ↓

Delivered

Delivered Gbit

Bundle DR

Bit DR

Ucontact

Masked Lag. PPO FIFO LIFO EDF Freshest-first

10.02 16.20 17.88 16.91 16.78

0.05 0.00 0.20 0.00 0.17

103.72 96.28 95.42 95.61 96.56

8.086 8.578 8.286 8.577 8.262

0.860 0.801 0.794 0.796 0.803

0.686 0.727 0.703 0.727 0.701

0.845 0.897 0.865 0.896 0.863

havior underlying the aggregate results in Table 3. Over the representative three-hour interval, Moon and Mars bundles arrive asynchronously with heterogeneous sizes of 20, 100, and 300 Mbit, while their transmission opportunities are restricted by the scheduled contacts. The relay consequently selects different buffered bundles for transmission and invokes drop decisions when incoming traffic creates buffer pressure. As seen in Fig. 3(d), the offered pre-eviction occupancy can exceed the Bmax = 1.2 Gbit capacity, whereas the post-decision occupancy is maintained within the buffer limit. The timeline therefore illustrates how scheduling and overflow-management decisions interact with intermittent connectivity, heterogeneous bundle sizes, and finite relay storage. 11.3 Fairness and Constraint Compliance Table 4 reports the empirical mean episodic constraint returns. The corresponding budgets are (17, 1, 0, 0). Masked Lagrangian PPO satisfies all four budgets on the empirical mean. LIFO is the only selected baseline that is not mean-feasible, because its mean drop return 17.88 exceeds the budget of 17. All five policies have negative mean source-specific fairness returns, indicating that their mean delivered freshness progress exceeds the required episodic service level for both sources. For additional finite-horizon context, the proposed controller achieves cumulative useful freshness progress of 14162.9 s for Moon and 13737.8 s for Mars, with terminal reflected deficits of 484.0 s and 237.2 s, respectively. These reflected deficits are diagnostic state variables rather than constraint returns; positive terminal values therefore do not imply fairness-constraint violation and do not establish or refute the long-run rate-stability condition of Lemma 6.1. 11.4 Overall Discussion The results show a consistent trade-off between freshness and transmitted volume. Masked Lagrangian PPO achieves the lowest aggregate and per-source excess AoI while satisfying the empirical mean operational and fairness budgets. Its advantage is observed on every trace against FIFO and EDF and on 59 of 64 traces against LIFO and Freshest-first. At the same time, the controller transmits fewer bits and uses less relay–Earth contact capacity than the deterministic baselines.

IAC–26–B2,5,8,x113206

Accordingly, the observed freshness improvement should be interpreted as an improvement in relay-side information selection rather than increased throughput. The results are descriptive statistics over the fixed held-out policy evaluation and do not constitute a claim of statistical significance or robustness to independent retraining. 12. Conclusion This paper developed a goal-oriented relay-control framework for deep-space DTNs with stochastic heterogeneous traffic, finite relay storage, intermittent contacts, and delayed Earth reception. Joint scheduling and dropping are formulated as a constrained event-driven SMDP and solved using Masked Lagrangian PPO with feasibility-aware action masking and variable-cardinality set encoding. The framework is implemented and evaluated in DTNSim. Across 64 common held-out traffic traces, the proposed controller achieves the lowest mean aggregate excess AoI among the evaluated policies: 887.63 s, compared with 993.57 s for LIFO, 1009.56 s for Freshest-first, 1692.38 s for EDF, and 1747.87 s for FIFO. It also satisfies the empirical mean drop, expiry, and source-specific fairness budgets. These results show that relay decisions based on destination-side freshness and system context can outperform fixed queue-ordering heuristics under the considered deep-space DTN scenario. The present study assumes a known contact plan, fixed routing, a single controlled relay, and non-preemptive noresume bundle transmission. Future work will consider component-wise ablation, contact-plan uncertainty, partialtransfer service, multi-relay control, and validation with operational DTN implementations. Acknowledgment This work was supported by the ERC Advanced Grant GOSPACE (No. 101122990), TÜBİTAK Grant 123C572 (GOREV), and the 1515 Frontier R&D Laboratories Support Program for the Türk Telekom 6G R&D Lab (Project No. 5249902). The views expressed are solely those of the authors and do not necessarily reflect those of the European Union or the European Research Council Executive Agency, neither of which can be held responsible for them. The authors used AI for language refinement and figure polishing. All AI-assisted material was verified by the authors for consistency with the system architecture.

Page 14 of 15

Table 4. Mean episodic constraint returns over the 64 held-out traces. A policy is mean-feasible when all four empirical means satisfy their corresponding budgets. Policy

JbCdrop

JbCexp

JbCfair,M

JbCfair,S

Mean-feasible

Budget

—

Masked Lag. PPO FIFO LIFO EDF Freshest-first

10.02 16.20 17.88 16.91 16.78

0.05 0.00 0.20 0.00 0.17

−2007.89 −1764.52 −2010.25 −1592.67 −2010.25

−1693.17 −1138.05 −1659.19 −1452.85 −1658.08

Yes Yes No Yes Yes

References [1] V. G. Cerf et al., Delay-tolerant networking architecture, RFC 4838, IRTF, Apr. 2007.

[14]

R. D. Yates et al., “Age of information: An introduction and survey,” IEEE J. Sel. Areas Commun., vol. 39, 2021.

Y. Zhu et al., “Fairness-aware age-of-information minimization in WPT-assisted short-packet data collection for mURLLC,” IEEE Trans. Mobile Comput., vol. 25, 2026.

[15]

R. D. Yates et al., “The age of information: Real-time status updating by multiple sources,” IEEE Trans. Inf. Theory, vol. 65, 2019.

A. Li et al., “Toward goal-oriented semantic communications: New metrics, framework, and open challenges,” IEEE Wireless Commun., 2024.

[16]

C. Caini et al., “Delay- and disruption-tolerant networking: An alternative solution for future satellite networking applications,” Proc. IEEE, vol. 99, 2011.

A. Li et al., “Goal-oriented tensor: Beyond age of information toward semantics-empowered goal-oriented communications,” IEEE Trans. Commun., 2024.

[17]

Y. Lyu et al., “Dynamic routing for integrated satellite-terrestrial networks: A constrained multi-agent reinforcement learning approach,” IEEE J. Sel. Areas Commun., vol. 42, 2024.

[18]

L. Yang et al., “Optimizing deep-space DTN congestion control via deep reinforcement learning,” Comput. Netw., vol. 255, 2024.

[19]

S. Huang et al., “A closer look at invalid action masking in policy gradient algorithms,” in FLAIRS, vol. 35, 2022.

[2]

S. Burleigh et al., Bundle protocol version 7, RFC 9171, IETF, Jan. 2022.

[3]

[4]

[5]

[6]

G. Araniti et al., “Contact graph routing in DTN space networks: Overview, enhancements and performance,” IEEE Commun. Mag., vol. 53, 2015.

[7]

J. A. Fraire et al., “Routing in the space internet: A contact graph routing tutorial,” J. Netw. Comput. Appl., 2021.

[8]

P. G. Madoery et al., “Routing heterogeneous traffic in delay-tolerant satellite networks,” IEEE J. Radio Freq. Identif., vol. 7, 2023.

[9]

T. Alhajj et al., Improved contact graph routing in delay tolerant networks with capacity and buffer constraints, 2024.

[10]

T. Le, “A unified utility-based framework for joint scheduling and buffer management in delay tolerant networks,” Comput. Commun., vol. 249, 2026.

[11]

X. Xie et al., “Scheduling for minimizing the age of information in multisensor multiserver industrial internet of things systems,” IEEE Trans. Ind. Informat., vol. 20, 2024.

[12]

[13] N. Jones et al., “Achieving AoI fairness in spatially distributed wireless networks: From theory to implementation,” in WiOpt, IEEE, 2024.

V. Tripathi et al., “Fresh-CSMA: A distributed protocol for minimizing age of information,” J. Commun. Netw., vol. 25, 2023.

IAC–26–B2,5,8,x113206

[20] J. Achiam et al., “Constrained policy optimization,” in ICML, vol. 70, 2017. [21]

Z. Sun et al., “Constrained reinforcement learning under model mismatch,” in ICML, vol. 235, 2024.

[22]

R. S. Sutton et al., “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning,” Artif. Intell., 1999.

[23]

J. Schulman et al., Proximal policy optimization algorithms, 2017.

[24]

J. Schulman et al., “High-dimensional continuous control using generalized advantage estimation,” in ICLR, San Juan, Puerto Rico, 2016.

[25]

M. S. Net, DtnSim: A Python-based simulator for delay tolerant networking, 2019.

Page 15 of 15

Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning

Analyze & explore related research in Conceptio

Related documents