Delay-Robust Deep Reinforcement Learning for Ranging-Free Channel Access under Mobility in Underwater Acoustic Networks Huaisheng Ye1 , Xiaowen Ye2 , and Liqun Fu1 1
arXiv:2605.06536v1 [cs.NI] 7 May 2026
2
School of Informatics, Xiamen University, Xiamen 361102, China College of Photonic and Electronic Engineering, Fujian Normal University, Fuzhou 350117, China Email: [email protected], [email protected], [email protected]
Abstract—Long propagation delays in underwater acoustic networks (UWANs) cause spatio-temporal uncertainty, constraining channel utilization in medium access control (MAC) protocols. Node mobility within autonomous underwater vehicle scenarios exacerbates these challenges by introducing dynamic propagation delays and varying spatial topologies. We present MobiU-MAC, a deep reinforcement learning (DRL)-based MAC protocol for mobile node access in UWANs that maximizes throughput via autonomous learning. MobiU-MAC incorporates CHILL-STER, a novel DRL algorithm optimized for UWANs that is both ranging-free and delay-robust. CHILL-STER employs a credit horizon-limited λ-return (CHILL-Return) mechanism to achieve stable learning under asynchronous delayed rewards, while the companion spatio-temporal experience replay (STER) mechanism addresses topological changes arising from node mobility. This work also demonstrates theoretically that DRL attains optimal policy learning equivalent to a standard Markov decision process under long propagation delays without requiring ranging. Performance evaluations indicate that MobiU-MAC outperforms existing DRL-based MAC protocols for UWANs by leveraging the maximum system delay boundary without ranging overhead, supporting the effectiveness of the proposed theory and algorithm in complex underwater dynamic environments. Index Terms—Underwater acoustic networks, Medium access control, Autonomous underwater vehicles, Deep reinforcement learning, λ-return.
I. I NTRODUCTION Underwater acoustic networks (UWANs) primarily utilize acoustic waves for long-range communication due to the severe attenuation of electromagnetic waves in marine environments. Given narrow spectrum and Doppler spread, TDMA is more robust than FDMA/CDMA and thus the most prevalent access method in UWANs [1]. However, the low propagation speed of acoustic waves ( 1500 m/s) induces long propagation delays that challenge the design of medium access control (MAC) protocols. This delay introduces spatio-temporal uncertainty [2], which reduces the efficiency of traditional handshake and carrier-sensing mechanisms. Consequently, intelligent access methods are required to sense the dynamic environment and facilitate adaptive decisions. Deep reinforcement learning (DRL) has proven effective in sequential decision-making for dynamic underwater enviCorresponding author: Liqun Fu. The source code of this work is available at https://github.com/HysonYe/CHILL-STER.
ronments, with demonstrated utility in link adaptation [3], routing [4], and AUV-assisted trajectory planning [5]. This trend is now driving MAC protocol designs toward intelligence for coordinated multi-node access. Early research employed tabular Q-learning to optimize access strategies [6], though this approach is limited by high-dimensional state spaces [1], [2]. With the rise of deep learning, DRL has gradually become a core tool for designing intelligent MAC protocols in UWANs. For instance, the DR-DLMA protocol incorporates a DR-DQN algorithm using propagation delay information for experience replay with slot alignment, addressing the problem of training failure caused by ACK feedback delay [2]. The DL-MAC protocol further explores asynchronous transmission, which reduces collisions caused by spatio-temporal uncertainty through flexibly adjusting transmission start times, thereby improving network throughput [1]. Furthermore, multi-agent reinforcement learning (MARL)-based MAC protocols for UWANs have recently emerged [7], [8]. However, existing DRL-based methods involve a trade-off regarding the decision interval. In time-division MAC protocols, the slot duration is often directly coupled with the decision time-step, making this contradiction especially prominent. Existing research falls into conservative and radical schemes. Conservative schemes use a time-step exceeding the maximum one-way propagation delay [1], avoiding delayed feedback but reducing time-domain utilization. Radical schemes use a timestep much smaller than the maximum one-way propagation delay [2], [7], [8], maximizing time-domain utilization and network performance at the cost of severe delays in both observation feedback and reward feedback. In dynamic topologies, reward feedback delay is particularly challenging and remains unresolved. In such networks, ranging-based reward alignment is impractical due to high overhead, leading to a credit assignment problem that hinders the convergence of DRL agents. To overcome these limitations, this paper adopts the radical scheme and proposes CHILL-STER, a novel DRL algorithm tailored for the underwater acoustic environments. Its core advantages are that it is ranging-free and exhibits delay-robust. The algorithm requires no real-time ranging overhead and enables efficient learning using only the maximum delay bound
Access Point D1
D2 D3
AUV TDMA
D4
D5
ALOHA
Fig. 1. System model of the heterogeneous UWANs.
of the system. Building on this, this paper develops MobiUMAC, an intelligent MAC protocol designed for mobile node access scenarios. The protocol can adaptively sense topology changes and significantly improve network throughput in dynamic environments through optimized access policies. 1) For the first time, we prove that learning the optimal policy in underwater environments with long propagation delays is equivalent to that in a standard Markov decision process (MDP). Specifically, by revealing the intrinsic mathematical relationship between delay bounds and return estimation in UWANs, we show that DRL algorithms based on multi-step returns (e.g., n-step return, λ-return [9]) retain convergence to the optimal strategy of a standard MDP, even without explicit delay compensation. 2) We introduce CHILL-STER, a ranging-free and delayrobust algorithm comprising a Credit Horizon-Limited λ-return (CHILL-Return) to enable convergence under asynchronous rewards and a Spatio-Temporal Experience Replay (STER) to mitigate node mobility-induced instability. 3) We develop and evaluate MobiU-MAC, an intelligent MAC protocol for mobile node access. Simulation results indicate that MobiU-MAC improves channel utilization and robustness in high-speed mobility scenarios. II. S YSTEM M ODEL We consider a heterogeneous UWAN with an Access Point (AP) and N underwater nodes (one mobile AUV and N − 1 static nodes) sharing an uplink channel, as illustrated in Fig. 1. The AUV, moving continuously to perform underwater operations, is equipped with a DRL-based MAC protocol and acts as a DRL agent. Concurrently, all underwater nodes continuously upload traffic data to the AP. To emulate a heterogeneous environment, static nodes each follow either TDMA (preallocated slots) or ALOHA (transmission probability q). Following the setup in [2], the system is time-slotted with duration τ , where each slot covers one uniform-length data packet and its corresponding ACK. In each time slot, a node decides to either transmit or wait according to its MAC protocol. The AP listens for data in the first part of each time slot and can successfully receive at most one packet. Multiple packets arriving in the same time slot cause a collision. Otherwise, the
AP receives a data packet from a node, it broadcasts an ACK packet containing the MAC address of that node in the latter part of the time slot. As underwater acoustic propagation is far slower than radio waves, propagation delay is significant in UWANs. The oneway propagation delay for node i at distance di is Di slots. Specifically, di , (1) Di = cτ where c is the underwater sound speed. As the AUV moves, its distance to the AP and the propagation delay D(t) vary, causing a time-varying topology and enhanced spatio-temporal uncertainty. The agent must then autonomously learn to access time slots not occupied by static nodes during AUV movement, maximizing total network throughput. III. D EEP R EINFORCEMENT L EARNING FOR UWAN S WITH D ELAYED R EWARDS A. Markov Decision Process and Preliminaries of DRL Standard Markov decision processes are defined by the tuple (S, A, P, R, γ), where S and A are finite sets of states and actions, P : S × A × S → [0, 1] satisfies P (s′ |s, a) = Pr(st+1 = s′ |st = s, at = a), while R : S × A → R denotes the immediate reward function and γ ∈ [0, 1) is the discount factor [9]. At step t, the agent executes at ∼ π(st ), receives rt+1 , and transitions to st+1 . The agent aims to find an optimal π ∗ maximizing the expected return P∞ policy k Gt = Eπ [ k=0 γ rt+k+1 ]. The optimal action-value function Q∗ (s, a) satisfies the Bellman equation Q∗ (s, a) = E[R(s, a) + γ maxa′ Q∗ (s′ , a′ )]. To handle large or continuous state spaces, the Q-function is approximated using a deep neural network Q(s, a; θ) with parameters θ. The deep Q-network (DQN) algorithm [10] minimizes the temporal-difference (TD) loss: h i 2 (2) L(θ) = E(s,a,r,s′ )∼D (Q(s, a; θ) − y) , where the target value is y = r + γ maxa′ Q(s′ , a′ ; θ− ), with D the experience replay buffer storing past transitions and θ− the target network parameters updated periodically to stabilize training. However, this standard formulation assumes that the reward r is observed immediately after taking action a, an assumption that is violated in UWANs due to long propagation delays. B. Challenges of Standard DRL: Reward Feedback Delay and Asynchrony While observation delay is addressed by incorporating historical observations into the state representation [1], [2], this paper focuses on the more critical reward delay problem. This delay stems from the round-trip propagation of ACKs, which violates the standard DRL assumption. Fig. 2 illustrates that under the radical scheme, the reward for at is only observed at time step t + M , where M = 2D(t) + 1. Crucially, rewards in the window [t + 1, t + M − 1] are independent of at and depend instead on the environment or
Data packet
ACK packet
Reliable Q estimation requires a multi-step return horizon H = M spanning the entire delay window, yielding the following multi-step return target:
Node i AP
ynstep =
Time
M −2 X
at
Reward delay 2D t
rt M
Fig. 2. Impact of long propagation delay on reward observation. Due to the round-trip propagation delay 2D(t) , the reward rt+M for action at is only observed at time step t + M , where M = 2D(t) + 1 .
other nodes. Single-step TD updates incorrectly attribute this noise to at , causing non-convergence. Existing DRL methods [2] align rewards using rangingbased estimates of D(t). Upon receiving the complete ACK at the end of time slot t + 2D(t), the agent constructs (st , at , rt+2D(t)+1 , st+1 ) for standard DRL updates, restoring the assumption that reward follows immediately after the action. However, ranging incurs high overhead and is impractical for mobile nodes with fast-changing topologies. C. Theoretical Analysis: Ranging-Free Optimal Policy Equivalence Under Delays 1) Optimal Policy Equivalence: We assume a node with a fixed one-way propagation delay D described in (1). The reward for action at is observed following a round-trip delay M = 2D + 1. The aligned Q-function for a standard MDP immediate rewards is defined as Qalign (st , at ) = Pwith ∞ Eπ [ n=0 γ n rt+M +n | st , at ]. In the actual delayed environment, the return associated with at includes M − 1 irrelevant rewards: M −2 X
Q(st , at ) = Eπ [
m=0
γ m rt+m+1 +
∞ X
γ M +n−1 rt+M +n ]st ,at
n=0
The first M − 1 rewards {rt+1 , . . . , rt+M −1 } are independent of at ; they depend only hPon state st . Denote this i state-dependent M −2 m bias as V (st ) = Eπ γ r s t+m+1 t . Thus Q decomm=0 M poses as Q(st , at ) = V (st ) + γ Qalign (st , at ). Since γ M > 0 and V (st ) does not vary with at , for any fixed st : arg max Q(st , a) = arg max Qalign (st , a). a
a
a
m=0
| Agent
Q(st+M , a′ ), γ m rt+m+1 +γ M −1 rt+M + γ M max ′
(3)
{z
Vnoise
where Vnoise is the action-independent noise and rt+M is the true feedback. Although Vnoise adds variance to the gradient, the presence of γ M −1 rt+M provides a genuine signal that guides the update toward the optimal policy. For mobile nodes, we utilize a prior delay bound Dmax = ⌈dmax /(cmin · τ )⌉ based on maximum distance. This yields the following design criterion. Design Criterion: Establishing the mathematical relationship between the multi-step return horizon H and the delay bound, H ≥ 2Dmax + 1, (4) ensures that the return contains the reward signal rt+2D(t)+1 for any delay D(t) ≤ Dmax . This enables stable convergence to the optimal policy without any real-time ranging overhead. Unlike [11], [12] which use multi-step returns empirically, this work establishes the theoretical policy equivalence under long delays. IV. M OBI U-MAC: A D ELAY-ROBUST AND R ANGING -F REE MAC P ROTOCOL D RIVEN BY CHILL-STER A. CHILL-STER Algorithm: Dual Mechanisms for Underwater Challenges Although Section III theoretically guarantees policy optimality with a multi-step return horizon (see (4)), naive multistep methods suffer from high variance and on-policy bias, while node mobility further introduces non-stationary topology dynamics. To address these challenges, we propose CHILLSTER, integrating Credit Horizon-Limited λ-return (CHILLReturn) for delay-robustness and Spatio-Temporal Experience Replay (STER) for robustness to topology changes. 1) Credit Horizon-Limited λ-Return: CHILL-Return balances the bias-variance trade-off by restricting the λ-return to a compact horizon Ĥ (satisfying (4)). Off-policy bias is mitigated via importance sampling. The Ĥ-step return is Ĥ Gλ, = t
Ĥ−1 X k=0
This equivalence directly implies the following theorem. Theorem 1 (Policy Equivalence): Maximizing the unaligned return without ranging or reward alignment yields the same optimal policy as the aligned standard MDP. 2) Multi-Step Return Horizon: Although Theorem 1 guarantees equivalence, single-step TD learning fails because its TD target y1step = rt+1 + γ maxa′ Q(st+1 , a′ ) (see the target y in (2)) uses rt+1 , which is uncorrelated with at , leading to high-variance gradients.
}
(γλ)k δt+k + (γλ)Ĥ max Q(st+Ĥ , a′ ), ′ a
(5)
where δt = rt + γ maxa′ Q(st+1 , a′ ) − Q(st , at ) is the TD error. We apply a clipped importance weight wt = min 1, ρβt+1:t+Ĥ−1 , β ∈ (0, 1], (6) based on trajectory-wise importance factor ρt+1:t+Ĥ−1 = QĤ−1 π(at+k |st+k ) k=1 µ(at+k |st+k ) , where π is the target policy and µ is the behavior policy. The hyperparameter β ∈ (0, 1] controls the degree of correction applied.
Combining the Ĥ-step return with importance sampling, CHILL-Return minimizes the following loss to update the value network: 2 L(θ) = E(st ,at )∼D wt Gtλ,Ĥ − Q(st , at ; θ) . (7) While we present CHILL-Return in the action-value form Q(s, a) for concreteness, the same credit horizon-limiting principles can be extended to other value-based frameworks (e.g., DDPG, or state-value V (s) with suitable bootstrapping). 2) Spatio-Temporal Experience Replay: AUV mobility introduces non-stationarity that traditional temporal-only experience replay (ER) fails to capture. We propose STER, which assigns an implicit spatial anchor zt ∈ Z ⊆ R (e.g., ranging, dead reckoning, or signal feature estimation) to each sample and stores transitions in independent buffers Dz based on the node’s relative position. To stabilize sampling, a smoothed spatial context ct is updated via ct = αct−1 + (1 − α)zt , where α ∈ [0, 1) is the smoothing factor. The smoothed context reflects the long-term spatial trend of the node. For a context c and radius R, we sample from neighboring anchors Znear (c) = {z ∈ Zvisited : |z − c| ≤ R}. The sampling quota b(z) for each anchor z follows a distance-decay principle: b(z) ∝ w(z) = f (|z − c|), where f (·) is a monotonically decreasing function (e.g., f (d) = 2−d ). Quotas are normalized to sum to batch size B, and b(z) segments of length Ĥ are sampled from each Dz to form the training batch, effectively mitigating boundary effects. B. MobiU-MAC Protocol Design: Integrating CHILL-STER for Channel Access under Mobility Algorithm 1 details the joint operation of CHILL-Return and STER. 1) MDP Formulation: We model the MAC interaction as a Partially Observable MDP (POMDP) with a decision interval τ matching the slot duration. • Action Space: The agent selects at ∈ {await , atx } per slot t, representing silent or transmission modes. • State Space: Given unobservable global states, the agent uses local observations ot = (at−1 , ξt−1 ), where ξt−1 ∈ {ξfail , ξsucc , ξbusy } represents the ACK feedback received at the end of the previous slot. The variables ξfail , ξsucc , and ξbusy denote an idle/collision slot, a successful selftransmission, and a successful transmission by another node, respectively. To mitigate observation delays, the agent constructs st by stacking the Lhist most recent observations st = (ot−Lhist +1 , ot−Lhist +2 , . . . , ot ) ∈ S. • Reward Function: To maximize utilization, rt+1 = 1 if ξt ∈ {ξsucc , ξbusy } and 0 otherwise, incentivizing successful access or yielding to other nodes. 2) Implicit Spatial Anchor Estimation: To support STER without ranging overhead, an implicit spatial anchor zt is estimated by exploiting the statistical correlation between actions and ACK feedback. The core principle holds that the propagation delay D(t) dictates the temporal coupling between
Algorithm 1 MobiU-MAC Protocol Require: Maximum delay bound Dmax ; hyperparameters Ĥ, γ, λ, β, α, B, Lhist , R 1: Initialize Qθ , Qθ− , STER buffer {Dz }, spatial context c ← 0, history buffer O (length Lhist ), initial state s0 2: for each time slot t do 3: Select action at ∼ ε-greedy(Qθ (st , ·)) 4: Execute at (i.e., transmit or wait), receive observation ot+1 = (at , ξt ), then compute reward rt+1 from ξt 5: Update O ← O ∪ {ot+1 }, construct state st+1 from O 6: ▶ STER: 7: Estimate implicit spatial anchor zt 8: Update spatial context c ← αc + (1 − α)zt 9: Store transition (st ,Sat , rt+1 , st+1 ) into Dzt 10: Sample B from Dz with distance-decay quotas z:|z−c|≤R
▶ CHILL-Return: for each sample i, take trajectory length Ĥ from i do Compute TD errors δi+k , k = 0, . . . , Ĥ − 1 Ĥ 14: Compute CHILL-Return Gλ, i QH−1 i+k |si+k ) 15: Compute importance factor ρ = k=1 π(a µ(ai+k |si+k ) 16: Compute importance weight wi = min 1, ρβ 17: end for 2 P 1 wi Gλ,H − Qθ (si , ai ) 18: Update θ by minimizing |B| i 19: Periodically update θ− ← θ 20: end for 11: 12: 13:
a transmission at and its corresponding ACK ξt+2D(t) . For candidate offsets m ∈ P {2, . . . , 2Dmax }, we maintain a cut−m mulative score S(m) = k=t−W ϕ(ak , ξk+m ) over a sliding window W . The scoring function ϕ(a, ξ) is 1 for alignment ((atx , ξsucc ) or (await , ξfail )), -1 for misalignment ((atx , ξbusy ) or (await , ξsucc )), and 0 otherwise. The optimal offset m̂t = arg maxm S(m) is smoothed via a moving average to filter instantaneous noise. Finally, the spatial anchor is derived as zt = ⌈mt /2⌉. This mechanism enables the agent to categorize experiences into spatial regions based on inherent communication patterns without active signaling overhead. V. E VALUATIONS A. Simulation Settings and Parameters Simulations are conducted in a 3D underwater acoustic network of size 1000 × 1000 × 100 m3 . The AP is located at the sea-surface center (0, 0, 0) with sound speed c = 1500 m/s, slot duration τ = 0.1 s, and Dmax = 5 slots. We evaluate four scenarios: Case 1 includes a mobile AUV coexisting with four TDMA nodes where three nodes occupy one fixed slot each and one node occupies two slots within a 10-slot frame; Case 2 features a mobile AUV and an ALOHA node (q = 0.2); Case 3 combines Cases 1 and 2; and Case 4 replicates Case 1 but introduces a random slot reallocation for the two-slot TDMA node at t = 2.5 × 104 slots to evaluate robustness. The AUV runs the proposed MobiU-MAC protocol to learn access policies autonomously. In stationary conditions, the AUV is located at (480, 480, 10). In mobile settings, the AUV
+ R U L ] R Q H
+ R U L ] R Q H
% R X Q G D U \ H = 2Dmax + 1
+ R U L ] R Q H
CV (Std/Mean)
' H O D \ % R X Q G Dmax
G & D V H 3 H U I R U P D Q F H 6 W D E L O L W \
F & D V H 3 H U I R U P D Q F H 6 W D E L O L W \
Mean Equivalence Ratio
' H O D \ % R X Q G Dmax
E & D V H ( T X L Y D O H Q F H 5 D W L R
D & D V H ( T X L Y D O H Q F H 5 D W L R
+ R U L ] R Q H
Fig. 3. Mechanistic analysis of equivalence ratio and performance stability across varying Dmax and Ĥ. The red dashed line represents the theoretical boundary Ĥ = 2Dmax + 1, beyond which the MobiU-MAC achieves near-optimal performance equivalent to a standard MDP as established in Theorem 1.
B. Benchmarks and Performance Metrics 1) Benchmarks: MobiU-MAC is evaluated against two key DRL-based MAC baselines and relevant variants. DR-DLMA and its Oracle variant: DR-DLMA [2] is a radical scheme baseline for static nodes. We implement an Oracle variant with ideal real-time delay D(t) and continuous training, which enables the extension of the protocol from static nodes to mobile AUV scenarios, to establish a performance upper bound. The learning rate is tuned to 6 × 10−4 . DL-MAC and its fine-tuned variants: async-DL-MAC [1] is reproduced with a decision cycle of 6 slots, denoted as DLMAC-P6. The longer decision cycle renders this method largely immune to delayed feedback. Furthermore, we provide finetuned versions with a learning rate of 1.2 × 10−3 and decision cycles of 6 and 3 slots, denoted as DL-MAC-P6-FT and DLMAC-P3-FT, respectively. All baselines adopt the same network architecture and ϵgreedy exploration strategy consistent with MobiU-MAC. Hyperparameters not specified follow their original papers. 2) Performance Metrics: We Pt evaluate Running Average Throughput ηavg (t) = L1win k=t−Lwin +1 1{ξk ∈{ξsucc ,ξbusy }} with Lwin = 2000 to track learning dynamics (1{·} is the indicator function); and Steady-state Throughput ηss averaged over the final 104 slots to assess converged performance. All experiments involved 10 independent runs. Performance curves depict the mean, where shaded regions indicate the standard deviation unless otherwise noted. C. Performance in Static Heterogeneous UWANs We evaluate MobiU-MAC in static heterogeneous environments by analyzing steady-state throughput and resilience to strategy shifts.
D + H W H U R J H Q H R X V & R H [ L V W H Q F H
E 5 H V L O L H Q F H W R 6 W U D W H J \ 6 K L I W V 5 X Q Q L Q J $ Y J 7 K U R X J K S X W
6 W H D G \ V W D W H 7 K U R X J K S X W
moves at constant velocities v ∈ {1, 2, 6, 10, 15, 20, 30} m/s along random waypoints. We evaluated the system in an online learning mode over 5 × 104 slots. MobiU-MAC parameters include: γ = 0.978 , λ = 0.971 , Ĥ = 12 , β = 0.2 , Lhist = 30 , B = 128 , lr = 6 × 10−4 , R = 2 , α = 0.95 , W = 100 , and |Dz | = 2048. The target network updates every 60 slots, with ε-greedy exploration decaying from 1.0 to 0.01 at a factor of 0.996. All DRL-based methods (MobiU-MAC and benchmarks) utilize the same lightweight network architecture as in [2], consisting of six linear layers with 64 neurons each.
& D V H
& D V H
6 F H Q D U L R V
& D V H
0 R E L 8 0 $ & 3 U R S R V H G 2 U D F O H
t = 5160
t = 6325
6 W U D W H J \ 6 K L I W
' 5 ' / 0 $ ' / 0 $ & 3
7 L P H 6 O R W ×104
' / 0 $ & 3 ) 7 ' / 0 $ & 3 ) 7
Fig. 4. Performance under heterogeneous coexistence and strategy shifts.
1) Steady-state Throughput Analysis: Fig. 4(a) shows that MobiU-MAC achieves near-optimal steady-state throughput across Cases 1–3, with a gap to Oracle within 0.2%–5.7%. This validates Theorem 1, demonstrating that ranging-free learning can attain optimal policies. Performance degrades for all protocols in Cases 2 and 3. Because ALOHA nodes lower the throughput ceiling and increase randomness, which particularly hinders DL-MAC due to its joint learning of timing and transmission decisions. This higher-dimensional action space slows convergence. Compared to DR-DLMA, MobiUMAC exhibits lower standard deviation than DR-DLMA in Cases 1 and 3, indicating more stable convergence. 2) Resilience to Strategy Shifts: In Case 4, MobiU-MAC recovers from strategy shifts at t = 2.5 × 104 in 6325 slots, comparable to Oracle’s 5160 slots. The slight delay is due to MobiU-MAC’s larger replay buffer retaining stale experiences longer. Conversely, DL-MAC recovers slower due to its decision cycle limiting sample efficiency. DR-DLMA exhibits a sharp throughput drop near the 2000th slot due to its heuristic training suspension, which may halt learning before convergence. D. Robustness against AUV Mobility Under AUV mobility (Case 1), MobiU-MAC maintains steady-state throughput above 80% across all speeds. Conversely, Oracle’s performance drops below MobiU-MAC when v > 2 m/s despite perfect delay info, while DR-DLMA and DL-MAC suffer significant degradation. This superiority stems from CHILL-STER: CHILL-Return ensures ranging-free convergence under long delays , whereas DR-DLMA’s ranging-dependency and heuristic training fail in
$ 8 9 0 R E L O L W \ P V
0 R E L 8 0 $ & 3 U R S R V H G 2 U D F O H
' 5 ' / 0 $ ' / 0 $ & 3
7 L P H 6 O R W ×104
' / 0 $ & 3 ) 7 ' / 0 $ & 3 ) 7
5 X Q Q L Q J $ Y J 7 K U R X J K S X W
Fig. 5. Throughput performance and convergence dynamics under AUV mobility. (a) Steady-state throughput vs. AUV velocity. (b) Running avg. throughput at v = 30 m/s.
H=9
H = 11
H = 2Dmax + 1
/ D U J H H K L J K Y D U L D Q F H
7 L P H 6 O R W ×104
IS , } SR R 5K R 1K QN D w/o /o {IS /o M w/ E w/ E w w 0 R E L 8 0 $ & ) X O O $ E O D W H G 9 D U L D Q W V
7 L P H 6 O R W ×104 w/ ER5K
D(t)
Fig. 7. Performance analysis: (a) Component ablation study with results normalized (Norm.) to MobiU-MAC (Full); (b) Comparison of mobility adaptation between STER and ER under time-varying propagation delay D(t).
VI. C ONCLUSIONS
H = 15
E 0 R E L O L W \ D G D S W D W L R Q 6 7 ( 5 Y V ( 5 5 X Q Q L Q J $ Y J 7 K U R X J K S X W
1 R U P 3 H U I R U P D Q F H
D & R P S R Q H Q W D E O D W L R Q D Q D O \ V L V
D(t) [slots]
E & R Q Y H U J H Q F H X Q G H U + L J K 0 R E L O L W \ 5 X Q Q L Q J $ Y J 7 K U R X J K S X W
6 W H D G \ V W D W H 7 K U R X J K S X W
D , P S D F W R I $ 8 9 0 R E L O L W \
Fig. 6. Convergence dynamics of running average throughput for different horizons Ĥ in Case 3 (Dmax = 4).
dynamic environments. STER further captures spatial characteristics via implicit anchors, which traditional experience replay misses. E. Mechanistic Analysis and Ablation Study 1) Mechanistic Analysis: We evaluate the Equivalence Ratio ηss /η ∗ (where η ∗ is the Oracle throughput) to assess rangingfree policy learning. Fig. 3 confirms a threshold effect: the equivalence ratio approaches 1.0 only when Ĥ ≥ 2Dmax + 1, empirically corroborating the policy equivalence in Theorem 1. Although Case 3 exhibits lower ratios due to the nonstationarity of ALOHA nodes, the criterion remains a critical inflection point for both throughput and stability, as evidenced by the Coefficient of Variation (CV) heatmap. However, excessive Ĥ (e.g., Ĥ = 15 for Dmax = 4) increases variance and slows convergence compared to a tighter bound Ĥ = 9 (Fig. 6), suggesting that Ĥ should be set slightly above the minimum integer bound. 2) Ablation Study: Ablations in Fig. 4(a) identify multi-step return (MSR) as the primary enabler for learning under delays, as disabling it (w/o MSR) reduces performance to DQN levels. Importance sampling (w/o IS) and λ-return (w/o λ) are also essential for off-policy correction and variance reduction. Replacing STER with standard ER (w/ ER) fails in dynamic scenarios ; even larger buffers (e.g., ER5K , K = |Dz | ) suffer from stale experience interference as D(t) varies (Fig. 4(b)), causing sharp throughput drops. This underscores STER’s efficacy in using implicit spatial anchors to filter spatiallyrelevant transitions for robust mobile access.
This paper proposed MobiU-MAC, a ranging-free DRLbased MAC protocol for mobile node access in UWANs. We proved that multi-step returns with H ≥ 2Dmax + 1 guarantee convergence to the standard MDP optimum under long delays, enabling optimal policy learning without ranging overhead. The integrated CHILL-STER algorithm effectively addresses asynchronous rewards and mobility-induced non-stationarity. Simulations show that MobiU-MAC significantly outperforms state-of-the-art DRL-based protocols in both static and highspeed mobile scenarios. R EFERENCES [1] X. Geng et al., “Exploiting propagation delay in underwater acoustic communication networks via deep reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 12, pp. 10 626–10 637, 2023. [2] X. Ye et al., “Deep reinforcement learning based mac protocol for underwater acoustic networks,” IEEE Trans. Mobile Comput., vol. 21, no. 5, pp. 1625–1638, 2022. [3] W. Shuangshuang et al., “Adaptive modulation and coding with feedback scheduling for an underwater acoustic link,” IEEE J. Ocean. Eng., vol. 50, no. 4, pp. 3054–3073, 2025. [4] Y. He et al., “Cadtr: Context-aware trust routing algorithm based on priority sampling ddpg for uasns,” IEEE Trans. Mobile Comput., vol. 24, no. 11, pp. 11 688–11 702, 2025. [5] A. Zhang et al., “A path planning method based on deep reinforcement learning for auv in complex marine environment,” Ocean Eng., vol. 313, p. 119354, 2024. [6] S. H. Park et al., “Reinforcement learning based mac protocol (uw-alohaq) for underwater acoustic sensor networks,” IEEE Access, vol. 7, pp. 165 531–165 542, 2019. [7] Y. Gao et al., “A multi-agent reinforcement learning-based transmission protocol for underwater acoustic networks,” in Proc. 17th Int. Conf. Underwater Netw. Syst., ser. WUWNet ’23. NY, USA: ACM, 2024. [8] J. Huang et al., “Leveraging propagation delays: A delay-aware multiagent reinforcement learning mac protocol for underwater acoustic networks,” IEEE Internet Things J., vol. 12, no. 20, 2025. [9] R. S. Sutton et al., Reinforcement learning: An introduction. MIT press Cambridge, 1998, vol. 1, no. 1. [10] V. Mnih et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015. [11] C. Su et al., “Auv path planning in complex 3d underwater environments based on an improved td3 algorithm,” Ocean Eng., vol. 345, 2026. [12] P. Padrao et al., “Towards learning ocean models for long-term navigation in dynamic environments,” in OCEANS, 2022, pp. 1–6.