1
Is DRL-based MAC Ready for Underwater Acoustic Networks? Exploring Its Practicality in Real Field Experiments
arXiv:2605.10144v1 [cs.NI] 11 May 2026
Jiani Guo, Bingwen Huangfu, Shanshan Song*, Member, IEEE, Nan Sun, Miao Pan, Senior Member, IEEE, Guangjie Han Fellow, IEEE,
Abstract—Medium Access Control (MAC) protocols rely on neighbor and environment information to design collision-free access rules for Underwater Acoustic Networks (UANs). Acquiring this information suffers from high communication overhead due to the unique underwater acoustic channel characteristics, such as long propagation delay, spatiotemporal variations in communication quality, and high attenuation. Deep Reinforcement Learning (DRL) is promising to circumvent the UANs’ physical constraints and provide a low-overhead solution for underwater MAC protocols, since it can decide access rules based on realtime observation without extra information exchange. However, the unique underwater acoustic channel characteristics impose significant challenges on observation acquisition, training time, and the balance of multiple reward factors for DRL-based MAC protocols. Most existing methods remain at the theoretical level: (1) they design partial intelligent agents failing to achieve fully autonomous access; (2) they assume unreasonable simulation scenarios, weakening the effects of underwater acoustic channel characteristics on MAC protocols. To enhance the practicality of DRL-based MAC protocols, we first analyze the application challenges of DRL in UANs through real field experiments. Based on the above challenges, we propose a DRL-based MAC protocol that considers observation loss and balances multiple reward factors to achieve efficient Entire Autonomous access in the UAN (EA-MAC). To further explore the feasibility of DRLbased MAC protocols, we implement EA-MAC and other stateof-the-art protocols on underwater acoustic modems and evaluate their performance in real field experiments. Experimental results demonstrate that EA-MAC can adaptively determine the scheduling sequence for each node, enabling high-throughput and fair communication in a straightforward manner for UANs. Index Terms—Underwater acoustic network, Medium access control, Adaptive networking
I. I NTRODUCTION Underwater Acoustic Networks (UANs) have attracted significant attention from researchers since they provide essential technical support for various underwater applications, including environmental observation, marine information collection, and resource exploration [1]–[3]. However, UANs suffer from unique underwater acoustic channel characteristics, such as Shanshan Song is the corresponding author. Shanshan Song, Bingwen Huangfu, Jiani Guo, and Nan Sun are with the College of Computer Science and Technology, Jilin University, Changchun 130012, China (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). Miao Pan is with the Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77204 USA (e-mail: [email protected]). GuangJie Han is with the Department of Internet of Things Engineering, Hohai University, Changzhou 213022, China (e-mail: [email protected]).
long propagation delay, spatiotemporal varying communication quality, and high attenuation, which make even basic point-to-point communication a significant challenge in UANs. Based on such a condition, Medium Access Control (MAC) protocols become crucial in UANs, as they define point-topoint access rules determining the performance bottleneck. To avoid complex implementation and packet collisions, traditional underwater MAC protocols generally allocate distinct sending time or frequency to different nodes [4]–[6]. Due to long propagation delay (underwater acoustic velocity is around 1500 m/s) and low bandwidth (ranging from tens of hertz to tens of kilohertz), this division approach further reduces the communication resources of a single node. In addition, underwater nodes are typically equipped with half-duplex acoustic modems, indicating that a node can not send and receive simultaneously. That renders protocols associated with carrier sensing ineffective. Under such conditions, researchers study complicated protocols to improve resource utilization [7]–[9]. These protocols rely on neighbor and environment information to allocate communication resources on demand. However, obtaining this information requires additional information exchanges, which is unaffordable for UANs where communication is unreliable. Can underwater MAC protocols simultaneously adopt a simple implementation scheme and improve resource utilization? To pursue this objective, researchers have started applying Deep Reinforcement Learning (DRL) to the underwater MAC design. By leveraging the sensing and learning capabilities of DRL, nodes acting as intelligent agents can adaptively determine their sending time while utilizing the full frequency spectrum, achieving collision-free communication without complex information exchanges [10]. However, most existing DRL-based underwater MAC protocols lack the consideration of real ocean scenarios and remain at the theoretical level, facing the following issues: (1) Partial Intelligent Access. Most existing protocols design partial nodes as intelligent agents, while other nodes still employ traditional protocols such as random access and Time Division Multiple Access (TDMA) [11] [12]. In such a condition, only a few nodes access the UAN autonomously, failing to improve the whole network performance. (2) Unrealistic Simulation Conditions. Some protocols ignore long propagation delay, assuming that rewards can be obtained immediately after an action is executed, which does not correspond to real ocean scenarios. Some protocols consider
2
the long propagation delay of UANs but assume it is an integer multiple of the transmission delay, neglecting the varying propagation delay from different nodes. Moreover, these protocols overlook the trade-off between overall throughput and individual fairness, failing to enhance the UAN performance in real ocean scenarios. To fully leverage DRL in designing simple and efficient underwater MAC protocols, we first analyze the effects of underwater acoustic channel characteristics on DRL through real field experiments. Targeting these practical problems, we design an Entire Autonomous MAC protocol (EA-MAC) based on DRL, which considers long propagation delay, high attenuation, and complex factor interactions. In addition, we implement EA-MAC and other state-of-the-art protocols on underwater acoustic modems and evaluate their performance in real field experiments, further exploring whether DRL-based MAC is ready for UANs. We summarize our contributions as follows: • We organize real field experiments, analyze the effects of underwater acoustic channel characteristics on DRL methods in real ocean environments, and explore the practical issues of DRL methods in underwater applications. These practical problems are summarized as uncertain reward acquisition delay, incomplete observations, and a trade-off between throughput and fairness. • Based on the above practical problems, we design DRLbased EA-MAC which 1) relaxes the assumption of fixed reward acquisition delay in existing algorithms, aligning more closely with real UANs; 2) employs Bayesian inference to deduce and compensate for missing data transmission observations, enhancing the learning capability of individual nodes; 3) designs a reward function balancing network throughput and transmission fairness among nodes to further improve UAN performance. • Extensive simulation and real field experiments demonstrate that the proposed MAC protocol enhances the adaptability of DRL to UANs, outperforming other DRLbased MAC protocols. Meanwhile, due to the integration of a DRL method, our EA-MAC can achieve collisionfree and fair transmission in a simple manner. II. R ELATED W ORK A. Underwater MAC Protocols without Equipment Constraints Traditional underwater MAC protocols are mostly modified based on fundamental terrestrial wireless network MAC protocols (such as random access-based protocols and TDMAbased protocols). These protocols only rely on the resources of the MAC layer to schedule multiple nodes, achieving network communication. [13] analyzed the influence of time and space on the underwater Slotted ALOHA MAC protocol and determined the spatiotemporal related interference regions, to improve the successful transmission probability in UANs. In [14], the authors designed a TDMA-based protocol, which allocates slots at a packet level to reduce transmission delay. [15] adopted random time slot scheduling created by seeds to improve the communication efficiency of UANs. After
exchanging seeds, nodes obtain the scheduling arrangements of other neighbors and transmit data in the same slot with those that do not cause spatiotemporal conflicts. These protocols are hardware-friendly and easy to implement, making them suitable for deployment across a wide range of underwater acoustic modems. However, random access-based protocols struggle to effectively resolve transmission conflicts, leading to increased retransmission delay, while TDMA-based protocols face low flexibility, resulting in higher queuing delay. B. Underwater Cross-layer MAC Protocols With the rapid advancement of underwater acoustic communication technology, researchers have begun exploring crosslayer underwater MAC protocols that integrate time resources at the MAC layer with communication resources at the physical layer (such as spectrum, power, and coding) to enable flexible, efficient, and collision-free node access. In [16], the authors proposed a Multi-Carrier Code Division Multiple Access (MC-CDMA)-based MAC protocol, which jointly allocates subcarrier-related parameters and transmission power to achieve efficient concurrent communication and optimize throughput, delay, and energy consumption in mobile UANs. [17] proposed a MAC protocol based on Orthogonal Frequency Division Multiplexing (OFDM) technology, adjusting the number of subcarriers and transmission power by considering dynamic traffic load to improve throughput. In [18], the authors proposed a protocol that combines power-domain and code-domain hybrid non-orthogonal multiple access (NOMA) technology to classify senders and allocate communication resources, achieving high-concurrency communication. Although this cross-layer design concept significantly enhances the flexibility of resource allocation and the capability of collision avoidance in UANs, it requires underwater acoustic modems to support corresponding modulation techniques and an extra handshake process to obtain neighbors’ information, thereby increasing both implementation and computational complexity. C. Underwater DRL-based MAC Protocols Based on the above analysis, DRL is promising for UANs to achieve both simple and efficient node access, since it treats the node as a DRL agent, and the agent can learn to find an optimal scheduling strategy without relying on detailed information from additional handshake processes. In this way, DRL-based MAC protocols can rely on a simple slotted random access mechanism to achieve high concurrency performance comparable to that of complex protocols. In [11], the authors proposed a DRL-based MAC protocol in which the agent can vary the start time in each transmission time slot to further exploit the spatiotemporal uncertainty of the UANs. [12] proposed a DON-based MAC protocol that incorporates the long propagation delay into the DRL framework and modifies the DRL algorithm accordingly to find the optimal scheduling strategy for limited intelligent nodes. In [19], the authors proposed a deep multi-agent reinforcement learning (MARL) strategy to maximize the number of successful communications while minimizing failures caused by conflicts or
3
(a) Test topology
(b) Test equipments
Fig. 2. Comparison of Round-Trip Time (RTT) delay between field experiments and theoretical value. RTT delay means the time from sending a data packet to receive the ACK packet. The absence of markers indicates packet loss. Theoretical Result is the twice propagation delay. TABLE I S UCCESSFUL RECEPTION RATIO AMONG DIFFERENT NODES (c) Test ships Fig. 1. A 5-node UAN deployed in Danjiangkou Reservoir, Henan, China, 2024. Node 1 to node 4 are senders, and node 5 is the sink node.
interference. Meanwhile, historical observations are leveraged to reduce decision uncertainty by bridging the gap between partial observations and global states. However, most existing DRL-based MAC protocols remain at the theoretical stage with only partial agents, relying on numerous unrealistic assumptions of the underwater acoustic environment, which limit their practical applicability in real UAN scenarios. To explore the practicality of DRL and enhance UANs’ performance, we analyze real-world field experimental results and propose a DRL-based MAC protocol, in which each node acts as an intelligent agent that adaptively determines its own scheduling strategy by considering the unique characteristics of the underwater acoustic channel. III. F IELD O BSERVATIONS To explore the effects of underwater acoustic channel characteristics on DRL-based MAC protocols, we deployed a 5node UAN in the Danjiangkou Reservoir in 2024, as represented in Fig. 1. Nodes 1 to 4 adopt the TDMA protocol and send packets to node 5 in their assigned slots. After receiving a data packet, node 5 replies with an acknowledgment (ACK) packet in the current slot to the sender. In such a scenario, we observe UAN communication results from different perspectives and summarize the following problems that affect the application of DRL in underwater MAC protocols. Uncertain reward acquisition delay. DRL-based MAC protocols for UANs typically use successful transmission counts as the primary basis for reward to enhance network performance. Therefore, senders as agents rely on ACK packets to obtain reward information since receiving a correct
Sender Receiver
Node 1
Node 2
Node 3
Node 4
Node 5
Node 1 Node 2 Node 3 Node 4 Node 5
94% 47% 89% 94%
95% 93% 30% 64%
69% 81% 42% 56%
24% 29% 59% 39%
100% 96% 35% 100% -
ACK packet indicates that the transmission is successful. However, existing protocols ignore long propagation delay or are limited by the simulation capabilities, assuming 1) senders can receive an ACK packet immediately after sending a data packet; 2) the propagation delay is an integer multiple of the transmission delay; or 3) the ACK acquisition delay is twice the propagation delay. These assumptions do not align with our field results. As represented in Fig. 2, we take node 1 as an example, recording its delay from sending a data packet to receiving the ACK packet over a period of time. We refer to the delay as Round-Trip Time (RTT), which represents the reward acquisition delay. Based on the recording results, we can observe that RTT is variable, indicating that it does not exhibit an integer multiple relationship with either propagation delay or transmission delay. To enhance the practicality of underwater DRL-based MAC protocols, such uncertain reward acquisition delay should be considered in the training process. Incomplete observations. In DRL-based MAC protocols, multiple senders, acting as collaborators, should observe each other’s transmission conditions to avoid collisions and enhance throughput performance. Therefore, for a given sender, observing the data transmission status of other senders is crucial for effectively training the scheduling strategy. To evaluate the quality of observations, the UAN is designed as a fullconnection network (every two nodes are within each other’s communication range) in our real field experiment. We record the successful reception ratio among different nodes over a period of time, and the results are presented in Table I. Due to
4
(a) Data transmission situation of (b) Data transmission situation of each node under a throughput-only each node under a punishment-only reward model fairness model Fig. 3. A real-world problem of balancing throughput and transmission fairness.
high attenuation and spatiotemporal variability of underwater acoustic channels, data loss occurs across most nodes and exhibits directional asymmetry on the same communication link. In such a scenario, senders as agents fail to obtain complete observation information. Existing DRL-based MAC protocols, which ignore severe data loss, cannot be applied to the actual underwater environment. Trade-off between throughput and fairness. In this set of experiments, instead of using the TDMA protocol, we employ a simple Deep Q-Network (DQN) that allows each sender to autonomously decide its own transmission slot. We perform offline training of the DQN based on the actual UAN deployment scenario. After that, we deploy the trained models on different nodes and test them in field experiments. As represented in Fig. 3(a), the first experiment’s reward is designed solely based on the senders’ ACK reception count, aiming to maximize throughput. Most existing protocols adopt a similar reward design. However, under such a condition, we can observe significant disparities in the number of packets sent by each node, indicating an unfair distribution of transmission opportunities among nodes. Based on the design of experiment 1, we introduce a suppression penalty for nodes with excessive transmission attempts, as shown in Fig. 3(b). Field results indicate that such a monotonic penalty fails to effectively address the issue of transmission fairness among nodes. Existing DRLbased MAC protocols typically overlook the trade-off between throughput and fairness, failing to achieve high-performance node scheduling in real-world UANs. IV. D ESIGN OF THE EA-MAC We design EA-MAC based on a mechanism similar to slotted ALOHA. In each slot, nodes utilize a DQN-based algorithm to adaptively decide their action. Due to simple and sparse UAN structures, a distributed DQN framework significantly reduces algorithmic complexity compared to a Multi-Agent Reinforcement Learning (MARL) framework, as each node independently learns its transmission policy without requiring coordination or policy sharing among agents. Meanwhile, leveraging the broadcast nature of the channel, nodes can acquire information from others by overhearing, enabling a certain degree of implicit collaboration and information sharing. In addition, using a distributed DQN framework allows for simple and rapid retraining when network dynamics occur. The above advantages make it well-suited for deployment in
resource-constrained UANs, meeting the robustness and easeof-maintenance demands of real-world systems. Specifically, as shown in Fig. 4, we consider a cluster UAN, which is a classical topology widely applied in various underwater scenarios. By leveraging local information and overheard data from neighboring nodes and the sink node, each sender considers unique underwater acoustic characteristics, deduces and completes the possibly lost observation information, and balances the trade-off between throughput and fairness to determine whether to send or not in the current slot. By exploiting the long propagation delay, EA-MAC allows multiple collision-free nodes to transmit data in the same slot without collisions. Therefore, we design an aggregated ACK mechanism, which combines ACK information into a bit sequence to confirm all transmissions in the current slot with low channel overhead. Based on the above descriptions, our DQN-based algorithm are designed as follows: Action and Observation Space In each time slot t, the EA-MAC node determines whether to access the channel or not. We define the action space of the EA-MAC node as: ait = {1, 0},
(1)
where ait = 1 means the node i chooses to transmit data in time slot t, and ait = 0 means the node remains idle. After executing action at in time slot t, the EA-MAC node will obtain the observation o( t + 1) at the end of the slot, which is demonstrated as: oit+1 = {oiaction , oidata , oiack },
(2)
where oiaction denotes the node’s action, oidata represents the overheard data packets from neighboring nodes, and oiack refers to the aggregated ACK information received from the sink node. However, due to the uncertainty and partial observability in underwater environments, observation within a single time slot may be insufficient to capture the temporal dynamics of nodes’ behavior. To release the delay constraints on observation acquisition and enhance the robustness of decision making, we construct observations over a sliding window of M time slots. However, an excessively large M can expand the observation space, increasing the complexity of training and execution. Adaptive Suppression-Promotion Fairness. Based on the analysis in Section III, transmission suppression alone is insufficient to achieve fairness among nodes. To address this problem, the proposed fairness index is designed to simultaneously motivate under-active nodes to transmit and suppress over-active nodes to reduce their transmission frequency. We define Nti as the number of senders confirmed by the broadcasting aggregated ACK received by node i in the current slot. The larger Nti means more concurrent senders. Although under-active nodes need to improve sending frequency, it is essential to reduce their interference with the existing high throughput. Meanwhile, we must suppress the sending frequency of over-active nodes to provide more opportunities for under-active nodes. Therefore, we define the fairness penalty
5
Fig. 4. The overview of EA-MAC.
Fig. 5. Nodes in UANs utilize long propagation delay, achieving concurrent transmission in the same slot. Sink node acknowledges multiple data using an aggregated ACK.
Fig. 6. Sink node maintains a transmission node IDs queue. The ACK generated by the sink carries the position of the sender’s ID.
as follows. First, we define the fairness penalty of node i in a single step as: ( p/Nti , if ait = 0; t fi = (3) p, if ait = 1, where p is the base penalty constant. We define hit as the number of consecutive times that node i selects the same action in the slot t, as: ( hit−1 + 1, if ait = ait−1 ; i ht = (4) 0, otherwise. The fairness penalty of node i in slot t is the cumulative fairness penalty over the past hit slots, demonstrated as: i
Fi (t) =
ht X
fi (t − j).
(5)
j=0
The fairness penalty Fi gradually increases with consecutive actions of the node, thereby suppressing over-active nodes and promoting under-active nodes. However, despite the ability to overhear data and aggregated ACKs to infer the transmission status of neighboring nodes,
EA-MAC also faces challenges in coordinating channel competition among nodes due to the decentralized nature of the DQN framework. We further enhance the aggregated ACK broadcast mechanism to assist EA-MAC in implicit channel access coordination. Specifically, the sink node maintains a queue qtx , initialized with all transmission node IDs in order, as shown in Fig. 6. Upon receiving a data packet, the sink node identifies the position of the sender ID i in qtx , demonstrated as ϕi , and moves it to the end of the queue. Then the sink node encodes the position information ϕi into the ACK packet and broadcasts it. We define the transmission order correction index Ci (t) as: ( 1 − w · ϕi , if ait = 1 and ACK received; Ci (t) = (6) 1, otherwise, where w is a weight factor to adjust the impact of the transmission order correction index, which is typically negatively correlated with the number of nodes. The transmission order correction index Ci (t) regulates the reward based on ϕi , providing an implicit scheduling order for channel access coordination among nodes, thereby reducing collision probability. Reward Function. For a sender i, if ait = 1 and it successfully receives the corresponding ACK, its reward is defined as R(t) = Ci (t) · (Rs − Fi (t)). If ait = 1 and it fails to receive the corresponding ACK, its reward is defined as R(t) = Rp − Fi (t). Rs denotes a positive reward, while Rp corresponds to a negative penalty. If ait = 0, the reward only considers the fairness index, denoted as R(t) = −Fi (t). The whole training process of EA-MAC is shown in Algorithm. 1 Observation Completion. During the actual testing phase, we consider data loss caused by both collisions and high packet errors in real UANs, completing incomplete observation data. There are several data loss conditions. • Data loss but ACK received: this condition means a node fails to overhear a neighbor’s information. Still, it can deduce the neighbor’s action based on the ACK packet to complete its observation. • Data loss and ACK loss: the current node has N neighbors, but it only received n neighbors’ information. In such a scenario, we can not determine whether other N − n nodes did not send data or if the data was lost.
6
Algorithm 1 EA-MAC Training Process 1: Initialize Q-networks Qi (o, a|θi ), target networks Q′i , and replay buffer Di for each node i; 2: Initialize sink’s transmission queue qtx . 3: for each episode do 4: for each time slot t do 5: for all nodes i = 1 to N in parallel do 6: Observe oit and select ait ∈ {0, 1} via ϵ-greedy from Qi , then take ait to interact with the environment. 7: end for 8: Sink node receives data, updates qtx and broadcasts aggregated ACK with ϕi . 9: for all nodes i = 1 to N in parallel do 10: Update consecutive action count hit ; 11: Compute cumulative fairness penalty Fi (t) using (3) and (5); 12: Parse ϕi from ACK (if any); 13: Compute transmission order correction index Ci (t); 14: Compute reward rti according to ait , Fi (t), Ci (t); 15: Generate next observation oit+1 ; 16: Sample minibatch from Di and update Q-network; 17: Periodically update target network: θi− ← θi . 18: end for 19: end for 20: end for
Therefore, we utilize historical data during the convergence phase of the training process to deduce the action probability of other N − n nodes as follows:
P (ain+1 , . . . , aiN | ai1 , . . . , ain ) = P (ai1 , . . . , ain | ain+1 , . . . , aiN )P (ain+1 , . . . , aiN ) , P (ai1 , . . . , ain ) (7)
where ain determines the n-th node’s action. It is worth noting that there is no data loss during the training process, and during the convergence phase, the neural network has already learned the conflict relationships between nodes. Therefore, we choose the data for this stage to deduce the unknown information. Finally, we select the action combination with the highest probability to supplement the unknown actions and ACK information. • ACK loss: a node sends a data packet but fails to receive an ACK packet. In such a scenario, we can not determine whether the data conflicts with other data or the ACK is lost. The derivation method is the same as that of condition 2. • Continuous ACK loss: If a node fails to receive ACK for a continuous period of time, we determine that the link between this node and the sink node is disconnected. The node enters a silence state until it re-overhears the sink node’s information.
TABLE II S UCCESSFUL RECEPTION RATIO AMONG DIFFERENT NODES Parameters
Values
Packet size Slot length Number of hidden layers Neuron number of each hidden layer Learning rate Batch size Episodes length Maximal steps per episode
10 B 9s 3 (128, 256, 128) 10−4 256 1000∼1800 150
V. P ERFORMANCE E VALUATION A. Experiment Settings Due to the limitations in the number of experimental modems and constraints of the marine environment, it is not feasible to evaluate the proposed protocol entirely through field experiments. Therefore, this paper adopts a combination of simulation and field experiments to comprehensively evaluate the performance of EA-MAC. Our underwater acoustic modems operate at 21k Hz31k Hz, with a transmission rate of 100 bps, a transmission power of 30 W, and a state transition time between two data transmissions of around 5 s. The simulation parameters are configured faithfully based on the aforementioned realworld modems. Meanwhile, we employ the fourth-generation ns3-based simulator for UWSNs, called Aqua-Sim Fg, to conduct simulation experiments. Aqua-Sim Fg is compatible with various programming languages, including MATLAB, C++, and Python, which provides a general environment to simulate communication technologies, network protocols, and AI models simultaneously. Moreover, Aqua-Sim Fg can integrate real marine information to simulate underwater signal propagation, packet errors, and packet collisions based on the event sequence. [20] has proved that Aqua-Sim Fg is close to real sea trials. In addition, other EA-MAC parameters are summarized in Table. II. From the perspective of MAC protocols, UANs have two types: 1) non-equidistant UAN - in such networks, some senders have different propagation distances and can exploit the long propagation delay to transmit packets within the same time slot without collisions, as illustrated in Fig. 5; 2) equidistant UANs – in such networks, senders have similar propagation distances, and selecting the same time slot can result in data collisions. The following experiments will consider both types of UANs to evaluate EA-MAC’s performance. B. Simulation Experiments In this set of experiments, we vary the number of UANs’ nodes from 3 to 6. Real UANs are generally of limited scale, with even their largest practical deployments typically involving only around ten to several tens of nodes. From the perspective of MAC protocols, the receiver is associated with a limited number of senders. Therefore, the setting is reasonable for sparse UANs. For non-equidistant UANs, senders are randomly deployed around the sink at a distance of
7
TABLE III T RADE - OFF OF THROUGHPUT AND FAIRNESS IN DIFFERENT TRANSMISSION CONDITIONS
Transmission Count
(a) Non-equidistant UANs’ reward (b) Equidistant UANs’ reward reresults sults
Received ACK Count
Node Index
1
2
3
4
1
2
3
4
No Packet loss Packet loss Observation completion
35 46
40 45
40 50
35 41
35 18
40 16
40 11
35 18
35
40
40
35
29
31
29
23
Fig. 7. The convergence of EA-MAC during training.
(a) UANs without packet loss
(a) Non-equidistant UANs
(b) Equidistant UANs
Fig. 8. The number of data packets sent by each sender in different UANs. (b) UANs loss 20% data packets and 30% ACK packets
1 km ∼ 5 km. For equidistant UANs, all senders are positioned uniformly at a fixed distance of 5 km from the sink. 1) Convergence: We simulate the convergence of EA-MAC as shown in Fig. 7. Simulation results show that the algorithm converges more easily and achieves a higher sum reward in non-equidistant UANs compared to equidistant UANs. This is because collision conditions are more complex in equidistant UANs, making it more challenging to balance the trade-off between throughput and fairness. Such differences in convergence also indicate that EA-MAC can adaptively adjust node access strategy to accommodate varying scenarios, enabling efficient and fair collision-free access. 2) Trade off of throughput and fairness: We set the test step as 150, and this set of experiments does not have packet loss caused by packet error. As represented in Fig. 8, we can observe that the trained EA-MAC adaptively balances throughput and fairness in both non-equidistant UANs and equidistant UANs. However, nodes in non-equidistant UANs transmit more packets than those in equidistant UANs. This is because nodes in equidistant UANs are unable to transmit concurrently due to high packet collisions, but nodes in non-equidistant UANs can employ the characteristic of long propagation delay to enable more concurrent transmissions. This demonstrates that the proposed DRL-based algorithm effectively captures and adapts to the distinct characteristics of different UAN topologies. In the 6-node non-equidistant UAN, the fifth sender can transmit packets concurrently with the other four senders. Under such a condition, the transmissions of node 5 do not compromise the fairness of data transmissions among the other senders; instead, they contribute to enhancing the overall network throughput. Therefore, node 5 transmits more packets than the other four senders, demonstrating the strong adaptability of EA-MAC to the network environment.
(c) UANs with packet loss and observation completion Fig. 9. Node transmission details under various test conditions in equidistant UANs. Each marker means that the node with the corresponding index sends a data packet in the current slot.
3) Observation Completion: In this set of experiments, we use a 5-node equidistant UAN—characterized by complex collision scenarios and unstable factors—as an example to evaluate EA-MAC over 150 steps under various conditions. As represented in Table. III, when the network occurs packet loss (we set 20% data loss and 30% ACK loss), although the fairness mechanism still plays a role, the lack of observation information can lead to inappropriate action decisions, resulting in a reduced number of packets received by the sink node. Further incorporate the details of Fig. 9(b), this is because some senders in such a condition decide to transmit data packets in the same slot, increasing data collisions. EAMAC with an observation completion strategy can overcome the above issue. As shown in Table. III and Fig. 9(c), EAMAC is able to determine collision-free actions for each node even under packet loss conditions. This demonstrates the effectiveness of the proposed observation completion strategy in supporting reliable operation in real-world UANs. C. Real-World Field Experiments We deployed the trained network on real underwater acoustic modems and conducted actual field experiments (a 5-node network) at Songhua Lake in May and October 2025, as
8
TABLE IV T RANSMISSION AND RECEPTION RESULTS OF VARIOUS PROTOCOLS IN THE EQUIDISTANT UAN TC1
RAC2
RDC3
Node Index
1
2
3
4
1
2
3
4
5
EA-MAC DR-DLMA(1T1A) 4 DR-DLMA(1T) 5 DL-MAC(1T1A) DL-MAC(1T) UW-ALOHA-Q S-FAMA TDMA ALOHA
23 7 13 61 93 33 3 25 18
25 0 16 86 97 33 7 25 24
23 25 35 25 100 34 3 25 29
24 49 25 54 25 33 2 25 28
4 0 2 2 3 1 0 6 4
19 0 3 16 40 12 5 21 7
8 8 9 2 14 2 1 6 5
10 12 4 10 0 8 1 13 6
84 55 49 72 105 72 13 88 56
1 TC means Transmission Count of each sender. 2 RAC means the Count of self-ACKs Received by each sender. 3 RDC means the Count of Data Received by the sink. 4 1T1A means that one node employs TDMA, one node employs ALOHA,
(a) Equidistant Topology
and the remaining two nodes employ DRL-based MAC. 5 1T means that one node employs TDMA, and the remaining three nodes
employ DRL-based MAC. TABLE V T RANSMISSION AND RECEPTION RESULTS OF VARIOUS PROTOCOLS IN THE NON - EQUIDISTANT UAN TC (b) Non-equidistant Topology
RAC
RDC
Node Index
1
2
3
4
1
2
3
4
5
EA-MAC DR-DLMA(1T1A) DR-DLMA(1T) DL-MAC(1T1A) DL-MAC(1T) UW-ALOHA-Q S-FAMA TDMA ALOHA
48 8 59 79 93 34 8 25 32
49 67 13 57 63 34 12 25 42
50 25 7 25 95 34 6 25 28
49 49 25 45 25 33 4 25 35
47 2 27 17 46 30 8 23 16
48 13 0 16 18 24 11 25 26
39 3 1 4 56 14 3 18 13
48 6 3 4 1 23 4 25 20
186 39 45 41 143 113 28 93 75
(c) Test Equipments Fig. 10. A 5-node UAN deployed in Songhua Lake, Jilin, China, 2025. Node 1 to node 4 are senders, and node 5 is the sink node.
represented in Fig. 10. EA-MAC and the baseline methods are evaluated in both equidistant and non-equidistant networks. 1) Baseline Methods: In real-world field experiments, we compare EA-MAC with the following protocols to evaluate its performance: • Delayed-reward Deep-reinforcement Learning Multiple Access (DR-DLMA) [11]: In DR-DLMA, only one node adopts DQN to determine its action: no transmission or transmission, while other nodes transmit packets with TDMA or ALOHA. Meanwhile, DR-DLMA assumes the reward can be exactly obtained after twice the one-way propagation delay following the agent’s action. • Deep-reinforcement Learning-based Medium Access Control (DL-MAC) [12]: DL-MAC still has only one node that adopts DQN like DR-DLMA. DL-MAC designs a time model, where the nodes are allowed to transmit packets with different delays in different time slots. Therefore, the agent has more actions to choose from: no transmission, or transmission with a choice of delay. • Underwater ALOHA Q-learning (UW-ALOHAQ) [21]: UW-ALOHA-Q employs Q-learning scheme to
select a slot for each sender. Meanwhile, it retains random back-off of ALOHA. After a collision occurs, the node randomly delays the start time of its next frame according to a uniform distribution. • Slotted-Floor Acquisition Multiple Access (S-FAMA) [22]: A classical handshake underwater MAC protocols. Before sending data packets, nodes should send control packets, called Request To Send (RTS) and Clear To Send (CTS), to reserve the channel. Only when the reservation is successful can nodes send data packets. • TDMA and ALOHA: We also select basic TDMA and ALOHA as representatives of traditional simple protocols to compare with other DRL-based protocols. 2) Performance Analysis of an Equidistant UAN: In this set of experiments, we test all protocols for 100 steps. The transmission and reception results of each protocol are shown in Table. IV. We can observe that the RDC results are larger than the RAC results, which indicates the communication quality from the sink to senders is poorer than that from senders to the sink. This is in line with our previous experimental explorations as represented in Table. I. In such a complex marine environment, the unreasonable assumptions of DR-DLMA and DL-MAC in reward acqui-
9
(a) EA-MAC
(b) DR-DLMA (1T1A)
(e) S-FAMA
(c) DL-MAC (1T1A)
(f) TDMA
(d) UW-ALOHA-Q
(g) ALOHA
Fig. 11. Node transmission details of various MAC protocols in an equidistant UAN. Each marker means that the node with the corresponding index sends a data packet in the current slot.
sition delay lead to failure of intelligent decisions. We take DL-MAC (1T) as an example, nodes 1 to 3 are all agent nodes. Due to the fact that its trained network is unable to adapt to the uncertain delay and packet losses in the real environment, the transmission of the three nodes is completely irregular like ALOHA, as shown in Fig. 11(c). Although the sink can receive 105 packets, the reception ratio is only 0.33 lower than EA-MAC and TDMA. Although UW-ALOHA-Q achieves relatively fair packet transmissions among nodes, this behavior mainly stems from its inherent protocol constraint, which requires each node to transmit once within every frame. Therefore, its fairness is enforced through a deterministic scheduling rule, rather than achieved through adaptive and intelligent decision-making. Meanwhile, UW-ALOHA-Q only relies ACK feedback to learn a collision-free transmission strategy. However, in real-world underwater environments, ACK loss can directly disrupt its Q-table update process, preventing nodes from maintaining a reasonable scheduling sequence. As a result, multiple nodes may select the same slot to transmit packets, leading to severe collisions. For handshake-based protocols, the main issue is severe collisions among control packets. We take S-FAMA as an example in our real-world experiments. Compared with other protocols, nodes in S-FAMA send fewer data packets. This is because S-FAMA encounters high RTS/CTS collisions and losses in real UANs, resulting in a low successful ratio of channel reservation, and senders have very few opportunities to send data. ALOHA is simple to implement, but has serious collisions. Our EA-MAC draws on the concept of its random access, expecting that nodes can adaptively decide their own scheduling action and avoid collisions without extra information exchange like other complex protocols. In our real-world UAN (an equidistant UAN), only one node is allowed to transmit in each time slot to avoid collisions. Based on the results of the Table. IV, Fig. 11(a), and Fig. 11(f), EA-MAC autonomously learns a TDMA-like transmission strategy with a randomaccess manner, which strongly demonstrates its effectiveness while simultaneously balancing protocol flexibility and overall network performance. However, EA-MAC is not equal to the TDMA protocol, it can improve concurrent transmissions in non-equidistant UANs, as shown in Fig. 8(a) and Fig. 12(a). In
addition, our real-world experiments occur continuous packet losses, and EA-MAC can enter a silence state to save energy. Take node 3 in Fig. 11(a) as an example, it fails to receive any ACK packets from step 36 to 52. Based on our design, the node determines that the communication link is disconnected if it fails to receive any ACK packets for eight consecutive steps. Therefore, node 3 stops sending data packets from step 44 until it successfully receives ACK packets again at step 53. Experimental results demonstrate that EA-MAC can save more energy when the communication links are unstable. Meanwhile, such a flexible and autonomous design does not affect the subsequent decisions of the neural network. 3) Performance Analysis of a Non-equidistant UAN: Compared with the equidistant UAN, the non-equidistant UAN performs better performance within the same number of test steps, as represented in Table. V and Fig. 12. This is because the non-equidistant topology introduces differences in propagation distance among nodes, which provides opportunities for concurrent transmissions by different nodes due to long propagation delay of underwater environments, and thereby improves channel utilization. However, the other baselines still perform worse than EAMAC. Their unsatisfactory performance in the non-equidistant UAN can be attributed to the same reasons observed in the previous equidistant-UAN experiment: 1) conventional protocols (S-FAMA, TDMA, ALOHA) lack the adaptability to respond to environmental variations; 2) existing DRL-based protocols (DR-DLMA, DL-MAC, UW-ALOHA-Q) make overly strong assumptions about the underwater environment, failing to effectively cope with delayed reward feedback and missing observations in practical underwater scenarios. Since these limitations are fundamental rather than topology-specific, their poor performance persists in the non-equidistant UAN. For EA-MAC, as most functionalities have already been validated in the previous experiment, we do not repeat the same analysis here. Instead, we focus on the concurrency performance to validate whether EA-MAC can adapt to underwater environmental variations and provide flexible access strategies accordingly. As represented in Fig. 12(a), EA-MAC allows a nearby node (node 1 or node 2) and a distant node (node 3 or node 4) to transmit packets in the same slot. This demonstrates that EA-MAC can capture environmental
10
(a) EA-MAC
(b) DR-DLMA (1T1A)
(e) S-FAMA
(c) DL-MAC (1T1A)
(f) TDMA
(d) UW-ALOHA-Q
(g) ALOHA
Fig. 12. Node transmission details of various MAC protocols in a non-equidistant UAN. Each marker means that the node with the corresponding index sends a data packet in the current slot.
variations, learn the underlying collision relationships among nodes, and adapt its access decisions accordingly to best match the current scenario. Meanwhile, as shown in Table. V, EAMAC maintains a satisfied transmission fairness while supporting concurrent transmissions. This indicates that EA-MAC’s adaptive suppression strategy not only regulates transmission fairness among nodes, but also preserves consistency among nodes with concurrency potential. In other words, for two nodes with concurrency potential, EA-MAC suppresses or promotes their transmissions simultaneously, thereby achieving a balance between fairness and throughput. VI. C ONCLUSION To enhance the flexibility and intelligence of underwater MAC protocols, we study DRL algorithms’ practicality in real-world UANs. Through real field experiments, we analyze the application challenges of DRL in UANs, including uncertain reward acquisition delay, incomplete observations, and the trade-off between throughput and fairness. Based on the above challenges, we propose EA-MAC that considers long propagation delay, observation loss, and balances multiple reward factors to achieve efficient entire autonomous access in UANs. Experimental results demonstrate that EA-MAC allows each sender in real-world UANs to act as an intelligent agent, making its own transmission decisions independently, thereby achieving high-throughput and fair communication in a straightforward manner. VII. ACKNOWLEDGMENT This work was supported in part by the National Natural Science Foundation of China under Grant 62501250 and Grant 62471201; in part by the Postdoctoral Science Foundation of China under Grant 2025M771509; in part by the Postdoctoral Fellowship Program of CPSF under Grant Number GZC20250178. R EFERENCES [1] E. Felemban, F. K. Shaikh, U. M. Qureshi, A. A. Sheikh, and S. B. Qaisar, “Underwater sensor network applications: A comprehensive survey,” International Journal of Distributed Sensor Networks, vol. 11, no. 11, p. 896832, 2015.
[2] S. Song, J. Liu, J. Guo, B. Lin, Q. Ye, and J. Cui, “Efficient data collection scheme for multi-modal underwater sensor networks based on deep reinforcement learning,” IEEE Transactions on Vehicular Technology, vol. 72, no. 5, pp. 6558–6570, 2022. [3] K. K. Gola and S. Arya, “Underwater acoustic sensor networks: Taxonomy on applications, architectures, localization methods, deployment techniques, routing techniques, and threats: A systematic review,” Concurrency and Computation: Practice and Experience, vol. 35, no. 23, p. e7815, 2023. [4] S. Jiang, “State-of-the-art medium access control (mac) protocols for underwater acoustic networks: A survey based on a mac reference model,” IEEE communications surveys & tutorials, vol. 20, no. 1, pp. 96–131, 2017. [5] R. Zhu, A. Boukerche, and Q. Yang, “An interference-aware and collision-free mac protocol for underwater wireless sensor networks,” ACM Transactions on Sensor Networks, vol. 21, no. 3, pp. 1–26, 2025. [6] H. Wang, G. Han, Y. Hou, M. Guizani, and Y. Peng, “A multichannel interference based source location privacy protection scheme in underwater acoustic sensor networks,” IEEE Transactions on Vehicular Technology, vol. 71, no. 2, pp. 2058–2069, 2021. [7] L. Liu, L. Cai, L. Ma, and G. Qiao, “Channel state information prediction for adaptive underwater acoustic downlink ofdma system: Deep neural networks based approach,” IEEE Transactions on Vehicular Technology, vol. 70, no. 9, pp. 9063–9076, 2021. [8] Z. Li, Z. Qi, and D. Pompili, “Ctsma: Cyclic time shift multiple access for underwater acoustic networks,” in 2024 IEEE 21st International Conference on Mobile Ad-Hoc and Smart Systems (MASS), 2024, pp. 261–269. [9] G. Ma, X. Shen, Y. Yan, H. Yao, and H. Wang, “Energy state sensing for robust mac protocol identification in underwater acoustic networks,” IEEE Transactions on Cognitive Communications and Networking, pp. 1–1, 2025. [10] Z. Wang, J. Du, X. Hou, J. Wang, C. Jiang, X.-P. Zhang, and Y. Ren, “Toward communication optimization for future underwater networking: A survey of reinforcement learning-based approaches,” IEEE Communications Surveys & Tutorials, pp. 1–1, 2024. [11] X. Geng and Y. R. Zheng, “Exploiting propagation delay in underwater acoustic communication networks via deep reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 12, pp. 10 626–10 637, 2022. [12] X. Ye, Y. Yu, and L. Fu, “Deep reinforcement learning based mac protocol for underwater acoustic networks,” IEEE Transactions on Mobile Computing, vol. 21, no. 5, pp. 1625–1638, 2022. [13] Y. Wang, Q. Guan, F. Ji, and W. Chen, “Impact and analysis of spacetime coupling on slotted mac in uans,” IEEE/ACM Transactions on Networking, vol. 32, no. 3, pp. 2099–2111, 2023. [14] M. Liu, X. Zhuo, Y. Wei, Y. Wu, and F. Qu, “Packet-level slot scheduling mac protocol in underwater acoustic sensor networks,” IEEE Internet of Things Journal, vol. 8, no. 11, pp. 8990–9004, 2021. [15] E. P. C. Júnior, L. F. Vieira, and M. A. Vieira, “Uw-seedex: A pseudorandom-based mac protocol for underwater acoustic networks,” IEEE Transactions on Mobile Computing, vol. 21, no. 9, pp. 3402–3413, 2021. [16] J. Guo, S. Song, J. Liu, L. Wan, Y. Yu, and G. Han, “An mccdma-based mac protocol for efficient concurrent communication in
11
mobile underwater acoustic networks,” IEEE Transactions on Mobile Computing, vol. 23, no. 12, pp. 12 428–12 443, 2024. [17] Y. Su, X. Liu, G. Han, and X. Fu, “A traffic load-aware ofdma-based mac protocol for distributed underwater acoustic sensor networks,” IEEE Transactions on Vehicular Technology, vol. 70, no. 10, pp. 10 501– 10 513, 2021. [18] J. Guo, S. Song, J. Liu, H. Chen, J.-H. Cui, and G. Han, “A hybrid noma-based mac protocol for underwater acoustic networks,” IEEE/ACM Transactions on Networking, vol. 32, no. 2, pp. 1187–1200, 2024. [19] T. Zhang, Y. Gou, J. Liu, and J.-H. Cui, “Traffic load-aware resource management strategy for underwater wireless sensor networks,” IEEE Transactions on Mobile Computing, vol. 24, no. 1, pp. 243–260, 2025. [20] J. Guo, S. Song, H. Chen, B. Huangfu, J. Liu, and J.-H. Cui, “Aquasim fourth generation: Toward general and intelligent simulation for underwater acoustic networks,” IEEE Internet of Things Journal, vol. 12, no. 15, pp. 30 203–30 214, 2025. [21] S. H. Park, P. D. Mitchell, and D. Grace, “Reinforcement learning based mac protocol (uw-aloha-q) for underwater acoustic sensor networks,” IEEE access, vol. 7, pp. 165 531–165 542, 2019. [22] M. Molins and M. Stojanovic, “Slotted fama: a mac protocol for underwater acoustic networks,” in OCEANS 2006-Asia Pacific, 2006, pp. 1–7.
Jiani Guo received the BS degree (2016) in computer science and technology from Beijing Jiaotong University, Beijing, China, received PhD degree (2024) in Jilin University, Changchun, China. She is currently a Postdoctoral Researcher with the Department of Computer science and technology, Jilin University. Her current research interests include protocols design, performance analysis, and machine learning for underwater acoustic networks.
Bingwen Huangfu received the B.E. degree in computer science and technology from Jilin University, Changchun, China, in 2022. He is currently working toward the PhD degree at the College of Computer science and technology at Jilin University, Changchun, China. His current research interests include network architecture, resource allocation, and machine learning for underwater acoustic networks.
Shanshan Song (Member, IEEE) received the BS degree (2011) and MS degree (2014) in computer science and technology from Jilin University, China, received PhD degree (2018) in Management science and engineering from Jilin University, China. She was a Post-Doctoral Researcher with the Department of Computer science and technology, Jilin University, Changchun, China. She is currently an associate professor with the Department of Computer science and technology, Jilin University. Her major research focuses on underwater data collection, localization and navigation and machine learning.
Nan Sun received the B.E. degree in software engineering from Yanshan University, Qinhuangdao, China, in 2023. He is currently working toward the M.S. degree with the College of Software Engineering, Jilin University, Changchun, China. His current research involves medium access control protocols for underwater acoustic networks.
Miao Pan (Senior Member, IEEE) received the B.Sc. degree in electrical engineering from the Dalian University of Technology, Dalian, China, in 2004, the MASc. degree in electrical and computer engineering from the Beijing University of Posts and Telecommunications, Beijing, China, in 2007, and the Ph.D.degree in electrical and computer engineering from the University of Florida, Gainesville, FL, USA, in 2012. He is currently an Associate Professor with the Department of Electrical and Computer Engineering, University of Houston, Houston, TX, USA. His research interests include wireless/AI for AI/wireless, deep learning privacy, cybersecurity, and underwater communications and networking. He was the recipient of the NSF CAREER Award in 2014, IEEE TCGCC (Technical Committee on Green Communications and Computing) Best Conference Paper Awards 2019, and Best Paper Awards in ICC 2019, VTC 2018, Globecom 2017 and Globecom 2015, respectively. Dr. Pan is the Editor of IEEE OPEN JOURNAL OF VEHICULAR TECHNOLOGY, an Associate Editor for ACM Computing Surveys and IEEE INTERNET OF THINGS Journal (Area 5: Artificial Intelligence for IoT), and was an Associate Editor for IEEE INTERNET OF THINGS Journal (Area 4: Services, Applications, and Other Topics for IoT) from 2015 to 2018. He is also a Technical Organizing Committee for several conferences such as TPC Co-Chair for Mobiquitous 2019 and ACM WUWNet 2019.
Guangjie Han (Fellow, IEEE) is currently a Professor with the Department of Internet of Things Engineering, Hohai University, Changzhou, China. He received his Ph.D. degree from Northeastern University, Shenyang, China, in 2004. In February 2008, he finished his work as a Postdoctoral Researcher with the Department of Computer Science, Chonnam National University, Gwangju, Korea. From October 2010 to October 2011, he was a Visiting Research Scholar with Osaka University, Suita, Japan. From January 2017 to February 2017, he was a Visiting Professor with City University of Hong Kong, China. From July 2017 to July 2020, he was a Distinguished Professor with Dalian University of Technology, China. His current research interests include Internet of Things, Industrial Internet, Machine Learning and Artificial Intelligence, Mobile Computing, Security and Privacy. Dr. Han has over 500 peer-reviewed journal and conference papers, in addition to 160 granted and pending patents. Currently, his H-index is 81 and i10-index is 381 in Google Citation (Google Scholar). The total citation count of his papers raises above 22300 times. Dr. Han is a Fellow of the UK Institution of Engineering and Technology (FIET). He has served on the Editorial Boards of up to 10 international journals, including the IEEE TII, IEEE TCCN, IEEE TVT, IEEE TNSM, IEEE Systems, etc. He has guest-edited several special issues in IEEE Journals and Magazines, including the IEEE JSAC, IEEE Communications, IEEE Wireless Communications, Computer Networks, etc. Dr. Han has also served as chair of organizing and technical committees in many international conferences. He has been awarded 2020 IEEE Systems Journal Annual Best Paper Award and the 2017-2019 IEEE ACCESS Outstanding Associate Editor Award. He is a Fellow of IEEE.