Curriculum-Guided Reinforcement Learning for Energy-Efficient UAV-ISAC in Post-Disaster Search-and-Rescue Operations

PREPRINT FOR IEEE JOURNAL

1

Curriculum-Guided Reinforcement Learning for Energy-Efficient UAV-ISAC in Post-Disaster Search-and-Rescue Operations

arXiv:2609.01764v1 [cs.NI] 1 Sep 2026

Tai-You Guo and Chuan-Chi Lai, Member, IEEE

Abstract—Uncrewed aerial vehicles (UAVs) are promising platforms for integrated sensing and communication (ISAC), but their limited onboard energy creates a strong coupling among sensing accuracy, communication quality, and propulsion cost. This paper proposes a curriculum-guided soft actor-critic (CGSAC) framework with propulsion-aware reward shaping for energy-efficient UAV-ISAC, jointly optimizing the 3D trajectory, communication-sensing power split, and per-user power allocation. A rotary-wing propulsion model is incorporated to derive a closed-form propulsion-economic cruising speed, which is used to construct a propulsion-aware speed-shaping term within a normalized composite reward together with navigation, node-visiting, energy-efficiency, and constraint-penalty terms. A log-linear curriculum progressively tightens the communication, sensing, and proximity requirements during training. Across 2000 randomized scenarios, CG-SAC achieves an average energy efficiency of 0.72 Mbits/J, substantially outperforming the evaluated DRL baselines. Among successfully completed missions, it requires 107.6 steps on average, corresponding to a 66%– 82% reduction in flight steps relative to the baselines. Crucially, the learned policy exhibits mission-aware speed adaptation by decelerating near service points and accelerating during transit, while achieving a 99.6% communication-rate satisfaction ratio at service instants. Ablation results further demonstrate the complementary roles of the reward components in balancing mission feasibility and energy efficiency. Index Terms—Uncrewed aerial vehicle (UAV), integrated sensing and communication (ISAC), energy efficiency, trajectory optimization, deep reinforcement learning, soft actor-critic, curriculum learning.

I. I NTRODUCTION S sixth-generation (6G) wireless systems evolve toward integrated sensing and communication (ISAC), the joint provision of communication and sensing services has emerged as an important direction for intelligent wireless networks [1], [2]. By sharing spectrum and radio-frequency hardware, ISAC enables a common platform to support both connectivity and

A

This work was supported by the National Science and Technology Council (NSTC), Taiwan, under Grant No. NSTC 115-2221-E-194-042-MY2, and was also supported in part by the Advanced Institute of Manufacturing with Hightech Innovations (AIM-HI) from the Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan. (Corresponding author: Chuan-Chi Lai.) Tai-You Guo is with the Department of Communications Engineering, National Chung Cheng University, Minxiong Township, Chiayi County 621301, Taiwan. Chuan-Chi Lai is with the Department of Communications Engineering, National Chung Cheng University, Minxiong Township, Chiayi County 621301, Taiwan, and also with the Advanced Institute of Manufacturing with Hightech Innovations (AIM-HI), National Chung Cheng University, Minxiong Township, Chiayi County 621301, Taiwan.

8$9WUDMHFWRU\ 6HQVLQJOLQNWUDQVPLW HFKR &RPPXQLFDWLRQOLQNGRZQOLQN 6HQVLQJWDUJHWYLFWLP &RPPXQLFDWLRQXVHUUHVFXHWHDP

w1

p1

pK z

y

w2 p2 x

wM

Fig. 1. UAV-ISAC mission for post-disaster search-and-rescue operations.

environmental awareness. This capability is particularly attractive for uncrewed aerial vehicles (UAVs), whose mobility and favorable line-of-sight (LoS) propagation can provide rapidly deployable sensing and communication services in areas where terrestrial infrastructure is unavailable or damaged [3], [4]. Among such applications, post-disaster search-and-rescue is especially compelling: a UAV can search for potential victims while simultaneously delivering the acquired information to ground rescue teams. In a post-disaster environment, however, the UAV is not merely required to establish wireless links or perform sensing at isolated locations. It must complete a mission over a sequence of spatially dispersed targets and communication users under a limited onboard energy budget. As illustrated in Fig. 1, the UAV jointly performs sensing of rescue targets p1 , . . . , p𝐾 and downlink communication to ground rescue teams w1 , . . . , w 𝑀 using a common transmit waveform. The resulting mission requires the UAV to determine where to fly, how fast to fly, and how to allocate its transmit power between communication and sensing while visiting the required nodes. Hence, the objective is inherently mission-oriented rather than being limited to optimizing an individual communication or sensing metric. The main difficulty arises from the strong coupling among mobility, sensing, communication, and propulsion energy. First, UAV propulsion accounts for a substantial portion of the onboard energy consumption, making inefficient flight trajectories or unnecessarily high speeds costly. Moreover, the propulsion cost depends nonlinearly on the flight speed, so

PREPRINT FOR IEEE JOURNAL

minimizing mission duration does not necessarily minimize energy consumption. Second, sensing and communication share a limited transmit-power budget: allocating more power to sensing improves the sensing quality but leaves less power for downlink communication. Third, improving communication or sensing quality generally requires favorable threedimensional positioning, which may increase the flight distance or force the UAV to deviate from an energy-efficient trajectory. These interactions create a tightly coupled continuouscontrol problem in which mission completion, service quality, and propulsion efficiency must be addressed simultaneously. This coupling also makes direct reinforcement-learning optimization challenging. The action space contains continuous mobility and radio-resource decisions, while the missioncompletion objective is sparse and the communication and sensing requirements impose stringent feasibility boundaries. A policy that focuses only on rapid node visiting may adopt unnecessarily aggressive flight speeds, whereas a policy that prioritizes service quality may spend excessive energy on prolonged or inefficient maneuvers. Therefore, an effective learning framework should not only explore the continuous control space, but also exploit the physical structure of UAV propulsion and progressively acquire feasible behaviors under increasingly stringent service requirements. To address these issues, we propose a curriculum-guided soft actor-critic (CG-SAC) framework with propulsion-aware reward shaping for energy-efficient UAV-ISAC. The proposed framework jointly optimizes the 3D UAV trajectory, the communication-sensing power split, and the per-user power allocation. A speed-dependent rotary-wing propulsion model is first used to derive a closed-form economic cruising-speed reference. Rather than treating this quantity as a standalone optimization target, we incorporate the resulting propulsion insight into a speed-shaping term within a normalized composite reward. In parallel, a log-linear curriculum progressively tightens the communication, sensing, and proximity requirements during training, allowing the agent to first learn basic spatial navigation and subsequently adapt to stricter mission constraints. The resulting design couples physical energy awareness with curriculum-based exploration while retaining the maximum-entropy characteristics of SAC. Under the proposed framework, the UAV sequentially visits the sensing targets and communication users, dynamically adapting its 3D motion and radio resources according to the spatial configuration and service requirements. In this way, the proposed method aims to learn a mission-aware policy that accelerates during transit while slowing down near service points when additional interaction time is beneficial. The sensing requirement is represented by an SNR-based sensingerror surrogate derived from the adopted echo model, while communication quality is characterized by the achievable downlink rate. The main contributions of this paper are summarized as follows. • Mission-oriented joint optimization for energy-efficient UAV-ISAC. We formulate a coupled optimization framework that jointly considers 3D trajectory control, communication-sensing power splitting, and

2

per-user power allocation under propulsion-energy, communication-QoS, sensing-accuracy, and kinematic requirements. The formulation explicitly captures the interaction between spatial mobility, radio-resource allocation, and mission-level energy efficiency. • Propulsion-aware reward shaping for continuous UAV control. A rotary-wing propulsion model is used to derive a closed-form horizontal propulsion-economic cruisingspeed reference. This physical insight is incorporated into the composite reward through a speed-shaping term. The propulsion-aware speed term provides an additional physical prior for shaping the learned flight-speed behavior, encouraging energy-efficient flight while retaining the flexibility to slow down near service locations and accelerate during transit. • Curriculum-guided SAC with synchronized entropy scheduling. A log-linear curriculum progressively tightens the communication, sensing, and proximity requirements during training. The curriculum determines task difficulty, while the entropy lower bound regulates exploration during curriculum transitions. Combined with propulsionaware reward shaping, the proposed CG-SAC framework mitigates exploration difficulty in the highly constrained continuous-control space. • Comprehensive evaluation under randomized mission conditions. Extensive simulations over 2000 randomized scenarios evaluate mission completion, energy efficiency, service quality, reward-component ablations, and sensitivity to the onboard energy budget. The results show that CG-SAC substantially improves the evaluated energy-efficiency and mission-completion performance over representative continuous-control DRL baselines, while maintaining high communication-rate satisfaction at service instants. The remainder of this paper is organized as follows. Section II reviews the related work. Section III establishes the UAV-ISAC system model, and Section IV formulates the joint optimization problem. Section V presents the proposed curriculum-guided SAC framework. Section VI reports the simulation results, and Section VII concludes the paper. II. R ELATED W ORK ISAC aims to achieve data transmission and environmental sensing cooperatively under shared spectrum and hardware. Early studies focused mainly on the interference and performance trade-offs induced by physical-layer waveform design. The work in [10] investigated frequency-hopping multipleinput multiple-output (FH-MIMO) architectures in dualfunction radar-communication (DFRC) systems and showed that embedding communication symbols through frequencyhopping code selection (FHCS) introduces undesirable sidelobes in the radar ambiguity function, thereby degrading the sensing resolution. To quantify this physical limit, the authors of [11] derived the Pareto-optimal boundary between the Cramér-Rao Bound (CRB) and the achievable rate in multiantenna ISAC systems, confirming that high-accuracy sensing is necessarily accompanied by a loss of communication

PREPRINT FOR IEEE JOURNAL

3

TABLE I C OMPARISON BETWEEN THIS WORK AND REPRESENTATIVE STUDIES Reference (Year) [5] (2020) [6] (2024) [7] (2023) [8] (2024) [9] (2026) This work

System scenario UAV-aided WSN data collection (non-ISAC) Cellular-connected UAV (non-ISAC) UAV-aided ISAC network

Objective Propulsion energy minimization Energy efficiency

Age of information (AoI) minimization UAV-ISAC with user- Sensing-communication position uncertainty trade-off Multi-UAV multi-static Joint resource and traUAV-ISAC jectory design 3D single-UAV ISAC EE maximization un(disaster relief) der QoS and sensing constraints

Sensing metric N/A

Jointly designed variables 3D flight trajectory

Methodology Convex optimization

N/A

Trajectory with map reconstruction

N/A

Trajectory and service order

Deep reinforcement learning Dynamic programming

CRB

Trajectory geometry

Artificial potential field

Sensing metric

Trajectory, communication/sensing power split 3D trajectory, communication/sensing power split, per-user power allocation

Hierarchical multi-agent DRL Curriculum-guided SAC

CRB-inspired sensing-error surrogate

throughput. As system complexity grows, auxiliary techniques and multi-dimensional resource allocation have also received attention. The study in [12] addressed the joint beamforming and reflection-matrix design for reconfigurable intelligent surface (RIS)-aided ISAC, whereas [13] proposed a predictive integrated sensing, communication, and computation over-theair (ISCCO) framework for vehicular networks, in which an extended Kalman filter (EKF) assists downlink beamforming and the convergence stability of reinforcement learning in dynamic resource allocation is preliminarily verified. Given the limited onboard battery capacity of a UAV, accurate propulsion power modeling serves as the physical foundation for trajectory optimization. A closed-form energy model covering initial speed, acceleration, and flight time for rotarywing UAVs was proposed in [14], improving upon earlier constant-speed assumptions. Through flight experiments, the authors of [15] decomposed the energy consumption into induced, profile, and parasitic power, and revealed a pronounced U-shaped curve relating EE to flight speed. The motion characteristics of a UAV also directly shape its energy profile: based on multi-rotor dynamics, [16] showed that frequent vertical maneuvers and horizontal accelerations substantially raise the energy cost, while [17] evaluated a proton-exchangemembrane fuel-cell (PEMFC) propulsion system and reduced the parasitic consumption through an integrated cathode-flowchannel design with airflow cooling. Collectively, these results indicate that dynamic propulsion energy must be treated as a primary consideration in UAV-ISAC trajectory planning. For highly nonlinear and time-varying environments, DRL offers advantages in handling high-dimensional non-convex optimization. In UAV-assisted mobile edge computing (MEC), model-agnostic meta-learning (MAML) was combined with DRL in [18] to accelerate environmental adaptation; Lyapunov optimization was integrated with DRL in [19] to guarantee long-term energy constraints under stochastic task arrivals; and multi-agent proximal policy optimization (MAPPO) was applied in [20] to jointly configure 3D trajectories, nonorthogonal multiple access (NOMA) power, and spectrum resources. Regarding stability and safety, existing studies continue to refine the actor-critic architecture originating from DDPG [21]. TD3 [22] may exhibit insufficient exploration or degraded stability under tightly constrained continuous

control, which was mitigated in [23] to optimize the EE of RIS-aided networks, and Safe-TD3 was proposed in [24] to convert energy limits into hard safety constraints. A broader account of how these algorithm families have been applied to UAV missions can be found in the recent survey [25]. However, when the sensing accuracy requirement of an ISAC system is both tightly coupled and stochastic, the exploration capability of deterministic policies in a complex action space remains insufficient. Table I positions this work against representative studies. As shown, existing efforts rarely reconcile ISAC operation with EE in trajectory design. Specifically, [5] and [6] build realistic propulsion energy models and pursue energy-efficient trajectories through conventional optimization and DRL, respectively, yet neither considers an ISAC network nor the joint design of sensing and communication functionalities. Conversely, existing UAV-ISAC trajectory studies largely neglect energy utilization: the work in [7] minimizes the AoI and determines the service order by dynamic programming, and [8] combines an artificial potential field to cope with user-position uncertainty; neither treats EE as a core objective nor addresses the communication-sensing power split. However, [9] focuses on multi-UAV cooperation and hybrid discrete/continuous control, whereas the present work considers single-UAV missionlevel energy efficiency with continuous control. In summary, although prior studies have separately advanced ISAC, energy modeling, trajectory design, and reinforcement learning, most remain confined to a single facet and have yet to integrate sensing, communication, EE, trajectory planning, and resource allocation into one tightly coupled nonconvex problem. This is particularly true for 3D UAV-ISAC scenarios with variable altitude, in which the sensing accuracy (surrogate threshold), the ground QoS rate, and the propulsion energy (characterized by the U-shaped curve) compete and interact. To address this gap, this study adopts the maximumentropy SAC algorithm and constructs an EE-centric joint optimization framework for UAV-ISAC that simultaneously considers the 3D trajectory, the communication-sensing power split, and per-user resource allocation, while incorporating a speed-dependent propulsion energy model and strict performance constraints.

PREPRINT FOR IEEE JOURNAL

4

III. S YSTEM M ODEL A. Network Architecture We consider a UAV-ISAC system in 3D space, in which a rotary-wing UAV equipped with 𝑁𝑡 antennas acts as an aerial base station over a mission period of duration 𝑇. For tractability, 𝑇 is discretized into 𝑁 equal time slots of length 𝛿𝑡 = 𝑇/𝑁, and the time-varying 3D coordinate of the UAV in slot 𝑛 ∈ N = {1, . . . , 𝑁 } is written as q[𝑛] = [𝑥 [𝑛], 𝑦[𝑛], 𝑧[𝑛]] 𝑇 ∈ R3 . The UAV employs a single transmit waveform to realize communication and sensing simultaneously. On the downlink, it serves 𝑀 single-antenna communication users (CUs) distributed on the ground, whose position vectors are w𝑚 = [𝑥 𝑚 , 𝑦 𝑚 , 0] 𝑇 , 𝑚 ∈ {1, . . . , 𝑀 }. In parallel, it exploits the echoes of its transmitted signal to estimate the parameters of 𝐾 sensing targets located at p 𝑘 = [𝑥 𝑘 , 𝑦 𝑘 , 0] 𝑇 , 𝑘 ∈ {1, . . . , 𝐾}. The overall architecture is depicted in Fig. 2. To ground the model in a concrete application, we adopt the disaster-relief scenario introduced in Section I: the sensing targets correspond to rescue points to be detected and localized (e.g., trapped survivors), and the communication users correspond to ground rescue teams awaiting notification. The UAV performs patrol-style sequential service, flying to each node in turn to complete an accuracy-compliant localization at a sensing target or a rate-compliant information delivery at a communication user. Only when all rescue points have been localized and all rescue teams have been notified is the mission considered complete; this is the physical meaning of adopting “all nodes visited” as the completion criterion. Accordingly, we set 𝑀 = 𝐾 = 3 so as to model a representative small relief unit that balances scenario realism against training tractability, noting that the framework extends directly to a larger number of nodes. In practice, a large-scale disaster area can be geographically partitioned into multiple micro-grids or clusters, with a single UAV sequentially assigned to patrol each small-scale unit. This clustering strategy ensures that the action and state spaces remain tractable while preserving the physical realism of the aerodynamic and QoS constraints. Considering the limited onboard battery and computing resources, the UAV is assumed to be connected to a ground core base station equipped with an edge-computing server through a wireless backhaul link. In the DRL workflow, the computationally intensive training stage (experience replay and gradient updates) is offloaded to the ground server; the UAV only receives the updated actor-network weights and performs lightweight forward-pass inference onboard. Such an edge-assisted deployment effectively relieves the onboard computing unit. B. Channel Model Air-to-ground (A2G) links are adopted between the UAV and the ground users. Let w𝑚 and q(𝑡) denote the 3D positions of the 𝑚-th user and the UAV in slot 𝑡, so that their geometric CU (𝑡) = kq(𝑡) − w k. To focus on system-level distance is 𝑑 𝑚 𝑚 trajectory and resource optimization, the random fluctuation of small-scale fading is neglected [20], i.e., ℎ̃ 𝑚 (𝑡) = 1; the channel power gain |ℎ 𝑚 (𝑡)| 2 is thus fully determined

Fig. 2. The considered UAV-ISAC system architecture.

by the reciprocal of the large-scale path loss 𝐿 𝑚 (𝑡), namely |ℎ 𝑚 (𝑡)| 2 = 𝜌 𝑚 (𝑡) = 1/𝐿 𝑚 (𝑡). The occurrence of LoS and non-LoS (NLoS) propagation on an A2G link depends on the elevation angle 𝜃 CU 𝑚 (𝑡) of the UAV as seen from the user. We adopt the widely used sigmoid LoS probability model originally proposed in [26] and subsequently employed in UAV trajectory studies such as [6], 1  
, 𝑃LoS (𝜃 CU (1) 𝑚 (𝑡)) = 1 + 𝛼L exp −𝛽L 𝜃 CU 𝑚 (𝑡) − 𝛼L where 𝛼L and 𝛽L are environment-dependent parameters. The average path loss of the A2G channel is accordingly
CU 𝐿 𝑚 (𝑡) = 𝑃LoS (𝜃 CU 𝑚 (𝑡)) 𝐿 LoS (𝑡) + 1 − 𝑃LoS (𝜃 𝑚 (𝑡)) 𝐿 NLoS (𝑡), (2) with 𝐿 LoS (𝑡) and 𝐿 NLoS (𝑡) denoting the path loss under LoS and NLoS conditions, respectively. This model captures how the 3D trajectory of the UAV dynamically shapes the link quality and serves as the basis of the subsequent resource allocation. C. Propulsion Energy Model and Economic Speed The onboard energy limit is the central factor constraining the mission duration and the trajectory design. According to the closed-form rotary-wing propulsion model of [27] and its subsequent refinements and experimental validation [14], [15], the propulsion power of a rotary-wing UAV is highly nonlinear and generally depends on its flight state and motion. We therefore adopt a computationally light and smooth approximation that preserves the dominant speed dependence. Specifically, the propulsion power is modeled as a hovering baseline plus a parasitic term growing with the cube of the speed, Pfly (𝑡) = 𝑃hover + 𝜅 kv 𝑥 𝑦 (𝑡) k 3 + 𝜅 𝑧 [𝑣 𝑧 (𝑡)] + , (3) where 𝜅 = 𝑃full𝑉−3𝑃hover is calibrated from the hovering power max 𝑃hover , the full-speed power 𝑃full , and the maximum horizontal speed 𝑉max , and kv 𝑥 𝑦 (𝑡) k is the horizontal speed; 𝜅 𝑧 is the proportionality coefficient for the vertical climbing power, and [𝑣 𝑧 (𝑡)] + = max(𝑣 𝑧 (𝑡), 0) ensures this additional energy cost is only counted when the UAV climbs. The total system power in slot 𝑡 is then Ptotal (𝑡) = Pfly (𝑡) + 𝑃max + 𝑃elec .

(4)

PREPRINT FOR IEEE JOURNAL

5

For the horizontal cruising-speed analysis, the vertical-climb term is omitted by assuming 𝑣 𝑧 = 0. Although simplified, (3) retains the essential trade-off between energy efficiency and flight speed. Measured in energy per unit horizontal displacement, it yields Pfly (𝑡) 𝑃hover = + 𝜅 kv 𝑥 𝑦 (𝑡) k 2 , (5) kv 𝑥 𝑦 (𝑡) k kv 𝑥 𝑦 (𝑡) k which is a typical U-shaped curve: at very low speeds the temporal accumulation of the hovering power inflates the energy per distance, whereas at very high speeds the cubic growth of the parasitic power does the same. Minimizing this convex function yields the model-derived horizontal propulsioneconomic cruising speed 𝑉eco , which is used as a physically motivated speed reference for reward shaping,  1/3  𝑃hover 𝑉eco = . (6) 2𝜅 During horizontal transit, encouraging operation near 𝑉eco reduces the propulsion energy per unit horizontal distance so as to avoid both high-energy branches of the U-shaped curve, all while adaptively splitting the transmit power between downlink communication and sensing. D. Communication and Sensing Metrics 1) Communication Metric: The achievable rate is adopted as the core communication measure. If the UAV transmits information to the 𝑚-th ground user in slot 𝑡, the downlink signal-to-interference-plus-noise ratio (SINR) is 𝑝 𝑚 (𝑡)|ℎ 𝑚 (𝑡)| 2 , (7) 𝛾𝑚 (𝑡) = Í 2 2 𝑗≠𝑚 𝑝 𝑗 (𝑡)|ℎ 𝑗 (𝑡)| + 𝜎 where 𝑝 𝑚 (𝑡) is the allocated power, ℎ 𝑚 (𝑡) the channel gain, and 𝜎 2 the noise power. The sequential-service rule determines when a node is credited as mission-served; it does not imply that non-served users are physically silent. During each slot, the available communication power is distributed among all users according to 𝜷(𝑡), and the resulting aggregate throughput contributes to the system EE. By the Shannon formula, the instantaneous rate is 𝑅𝑚 (𝑡) = 𝐵 log2 (1 + 𝛾𝑚 (𝑡)) [11] with 𝐵 the system bandwidth. 2) Sensing Metric: A rigorous CRB requires deriving the Fisher information matrix under a specific echo model, which would substantially increase the modeling and computational complexity of the sensing evaluation. Under fixed waveform and radar parameters, we therefore adopt a simplified accuracy surrogate that is inversely proportional to the echo signal-tonoise ratio (SNR) [12]. For a target 𝑘 at distance 𝑑 tar 𝑘 (𝑡), the expected echo SNR is 𝑃sens (𝑡) 𝐺 𝑡 𝐺 𝑟 𝜆2 𝜎𝑘 SNRecho,𝑘 (𝑡) = , (8) 4 2 (4𝜋) 3 (𝑑 tar 𝑘 (𝑡)) 𝜎𝑛 where 𝑃sens (𝑡) is the total sensing transmit power, 𝜎𝑘 the radar cross section (RCS), and 𝜎𝑛2 the receiver noise power. Since the estimation-error lower bound is inversely proportional to the echo SNR, the sensing-error surrogate, denoted by 𝜖 𝑘sens (𝑡) = 𝜉 𝑘 /SNRecho,𝑘 (𝑡), leads to the constraint 𝜖 𝑘sens (𝑡) =

4 2 𝜉 𝑘 (4𝜋) 3 (𝑑 tar 𝑘 (𝑡)) 𝜎𝑛

𝑃sens (𝑡) 𝐺 𝑡 𝐺 𝑟 𝜆2 𝜎𝑘

≤ Γsens ,

∀𝑡,

(9)

in which 𝜉 𝑘 is a proportionality constant related to the radar hardware. The metric decreases, i.e., the resolution improves, when the sensing power increases or the observation distance shrinks. In post-disaster environments filled with debris, the radar cross section (RCS) 𝜎𝑘 of trapped targets is typically weak and heavily attenuated. Consequently, maintaining a stringent sensing-error-surrogate threshold is important for ensuring reliable detection, explicitly forcing the UAV to physically approach the targets despite the aerodynamic energy cost. IV. P ROBLEM F ORMULATION The system EE is defined as the ratio of the total number of bits delivered to the ground users over the mission period to the total energy consumed by the UAV, Í𝑁 Í𝑀 𝑚=1 𝑅 𝑚 [𝑛] 𝛿𝑡 , (10) 𝜂EE (x) = 𝑛=1 Í𝑁 𝑛=1 Ptotal [𝑛] 𝛿𝑡 where the variable set x = {q[𝑛], 𝜔𝑟 [𝑛], 𝜷[𝑛]} encompasses the 3D positions, the power-split ratios, and the powerallocation vectors across the trajectory. Because the magnitude of 𝜂EE depends on the bandwidth and power settings and hence has no fixed range, we further define the normalized energy efficiency 𝜂EE (x) , (11) 𝜂ˆEE (x) = ref 𝜂EE ref is a fixed reference scale (2.0 bits/J in this work) where 𝜂EE ref is a positive constant, used for reward normalization. Since 𝜂EE (11) is a monotonically increasing linear map that does not move the optimum, i.e., argmax 𝜂ˆEE (x) = argmax 𝜂EE (x). (12) x

x

Its magnitude no longer fluctuates with the system parameters, which improves cross-scenario comparability. Guaranteeing the feasibility of the solution in practice requires satisfying communication, sensing, and kinematic constraints simultaneously. Let T𝑚serv and T𝑘serv denote the sets of slots at which user 𝑚 and target 𝑘 are actively served, respectively. The instantaneous rate of every user must meet a minimum requirement during its service slots, 𝑅𝑚 (𝑡) ≥ 𝑅min , ∀𝑚, 𝑡 ∈ T𝑚serv , (13) and the simplified accuracy metric of the currently tracked target must not exceed the tolerable threshold as in (9), 𝜖 𝑘sens (𝑡) ≤ Γsens , ∀𝑘, 𝑡 ∈ T𝑘serv . (14) The horizontal and vertical speeds kv 𝑥 𝑦 (𝑡) k and |𝑣 𝑧 (𝑡)| are bounded by 𝑉max and 𝑉𝑧,max , the acceleration ka(𝑡) k by 𝑎 max , and the trajectory must satisfy the initial boundary condition q(0) = qstart . Let Afly denote the prescribed 3D flight region, whose horizontal projection is the 1000 × 1000 m2 mission area and whose altitude range is [𝐻min , 𝐻max ]. The horizontal coordinates are constrained within the flight area such that q(𝑡) ∈ Afly . Limited by the onboard radio-frequency hardware, the communication and sensing powers additionally satisfy 𝑃comm (𝑡) + 𝑃sens (𝑡) ≤ 𝑃max . Beyond the instantaneous power budget, the mission is constrained by the onboard battery capacity. Over the whole

PREPRINT FOR IEEE JOURNAL

6

mission the consumed energy must not exceed the strict energy budget 𝐸 max , 𝑁 Õ Ptotal [𝑛] 𝛿𝑡 ≤ 𝐸 max . (15) 𝑛=1

It is worth emphasizing that, because Ptotal [𝑛] varies nonlinearly with the flight speed through (3), the same flight duration consumes different amounts of energy at different speeds. We therefore adopt energy, rather than time, as the criterion of this constraint. In the implementation, the episode terminates if the accumulated energy exceeds the strict budget 𝐸 max or if the maximum step horizon 𝑁max is reached. An episode that terminates without completing all visits is declared unsuccessful. The precise derivation of the nominal 𝐸 max capacity and its relationship with the horizontal flight power is further elaborated in Section VI-A. The optimization variables are the trajectory positions and radio-resource variables, with velocity and acceleration determined by the discrete-time kinematic model. Collecting the objective and the constraints, the joint optimization problem is formulated as (P1) : max 𝜂ˆEE (x) (16a) {q(𝑡 ), 𝜔𝑟 (𝑡 ), 𝛽𝑚 (𝑡 ) }

s.t.

𝑅𝑚 (𝑡) ≥ 𝑅min ,

∀𝑚, 𝑡 ∈ T𝑚serv

(16b)

𝜖 𝑘sens (𝑡) ≤ Γsens,

∀𝑘, 𝑡 ∈ T𝑘serv

(16c)

q(𝑡 + 1) = q(𝑡) + v(𝑡)𝛿𝑡, 𝐻min ≤ 𝑧(𝑡) ≤ 𝐻max q(𝑡) ∈ Afly kv 𝑥 𝑦 (𝑡) k ≤ 𝑉max ,

𝑡 = 0, . . . , 𝑁 − 1

|𝑣 𝑧 (𝑡)| ≤ 𝑉𝑧,max

ka(𝑡) k ≤ 𝑎 max 𝑃comm (𝑡) + 𝑃sens (𝑡) ≤ 𝑃max , 𝑁 Õ Ptotal [𝑛] 𝛿𝑡 ≤ 𝐸 max

∀𝑡

(16d) (16e) (16f) (16g) (16h) (16i) (16j)

𝑛=1

q(0) = qstart . (16k) The RF power-budget constraint is automatically satisfied by the action parameterization. Since (P1) couples nonlinear trajectory-dependent channel conditions, propulsion energy, and resource allocation under sensing and QoS constraints, obtaining a globally optimized solution with low online computational cost is challenging. This motivates the DRL framework developed next. V. C URRICULUM -G UIDED SAC FOR J OINT T RAJECTORY AND R ESOURCE D ESIGN To obtain a low-complexity online policy for the continuous control problem in (P1), we formulate the problem as an MDP and adopt SAC as the underlying off-policy continuous-control algorithm. Built on the maximum-entropy framework, SAC maintains favorable convergence and robustness in complex continuous action spaces. Importantly, the proposed algorithm is not a plain application of vanilla SAC: on top of its maximum-entropy core we integrate two mechanisms tailored to the high-dimensional non-convex nature of the UAV-ISAC task, namely a curriculum-learning schedule and a normalized

composite reward, which together turn it into a curriculumguided joint optimization framework. A. MDP Formulation Problem (P1) is first cast as a Markov decision process (MDP). 1) State Space: The state s𝑡 ∈ S collects the principal observable features affecting EE, rate, and sensing accuracy, s𝑡 = [q(𝑡), v(𝑡), dCU (𝑡), dtar (𝑡), 𝜽 CU (𝑡), b(𝑡), Rem𝑇 ], (17) where q(𝑡) and v(𝑡) are the 3D position and instantaneous velocity, which jointly determine the displacement and the propulsion power; dCU (𝑡) = [𝑑 1CU (𝑡), . . . , 𝑑 CU 𝑀 (𝑡)] and dtar (𝑡) = [𝑑 1tar (𝑡), . . . , 𝑑 tar (𝑡)] are the distance vectors to the 𝐾 communication users and the sensing targets, respectively; 𝜽 CU (𝑡) = [𝜃 1CU (𝑡), . . . , 𝜃 CU 𝑀 (𝑡)] is the vector of elevation angles to the users; b(𝑡) is the binary visited mask indicating the service status of each node; and Rem𝑇 is the remaining mission time, which endows the agent with temporal awareness. The mission-tracked node is implicitly determined as the nearest unserved communication user or sensing target. When a sensing penalty is evaluated, 𝑘 ★ (𝑡) is separately defined as the nearest unserved sensing target. It is worth noting that the state vector intentionally excludes explicit feedback on the remaining energy or battery capacity. Instead, the framework relies on temporal awareness (Rem𝑇 ) coupled with the EEcentric reward structure. This design choice avoids explicitly conditioning the policy on a particular initial battery level. Its performance across different externally imposed energy budgets is evaluated empirically in Section VI-E. 2) Action Space: The action a𝑡 ∈ A gathers all continuous decision variables,   a𝑡 = 𝑣 𝑥 (𝑡), 𝑣 𝑦 (𝑡), 𝑣 𝑧 (𝑡), 𝜔𝑟 (𝑡), 𝜷(𝑡) , (18)

where 𝑣 𝑥 , 𝑣 𝑦 , 𝑣 𝑧 are the velocity commands along the three axes, 𝜔𝑟 (𝑡) ∈ [0, 1] is the power-split ratio between communication and sensing, and 𝜷(𝑡) = [𝛽1 (𝑡), . . . , 𝛽 𝑀 (𝑡)] is the relative power-allocation vector satisfying 𝑀 Õ 𝛽𝑚 (𝑡) = 1, 𝛽𝑚 (𝑡) ≥ 0, ∀𝑚. (19) 𝑚=1

The communication and sensing powers are thus 𝑃comm (𝑡) = 𝜔𝑟 (𝑡)𝑃max and 𝑃sens (𝑡) = (1 − 𝜔𝑟 (𝑡))𝑃max , and the power actually assigned to user 𝑚 is 𝑝 𝑚 (𝑡) = 𝛽𝑚 (𝑡)𝑃comm (𝑡). On the sensing side we do not allocate a dedicated power per target; instead a single aggregate sensing power 𝑃sens (𝑡) is applied to the currently tracked target. Because SAC operates on continuous actions, the velocity and split outputs are mapped through tanh(·) to [−1, 1] and then linearly scaled, while 𝜷(𝑡) is passed through a softmax so that the simplex constraint is strictly enforced. To enforce the kinematic acceleration constraint ka(𝑡) k ≤ 𝑎 max defined in (P1), the raw velocity commands vcmd (𝑡) from the policy are passed through a first− v𝑡 . The filtered velocity order motion filter. Let Δv𝑡 = vcmd 𝑡 is given by   𝑎 max 𝛿𝑡 Δv𝑡 , (20) v𝑡+1 = v𝑡 + min 1, kΔv𝑡 k where the scaling factor is defined as 1 when kΔv𝑡 k = 0. This construction explicitly guarantees kv𝑡+1 − v𝑡 k/𝛿𝑡 ≤ 𝑎 max .

PREPRINT FOR IEEE JOURNAL

B. Propulsion-Aware Composite Reward Shaping To promote the mission-level EE objective while encouraging constraint satisfaction, we construct a composite reward. To provide step-wise feedback without requiring future trajectory information, we define an instantaneous EE surrogate Í𝑀 𝑅𝑚 (𝑡 ) , and its normalized counterpart 𝜂ˆinst (𝑡) = 𝜂inst (𝑡) = 𝑚=1 Ptotal (𝑡 ) 𝜂inst (𝑡 ) . Thus, at each step 𝑡 the agent receives 𝜂 ref EE

𝑟 𝑡 = 𝜆nav 𝑅nav (𝑡) + 𝜆visit 𝑅visit (𝑡) + 𝜆EE 𝜂ˆinst (𝑡) + 𝜆spd 𝑅spd (𝑡)
− 𝑤 pen (𝑒) Penaltyrate (𝑡) + Penaltysens (𝑡) + 𝑅term (𝑡), (21) where the first line collects the positive guidance terms and the second the constraint penalties and the terminal settlement. The coefficients 𝜆nav , 𝜆visit , 𝜆EE , 𝜆spd > 0 balance the relative importance of the objectives. 𝜂ˆinst (𝑡) provides a step-wise proxy aligned with, but not identical to, the mission-level EE objective in (11), 𝑤 pen (𝑒) is a penalty weight that grows with the curriculum progress, and 𝑅term (𝑡) acts only at the end of an episode. The navigation, node-visiting, EE, speed, and constraint-violation terms are normalized to [0, 1] or [−1, 1] before weighting, while the terminal settlement uses explicitly specified bonus and penalty magnitudes; the tuned values are listed in Table II. Each component has a distinct role in connecting the learning objective with the mission requirements. The instantaneous EE term provides a step-wise proxy for the core missionlevel objective; the economic-speed reward 𝑅spd steers the UAV toward 𝑉eco ; the rate and sensing penalties incorporate the corresponding violations into the reward through penalty terms; the terminal settlement 𝑅term encodes the boundary and completion constraints. Finally, the distance-progress navigation shaping 𝑅nav provides dense directional feedback and is designed to encourage movement toward the nearest unserved node. 1) Navigation Shaping: To provide dense directional feedback in an otherwise sparse-reward environment, let 𝑑near (𝑡) be the distance from the UAV to the nearest unserved node. The distance-progress navigation shaping takes its per-step decrement, normalized by the maximum horizontal displacement in one slot, 𝑑near (𝑡 − 1) − 𝑑 near (𝑡) , (22) 𝑅nav (𝑡) = 𝑉max 𝛿𝑡 which is subsequently clipped to [−1, 1] before weighting. 2) Node-Visiting Reward: When node 𝑗 is effectively served for the first time at step 𝑡, i.e., the UAV enters its service range and the corresponding rate or sensing requirement is met, a reward is granted, Õ   𝑅visit (𝑡) = 1 𝑏 𝑗 (𝑡 − 1) = 0 ∧ 𝑏 𝑗 (𝑡) = 1 𝜌 𝑗 (𝑡), (23) 𝑗

where 𝑏 𝑗 ∈ {0, 1} is the visited mask of node 𝑗, 1[·] the indicator function, and 𝜌 𝑗 (𝑡) ∈ (0, 1] a scaling factor increasing with the achieved service quality. 3) Economic-Speed Reward: To keep the UAV near the economic speed derived in (6), we use kv 𝑥 𝑦 (𝑡) k − 𝑉eco , (24) 𝑅spd (𝑡) = 1 − 𝑉max

7

TABLE II W EIGHTS OF THE COMPOSITE REWARD COMPONENTS Weight 𝜆nav 𝜆visit 𝜆EE 𝜆spd 𝑤pen (𝑒) 𝑟done 𝜆term 𝜁bnd 𝜁unv

Component Navigation shaping reward Node-visiting reward Energy-efficiency reward Economic-speed reward Constraint penalty (curriculum) Mission-completion bonus Terminal EE bonus weight Out-of-boundary penalty Unserved-node penalty (per node)

Value 2.0 150 20 50 0.5 → 5.0 500 300 300 80

which is already normalized by 𝑉max and encourages operation near the analytically derived economic speed. 4) Constraint Penalties: When the instantaneous rate of a user falls below 𝑅min , a penalty proportional to the relative rate deficit is imposed and averaged over the users so as to lie in [0, 1],
𝑀 1 Õ max 0, 𝑅min − 𝑅𝑚 (𝑡) Penaltyrate (𝑡) = ∈ [0, 1]. (25) 𝑀 𝑚=1 𝑅min Although the hard communication constraint in (P1) is imposed only at service instants, the rate-violation penalty is evaluated at every step as a dense shaping signal to guide the policy toward favorable communication conditions. For the sensing side, the penalty is continuously evaluated against the nearest unserved sensing target. Let 𝑘 ★ (𝑡) = arg min𝑘 ∈ Kunserved 𝑑 tar 𝑘 (𝑡) denote the currently tracked target. When all sensing targets have been served, we set Penaltysens (𝑡) = 0. Otherwise, the sensing violation is measured in orders of magnitude beyond the threshold and
squeezed into [0, 1]. Let Δsens (𝑡) = log10 𝜖 𝑘sens ★ (𝑡 ) (𝑡)/Γsens denote the logarithmic sensing violation. The penalty is given by  
 max 0,Δsens (𝑡 )    clip , 0, 1 , Kunserved ≠ ∅, 𝐿sens Penaltysens (𝑡) =   0, Kunserved = ∅,  (26) where 𝐿 sens is a normalization scale spanning roughly two decades. Since 𝜖 𝑘sens ★ (𝑡 ) (𝑡) depends jointly on the echo quality, the aggregate sensing power, and the UAV-target geometry, this penalty simultaneously drives the trajectory and the powersplit policy. 5) Terminal Settlement: At the end of an episode we apply   + 𝑟 done + 𝜆term 𝜂¯ˆinst , all nodes served,     𝑅term = − 𝜁bnd , out of the service area, (27)     − 𝜁unv 𝑁unv , otherwise,  where 𝑁unv is the number of unserved nodes. Upon completion, an extra bonus proportional to the episode-average Í 𝑁𝑒 normalized EE is granted, defined as 𝜂¯ˆinst = 𝑁1𝑒 𝑡=1 𝜂ˆinst (𝑡) where 𝑁 𝑒 is the duration of the episode, to further reinforce energy-saving behavior.

C. SAC Update and Curriculum Schedule Unlike conventional RL, which maximizes the cumulative reward alone, SAC incorporates the policy entropy H (𝜋(·|s𝑛 ))

PREPRINT FOR IEEE JOURNAL

8

Algorithm 1 Curriculum-Guided SAC for Joint UAV-ISAC Optimization

UAV-ISAC Environment

Explore

Fig. 3. Architecture and update flow of the SAC agent.

into the objective [28], 𝑁Õ max −1   E (s𝑛 ,a𝑛 )∼𝜌 𝜋 𝑟 𝑛 + 𝛼H (𝜋(·|s𝑛 )) , 𝐽 (𝜋) =

(28)

𝑛=0

where the temperature 𝛼 controls the trade-off between expected reward and policy entropy and is adjusted automatically during training [29] to navigate the complex boundary formed by the communication and sensing constraints. As shown in Fig. 3, the architecture comprises an actor 𝜋 𝜃 and twin critics 𝑄 𝜙1 , 𝑄 𝜙2 . Given the limited onboard computing capability, the replay buffer and the network training reside on the ground edge server, and only the actor is kept onboard for online inference. Since the actor network consists of lightweight fully connected layers, the onboard forward-pass inference requires less than a millisecond per step, fully satisfying the real-time control requirements of highly dynamic UAV systems while effectively bypassing the onboard computational bottleneck. The critics are updated by minimizing the Bellman error h
2i 𝐽𝑄 (𝜙 𝑐 ) = E (s𝑖 ,a𝑖 )∼D 1 𝑄 𝜙𝑐 (s𝑖 , a𝑖 ) − (𝑟 𝑖 + 𝛾𝑉ˆ (s𝑖+1 )) , 2

(29)

and the actor by the reparameterization trick with   𝐽 𝜋 (𝜃) = Es𝑖 ∼D 𝛼 log 𝜋 𝜃 (a𝑖 |s𝑖 ) − 𝑄 𝜙min (s𝑖 , a𝑖 ) .

(30)

The second mechanism is the curriculum-learning schedule, which follows the principle of presenting progressively harder training instances [30]. With a progress factor 𝑐(𝑒) = min(𝑒/𝐸 𝑐 , 1) over training episodes 𝑒, the minimum rate requirement 𝑅min , the sensing threshold Γsens, and the proximity threshold 𝑑th are tightened toward their final targets, while the violation penalty weight 𝑤 pen (𝑒) is linearly amplified. However, relying solely on the standard automatic temperature adjustment of SAC is insufficient under a nonstationary curriculum. Under progressively tightened constraints, the automatically adapted entropy temperature may decrease substantially, reducing policy stochasticity during difficult curriculum transitions. To mitigate this effect, we implement a curriculum-synchronized entropy-temperature scheduling mechanism by imposing an episode-dependent lower bound on 𝛼, which decays stepwise in accordance with the curriculum progression (e.g., stepping down from 0.05 during the early loose-constraint phases to 0.01 in the final stringent phase). This heuristic mechanism helps preserve policy stochasticity during curriculum transitions. The complete procedure is summarized in Algorithm 1.

1: Initialize actor 𝜋 𝜃 , critics 𝑄 𝜙1 , 𝑄 𝜙2 , targets 𝑄ˆ 𝜙𝑐 ← 𝑄 𝜙𝑐 , replay buffer D, temperature 𝛼 0 → 𝑅★ , Γ 0 2: Curriculum setup: horizon 𝐸 𝑐 ; ranges 𝑅min sens → min ★ , 𝑑 0 → 𝑑★ , 𝑤 0 ★ , and exploration lower bound Γsens → 𝑤 pen pen th th schedule 𝛼min : 0.05 → 0.01 3: for each episode 𝑒 = 1, . . . , 𝐸 do 4: 𝑐 ← min(𝑒/𝐸 𝑐 , 1) ★ 0 Γsens (𝑒) ← 5: 𝑅min (𝑒) ← clip(𝑐 𝑅★ min , 𝑅min , 𝑅min ); 10 𝑔0 −𝑐(𝑔0 −𝑔1 ) 0 ★ 0 0 0 − 𝑑★ − 𝑐(𝑑th 6: 𝑑th (𝑒) ← 𝑑th th ); 𝑤 pen (𝑒) ← 𝑤 pen + 𝑐(𝑤 pen − 𝑤 pen ) 7: Update the temperature lower bound 𝛼min based on the current curriculum phase 8: Reset environment; initialize s0 and visited mask b ← 0 9: for each step 𝑛 = 0, . . . , 𝑁max − 1 do 10: Sample a𝑛 ∼ 𝜋 𝜃 (·|s𝑛 ); execute to obtain s𝑛+1 , service metrics, and Ptotal 11: Evaluate the composite reward 𝑟 𝑛 in (21) with the current curriculum weights 12: Update b; store (s𝑛 , a𝑛 , 𝑟 𝑛 , s𝑛+1 ) in D 13: if update conditions are met then 14: Sample a mini-batch from D 15: Critics: for each sampled transition 𝑖, 𝑦 𝑖
= 𝑟 𝑖 + 𝛾 min𝑐=1,2 𝑄ˆ 𝜙𝑐 (s𝑖+1 , ã𝑖+1 ) − 𝛼 log 𝜋 𝜃 ( ã𝑖+1 |s𝑖+1 ) ; update 𝜙𝑐 by minimizing 21 (𝑄 𝜙𝑐 (s𝑖 , a𝑖 ) − 𝑦 𝑖 ) 2 16: Actor: update 𝜃 by maximizing min𝑐=1,2 𝑄 𝜙𝑐 (s𝑖 , ã 𝜃 ) − 𝛼 log 𝜋 𝜃 ( ã 𝜃 |s𝑖 ) 17: Temperature and targets: update 𝛼 by minimizing 𝐽 (𝛼) subject to 𝛼 ≥ 𝛼min ; soft update 𝜙ˆ𝑐 ← 𝜏𝜙𝑐 + (1 − 𝜏) 𝜙ˆ𝑐 18: end if 19: end for 20: end for

Note that the curriculum and the synchronized annealing are treated strictly as external training-time scheduling mechanisms rather than part of the environmental state, ensuring the policy observes only the physically available system information. Because the curriculum parameters vary across training episodes, transitions in the replay buffer may be associated with different curriculum levels. We retain these transitions without retroactive reward recomputation; the finite FIFO buffer reduces the persistence of transitions generated under substantially earlier curriculum conditions. D. Computational Complexity Analysis To evaluate the practical feasibility of the proposed CGSAC framework, we analyze its computational and space complexities. The architecture relies on an asymmetric edgeassisted deployment: training is offloaded to the ground server, while only the actor network operates onboard the UAV. Let |S| and |A| denote the dimensions of the state and action spaces, respectively. Suppose the actor and critic networks are implemented as multi-layer perceptrons (MLPs) with 𝐻 hidden layers, each containing 𝑈 neurons. During the training phase at the ground edge server, the time complexity per update step is dominated by the forward and backward propagation through one actor and two critic networks, alongside the target network soft updates. The time complexity is bounded by O(|S|𝑈 + (𝐻 − 1)𝑈 2 + 𝑈|A|) per network. The space complexity is primarily dictated by

PREPRINT FOR IEEE JOURNAL

9

TABLE III S IMULATION PARAMETERS OF THE UAV-ISAC SYSTEM AND THE SAC ALGORITHM Symbol 𝐷area 𝐻min 𝐻max 𝑀 𝐾 𝑉max 𝑉𝑧,max 𝑃hover 𝑃full 𝑃elec 𝑃max 𝜅𝑧 𝐸 𝛿𝑡

Description Simulation area Minimum flight altitude Maximum flight altitude Number of communication users Number of sensing targets Maximum horizontal speed Maximum vertical speed Hovering power Full-speed flight power Hardware circuit power Maximum transmit power budget Vertical climbing power coef. Total training episodes Time slot duration

Value 1000 × 1000 m2 50 m 150 m 3 3 40 m/s 20 m/s 80 W 250 W 10 W 5W 12.0 J/m 5000 1s

the experience replay buffer D with capacity 𝐵size , requiring O(𝐵size (|S| + |A|)) memory. These requirements are easily handled by standard ground servers. During the online inference phase onboard the UAV, the critics and the replay buffer are entirely discarded. During inference, the mean action of the trained actor is used for deterministic control. Thus, the onboard time complexity per control step is O(|S|𝑈+(𝐻−1)𝑈 2 +𝑈|A|), and the space complexity is reduced to storing the actor network weights, which is also O(|S|𝑈 + (𝐻 − 1)𝑈 2 +𝑈|A|). On our test platform, the measured actor inference latency is below 1 ms per control step. For the adopted 2-layer, 256-unit multi-layer perceptron architecture, the resulting actor inference requires only a small number of matrix operations. The measured latency indicates that the trained actor can support sub-millisecond control updates on the tested platform. VI. P ERFORMANCE E VALUATION A. Simulation Setup A 3D simulation environment is built with 3 communication users and 3 sensing targets randomly distributed in the area, corresponding to the small relief unit described in Section III. The flight altitude is confined to 50 ∼ 150 m and the system bandwidth is 10 MHz. Each slot lasts 𝛿𝑡 = 1 s and an episode is capped at 𝑁max = 800 slots. From the energy model of Section III-C, the nominal upper bound of the instantaneous total power during horizontal flight max is Ptotal ≈ 265 W, attained when flying at 𝑉max . The onboard energy budget is strictly set to 𝐸 max = 212 kJ (≈ 58.9 Wh). This value is established as a nominal energy-budget baseline corresponding to the energy required to fly at the maximum horizontal speed 𝑉max for 𝑁max = 800 slots without climbing (i.e., ≈ 265 W × 800 s). While vertical climbing incurs additional instantaneous power according to (3), the overall energy budget remains hard-capped at 212 kJ, forcing the agent to manage its 3D maneuvers efficiently to avoid premature energy exhaustion. An episode that reaches 𝑁max slots or exceeds 𝐸 max without completing all visits is declared unsuccessful. To improve exploration in the non-convex space, a curriculum tightens the minimum rate to 25 Mbps, the sensing threshold to 1010 , and the proximity threshold to 60 m, while amplifying

Symbol 𝐵 𝜎2 𝑅min Γsens 𝑔0 , 𝑔1 0 , 𝑑★ 𝑑th th 𝐸𝑐 𝛼min 𝜂 𝛾 𝜏 𝐵size 𝑁max 𝐿sens

Description System bandwidth Noise power Minimum rate requirement Sensing accuracy threshold Sensing threshold log-exponents Initial/final proximity thresholds Curriculum horizon (episodes) Entropy temp. lower bound Learning rate Discount factor Soft-update coefficient Batch size Maximum steps per episode Sensing penalty norm. scale

Value 10 MHz 10−11 W 5 → 25 Mbps 1012 → 1010 12, 10 80, 60 m 3000 0.05 → 0.01 3 × 10−4 0.99 0.005 512 800 2

the penalty weight and the EE reward. All parameters are listed in Table III. For performance comparison, we evaluate CG-SAC against four continuous-control DRL baselines: DDPG [21], TD3 [22], PPO [31], and the standard maximum-entropy Vanilla SAC [28]. Vanilla SAC serves as a baseline for evaluating the combined effect of curriculum guidance and synchronized entropy scheduling in this highly constrained environment. All non-curriculum baselines use the fixed final task constraints, whereas curriculum-enabled methods follow identical prescribed schedules and episode budgets to ensure fairness. The subsequent performance evaluation is systematically structured into four parts. First, we analyze the training convergence and policy stability against the baseline algorithms. Second, the macro-level inference performance is evaluated across randomized test scenarios. Third, a micro-level analysis investigates the service quality distributions and parameter sensitivity. Finally, an ablation study examines the roles of four selected reward components (including shaping terms and penalties). B. Convergence and Baseline Comparison Fig. 4 presents a holistic view of the training trajectories over 5000 training episodes across four key metrics, merging the behavioral analysis of the proposed method and the baselines. As shown in Fig. 4(a), the average number of served nodes rises steadily and approaches the maximum of six nodes per episode. Among the baselines, Vanilla SAC demonstrates the most resilience, plateauing at approximately 4 nodes per episode. Without the proposed curriculum and synchronized entropy-temperature scheduling, Vanilla SAC converges to a conservative sub-optimal operating regime. In contrast, PPO and TD3 initially show moderate coverage but suffer severe policy collapses after episode 1500, dropping to approximately 1 to 2 nodes, while DDPG exhibits persistently poor coverage throughout the training. These behavioral patterns are directly reflected in the energy efficiency trends depicted in Fig. 4(b). The EE of CG-SAC robustly climbs and stabilizes around 0.8 to 0.9 Mbits/J. Vanilla SAC settles into a sub-optimal plateau around 0.6

PREPRINT FOR IEEE JOURNAL

10

6

1.5

Energy Efficiency (Mbits/J)

Number of Served Nodes

5 4 3 2 1

1

0.5

0

0

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

0

500

1000

1500

2000

Episode

(a) Task completion stability.

3000

3500

4000

4500

5000

(b) Energy efficiency trend.

8

40

7

35

600



500

25

400

 20

300

15

 30

    

10

200 v

A

Mission Completion Steps / Episode

2500

Episode

5

100

T

O   !" #$%

0

0 0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

0

500

(c) Path efficiency trend.

1000

1500

2000

2500

3000

3500

4000

4500

5000

Episode

Episode

(d) Episode-average horizontal speed.

Fig. 4. Training convergence behavior and physical state adaptation of the proposed CG-SAC and baseline algorithms. (a) Number of served nodes. (b) Energy efficiency. (c) Mission completion steps per episode. (d) Episodeaverage horizontal flight speed and the model-derived propulsion-economic speed reference.

to 0.7 Mbits/J, establishing itself as the strongest baseline but still falling significantly short of the proposed method. Meanwhile, the EE of TD3 declines in tandem with its completion rate, and PPO drops to a low plateau near 0.1 to 0.2 Mbits/J, demonstrating the inability of these standard methods to maintain efficient flight profiles under strict constraints. The underlying causes of these performance gaps are fundamentally illustrated by the path efficiency in Fig. 4(c) and the flight speed adaptation in Fig. 4(d). CG-SAC progressively reduces the episode length toward approximately 200 steps during training, while the fully trained policy achieves 107.6 steps on the unseen test scenarios. Vanilla SAC stagnates at approximately 400 steps, whereas TD3 and DDPG exhibit substantially longer and more unstable trajectories, often approaching the episode limit. Furthermore, Fig. 4(d) reveals that under the explicit economic-speed shaping term, CGSAC learns a policy that closely aligns its flight speed with the model-derived propulsion-economic speed reference of 24.69 m/s. Vanilla SAC exhibits a more conservative speed profile, which prolongs mission duration and is consistent with its lower EE. PPO adopts a high-consumption strategy of sprinting near the maximum speed (nearly 38 m/s), which severely penalizes its propulsion energy and leads to frequent early terminations, resulting in fewer steps per episode despite its poor mission completion. C. Inference Performance Evaluation 1) Macro-Level Performance Evaluation: To validate generalization, the trained policies are deployed across 2000 randomized test scenarios, with macro-level statistics summarized in Fig. 5. In Fig. 5(a), the proposed CG-SAC demonstrates a substantial advantage in energy efficiency, achieving an average EE of 0.72 Mbits/J. This performance is over three times higher than the best baseline, TD3, which only yields 0.21 Mbits/J. The boxplot distribution reveals that CG-SAC

flexibly adapts its energy expenditure to various topological layouts, whereas the baseline policies exhibit less efficient flight behaviors across the tested layouts. This difference is further reflected by the mission completion steps in Fig. 5(b). For each method, the average mission completion steps are computed exclusively over successfully completed missions. CG-SAC requires an average of 107.6 mission steps over successfully completed episodes. Conversely, DDPG and TD3 suffer from severe navigational wandering, consuming over 500 steps, while PPO requires 319.1 steps, leading to substantially higher propulsion-energy consumption. Under the final stringent constraints, Vanilla SAC exhibits an insufficient full-mission completion rate; therefore, its successful-missionconditioned metrics are reported as N/A in the subsequent macro- and micro-level analyses. Its role in the evaluation is primarily to isolate the combined contribution of curriculum guidance and synchronized entropy-temperature scheduling in the training convergence analysis. Beyond navigational efficiency, satisfying the QoS requirement at the service instants is a key mission objective. Fig. 5(c) evaluates the communication satisfaction rate, calculated exclusively at the visit instants to reflect the true service reliability. CG-SAC successfully satisfies the strict rate threshold (𝑅min ≥ 25 Mbps) in 99.6% of the service events. The baselines are less effective at adapting their 3D positions to favorable channel conditions, resulting in severe degradation. TD3 achieves 66.4%, PPO achieves 56.4%, and DDPG fails in more than half of its attempts with a mere 41.4% success rate. This contrast highlights that navigating to a two-dimensional coordinate alone is insufficient in this setting, because the achievable service quality also depends on the UAV altitude and flight state; the agent must actively manage its three-dimensional altitude and speed to optimize the dynamic channel capacity. Finally, Fig. 5(d) indicates that all tested policies satisfy the basic sensing-feasibility criterion at service instants (𝜖 sens ≤ 1010 , where time and target indices are omitted for brevity). This uniform success is largely attributable to the spatial distance ceiling enforced during the state transitions, which naturally guarantees a baseline level of radar return power once a visit is triggered. However, simply crossing this conservative threshold does not imply identical sensing precision, a nuance that necessitates the subsequent microlevel distribution analysis. 2) Micro-Level Service Quality and Sensitivity Analysis: To investigate the fine-grained performance and sensitivity to sensing-channel variations, we conduct a micro-level analysis focusing on the cumulative distributions and sensitivity behaviors depicted in Fig. 6. In Fig. 6(a), the cumulative distribution function (CDF) of the served user rate reveals the distinct performance limitations of the baseline algorithms. While CG-SAC maintains a sharp, right-shifted distribution where nearly all visit-instant rates exceed the 25 Mbps QoS threshold, the baselines exhibit substantial probability mass near zero rate. Specifically, DDPG and PPO experience immediate probability jumps of over 55% and 35% near zero, indicating that more than half of their attempted service interactions fail to establish meaningful

PREPRINT FOR IEEE JOURNAL

11

UMgfR \ UMM

SRMfR

ac d e ;<

^ SMM ] d[c

1 5 :

b

2

a RMM 93 9:

` /

^ IMM

INhfN

] 66 _ 5 -

0*E1

34 0*+

] \[\

0*1F [

Z QMM

2 / .

0.11 -

A 0

&'(

CG-SAC

''D)

u ~

”

| { 

”ŽŽŽ¡

”ŽŽŽ¡

”ŽŽŽ¡

› Ž’ çìê

éí

œ

é Þßá š Ž‘

æ

—

çè •

ã êç

é 0.4 – æ å

Ž ä

ä Þßà ã o

CG-SAC

ijk

jjlm

lln

Ž

10 ·¸¹º»¹¼ ½¾¿

10

10

11

10

1

ë

w

(c) Communication satisfaction rate.

9

10

(b) CDF of sensing-error surrogate.

è

˜™

y

8

¢¥£

1

ž

{

y opq

¢¤£

î

•

x

60 80 ¢££ ¦§¨©§ª «¬§¨ ­®¯§ °±²³¬´

ã 0.8

™—˜

TD3 DDPG PPO CG-SAC (Proposed) ÅÆÇ Threshold

ÀÁÂ

é

›

z

40

êç

|

ÀÁÃ

0

20

(a) CDF of served user rate.

æ

~} opr

ÀÁÄ

”ŽŽŽ¡ ïê

rupr† x|

£µ¤

0

™Ÿœ

‡spr†

~ ops

TD3 DDPG PPO CG-SAC (Proposed) QoS Threshold

0

JJL

› Ž“

sspr†

€

HHJK

£µ¥

—

{ ‚~

GHI

£µ¶

œ–

ƒ |x opt

CG-SAC

1

0.8

(b) Average mission completion steps.

ps†

}

NMVfU

NMM M

DDP

(a) Distribution of average EE. „

X W

A

,

Y

1

0.8

Sensing Satisfaction Rate

=

Cumulative Probability

0*CE 9>

Cumulative Probability

VMM

1*+ B @A?

â

CG-SAC

ˆ‰Š

‰‰‹Œ

‹‹

(d) Sensing satisfaction rate.

Fig. 5. Macro-level performance statistics of the proposed CG-SAC and baseline algorithms. (a) Distribution of average energy efficiency. (b) Average mission completion steps. (c) Communication satisfaction rate at visit instants (𝑅min ≥ 25 Mbps). (d) Sensing satisfaction rate at visit instants (𝜖 sens ≤ 1010 ).

communication links due to poor spatial alignment. A complementary microscopic perspective is illustrated by the distribution of the sensing-error surrogate in Fig. 6(b). Although all algorithms manage to stay below the basic threshold of 1010 , PPO and CG-SAC achieve substantially lower surrogate values across the entire distribution compared to TD3 and DDPG. While PPO also attains low surrogate values, it does so at the expense of communication performance (as evidenced by its high failure rate in Fig. 6(a)), whereas CG-SAC effectively balances both objectives. This indicates that CG-SAC not only fulfills the loose constraint but actively optimizes the geometric observation angle and proximity without compromising ground user service quality. Furthermore, Fig. 6(c) examines the communication satisfaction probability under stringent rate threshold sweeps ranging from 10 to 85 Mbps. As the requirement tightens, CGSAC exhibits remarkable stability, preserving a satisfaction rate above 95% up to 55 Mbps and retaining over 73% even at an extreme threshold of 85 Mbps. In contrast, the baseline algorithms suffer rapid performance degradation, dropping below 50% as soon as the threshold exceeds 55 Mbps. Finally, Fig. 6(d) examines the sensitivity of the trained policies to variations in the sensing-channel coefficient by scaling 𝜁sens from 0.3× to 3.0× of its baseline value. All algorithms maintain stable sensing satisfaction across the tested range, indicating that the sensing-feasibility metric is relatively insensitive to these channel-coefficient variations in the considered setting. D. Ablation Study of the Reward Design In a single-UAV ISAC system, the agent cannot distribute tasks among multiple units; it must sequentially satisfy stringent spatial, temporal, and communication constraints within a strictly limited onboard energy budget. To verify how the proposed reward framework guides the single UAV to untangle these highly coupled constraints, we conduct an ablation study to examine the roles of four selected reward shaping

0.8

0.6

0.4 TD3 DDPG PPO CG-SAC (Proposed) Baseline

0.2

PPO CG-SAC (Proposed)

0 10

25

40 55 70 ÊËÌÌÍÎÏÐÑÒÏËÎ ÓÑÒÔ ThrÔÕÖË×Ø ÙÚÛÜÕÝ

(c) Rate threshold sensitivity.

ÈÉ

0 0

ðñò

1

óñò

ô

ôñò

3

Sensing Channel Coefficient (x Baseline)

(d) Sensing channel sensitivity.

Fig. 6. Micro-level service quality and sensitivity evaluation of the proposed CG-SAC and baseline algorithms. (a) Cumulative distribution function of the served communication rate across all service attempts. (b) Cumulative distribution function of the sensing-error surrogate at visit instants. (c) Communication satisfaction probability under varying data rate thresholds. (d) Sensing satisfaction probability under varying sensing channel coefficients.

and penalty components: (i) w/o Nav removes the navigation shaping 𝑅nav ; (ii) w/o Rate removes the rate penalty; (iii) w/o Sens removes the sensing penalty; and (iv) w/o EE removes the EE reward. The network architectures, hyper-parameters, and curriculum schedules remain identical across all variants. Because the ablation variants are evaluated on a separate set of 100 unseen scenarios, the performance of the full model in Table IV is not directly identical to the 2000-scenario main evaluation. The evaluation metrics are summarized in Table IV. The completion rate represents the fraction of episodes where all six nodes are sequentially served. The rate and sensing satisfaction ratios are measured exclusively at the service instants, with the latter strictly evaluated under the 𝜖 sens ≤ 1010 criterion. The results in Table IV reveal two distinct categories of performance degradation: a total collapse of mission feasibility (w/o Nav and w/o Sens) and a severe decline in energy efficiency (w/o Rate and w/o EE). 1) Foundations of Mission Feasibility in 3D Space: For a single UAV tasked with sequentially visiting multiple dispersed nodes, the continuous 3D state space is immense, and the terminal completion reward is inherently sparse. The w/o Nav variant completely fails to establish an effective routing behavior, yielding a 0.0% completion rate and wandering for an average of 647.4 steps. As illustrated in Fig. 7(a), without the dense gradient field provided by the distance-progress navigation shaping, the agent cannot learn the basic geometric concept of approaching a target. Conversely, the w/o Sens variant exhibits a progressive structural failure. Its task coverage initially rises but sharply degrades as the curriculum tightens. In an ISAC context, radar sensing is highly sensitive to the 3D relative distance and observation angle. Without the explicit sensing penalty, the single UAV optimizes its trajectory solely for proximity or communication, failing to satisfy the sensing-accuracy require-

PREPRINT FOR IEEE JOURNAL

12

TABLE IV T EST- STAGE PERFORMANCE OF THE REWARD - ABLATION VARIANTS OVER 100 UNSEEN SCENARIOS Rate Satisfaction (%) 99.7 99.3 97.8 99.3 98.3

Average Steps 97.4 39.1 597.8 77.8 647.4

< 0.8 :;9

v 4

56 0%(

ed

Average EE (Mbits/J) 0.739 0.613 0.664 0.628 0.675

0.9

6

õ

0%=>?

38 0.7

0%(@> AB@=%@CD

7



0%((E AB@0%@CD

0%(&F AB@'%0CD

GHIJKG LMNoOoPQRT 0%(=' ABF%=CD

41 -

3.4 0%'

 3 a

232

, 0.4

T

v 2

1*

./ 0.3

CG-SAC

w   w  S w   w  

1 0 0

õöö

1000

-

,+ 0%& *)

A

ment represented by the surrogate metric. Consequently, its completion rate plummets to 14.0%, and its sensing satisfaction ratio degrades to 59.8% (Table IV), suggesting that spatial navigation alone is insufficient to guarantee sensing feasibility. This behavior is corroborated by Fig. 7(d), where w/o Sens demonstrates severe policy instability, with episode completions randomly distributed across the 1 ∼ 3 (49%) and 4 ∼ 5 (33%) ranges. 2) Energy Efficiency and the Aerodynamic Dilemma: The variants w/o Rate and w/o EE successfully complete the mission with high probabilities (95.0% and 91.0%), demonstrating that they are capable of navigating the topology. However, their fundamental flaw lies in energy blindness, which is particularly detrimental under a strict single-UAV energy budget. As shown in Fig. 7(b), compared to the Full model (0.739 Mbits/J), the average EE drops significantly by 17.1% for w/o Rate (0.613 Mbits/J) and 15.0% for w/o EE (0.628 Mbits/J). The mechanical root of this inefficiency is captured in Fig. 7(c). The proposed Full model yields an episode-average horizontal speed of 22.7 m/s, operating efficiently just below the model-derived propulsion-economic speed reference of 24.7 m/s. In contrast, the w/o Rate and w/o EE variants aggressively accelerate to 26.8 m/s and 26.1 m/s, respectively. This reveals the core physical dilemma of single-UAV control: the trade-off between mission duration and aerodynamic power consumption. Without the rate penalty, the UAV is not heavily penalized for establishing brief, lowquality communication links; thus, it adopts a “sprinting” strategy to rapidly finish the mission (averaging merely 39.1 steps). Similarly, without the explicit EE reward, standard reinforcement learning simply maximizes the unnormalized step-wise returns by finishing as quickly as possible. Because the propulsion power of a rotary-wing UAV grows with the cube of its horizontal speed (𝑉 3 ), sprinting leads to substantially higher propulsion-energy consumption. The full CGSAC model explicitly encourages the UAV to manage this aerodynamic profile: decelerating to loiter within effective coverage areas to secure sufficient communication rates, while maintaining an economic cruising speed between targets. 3) Microscopic Speed Adaptation: To substantiate this aerodynamic trade-off at a microscopic level, Fig. 8 contrasts the per-step speed distributions of the Full model and the w/o Rate variant over 100 test episodes. It is important to note that while the speed statistic in Fig. 4(d) represents the episode-level average cruising speed, Fig. 8 captures the full per-step speed distribution, including the low-speed periods required for service interactions. For a single UAV, optimizing service quality necessitates physical deceleration to prolong

Sensing Satisfaction (%) 89.9 93.3 59.8 93.5 86.2

A

Completion (%) 91.0 95.0 14.0 91.0 0.0

÷õöö

2000 øõöö 3000 Tûüýþýþÿ r ý


ùõöö

4000

úõöö

0.1

õööö

0 CG-SAC

(a) Task completion stability. €‚ƒ

30

{z ec y ux

e~|

„ †‡ ˆ‰Š

  !

 ""

 #$

ªisited ° 0 ªisited ° ·µ¸

›˜™

ªisited ° ´µ¶ ª«¬«­®¯ ° ±²² ³

© 100 §¨

ed|

sts ef

 

(b) Average energy efficiency.

e~|d

ee|}

¦

¥ž¤ 80

q



p n

m dc lhk

¤ š™ £ ¢

1d|~

ji

hg 10

¡

žŸ 40  œ

A

Variant Full (proposed) w/o Rate (no rate penalty) w/o Sens (no sensing penalty) w/o EE (no EE reward) w/o Nav (no navigation shaping)

c 0 CG-SAC

UVW XYZ[

UVW \[]^

UVW __

UVW `Yb

(c) Flight speed adaptation.

˜™ 0 CG-SAC

‹Œ Ž‘

‹Œ ’‘“”

‹Œ ••

‹Œ –—

(d) Task completion breakdown.

Fig. 7. Ablation study of the proposed CG-SAC framework under different reward components and architectural variants. (a) Training convergence trajectories of average targets covered. (b) Average energy efficiency and performance degradation relative to the full model. (c) Average flight speed compared to the model-derived propulsion-economic speed reference 𝑉eco . (d) Percentage breakdown of target completion proportions per episode across variants.

the interaction time with ground nodes. As shown in the box plot and CDF, the per-step speed distribution has a median of approximately 5 m/s with a wide interquartile range and an overall per-step mean speed of 12.6 m/s. This indicates an intelligent, highly dynamic policy: the UAV decelerates when approaching a service point to improve the achievable service quality, while accelerating during transit. Conversely, without the rate penalty, the median speed of w/o Rate shifts aggressively to 36 m/s, with its distribution heavily concentrated above the economic speed 𝑉eco and pushing the mean speed up to 29.1 m/s. A Mann-Whitney U test confirms that this behavioral divergence is statistically significant (𝑝 < 0.001). Consequently, the per-step instantaneous EE of the Full model (0.822 Mbits/J) substantially outperforms that of the w/o Rate variant (0.706 Mbits/J), demonstrating that explicit QoS constraints are required to suppress energywasting sprinting behaviors. In summary, the ablation study confirms that the selected reward components play complementary roles in mission feasibility and energy efficiency for navigating the highly coupled constraints of a single-UAV ISAC mission. The navigation shaping 𝑅nav provides the primary spatial guidance; the sensing penalty Penaltysens promotes stable targetoriented behavior under the sensing requirement; and the rate penalty Penaltyrate together with the EE reward 𝜆EE encourages energy-efficient flight. Together, these components provide a practical balance among mission completion, service quality,

PREPRINT FOR IEEE JOURNAL

13

24.7 m/s

Flight Speed (m/s)

Veco

Ï×Ð ÆÇÉ

24.7 m/s



80

ÐÍÖ ÎÖ



Ó





 60

ÔÕ ÆÇÈ

30

Ò



ÍÎË 0.4

 40

Ì Ë Ê

10

C

0.2

20

0

0

0

CG-SAC

¹º» ¼½¾¿

(a) Box plot of per-step speed.

PP T PD D



ÐÑÏ

ÀÁ

1.2

100

ØÙÚÛÜØ ÝÞßàáâãäåæç èéêâãëåìí îïð ñàéß ÝÞßàáâäòåãç èéêâãìåóí

1 Veco

40

Avg. Energy Efficiency (Mbits/J)

50

Â

10

ÃÂ

20

ÄÂ

30

ÅÂ

40

Flight Speed (m/s)

(b) CDF of per-step speed.

0

ôô

60

õô

80 ö÷ öô øô úûüýþÿ E
þü ü 

ù÷

0.8 0.6 0.4 0.2 0

ùô

100

55

60

65

7

75

80

85

9

95

100

Energy Budget Scale (%)

(a) Mission completion rate.

Fig. 8. Microscopic per-step flight-speed distributions of the Full model and the w/o Rate variant over 100 test episodes. (a) Box plot illustrating the interquartile speed ranges. (b) Cumulative distribution function showing the shift toward high-consumption flight regimes in the ablated variant.

PPO TD3 DDPG CG-SAC

1

(b) Average energy efficiency.

40 35

43 2

01 30 / ..-

, 25 + *% ()

&'% 20

::; <=> ==:? @?ABF@

v

A

and propulsion efficiency.

15

Veco

E. Sensitivity Analysis of Onboard Energy Budget To evaluate the sensitivity of the proposed framework to onboard energy limitations, we conduct a sensitivity analysis on the onboard energy budget 𝐸 max . For this evaluation, 𝐸 max is directly enforced as an additional episode-termination condition; an episode terminates with failure once the accumulated energy exceeds the prescribed budget, while the nominal 𝑁max = 800 step cap remains unchanged. We scale the nominal battery capacity (212 kJ) down from 100% to approximately 56% (i.e., 120 kJ) and evaluate the policies over 100 unseen scenarios. Error bars represent the 95% confidence intervals. The performance of CG-SAC exhibits a non-monotonic trend over the tested energy-budget range. When the energy budget is abundant (100%), CG-SAC achieves its peak energy efficiency of 0.72 Mbits/J. As the budget is restricted to 71%, the overall EE experiences a dip to roughly 0.53 Mbits/J. Interestingly, under the most severe constraint of 56%, the completion rate returns to nearly 90% and the EE recovers to peak at approximately 0.9 Mbits/J. The apparent recovery of EE at the most restrictive budget is partly attributable to early termination of energy-infeasible trajectories, which concentrates successful episodes on efficient flight regimes. Throughout this restriction, CG-SAC consistently regulates its average flight speed near the model-derived propulsioneconomic speed reference of 24.7 m/s (Fig. 9(c)), indicating stable adaptation around the economic-speed region. In contrast, the baseline algorithms exhibit structural instabilities. As shown in Fig. 9(c), PPO adopts an energywasting sprinting strategy near the maximum speed (37 ∼ 38 m/s) regardless of the energy budget, resulting in nearzero completion rates and negligible EE. DDPG similarly struggles with excessive speeds and fails to complete the sequential visits. Interestingly, TD3 exhibits an anomalous behavioral shift: while it completely fails at energy budgets of 71% and above due to severe navigational wandering, the extreme energy constraint at the 56% scale appears to inadvertently restrict its wandering, resulting in an unexpected spike in completion rate (∼ 95%) and EE. Nevertheless, the proposed curriculum-guided framework maintains the most favorable overall performance across the tested nominal and reduced-energy budgets. Notably, the policy remains effective across these externally imposed energy budgets without explicit remaining-energy feedback in the state. This result

10

60

!"

!

80

#

$"

$

24.7 m/s

100

Energy Budget Scale (%)

(c) Average flight speed. Fig. 9. Sensitivity analysis of the proposed CG-SAC and baseline algorithms under varying onboard energy budgets (𝐸max ) with 95% confidence intervals. (a) Mission completion rate. (b) Average energy efficiency. (c) Average flight speed.

indicates that the learned policy is relatively insensitive to the tested budget variations. This behavior is consistent with the combined design of maximum-entropy exploration and the EEcentric composite reward, allowing the agent to dynamically reshape its aerodynamic and trajectory profile under severe hardware limitations. F. Limitations and Open Directions The proposed curriculum-guided reinforcement learning framework establishes a baseline for energy-efficient UAVISAC systems by jointly optimizing the 3D trajectory and resource allocation of a single aerial platform. Building upon this foundation, several directions can be extended for future research. First, while this study resolves the highly coupled aerodynamic and QoS constraints for a single UAV, it provides a scalable algorithmic structure for multi-UAV cooperative ISAC systems. Future work can extend the current singleagent continuous control to multi-agent frameworks, addressing inter-UAV interference and joint task assignment in largescale disaster areas [1], [4]. Second, the designed spatial-temporal navigation policy relies on ideal CSI and static sensing targets to validate the theoretical aerodynamic trade-offs. This deterministic environment modeling serves as a basis for investigating robust policies under imperfect CSI and tracking mobile targets, which are critical for vehicular and low-altitude applications. Third, the current model leverages the 3D mobility of the UAV to proactively optimize channel capacity. This geometric optimization can be further synergized with RIS technologies. By jointly optimizing the UAV trajectory and RIS phase shifts, future studies can alleviate the sensing and communication bottleneck in severely obstructed environments. Finally, the formulation of the normalized composite reward successfully disentangles the competing ISAC constraints with empirically tuned weights. This modular reward structure naturally paves the way for automated reward shaping or

PREPRINT FOR IEEE JOURNAL

multi-objective reinforcement learning, enabling the agent to adaptively balance communication, sensing, and energy objectives for online deployment. VII. C ONCLUSION This paper investigated energy-efficient mission planning for UAV-ISAC systems through a curriculum-guided soft actor-critic (CG-SAC) framework with propulsionaware reward shaping, jointly optimizing the 3D trajectory, communication-sensing power split, and per-user power allocation under coupled mobility, communication, sensing, and energy requirements. A speed-dependent rotary-wing propulsion model was used to derive a closed-form horizontal propulsion-economic cruising-speed reference, while a loglinear curriculum progressively tightened the service requirements during training. Extensive evaluations over 2000 randomized scenarios show that CG-SAC achieves an average energy efficiency of 0.72 Mbits/J and requires 107.6 steps on average among successfully completed missions, corresponding to a 66%–82% reduction in flight steps relative to the evaluated DRL baselines. At service instants, the learned policy achieves a 99.6% communication-rate satisfaction ratio while satisfying the adopted sensing-feasibility threshold. The results further reveal mission-aware speed adaptation, with the UAV decelerating near service points and accelerating during transit, highlighting the benefit of jointly accounting for propulsion cost and service requirements in UAV-ISAC mission control. R EFERENCES [1] S. Lu, F. Liu, Y. Li, K. Zhang et al., “Integrated sensing and communications: Recent advances and ten open challenges,” IEEE Internet Things J., vol. 11, no. 11, pp. 19 094–19 120, 2024. [2] J. Zhang, W. Lu, C. Xing et al., “Intelligent integrated sensing and communication: A survey,” Science China Information Sciences, vol. 68, no. 3, p. 131301, 2025. [3] K. Meng, Q. Wu, J. Xu, W. Chen, Z. Feng, R. Schober, and A. L. Swindlehurst, “UAV-enabled integrated sensing and communication: Opportunities and challenges,” IEEE Wireless Commun., vol. 31, no. 2, pp. 97–104, 2024. [4] Y. Hu, T. Zhang, B. Li, and D. Yang, “A survey on UAV-enabled integrated sensing and communication networking and technologies,” Journal of Electronics & Information Technology, vol. 47, no. 4, pp. 859–875, 2025. [5] D. B. Licea, E. Nurellari, and M. Ghogho, “Energy-efficient 3D UAV trajectory design for data collection in wireless sensor networks,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8329–8333, 2020. [6] H. Zhao, Q. Hao et al., “Online trajectory optimization for energyefficient Cellular-Connected UAVs with map reconstruction,” IEEE Trans. Veh. Technol., vol. 73, no. 3, pp. 3445–3456, 2024. [7] Q. Zhu, R. Liu et al., “AoI-optimal trajectory planning in UAV-assisted ISAC networks,” IEEE 23rd International Conference on Communication Technology (ICCT), pp. 428–433, 2023. [8] Y. Li, X. Li et al., “Continuous UAV trajectory design with uncertain user location in ISAC networks,” IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6, 2024. [9] Z. Wang, J. Zhao et al., “A hierarchical MADRL framework for joint resource allocation and trajectory design in UAV enabled Multi-Static ISAC networks,” IEEE Wireless Commun. Lett., vol. 15, pp. 1325–1329, 2026. [10] R. H. Kumar, E. Aboutanios et al., “Optimizing the ambiguity function profile of FHCS: A sensing-communication trade-off,” 25th International Conference on Digital Signal Processing (DSP), pp. 1–5, 2025.

14

[11] Z. Ren, Y. Peng et al., “Fundamental CRB-rate tradeoff in MultiAntenna ISAC systems with information multicasting and Multi-Target sensing,” IEEE Trans. Wireless Commun., vol. 23, no. 4, pp. 3870–3885, 2024. [12] R. Liu, M. Li et al., “SNR/CRB-constrained joint beamforming and reflection designs for RIS-ISAC systems,” IEEE Trans. Wireless Commun., vol. 23, no. 7, pp. 7456–7470, 2024. [13] Y. Fu, Z. Li et al., “A predictive integrated sensing, communication, and computation Over-the-Air approach for IoV: Optimization and trade-off analysis,” IEEE J. Sel. Areas Commun., vol. 44, pp. 278–294, 2026. [14] H. Yan, Y. Chen, and S.-H. Yang, “New energy consumption model for Rotary-Wing UAV propulsion,” IEEE Wireless Commun. Lett., vol. 10, no. 9, pp. 2009–2012, 2021. [15] N. Gao, Y. Zeng et al., “Energy model for UAV communications: Experimental validation and model generalization,” China Communications, vol. 18, no. 7, pp. 253–264, 2021. [16] K. Wu, M. Feng et al., “Trajectory planning for Multi-rotor UAV based on energy cost model,” 41st Chinese Control Conference (CCC), pp. 1–6, 2022. [17] Z. Kehan, L. Zhiwei et al., “Performance assessment of a LowTemperature PEMFC powered integrated propulsion system for UAV with inclined cathode flow structure,” IEEE Vehicle Power and Propulsion Conference (VPPC), pp. 1–5, 2023. [18] X. Wu, L. Liang et al., “DRL-based trajectory optimization and computation-aware resource allocation for UAV-assisted edge computing networks,” IEEE Internet Things J., vol. 12, no. 20, pp. 43 540–43 558, 2025. [19] Y. Chen, Y. Yang et al., “Joint trajectory optimization and resource allocation in UAV-MEC systems: A Lyapunov-assisted DRL approach,” IEEE Trans. Serv. Comput., vol. 18, no. 2, pp. 854–867, 2025. [20] P. Qin, Y. Fu et al., “DRL-based resource allocation and trajectory planning for NOMA-enabled multi-UAV collaborative caching 6G network,” IEEE Trans. Veh. Technol., vol. 73, no. 6, pp. 8750–8764, 2024. [21] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in Proc. International Conference on Learning Representations (ICLR), 2016. [22] S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in Proc. International Conference on Machine Learning (ICML), 2018, pp. 1587–1596. [23] B. Adhikari, A. S. Khwaja et al., “Energy efficient RIS-assisted UAV networks using Twin Delayed DDPG technique,” IEEE Trans. Wireless Commun., vol. 23, no. 12, pp. 18 423–18 439, 2024. [24] H. Sun, Y. Zhou et al., “Average AoI-minimal trajectory design for UAV-assisted IoT data collection system: A Safe-TD3 approach,” IEEE Wireless Commun. Lett., vol. 13, no. 2, pp. 530–534, 2024. [25] O. A. Amodu et al., “A comprehensive survey of deep reinforcement learning in UAV-assisted IoT data collection,” Vehicular Communications, vol. 54, p. 100949, 2025. [26] A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal LAP altitude for maximum coverage,” IEEE Wireless Commun. Lett., vol. 3, no. 6, pp. 569–572, 2014. [27] Y. Zeng, J. Xu, and R. Zhang, “Energy minimization for wireless communication with Rotary-Wing UAV,” IEEE Trans. Wireless Commun., vol. 18, no. 4, pp. 2329–2345, 2019. [28] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor,” 2018. [Online]. Available: https://arxiv.org/abs/1801.01290 [29] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft actor-critic algorithms and applications,” 2019. [Online]. Available: https://arxiv.org/abs/1812.05905 [30] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proc. International Conference on Machine Learning (ICML), 2009, pp. 41–48. [31] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.

Related documents

• Performance Analysis of Reinforcement Learning-Optimized Firefighting Helicopter Routing Strategies in Simultaneous Multi-Focus Wildfire Scenarios#11831
• Performance Analysis of Reinforcement Learning-Optimized Firefighting Helicopter Routing Strategies in Simultaneous Multi-Focus Wildfire Scenarios#11830
• 6 U.S.C. § 321a — Evacuation plans and exercises#559646
• World-Action Models for Robot Learning and Control: A Survey#1110464
• WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory and In-Context Learning#1048747