HQARRF: Hierarchical Q-learning and Force-aware Routing for Multi-Charger Scheduling in Wireless Rechargeable Sensor Networks
arXiv:2609.13901v1 [cs.NI] 12 Sep 2026
Liang-Ching Tao and Pi-Chung Wang⋆ Department of Computer Science and Engineering, National Chung Hsing University, 145 Xingda Rd., South District, Taichung 402, Taiwan [email protected], [email protected]
Abstract. Multi-charger scheduling in wireless rechargeable sensor networks must weigh sensor death risk, charger energy, travel cost, returnto-base feasibility and inter-charger coordination at once, and schedulers driven by local urgency alone duplicate service and leave whole regions unattended. We present HQARRF, a two-level scheduler. Below, an interpretable ARR-F score ranks candidate clusters through an attraction term for local urgency, a repulsion term against charger crowding and a force bonus from nearby critical sensors. Above, adaptive zones compress regional state into a deadline-based risk estimate, and a gated Q-learning controller decides only whether to redirect service to a high-risk, underserved zone. Over 27 parameter points HQARRF attains the highest mean survival rate at 26, improving survival by 20.7 percentage points over the mean of five baselines and 9.2 over the strongest baseline at each point. An ablation isolates the upper level: its gain tracks how often the controller fires. Keywords: wireless rechargeable sensor networks, Q-learning, multicharger scheduling, force-aware routing, zone-level risk
1
Introduction
Wireless rechargeable sensor networks (WRSNs) keep battery-powered sensor nodes alive after deployment by delivering energy wirelessly rather than by replacing batteries [22, 7, 14]. A node that forwards traffic for many others drains far faster than one reporting only its own readings, so residual energy becomes uneven over time, and a node not recharged in time opens a coverage hole or disconnects the subtree behind it. Mobile chargers address this by touring the field and replenishing nodes in place [23, 1]. Deploying several chargers raises service capacity but creates a scheduling problem harder than route-length minimisation. Each charger has a ⋆
Corresponding author.
2
L.-C. Tao and P.-C. Wang
finite movement budget and must reserve enough energy to return to the base station, so a locally attractive target is a poor choice if reaching it strands the charger or starves another region; and when chargers evaluate targets independently, several may converge on one neighbourhood while other high-risk regions receive no service at all [2, 5]. Two observations motivate our design. Urgency is not purely local: several high-risk regions can appear at once, and a scheduler that always picks the single most urgent sensor allocates the fleet poorly. Yet the corrective signal needed to fix that is coarse — knowing which region is under-served is enough, and no learned ranking over sensor–charger pairs is required. We therefore propose HQARRF, which separates these concerns into two levels. At the lower level, an interpretable score named ARR-F ranks candidate clusters using an electrostatic analogy: attraction drawn from local urgency, repulsion that discourages charger crowding, and a force bonus contributed by nearby critical sensors. At the upper level, adaptive zones aggregate regional state into a risk estimate, and a gated Q-learning controller decides only whether scheduling focus should be redirected toward a high-risk, under-served zone. Confining learning to that one gated decision keeps the table under two hundred entries in every configuration we ran and leaves the routing behaviour explainable. Our contributions are threefold. (i) We formulate ARR-F, a local routing score that unifies urgency-driven attraction, anti-crowding repulsion and forceaware critical-sensor selection under one electrostatic analogy, with target reservation enforced separately as a hard constraint. (ii) We develop a zone-level risk estimator that converts per-sensor time-to-death slack into a bounded deadline risk and aggregates the most endangered sensors per zone, giving a regional view that candidate-cluster scoring alone cannot provide. (iii) We evaluate against five baselines over 27 parameter points and, rather than reporting the aggregate margin alone, use a four-step ablation to attribute it: the variant that differs from HQARRF only by the absence of its upper level decays with network size exactly as the baselines do, and the survival it recovers scales with how often its controller fires.
2
Related Work
Charging scheduling. Early WRSN work optimised the tour of a single charger, establishing that the optimal charging path is a shortest Hamiltonian cycle [22] and that energy provisioning bounds network lifetime [7]. On-demand schemes let nodes request service once their energy falls below a threshold [11, 18], which responds to actual demand but leaves the threshold as a fixed global parameter; utility-based selection ranks requests by a scalar value [15, 3], and partialcharging schemes trade per-node completeness for shorter queues [19]. Greedy service disciplines such as nearest-job-next were characterised analytically by He et al. [6].
HQARRF: Hierarchical Q-learning and Force-aware Routing for WRSNs
3
Multiple chargers. Once several chargers operate together, task allocation and coordination dominate. Wei et al. [21] schedule multiple chargers under time windows; Han et al. propose uneven cluster-based allocation [4] and explicit multi-charger cooperation [5]; dual-partition [8] and hierarchical [16] designs split the field to reduce interference between chargers. These works establish that regional structure helps, but the region assignment is generally static, so it cannot follow risk as it migrates across the field. Learning-based scheduling. Reinforcement learning has been applied to charging decisions in several forms [10, 9, 12, 17]. The common pattern is to learn the scheduling policy end to end, so the state space grows with the number of sensor– charger pairs and the resulting routes are hard to explain. The boundary we draw is one of scope rather than of technique: the learned component here never selects a target, and the routing that does remains a closed-form score.
3
System Model and Problem Statement
A set V of static sensors is deployed over a square field of side L with one base station. Each sensor i has capacity E max and residual energy ei (t) normalised to [0, 1], and dies at the critical threshold. Sensors report over a multi-hop tree, so a node’s consumption rate ρi covers its own sensing and transmission plus the relay load of its subtree, and nodes closer to the base station drain fastest. Sensors are grouped into candidate clusters C by minimum circle cover, each cluster c owning a docking point dc at which a charger parks to serve the alive sensors within charging radius Ra (Fig. 1); deciding at cluster rather than sensor granularity keeps the decision space manageable as the field grows [4]. max and moveA fleet of M mobile chargers, each with speed vm , capacity Em move ment power Pm , serves the field. Charging power decays with distance from the docking point. An assignment of charger m to cluster c is feasible only if Em (t) ≥
move Pm dm,c + ∥dc − pBS ∥ + Ecserve , vm
(1)
where Ecserve is the energy the cluster’s alive sensors need: the charger must be able to reach dc , serve it, and still return to the base station. Otherwise it returns to recharge. The constraint is enforced on every assignment and is what makes a locally attractive target sometimes inadmissible. Problem. Let A(T ) be the number of sensors still alive at the end of a horizon T . The scheduler chooses, at each decision epoch and for each idle charger, one feasible cluster to serve (or the base station), so as to maximise A(T )/|V | subject to (1) holding for every assignment and to each cluster being served by at most one charger at a time. Travel distance and base-station recharge time are not constrained but are reported, because a scheduler that maximises survival by spending without limit is not a useful one; Section 5.5 states what HQARRF spends.
4
L.-C. Tao and P.-C. Wang 1000
Sensors Docking points Base station Charging clusters (r = 30 m) Selected clusters Chargers
Y coordinate (m)
800
600
400
200
0 0
200
400
600
800
1000
X coordinate (m)
Fig. 1. A deployment at t = 0: sensors, the candidate clusters produced by minimum circle cover with their docking points, the base station, and the five chargers with the clusters they have currently selected. Scheduling decisions are made over docking points rather than over individual sensors.
4
The HQARRF Scheduler
HQARRF is a two-level framework. The lower level answers which docking point is worth serving now ; the upper level answers whether the fleet’s focus should be redirected to an under-served region. Whichever level produces the target, the assignment must still satisfy (1). Algorithm 1 gives one decision epoch in full.
4.1
ARR-F Local Routing Score
For candidate cluster c, the local score combines three terms, ec − β Φ ec + λf Fc , Sc = A
c⋆ = arg max′ Sc , c∈Cm
(2)
ec and Φ ec are the attraction and repulsion terms below, each min–max where A normalised over the candidate set of the current epoch so that the two are commensurable, and β, λf weight repulsion and the force bonus. Reservation does not appear in (2): a cluster already claimed by another charger is removed ′ from Cm outright. Repulsion and reservation are therefore distinct mechanisms, and the ablation of Section 5.3 isolates repulsion by setting β = 0 while leaving reservation in place.
HQARRF: Hierarchical Q-learning and Force-aware Routing for WRSNs
5
Algorithm 1 HQARRF, one decision epoch (interval ∆) Require: clusters C with docking points dc ; zones Z; chargers M; Q-table Q 1: recompute per-sensor deadline risk ui and zone risk Riskzsmooth 2: z ⋆ ← arg maxz Riskzsmooth ; form state st = (z ⋆ , gt , bt ) 3: at ← ε-greedy(Q, st ) ▷ at ∈ {0, 1}: redirect or not 4: if at = 1 and z ⋆ passes the gate of (11) then 5: C ′ ← {c ∈ C : dc ∈ z ⋆ } ▷ intervention active 6: else 7: C′ ← C 8: end if 9: for all idle chargers m ∈ M do ′ 10: Cm ← {c ∈ C ′ : c unreserved and (m, c) satisfies (1)} ′ 11: if Cm = ∅ then 12: send m to the base station; continue 13: end if ′ 14: c⋆ ← arg maxc∈Cm Sc ▷ ARR-F score, (2) 15: reserve c⋆ for m and dispatch 16: end for 17: observe reward rt ; Q(st , at ) ← Q(st , at ) + α rt + γ maxa Q(st+1 , a) − Q(st , at )
Attraction. Cluster urgency is a weighted sum of the cluster’s normalised energy deficit, mean consumption rate, criticality and alive count, together with an asymmetric term in the blended residual-energy ratio that is positive below a cutoff of 0.8 and sharply negative above it. That sum is min–max normalised bc ∈ [0, 1] and compressed into a scalar charge Qc = over the candidates to U b 1 − 2Uc ∈ [−1, 1]. Attraction then follows an inverse-square law between the charger at pm (t) and the docking point, ka qm Qc Ac = − 2 , dm,c + ϵ2a
dm,c = ∥pm (t) − dc ∥,
(3)
with charge gain ka , charger charge qm and a softening radius ϵa = max( 12 Ra , 1) that keeps the score finite at contact. The sign convention carries the intent: an bc → 1, hence Qc → −1, which the leading minus sign turns urgent cluster has U into strong attraction, while a well-charged cluster has Qc → +1 and becomes mildly repulsive, suppressing repeated visits without any explicit exclusion rule. Repulsion. Φc is a Coulomb-like penalty accumulated from the current positions of the other chargers, X 1 Φc = , ϵr = max( 12 Ra , 1). (4) 2 + ϵ2 ′ ∥p (t) − d ∥ m c r ′ m ̸=m
Independent evaluation of the same urgency signal makes crowding the default failure mode even when reservation prevents outright duplication, since chargers converge on adjacent docking points around one hot region. Repulsion makes a cluster progressively less attractive as other chargers approach it, so the fleet spreads without centralised assignment.
6
L.-C. Tao and P.-C. Wang
Force bonus. Let TKf (c) be the set of at most Kf alive sensors whose residual energy ratio does not exceed a force threshold θf and that lie within radius Rf of dc . Then X 1 − e (t) i , Fc = log1 + (5) ∥pi − dc ∥2 + ϵf i∈TKf (c)
where ϵf is a numerical stabiliser. A docking point thus inherits value from critical sensors that sit near it without belonging to it — which matters at cluster boundaries, where the nearest endangered sensor is often just outside the cluster that would serve it — while the logarithm stops a dense pocket of critical nodes dominating Sc . 4.2
Adaptive Zones and Zone-Level Risk
Zones aggregate regional state, and are coarser than clusters: in our runs the field holds 124–633 candidate clusters but only 7–11 zones. Their P number P is derived once per deployment rather than tuned. Write nef f = ( i ρi )2 / i ρ2i for the effective network size under per-node consumption rates ρi , and λd = nef f πRa2 /L2 for the coverage density. A node-driven requirement grows with both, znode = (1 + ln nef f ) max(1, λd ) , (6) 2 )⌉ follows from the distance a space-driven requirement zspace = ⌈L2 /(πrresp rresp a charger can cover before sensors start dying, a learnability cap zlearn keeps the Q-table small enough to fill within the horizon, and |Z| = max 1, min |C|, zlearn , max(2, znode , zspace ) , (7)
with boundaries then obtained by k-means over sensor positions. In every configuration we ran the binding term was znode , giving |Z| = 7 to 11; zspace and zlearn acted only as guards. Risk is built from deadlines rather than from energy alone. For sensor i, let T T Di be its time to death at the current consumption rate and ET Ai = minm ∥pi − pm (t)∥/vm the earliest arrival of any charger; the slack slacki = T T Di − ET Ai is mapped to a bounded per-sensor risk ( exp(−slacki /τ ), slacki ≥ 0, ui = (8) 1 + δ, slacki < 0, bounded above by 1 + δ, so that a sensor no charger can reach in time saturates the scale instead of competing on the same continuum with sensors that are merely far away; τ sets how fast comfortable slack decays and δ is the misseddeadline bonus. When the adaptive threshold of Section 4.3 is active it multiplies ui by up to 1.5 for sensors in locally stressed neighbourhoods. Zone risk aggregates only the most endangered members, X 1 ui , (9) Riskzraw = alive min (Kz , |Vz |) i∈TKz (z)
HQARRF: Hierarchical Q-learning and Force-aware Routing for WRSNs
7
where TKz (z) holds the Kz alive sensors in z with the largest ui . Averaging over the top-Kz rather than the whole zone stops a small pocket of genuinely endangered sensors being diluted by healthy neighbours, and normalising by min(Kz , |Vzalive |) stops a nearly depleted zone being assigned an artificially low risk merely because too few nodes survive to fill the set. The value used downstream is smoothed over the k nearest zones, Riskzsmooth = (1 − λs )Riskzraw + λs Risk N (z) , so that zone boundaries do not act as hard discontinuities. 4.3
Gated Q-learning Intervention
The upper level observes a compact state and encodes it as a single index, st = (zt⋆ , gt , bt ),
zt⋆ = arg max Riskzsmooth , z
id(st ) = 9zt⋆ + 3gt + bt , (10)
where gt , bt ∈ {0, 1, 2} are the service capacity of zt⋆ (how many chargers could reach it feasibly) and its coverage ratio (the share of its critical sensors already inside some charger’s service radius), each discretised into three levels. The action is binary: intervene or not. Tabular Q-learning [20] is sufficient at this granularity: with |Z| ≤ 11 the table has at most 9 · 11 · 2 = 198 entries, of which between 22 and 186 were ever updated across our runs, so the learned policy can be printed and inspected. Intervention is additionally gated. A zone is eligible only if Riskzsmooth ≥ θrisk , ⋆
gt > 0,
coverage(z ⋆ ) < θcov ,
(11)
so the controller cannot fire on a low-risk zone, a zone no charger can reach, or one that is already sufficiently covered; across the 27 parameter points 92% of the epochs in which the policy chose to intervene passed this gate. When intervention is active, ARR-F is restricted to candidate clusters inside zt⋆ ; otherwise routing proceeds unrestricted. Learning is driven by a reward combining the change in alive nodes, requests served, critical-node count and delivered energy, against penalties on newly dead nodes, extra movement and the act of intervening. Two smaller upper-level components are enabled alongside the controller, and Section 5.3 therefore measures all three as one step: a target selector that ranks by time-to-death rather than urgency alone, and a soft adaptive threshold that raises a sensor’s effective urgency in locally stressed neighbourhoods.
5
Evaluation
5.1
Setup
Five scenario groups vary sensor count, field size, sensor battery capacity, charger speed and consumption rate, giving 27 parameter points. Each group passes through the same default configuration, so the 27 points cover 24 distinct ones;
8
L.-C. Tao and P.-C. Wang
Table 1. Simulation parameters. Swept values are given as ranges with the default, used when another parameter is swept, in bold. max Field side L 1000–3000 m (1000) Charger capacity Em 104 J † Sensors |V | 300–500 (250) Charger speed vm 1–10 m/s (5) move Base station field centre Movement power Pm 0.1 W Sensor battery E max 100–400 J (150) Charging radius Ra 30 m Consumption ρi 0.5–1.5 mJ/s (1.0) Charging power 10 W Sensing range 40 m Charging efficiency 0.9 Comm. range 80 m Chargers M 5 (2–7 in §5.4)
Repulsion weight β Force weight λf Force radius Rf Force cap Kf Force threshold θf Decision interval ∆ Q-learning α, γ Horizon T
0.2 1.0 150 m 10 0.3 30 s 0.1, 0.9 105 s
Risk top-Kz Risk decay τ Missed-deadline δ Risk gate θrisk Coverage gate θcov Smoothing λs (k=3) ε (start/min/decay) Runs per point
5 8000 s 0.5 0.45 0.8 0.3 0.3 / 0.05 / 0.995 5
†
The field-size sweep scales |V | with area, from 250 at 1000 m to 750 at 3000 m, so node density stays constant; the sensor-count sweep varies |V | from 300 to 500 at L = 1000 m.
collapsing the repeats moves the aggregate margins below by less than 0.3 percentage points. Every point is run on five independent network instances from one fixed base seed, and every method sees the same five instances, so every comparison in this campaign is paired. Table 1 lists the parameters. The primary metric is sensor survival rate at the end of the horizon; travel distance, delivered energy, base-station recharge time and per-charger load balance are reported alongside it. We compare against five baselines spanning the main design families: a genetic algorithm with 2-OPT tour improvement under time windows [21], multicharger cooperation (MCCA) [5], uneven cluster-based charging (UCMC) [4], a k-way earliest-deadline-first heuristic and nearest-job-next. The first three follow their published designs. The last two have no single canonical multi-charger formulation, so we implement them as family representatives rather than as reproductions of a specific paper: K-EDF sorts pending requests by residual lifetime in the classical earliest-deadline-first order [13] and assigns the top k to available chargers by minimum total travel; NJNP sends each charger to its nearest pending request, following the greedy nearest-job discipline analysed for on-demand charging by He et al. [6] but without their preemption rule. All five share the same simulation core — energy model, distance-decay charging model, charging radius and charger speed — and differ only in target selection and scheduling policy. Baselines whose original formulations do not model the charger’s own energy budget were extended with the same return-to-base check (1) that HQARRF must satisfy.
HQARRF: Hierarchical Q-learning and Force-aware Routing for WRSNs
9
Survival rate (%)
100 80 60 40 20
HQARRF Genetic+2-OPT
UCMC K-EDF
MCCA NJNP
0 0.50
0.75
1.00
1.25
1.50
Sensor consumption rate (mJ/s)
Fig. 2. Survival rate versus sensor consumption rate. Whiskers are ±1 standard error over the five runs at each point. At 0.5 mJ/s the leading methods lie within each other’s error bars; as workload rises HQARRF degrades gracefully while the baselines fall away.
5.2
Survival Rate
Across the 27 parameter points, HQARRF attains the highest mean survival rate at 26. Relative to the mean of the five baselines the improvement is 20.7 percentage points; relative to the strongest baseline at each individual point it is 9.2 points. The margin is not uniform across the sweeps, and in the lightestload regimes the leading methods sit within each other’s run-to-run variation — including the single point HQARRF does not lead, where the gap is 0.16 points. The figures carry error bars throughout. The shape of the margin is informative: 26.1 points in the sensor-count group, 3.7 in the battery-capacity group. Fig. 2 makes the pattern concrete along the workload axis. There is little for zone-level intervention to contribute while charging pressure is low, but as consumption rises the methods separate monotonically: HQARRF degrades gently from 86.5% to 78.2%, the strongest baseline falls to 66.0% and the weakest collapses to 5.5%, opening the margin from −0.2 to +12.2 points without reversal. Fig. 3 shows the same ordering along network size, and adds the evidence that makes the reading causal rather than suggestive. HQARRF loses only 1.5 points between 300 and 500 sensors while the three strongest baselines lose 14.7 to 16.8 — which by itself would not say which part of HQARRF is responsible. The dotted line is ARR-F, which is HQARRF with its upper level removed and the routing score unchanged: it decays by 15.8 points, indistinguishably from the baselines, placing the flatness with the regional view rather than the local score. NJNP is also nearly flat, at −1.3 points, but flat at roughly 30% throughout, having already collapsed at the smallest network.
10
L.-C. Tao and P.-C. Wang
Survival rate (%)
100 80 60 40 HQARRF Genetic+2-OPT UCMC
20
K-EDF MCCA
NJNP ARR-F (routing only)
0 300
350
400
450
500
Number of sensors
Fig. 3. Survival rate versus network size. HQARRF holds near 80% while every other method decays. ARR-F is HQARRF with its upper level removed and the routing score unchanged; it decays by 15.8 points, placing the flatness in the upper level rather than the routing score.
5.3
Component Ablation
Four variants isolate the contributions. AR uses attraction with hard reservation only (β = 0); ARR adds the repulsion term; ARR-F adds the force bonus; HQARRF adds the gated zone-level controller together with the two upperlevel components named at the end of Section 4.3. Averaged over the 27 points, survival rises 48.8 → 63.9 → 69.8 → 81.6%, and each step is an improvement at every one of the 27 points individually (mean increments +15.1, +5.9 and +11.8 points). Travel does not behave as a spreading argument would predict. Each added component reduces average travel rather than raising it, from 2240 km for AR to 1980, 1950 and 1760 km; the reductions from repulsion and from the controller hold at all 27 points, while the force bonus is travel-neutral (lower at 10 of 27). Better target choice removes wasted trips: a charger not sent to a cluster another charger is about to serve, or to a region that will be covered anyway, does not pay the round trip. Survival and travel improve together here; the trade-off HQARRF does pay appears in Section 5.5. The last step carries the paper’s claim, and the ablation supports it more directly than an increment can. The controller is dormant by design — the gate of (11) keeps it idle unless a zone is both at risk and under-served — so what it recovers over ARR-F should scale with how often it actually engages. It does: across the 27 points the survival it adds rises with its intervention count (Fig. 4(b), r = 0.84). It is monotone in the battery-capacity and consumptionrate sweeps and near-monotone across charger speed — the three that vary charging pressure at a fixed topology. The two sweeps that change the field itself instead sit at the top of both axes: in the largest networks the controller engages almost constantly and recovers up to 36.6 points, the most it contributes anywhere. Because ARR-F differs from HQARRF only in that upper level, the
(a) component ablation
Survival rate (%)
90
HQARRF
80 ARR-F
70 60
ARR
50
AR
40 1800
2000
2200
Travel distance (km)
Survival gain over ARR-F (pp)
HQARRF: Hierarchical Q-learning and Force-aware Routing for WRSNs
40
11
(b) how often it fires vs what it buys battery capacity consumption rate sensor count field size charger speed
30 20 10 0 0
2000
4000
Zone-controller interventions per run
Fig. 4. (a) The four variants on the survival–travel plane, averaged over the 27 parameter points; whiskers are ±1 standard error across those points, so they show how much the scenarios differ, not run-to-run noise. The survival ordering holds at all 27 points individually. (b) For each point, how often the zone controller engaged against the survival it recovered relative to ARR-F. The relationship is monotone within the battery and consumption sweeps and near-monotone across charger speed; the two sweeps that change the field itself sit at the high end of both axes.
gain tracks the mechanism rather than the routing score, and it explains the shape of the margin reported above: the advantage is largest exactly where sensor death would otherwise be irreversible. 5.4
Fleet Size
A separate sweep varies the fleet from two to seven chargers at 300 sensors (Fig. 5). Survival rises steeply from two to four and is essentially flat thereafter for every method, so the fleet saturates — but the methods do not converge as they saturate. HQARRF leads at every fleet size tested, by 13.7 points at two chargers and by 16.3 to 18.7 from three upward, with ARR-F again tracking the baselines rather than HQARRF. What this sweep establishes is the separation between methods; the small variations along each curve are not meant to be read. 5.5
What the Gain Costs
The survival advantage is not free, but the cost is not where a route-length argument would put it. Averaged over the 27 points HQARRF travels 1760 km, the lowest of the five non-route-optimising methods (UCMC 1820, MCCA 1830, K-EDF 1900, NJNP 2440 km). Only the genetic baseline with 2-OPT travels less, and by a wide margin: at 367 km it covers 4.8 times less ground, which is what a scheduler optimising tours for their own sake should do, and it pays 13.8 points of survival for it.
12
L.-C. Tao and P.-C. Wang
Survival rate (%)
100 80 60 40 HQARRF Genetic+2-OPT UCMC
20
K-EDF MCCA
NJNP ARR-F (routing only)
0 2
3
4
5
6
7
Number of mobile chargers
Fig. 5. Survival rate versus fleet size, at 300 sensors. Every method saturates by four chargers without the methods converging.
The real cost is service throughput. HQARRF delivers the most energy of any method (27.6 kJ against 21.6 kJ for the genetic baseline) and so must refuel most often: base-station recharge time averages 76.0% of the horizon against 64.3% for UCMC and 61.2% for the genetic baseline, the highest of the six at 20 of the 27 points. A charger that is refuelling is not serving, so this is the first thing a deployment with slow base-station charging should check. Load balance is unremarkable: the coefficient of variation of travel across the fleet averages 0.12, matching UCMC and far below the genetic baseline’s 0.73, though above the near-uniform 0.01–0.04 of the round-robin heuristics. The case for HQARRF is therefore not that it dominates every axis, but that the throughput it sustains buys a survival improvement the cheaper methods do not achieve, and that the exchange is favourable precisely in the high-pressure regimes where sensor death would otherwise be irreversible. Where charging pressure is low the controller barely fires, the margin falls inside the noise, and a simpler scheduler is the better choice.
5.6
Reproducibility
Every number above comes from one simulation campaign, exported to flat CSV at three granularities — per run, per point and per charger — together with the per-point intervention counts extracted from the run summaries. One script regenerates all four result figures from those files, so each plotted value is traceable to a stored run rather than read back off an image. The simulator writes a snapshot of its full configuration with each campaign, and Table 1 is taken from the snapshot of the campaign reported here rather than from the current defaults.
HQARRF: Hierarchical Q-learning and Force-aware Routing for WRSNs
6
13
Conclusion
We presented HQARRF, a two-level scheduler for multi-charger WRSNs that keeps routing interpretable and confines learning to a gated, zone-level intervention decision. Across 27 parameter points it leads at 26, improving survival by 20.7 percentage points over the baseline mean and 9.2 points over the strongest baseline at each point. The ablation does more than rank the components: the variant without the upper level decays with network size exactly as the baselines do, and the survival it recovers scales with how often the controller fires, which locates the gain in the regional view rather than the local score. The framework should be read as risk-aware resource allocation rather than route-distance minimisation — it travels less than the comparable heuristics, but it spends three quarters of the horizon refuelling to sustain the energy it delivers, and that is the cost to weigh. Three directions follow. The first is validation outside simulation, through hardware-in-the-loop testing or a small physical deployment, which would show how the charging model and the return-to-base design behave under real localisation and terrain error. The second is automatic parameter adaptation: the risk threshold, coverage threshold and zone count are fixed per scenario here, and could instead be mapped from observable scenario features. The third follows from the cost analysis — since base-station refuelling rather than travel is what limits the method, charger-to-charger transfer or additional depots are the natural next lever. Acknowledgements. This work was supported by the National Science and Technology Council, Taiwan, under Grant No. NSTC 114-2221-E-005-043-MY3. Parts of this work appeared in the first author’s master’s thesis at National Chung Hsing University.
References 1. Aziz, S.A., Wang, X., Hawbani, A., Qureshi, B., Alsamhi, S.H., Alabsi, A., Zhao, L., Al-Dubai, A., Ismail, A.S.: Wireless rechargeable sensor networks: Energy provisioning technologies, charging scheduling schemes, and challenges. IEEE Transactions on Sustainable Computing 10(5), 873–890 (2025). DOI 10.1109/TSUSC. 2025.3549414 2. Beigel, R., Wu, J., Zheng, H.: On optimal scheduling of multiple mobile chargers in wireless sensor networks. In: Proceedings of the First International Workshop on Mobile Sensing, Computing and Communication, pp. 1–6 (2014). DOI 10.1145/ 2633675.2633676 3. Chen, L., Lin, S., Huang, H.: Charge me if you can: Charging path optimization and scheduling in mobile networks. In: Proceedings of the 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing, pp. 101–110 (2016). DOI 10.1145/2942358.2942364 4. Han, G., Guan, H., Wu, J., Chan, S., Shu, L., Zhang, W.: An uneven clusterbased mobile charging algorithm for wireless rechargeable sensor networks. IEEE Systems Journal 13(4), 3747–3758 (2019). DOI 10.1109/JSYST.2018.2879084
14
L.-C. Tao and P.-C. Wang
5. Han, G., Wang, H., Guan, H., Guizani, M.: A mobile charging algorithm based on multicharger cooperation in internet of things. IEEE Internet of Things Journal 8(2), 684–694 (2021). DOI 10.1109/JIOT.2020.3006851 6. He, L., Kong, L., Gu, Y., Pan, J., Zhu, T.: Evaluating the on-demand mobile charging in wireless sensor networks. IEEE Transactions on Mobile Computing 14(9), 1861–1875 (2015). DOI 10.1109/TMC.2014.2368557 7. He, S., Chen, J., Jiang, F., Yau, D.K.Y., Xing, G., Sun, Y.: Energy provisioning in wireless rechargeable sensor networks. IEEE Transactions on Mobile Computing 12(10), 1931–1942 (2013). DOI 10.1109/TMC.2012.161 8. Jia, Y., Wang, J., Ji, Z., Peng, R.: Multiple mobile charger charging strategy based on dual partitioning model for wireless rechargeable sensor networks. IEEE Access 10, 93,731–93,744 (2022). DOI 10.1109/ACCESS.2022.3203410 9. Jiang, C., Chen, S., Li, J., Wang, H., Wang, J., Xu, T., Xiao, W.: Mobile charging scheduling approach for wireless rechargeable sensor networks based on multiple discrete-action space deep q-network. Applied Sciences 13(14), 8513 (2023). DOI 10.3390/app13148513 10. Jiang, C., Wang, Z., Chen, S., Li, J., Wang, H., Xiang, J., Xiao, W.: Attentionshared multi-agent actor–critic-based deep reinforcement learning approach for mobile charging dynamic scheduling in wireless rechargeable sensor networks. Entropy 24(7), 965 (2022). DOI 10.3390/e24070965 11. Jiang, L., Dai, H., Wu, X., Chen, G.: On-demand mobile charger scheduling for effective coverage in wireless rechargeable sensor networks. In: Mobile and Ubiquitous Systems: Computing, Networking, and Services, pp. 732–736. Springer International Publishing (2014). DOI 10.1007/978-3-319-11569-6 62 12. Li, J., Wang, H., Jiang, C., Xiao, W.: A deep reinforcement learning approach for online mobile charging scheduling with optimal quality of sensing coverage in wireless rechargeable sensor networks. Ad Hoc Networks 156, 103,431 (2024). DOI 10.1016/j.adhoc.2024.103431 13. Liu, C.L., Layland, J.W.: Scheduling algorithms for multiprogramming in a hardreal-time environment. Journal of the ACM 20(1), 46–61 (1973). DOI 10.1145/ 321738.321743 14. Lu, X., Wang, P., Niyato, D., Kim, D.I., Han, Z.: Wireless charging technologies: Fundamentals, standards, and network applications. IEEE Communications Surveys & Tutorials 18(2), 1413–1452 (2016). DOI 10.1109/COMST.2015.2499783 15. Ma, Y., Liang, W., Xu, W.: Charging utility maximization in wireless rechargeable sensor networks by charging multiple sensors simultaneously. IEEE/ACM Transactions on Networking 26(4), 1591–1604 (2018). DOI 10.1109/TNET.2018.2841420 16. Madhja, A., Nikoletseas, S., Raptis, T.P.: Hierarchical, collaborative wireless energy transfer in sensor networks with multiple mobile chargers. Computer Networks 97, 98–112 (2016). DOI 10.1016/j.comnet.2016.01.007 17. Mo, L., Kritikakou, A., He, S.: Energy-aware multiple mobile chargers coordination for wireless rechargeable sensor networks. IEEE Internet of Things Journal 6(5), 8202–8214 (2019). DOI 10.1109/JIOT.2019.2918837 18. Wang, C., Li, J., Ye, F., Yang, Y.: Recharging schedules for wireless sensor networks with vehicle movement costs and capacity constraints. In: 2014 Eleventh Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp. 468–476 (2014). DOI 10.1109/SAHCN.2014.6990385 19. Wang, C., Li, J., Ye, F., Yang, Y.: A mobile data gathering framework for wireless rechargeable sensor networks with vehicle movement costs and capacity constraints. IEEE Transactions on Computers 65(8), 2411–2427 (2016). DOI 10.1109/TC.2015. 2490060
HQARRF: Hierarchical Q-learning and Force-aware Routing for WRSNs
15
20. Watkins, C.J.C.H., Dayan, P.: Q-learning. Machine Learning 8(3–4), 279–292 (1992). DOI 10.1007/BF00992698 21. Wei, Z., Li, M., Zhao, Q., Lyu, Z., Zhu, S., Wei, Z.: Multi-mc charging schedule algorithm with time windows in wireless rechargeable sensor networks. IEEE Access 7, 156,217–156,227 (2019). DOI 10.1109/ACCESS.2019.2949284 22. Xie, L., Shi, Y., Hou, Y.T., Sherali, H.D.: Making sensor networks immortal: An energy-renewal approach with wireless power transfer. IEEE/ACM Transactions on Networking 20(6), 1748–1761 (2012). DOI 10.1109/TNET.2012.2185831 23. Zhang, F., Zhang, J., Qian, Y.: A survey on wireless power transfer based charging scheduling schemes in wireless rechargeable sensor networks. In: 2018 IEEE 4th International Conference on Control Science and Systems Engineering (ICCSSE), pp. 194–198 (2018). DOI 10.1109/CCSSE.2018.8724809