A Deep Reinforcement Learning Framework for Real-Time Electric Vehicle Charging Management in IoEV-Enabled Smart Cities

A Deep Reinforcement Learning Framework for Real-Time Electric Vehicle Charging Management in IoEV-Enabled Smart Cities

arXiv:2611.01751v1 [cs.LG] 1 Nov 2026

Nour-Eddine Moumni1 [0009-0008-5803-0604] , Rachid Alaoui1,3 [0000-0001-9948-329X] , and Driss Kiouach2 [0000-0002-1497-3028] 1

Laboratory of Research in Computer Science and Telecommunications (LRIT), Mohammed V University, Rabat, Morocco noureddine_moumni@um5.ac.ma 2 Laboratory of Probability, Statistics and Intelligent Systems (LPAIS), Sidi Mohamed Ben Abdellah University, Fez, Morocco 3 Higher School of Technology of Salé (EST Salé), Mohammed V University, Rabat, Morocco

Abstract. The rapid growth of electric vehicles (EVs) in urban environments demands intelligent, decentralised charging management strategies that operate under strict real-time constraints. This paper proposes an Edge-Intelligent architecture in which each EV would select a charging station (CS) by querying a shared, centrally trained Deep Q-Network (DQN) policy over a lightweight MQTT publish–subscribe backbone; independent per-EV edge inference is the deployment target this architecture is designed for, not the configuration evaluated here. The reported simulation evaluates that shared DQN policy applied sequentially to each vehicle’s observations within a single SUMO run, using MQTT to carry KPI telemetry while observations are obtained through TraCI. The system is evaluated on two real-world road networks extracted from OpenStreetMap (Rabat Agdal–Hay Riad and Tangier, Morocco); the primary network carries four geolocated public charging stations offering heterogeneous power levels (50 kW and 150 kW). Three traffic demand profiles (morning, midday, and evening peak) are simulated using SUMO, and four decision policies are compared: an Adaptive Heuristic baseline, a Tabular Q-Learning agent, the proposed DQN agent, and a Double DQN variant. A multi-seed statistical protocol (7 seeds, Wilcoxon signed-rank test, Cohen’s d) evaluates the results. The DQN agent achieves a 6.8% relative improvement in charge rate and a 5.4% increase in delivered energy over the Adaptive Heuristic under Evening Peak demand (p = 0.016, Cohen’s d = 0.99 and 1.51), while showing no advantage in waiting time; This is the authors’ accepted manuscript (paper ID 733916), accepted as a regular paper and presented at the 10th International Conference on Advanced Intelligent Systems for Sustainable Development (AI2SD’2025 cycle), carried over for publication in the AI2SD 2026 proceedings, scheduled to appear in an indexed Springer Lecture Notes in Networks and Systems volume. This version is posted as a preprint ahead of that publication.

2

Moumni et al. on the second network the heuristic in turn leads on waiting time, so the ranking is not stable across geographies. A dedicated scalability analysis shows linear growth of the state space with the number of stations and sub-millisecond inference, and all runs are obtained under intermittent station outages driven by a Markov failure process. Keywords: Electric Vehicle Charging · Deep Q-Network · Reinforcement Learning · MQTT · Edge Intelligence · SUMO · Internet of Electric Vehicles

1

Introduction

Morocco has placed transport decarbonisation among its policy priorities [9], with battery-electric-vehicle adoption barriers analysed in their own right [22]; more broadly, rapid worldwide EV-fleet growth strains urban charging infrastructure and grid stability [3,21,25]. Charging infrastructure nonetheless remains thin: a survey of the public charging points listed for the greater Rabat area, carried out in June 2026 against Google Maps and Chargemap (see Section 6), returned fewer than twenty, creating acute risk of station congestion during peak commuting hours. Two solution families have emerged: centralised approaches, which compute globally optimal assignments but can incur communication latency incompatible with sub-second IoT requirements as system size grows [30] — related to the coordinated charging-scheduling optimisation addressed by Usman et al. [29] — and rule-based heuristics, which react quickly but encode a fixed, non-adaptive priority (distance or occupancy) whose adequacy depends on topology and demand [19]. We address this gap through an Edge-Intelligent architecture: proposed for deployment, each EV would query a DQN policy, from station observations exposed by the environment, over an MQTT publish–subscribe backbone [1, 2], requiring no centralised controller at inference time. The evaluation reported in this paper runs that policy shared and sequential within a single SUMO process, not as independent per-EV instances (Section 3). The policy maps a 13-dimensional continuous state vector—encoding battery state of charge (SoC), distances to stations, occupancy levels, and queue lengths—directly to a discrete action (choose CSk , k ∈ {1, 2, 3, 4}). The main contributions of this work are: 1. A DQN-based charging assignment framework, designed for decentralised edge deployment and evaluated here as a shared policy in simulation, with a formal MDP formulation incorporating a queue-aware state representation. 2. The first RL evaluation on real OpenStreetMap road networks in Morocco — Rabat and Tangier — using four verified real-world EV charging stations. 3. An empirical comparison of four policies — Adaptive Heuristic, Tabular QLearning, DQN and DDQN — across three demand scenarios on two road networks, with statistically validated results (Wilcoxon signed-rank, n = 7 seeds). The three structurally distinct policies carry the comparison; DDQN

Title Suppressed Due to Excessive Length

3

is reported as a variant of DQN, and the two are contrasted as a family in Section 8.2. 4. A scalability and robustness analysis addressing large-scale deployment. The remainder of the paper is structured as follows. Section 2 reviews related work. Section 3 describes the system architecture. Section 4 formalises the MDP. Section 5 details the three algorithms. Section 6 describes the experimental setup. Section 7 presents and discusses results. Section 8 analyses scalability and robustness. Section 9 concludes.

2

Related Work

2.1

RL-Based EV Charging Management

RL and Deep RL (DRL) have been extensively investigated for EV charging management across system scales, but centralised or grid-side works assume centralised state observation and do not incorporate real-time IoT communication constraints. Wan et al. [30] pioneered DQN for EV routing on synthetic networks; Ma et al. [15], Han et al. [8], and, in the related but distinct domain of trafficsignal control, Pang et al. [23], extend deep Q-learning to large-scale scheduling under communication delays. From a grid-side perspective, Da Silva et al. [6], Cui et al. [5], and Li et al. [12] formulate multi-agent or multi-objective DRL for station scheduling and renewable integration without modelling vehicle mobility, while Lee and Choi [11] and Michailidis et al. [16] take the operator’s perspective with federated DRL for dynamic pricing and solar/storage management. Vehicle-side travel decisions have been addressed directly in smart-city settings: Lee et al. [10] jointly select a route and station; Farhani et al. [7] formulate assignment from SUMO traffic data; Zhang et al. [33] recommend stations via city-scale MARL; Chifu et al. [4] explore Q-learning for demand response; Suanpang et al. [27] propose MARL for recommendations; and Liu et al. [13] schedule mobile IoEV charging. These works establish the vehicle-side formulation adopted here. What distinguishes the present study is the combination evaluated: a real OpenStreetMap Moroccan-city network, an MQTT deployment path, four policies under an identical protocol, and a multi-seed statistical analysis. 2.2

IoT-Enabled EV Coordination

The simulation platform used here originates in an earlier study of EV routing to charging points on a Moroccan case [20, 21]. The integration of IoT architectures and MQTT into EV management was explored in foundational surveys by Al-Fuqaha et al. [1] and Belli et al. [2]. Closer to the EV domain, Savio Abraham et al. [26] review EV charging architectures and control strategies, Urooj et al. [28] address IoT-based EV and battery management, and Rimal et al. [24] review smart EV charging frameworks; none of the three reports a deployed MQTT-based charging-assignment control loop, which this paper

4

Moumni et al.

Table 1. Comparison with related work on EV charging assignment. R=Real OSM, S=Synthetic, ✓=present, ×=absent. Work Wan et al. (2019) [30] Lee et al. (2020) [10] Zhang et al. (2021) [33] Farhani et al. (2023) [7] Chifu et al. (2024) [4] Liu et al. (2024) [13] Moumni (2026a) [19] Moumni (2026b) [18] This work

Network DQN MQTT Multi-seed S S S S S R S S R

✓ ✓ ✓ × ✓ ✓ × × ✓

× × × × × × ✓ ✓ ✓

× × × × × × × ✓ ✓

also does not evaluate: MQTT carries KPI telemetry only in the reported runs (Section 3), and is proposed, not evaluated, as the control-loop transport for a future edge deployment. In our prior work [19], MQTT messaging demonstrated a 97% improvement in charging success over static assignment using an Adaptive Heuristic on a synthetic SUMO network. The present work upgrades that baseline with DQN, a real OSM network, and formal statistical validation. Cooperative multi-agent formulations of the same problem are examined in a companion study by the present authors [18], a preprint reporting a controlled comparison of centralised-training decentralised-execution against independent Q-learning on the station side. The present paper targets the decentralised-inference paradigm as its deployment design but applies it, in the evaluation reported here, to a shared vehicle-side agent (Section 3). 2.3

Gap Analysis

Table 1 summarises the positioning of this work against representative studies in the literature. To the best of our knowledge, no prior study combines (i) formal DQN training, (ii) a real OSM network, (iii) MQTT IoT backbone, and (iv) multi-seed statistical validation in the Moroccan urban context.

3

System Architecture

The proposed framework comprises four interacting components (Fig. 1): SUMO Traffic Simulator. Microscopic traffic simulation is performed with SUMO 1.19 [14] on a real OSM road network (Section 6). The simulation timestep is ∆t = 1 s; vehicle states are exposed via the TraCI API. MQTT Broker (Mosquitto). An Eclipse Mosquitto broker provides the publish–subscribe backbone of the architecture, with three topic families: sumo/ vehicles (SoC, position), sumo/stations (occupancy, queue, power), and sumo/ kpi (per-step reward signals). We state explicitly which of these are exercised by the experiments reported here: the broker carries sumo/kpi telemetry only, while

Title Suppressed Due to Excessive Length

5

AI2SD Architecture — Edge-Intelligent IoEV Charging Framework SUMO microscopic simulation · TraCI observation path · MQTT KPI telemetry · single-agent MDP (13D state) · edge policies 1. TRAFFIC ENVIRONMENT

2. IOEV MIDDLEWARE

SUMO Simulator

TraCI API

3. MDP & EDGE INTELLIGENCE

Edge Policy

Microscopic Traffic Engine

1 Hz control loop

NumPy inference, no DL framework

OSM: Rabat & Tangier Δt = 1 s, 3600 s per run

Observation path of the reported runs

Adaptive Heuristic · Tabular Q-Learning · DQN · Double DQN

TraCI call

4. CHARGING STATIONS

Charging Stations

action at ∈ {CS1. . CS4} CS1–CS4, capacity 4 2 × 150 kW, 2 × 50 kW geolocated, real sites

state st station observations

EV Fleet Model

MQTT Broker

Single-agent MDP encoder

Demand Profiles

Pub/Sub backbone

State st ∈ ℝ 13

Markov failure process

Availability

Morning / Midday / Evening Initial SoC 10–80 % Queueing at CS nodes

sumo/vehicles · sumo/stations sumo/kpi reported runs: KPI telemetry

1 + 3N with N = 4: SoC, distance ×4, occupancy ×4, queue length ×4

pfail = 2 × 10−4 / step symmetric recovery failed stations excluded before the decision

occupancy · queue

KPI telemetry

per-run metrics

5. EVALUATION & KPIS

Primary : Queue Wait QW (s) ⋅ Completion Rate CR (%) ⋅ Energy Delivered ED (kWh) Secondary : Balance Index BI ⋅ First-Step Assignment Rate FSAR ⋅ Vehicles Served

Fig. 1. Edge-Intelligent EV charging framework architecture.

agents read station observations from the environment through the TraCI API. The vehicle and station topics specify the interface intended for a physical deployment, in which the simulator is replaced by real charging infrastructure; they are not on the critical path of the reported runs, and no latency measurement over a physical broker is claimed here. Edge EV Agents. In the reported simulation, a single shared DQN policy πθ is queried sequentially, once per vehicle that needs a decision at each simulation step: for each such vehicle, the current station observations are read, a local state vector st ∈ R13 is constructed, the shared policy is queried, and the chosen action is applied in the simulator. No inter-vehicle communication occurs, and no vehicle’s decision depends on another vehicle’s state at the same step, but the policy itself is not independently instantiated per vehicle in this evaluation. Deploying one independent policy instance per EV, running as its own edge process, is the target architecture this design is intended for and is left to future work. Charging Stations CS1 –CS4 . Each station continuously publishes its occupancy ratio occi ∈ [0, 1], queue length qi ∈ N, and available power Pi ∈ {50, 150} kW. The decision loop executes once per second: SUMO → Agent observe → Policy query → SUMO execute → reward → next state, with the resulting KPI published on sumo/kpi.

4

MDP Formulation

We model the charging assignment problem as a Markov Decision Process M = (S, A, R, γ) from the perspective of a single EV agent.

6

Moumni et al.

4.1

State Space

The state vector st ∈ R13 encodes:   st = SoCt , dt,1 , . . . , dt,4 , occt,1 , . . . , occt,4 , qt,1 , . . . , qt,4

(1)

where dt,i is the road distance to station i, occt,i its occupancy ratio, and qt,i the current queue length. The queue signal is distinct from occupancy: occt,i counts vehicles actively charging, while qt,i counts vehicles waiting—providing complementary decision signals. 4.2

Action Space

A = {1, 2, 3, 4}, representing the four candidate stations. 4.3

Reward Function

All penalty terms are normalised to [0, 1] before weighting. Writing q̃t , s̃t , d˜t and c̃t for the normalised queue wait, service time, detour and queue length at the assigned station, the reward rt at each decision step is: rt = wch 1[chargedt ] − wq q̃t2 − ws s̃t − wd d˜t − wc c̃t

(2)

with wch =5, wq =5, ws =1.5, wd =1 and wc =2. Normalisation uses fixed caps — 1 800 s for queue wait, 900 s for service time, 10 000 m for detour — and the station capacity for c̃t , each term being clipped at 1. The queue penalty is quadratic, so that severe congestion is penalised disproportionately rather than linearly. The discount factor is γ = 0.99. Equation 2 specifies the target reward design. The results reported in this paper are those of the v2 implementation; the equation should therefore be read as a design specification rather than a guaranteed transition-level execution trace. Two further points deserve emphasis, since both are frequent sources of confusion between a reward and a metric. First, the Balance Index BI = 1−σ(u)/µ(u), where u is the vector of station utilisation rates, is an evaluation KPI only: it is reported in Section 7 but never enters Equation 2. Any load balancing the agents achieve is therefore an indirect consequence of the congestion and queue penalties, not a directly optimised objective. Second, the objective of Section 4.4 describes the system-level criterion against which policies are compared, not the per-step signal the agents receive. 4.4

Optimisation Objective   min Eπ w1 · W T − w2 · ED + w3 · CI π

(3)

with weights w1 =0.5, w2 =0.3, w3 =0.2, subject to capacity (occi ≤ Ci ), autonomy (SoC ≥ SoCmin ), uniqueness (one station per step), and real-time (< 1 s decision latency) constraints.

Title Suppressed Due to Excessive Length

5

Algorithms

5.1

Adaptive Heuristic (Baseline)

7

Each vehicle maintains a per-station preference vector p ∈ R4 , updated after each interaction: pi ← (1 − ρ) pi + ρ rt ,

ρ = 0.2

(4)

Station selection follows a∗ = arg maxi (pi − λ occi − µ di ) with ε=0.15 exploration. This O(4) operation satisfies the real-time constraint trivially. The heuristic carries no formal convergence guarantee but serves as a strong practical baseline (+97% vs. static assignment in prior work [19]).

5.2

Tabular Q-Learning

The Bellman update is:   Q(s, a) ← Q(s, a) + αQ r + γ max Q(s′ , a′ ) − Q(s, a) ′ a

(5)

with αQ =0.1 and ε-greedy decay from 1.0 to 0.05. The continuous state is discretised into five aggregated variables from the vehicle’s perspective: SoC (4 bins: <0.15 / 0.15–0.30 / 0.30–0.50 / >0.50), distance to the nearest station (5 bins), mean occupancy across stations (3 bins), queue length at the least-congested reachable station (3 bins), and the number of vehicles already assigned/in transit to that station (3 bins), yielding 4 × 5 × 3 × 3 × 3 = 540 states and a 540 × 4 = 2,160-entry Q-table. Convergence to Q∗ is guaranteed under standard conditions [31].

5.3

Deep Q-Network (DQN)

The DQN replaces the lookup table with a feed-forward network Qθ : R13 → R4 of architecture Dense(13 → 64 → 32 → 4) with ReLU activations. The loss minimised at each training step is: L(θ) = E(s,a,r,s′ )∼B



2  r + γ max Qθ− (s , a ) − Qθ (s, a) ′ ′

a

′

(6)

Two stabilisation mechanisms are employed: (i) an experience replay buffer B of capacity 10 000 transitions sampled in mini-batches of 64, breaking temporal correlations; (ii) a frozen target network Qθ− , synchronised with Qθ every 100 steps, preventing oscillation of the regression target [17]. The 13-dimensional continuous state is fed directly, eliminating all discretisation error. Forward propagation, backpropagation and the Adam optimiser are implemented directly on top

8

Moumni et al. Table 2. Real EV charging stations in Rabat Agdal / Hay Riad. ID Name CS1 Tesla Supercharger CS2 Fastvolt Al Melia CS3 EBORNE Al Massira CS4 FASTVOLT Marjane Hay Riad

Power (kW) Capacity 150 50 50 150

4 4 4 4

GPS 33.9912, −6.8368 33.9693, −6.8642 33.9738, −6.8860 33.9564, −6.8510

of NumPy arrays, without relying4 on an external deep-learning framework (PyTorch/TensorFlow). This keeps the trained policy free of heavyweight runtime dependencies, consistent with the edge-deployment requirement of Section 6, where inference must run on a commodity edge device within the MQTT decision budget.

6

Experimental Setup

6.1

Road Network

The simulation network was extracted from OpenStreetMap for the Rabat Agdal / Hay Riad district (bounding box: 33.948◦ –33.997◦ N, 6.891◦ –6.832◦ W) using netconvert with a primary/secondary/tertiary road filter, yielding 2 176 edges and 648 junctions (2.44 MB). This constitutes, to our knowledge, the first RLbased EV charging study on a real Moroccan OSM network. 6.2

Charging Stations

Four real public EV charging stations, manually verified via Google Maps and Chargemap (June 2026), are used (Table 2). 6.3

Traffic Demand Profiles

Three hourly demand profiles are simulated with randomTrips.py using 7 fixed seeds (42–48) to ensure reproducibility: Morning (07h–09h, 50 EVs, mean interdeparture 70 s, theoretical utilisation ≈ 117%); Midday (12h–14h, 35 EVs, 100 s, ≈ 88%); and Evening peak (17h–20h, 75 EVs, 47 s, ≈ 180%) — queue formation guaranteed, constituting the stress scenario for policy discrimination. Each simulation targets a 3 600 s (one-hour) demand window at ∆t = 1 s, but does not stop at that fixed duration: the episode continues, capped at 4

A Double DQN (DDQN) variant, which decouples action selection from action evaluation in the target term to reduce the overestimation bias of the max operator, was trained under the identical protocol and is included in the experimental campaign. Its results are reported in Section 8.2; it tracks DQN closely throughout, and for clarity the comparison tables of Section 7 report the three structurally distinct policies.

Title Suppressed Due to Excessive Length

9

MAX_STEPS_PER_EPISODE, until SUMO reports no vehicle remaining active or expected. Observed episode lengths accordingly range beyond the nominal window (e.g. 5 325–6 184 steps observed for individual Rabat-evening episodes), reflecting this clearance behaviour rather than a fixed 3 600 s cutoff. 6.4

Evaluation Metrics

Six complementary KPIs are recorded: W T , mean waiting time per vehicle (primary KPI, seconds); ED, total energy delivered (kWh); CR, charge completion rate (%); BI, Balance Index = 1 − σ(u)/µ(u) ∈ [0, 1]; F SAR, first-step assignment rate, the share of vehicles assigned at their first request (%); and Served, the number of vehicles that complete a charge. 6.5

Statistical Protocol

Each agent–demand combination is run with 7 independent seeds. Pairwise comparisons use the Wilcoxon signed-rank test [32] at α = 0.05: this non-parametric test requires no normality assumption and is valid for n = 7 [32]. Effect sizes are quantified with Cohen’s d; 95% confidence intervals are bootstrapped over the 7 seeds. One limitation of this protocol should be noted, as it bears on every p-value reported below. The signed-rank test assumes paired observations: at a given seed, the two policies being compared should face the same environment. Traffic demand is matched across agents at each seed, but the station-failure process shares its random stream with each policy’s own exploration, so the failure sequence a policy observes can differ slightly from another’s at the same seed. This imperfect environmental pairing adds variance to the policy differences and may reduce the test’s sensitivity. Non-significant comparisons therefore do not establish equivalence; all nominally significant comparisons should also be interpreted under this pairing limitation. The total experimental campaign comprises four agents (Heuristic, Tabular Q-Learning, DQN and DDQN) over three demand profiles and seven seeds, i.e. 4 × 3 × 7 = 84 simulation runs per road network, to which the reward ablation of Section 7.5 adds three configurations over six scenarios and seven seeds (126 further runs).

7

Results and Discussion

7.1

Waiting Time Reduction

Table 3 reports mean waiting time W T (seconds) per agent and demand profile. Table 3 summarises these results. Under Morning demand, DQN and Heuristic achieve comparable W T (347 s vs. 347 s). Q-Learning achieves lower W T under Evening Peak (338.4 s vs. 353.9 s for DQN, p = 0.016, Cohen’s d = 1.12), prioritising route-efficiency over energy throughput. No statistically significant difference between DQN and Heuristic is observed for W T (p = 0.375), confirming hypothesis H3: RL gains under congestion emerge primarily in energy delivery metrics rather than waiting time.

10

Moumni et al.

Table 3. Mean waiting time W T (s), mean ± std over 7 seeds (Rabat network). p < 0.05 vs. DQN (Wilcoxon signed-rank, two-sided).

†

Agent

Morning

Midday

Evening Peak

Adaptive Heuristic 347.1±14.1 350.5±14.1 356.6±13.0 Tabular Q-Learning 349.9±17.1 336.7±22.2 338.4±11.7† DQN (proposed) 347.0±26.5 347.1±17.9 353.9±15.7

Energy Delivered and Balance Index — Rabat Network (mean ± std, 7 seeds) 1000

Energy Delivered — Rabat Network 0.800

950 900

0.775

850

0.750

Balance Index

Energy Delivered (kWh)

Load Balance Index — Rabat Network +5.4%* (d=1.51)

800 750 700 650

Heuristic Q-Learning DQN

600 550

Morning

Midday

Evening Peak

Heuristic Q-Learning DQN

0.725 0.700 0.675 0.650 0.625 0.600

Morning

Midday

Evening Peak

Fig. 2. Energy delivered (kWh) and Balance Index per agent and demand profile (Rabat, mean ± std, 7 seeds).

7.2

Energy Delivered and Balance

Figure 2 shows cumulative energy delivered and Balance Index per agent. The DQN agent’s queue-aware state representation enables it to route vehicles preferentially toward CS1 and CS4 (150 kW) when occupancy permits, yielding a 5.4% increase in ED over the Heuristic under Evening Peak (919.5 kWh vs. 872.1 kWh, p = 0.016, Cohen’s d = 1.51) and a 6.6% improvement over QLearning (p = 0.016, Cohen’s d = 2.49). The same pattern holds for charge completion: DQN reaches CR = 51.9% against 48.6% for the Heuristic under Evening Peak, a 6.8% relative improvement (p = 0.016, Cohen’s d = 0.99), and serves 29.5 vehicles against 27.6 (p = 0.016, Cohen’s d = 1.04). The Balance Index confirms that the DQN policy distributes load more equitably across all four stations (BI = 0.760 vs. 0.736 for Q-Learning and 0.743 for Heuristic under Evening Peak).

Title Suppressed Due to Excessive Length

7.3

11

Convergence Analysis

Each RL agent is trained for 40 episodes per demand period (120 cumulative episodes per city across the sequential Morning→Midday→Evening curriculum, sharing a single ε-greedy schedule and Q-table/network across periods; see Section 6). Within this budget, both DQN and Tabular Q-Learning show a modest but consistent reduction in mean queue-wait time between the first and last training episodes of each period (Rabat: 2–4% relative reduction; Tanger: up to 8% for Tabular Q-Learning), rather than full convergence to a stable optimum. Tabular Q-Learning improves more consistently than DQN/DDQN on the highest-demand Evening period, a pattern examined in Section 7.4. The Heuristic baseline requires no training phase, as its EWA preference table adapts online within each episode. 7.4

Limitations of the Sequential Curriculum

Because RL agents share one checkpoint per city across the three demand periods (Section 6), the ε-greedy exploration budget is not evenly split: Morning (trained first) receives most of the high-ε exploration, while Evening (trained last) inherits a near-exhausted budget. This design reflects a single agent operating continuously across a real day rather than three independently trained policies, but it confounds “harder demand period” with “less residual exploration” when interpreting per-period results, and it means per-period comparisons are not statistically independent samples for a given seed. A round-robin (interleaved) training curriculum, which distributes each period’s episodes evenly across the full εdecay schedule while still sharing one policy, is under evaluation as a follow-up to disentangle these two effects. 7.5

Reward Design Ablation

To isolate the contribution of each reward component, we trained DQN under three progressive configurations across all six scenarios (7 seeds each): A0 applies the queue-wait term R1 alone, with a linear queue penalty of weight 3 and γ = 0.95; A1 adds the rejection penalty R2 , incurred when a vehicle is turned away from its assigned station; A2 (the full-reward configuration of Equation 2, and nominally that of all results reported above) further adds R3 — the quadratic queue penalty with its weight raised from 3 to 5 — and R4 , which raises the discount factor from 0.95 to 0.99 for better anticipation of future congestion. Figure 3 reports the four primary KPIs. Adding R2 changes Charge Rate by +0.61 pp on average across the six scenarios, but the effect is neither large nor consistent: it ranges from −1.18 pp (Tangier evening) to +3.25 pp (Tangier midday) and is negative in three of the six scenarios. On this evidence the rejection penalty does not provide a decisive performance benefit; what the ablation supports is the weaker statement that no single reward term dominates the others. The step from A1 to the full reward A2 does not behave as a performance improvement either, and we report it as measured. On energy delivered,

12

Moumni et al. Ablation Study — DQN Reward Fixes A0 vs A1 vs A2 (mean ± std, 7 seeds)

950

1.0 A0 (R1) A1 (R1+R2) A2 (full)

0.9

900

Charge Rate

Energy Delivered (kWh)

1000

850 800

A0 (R1) A1 (R1+R2) A2 (full)

0.8

0.7

0.6 750 0.5 700 Rab Morn

Rab Mid

Rab Eve

Tang Morn

Tang Mid

Tang Eve

Rab Morn

Rab Mid

Rab Eve

Tang Morn

Tang Mid

Tang Eve

0.80 A0 (R1) A1 (R1+R2) A2 (full)

400

0.75

Balance Index

Queue Wait (s)

350

300

250

200

A0 (R1) A1 (R1+R2) A2 (full)

0.70

0.65

0.60

150 Rab Morn

Rab Mid

Rab Eve

Tang Morn

Tang Mid

Tang Eve

0.55 Rab Morn

Rab Mid

Rab Eve

Tang Morn

Tang Mid

Tang Eve

Fig. 3. Reward design ablation: DQN under A0 (queue-wait only), A1 (+ rejection penalty), and A2 (full-reward configuration) across four KPIs and six scenarios (mean ± std, 7 seeds).

A2 is above A1 in two of the six scenarios; on the Balance Index, in one of six. The single comparison that reaches nominal significance is adverse: energy delivered under Tangier morning falls from 820.5 to 785.8 kWh (p = 0.031). On the queue-wait metric that the R3 changes were designed to target, no scenario reaches significance (p from 0.109 to 0.938). We therefore make no performance claim for R3 and R4 here: on this evidence they act as design choices whose effect this campaign cannot resolve. The p-values quoted above are per-comparison and unadjusted; the ablation spans a large family of scenario–KPI pairs, and they should be read as exploratory rather than confirmatory.

8

Scalability and Robustness Analysis

8.1

Scalability to Larger Station Networks

The DQN architecture scales to a network of N stations by extending the state vector to 1 + 3N dimensions and the output layer to N neurons: the input grows linearly in N while the policy retains per-station resolution. Tabular QLearning does not suffer a combinatorial explosion either, since its aggregated discretisation (Section 5) keeps the table at 540 states regardless of N ; it instead pays for that aggregation by discarding which station is which, a loss that grows with N . The DQN’s advantage at scale is therefore one of fidelity, not table size.

Title Suppressed Due to Excessive Length

13

For the reference case N = 4 (this study), inference requires <0.5 ms on a commodity laptop CPU (single timing measurement)—well within the 1 s MQTT decision budget. Preliminary estimates for N = 20 yield an inference time of <2 ms with a Dense(1+80 → 128 → 64 → 20) architecture, still compatible with real-time constraints. A full multi-station ablation (4, 8, 16, 32 stations on synthetic networks) is planned as future work to provide empirical scaling curves. 8.2

Generalisation to a Second Road Network

To assess geographic robustness, we re-ran the full protocol on a second, independently trained road network: Tanger (3 demand profiles, 7 seeds). Each RL agent is trained from scratch on Tanger — separate Q-table/network checkpoints from Rabat, same curriculum and hyperparameters (Section 6) — rather than transferred zero-shot from the Rabat-trained policy. Figure 4 reports Queue Wait and Energy Delivered. The Heuristic baseline dominates on Queue Wait in Tanger, reflecting its hard-coded distance priority which matches the more dispersed topology of this network. DQN recovers competitive Energy Delivered across all profiles, and Tabular Q-Learning shows a more consistent Queue-Wait improvement over training than DQN/DDQN on the Evening period (Section 7.4), confirming that the relative agent ranking observed on Rabat is not fully stable across geographies. DDQN tracks DQN closely throughout the campaign: only two of all agent–scenario–KPI comparisons reach nominal significance (energy delivered under Tangier morning, p = 0.047; Balance Index under Tangier midday, p = 0.031), both unadjusted, isolated, and without a Rabat counterpart, so we report no evidence that the double estimator changes behaviour here. 8.3

Robustness to Failures and Noise

Station unavailability. Station failure is not a separate experiment but a permanent feature of every run reported above. Each station follows a two-state, discrete-time, per-step Markov process with asymmetric transition probabilities: P (available → failed) = 2 × 10−4 per step (mean time to failure ≈ 5 000 steps, about 83 minutes of simulated time) and P (failed → available) = 8 × 10−3 per step (mean time to recovery ≈ 125 steps, about 2 minutes) — a 40-fold asymmetry, not a symmetric process. A station in the failed state is removed from the candidate set before the decision is taken, so all agents — learned and heuristic alike — act on the set of currently available stations. We stress that this exclusion is an environment rule applied uniformly, not a behaviour learned by the DQN: the results of Section 7 are therefore already obtained under intermittent station outages, but they do not establish that any agent learns to avoid failing stations. Isolating that question requires a controlled comparison against an environment without exclusion, which we have not run. Communication faults and observation noise. Two further robustness questions follow naturally from the deployment architecture: the behaviour of the agents when the broker becomes unreachable, and their sensitivity to noisy

14

Moumni et al. Tanger Network — 4 Agents × 3 Demand Profiles (mean ± std, 7 seeds)

Heuristic Q-Learning DQN DDQN

Mean Queue Wait (s)

220 200 180 160 140

1000

Energy Delivered — Tanger Network Heuristic Q-Learning DQN DDQN

900

Energy Delivered (kWh)

240

Queue Wait — Tanger Network

800

700

600

120 500 100

Morning

Midday

Evening Peak

Morning

Midday

Evening Peak

Fig. 4. Queue Wait (s) and Energy Delivered (kWh) on the Tanger network — 4 agents × 3 demand profiles (mean ± std, 7 seeds).

SoC estimates, which in a physical fleet would be reported by heterogeneous battery-management systems rather than read exactly. Neither is evaluated in this paper. Both require the physical-deployment path of Section 3, where the vehicle and station topics carry the observations, and they are the first items of the testbed work announced in Section 9. We prefer to state this boundary rather than report robustness that our simulation protocol did not measure.

9

Conclusion

This paper presented an Edge-Intelligent DQN framework designed for realtime EV charging assignment, evaluated on two real OpenStreetMap road networks in Morocco — Rabat Agdal / Hay Riad and Tangier. A queue-aware 13-dimensional MDP formulation drives a DQN policy that this paper evaluates as a single shared instance applied sequentially per vehicle in simulation; deployment as an independently instantiated, locally executed policy per EV over a lightweight MQTT publish–subscribe backbone is the target architecture, not the evaluated configuration. Comparison against an Adaptive Heuristic and Tabular Q-Learning under three demand profiles and seven independent seeds supports the following, and no more: (H1) under peak demand DQN improves charging throughput over the Heuristic — energy delivered +5.4% (d = 1.51), charge rate +6.8% relative (d = 0.99), vehicles served +6.9% (d = 1.04), all at p = 0.016 — whereas it shows no significant advantage in waiting time (p = 0.375 against the Heuristic, and Tabular Q-Learning is significantly faster than DQN under Evening Peak); (H2) the gain is specific to DQN and does not extend to reinforcement learning as a family: Tabular Q-Learning delivers less energy than

Title Suppressed Due to Excessive Length

15

the Heuristic under the same profile; (H3) statistical significance emerges only under congestion stress. A scalability analysis shows linear state-space growth with the number of stations and sub-millisecond inference time (single timing measurement, Section 8). All reported runs are obtained under intermittent station outages driven by a Markov failure process, so the comparison holds in the presence of unavailable infrastructure; the unevaluated communication-fault and SoC-noise boundary is stated in Section 8. Future work will extend the campaign to N ∈ {8, 16, 32} synthetic stations, investigate Double DQN and Dueling architectures, and deploy the framework — including the communication-fault and SoC-noise testing this paper does not cover — on a physical MQTT testbed with real EV charging hardware in the Rabat metropolitan network.

Acknowledgements The authors acknowledge the support of the Mohammed V University research laboratory LRIT and the LPAIS laboratory at USMBA Fez. Simulation infrastructure was provided by the SUMO open-source community [14].

References 1. Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., Ayyash, M.: Internet of things: A survey on enabling technologies, protocols, and applications. IEEE Commun. Surv. Tutorials 17(4), 2347–2376 (2015). https://doi.org/10.1109/ COMST.2015.2444095 2. Belli, L., Cilfone, A., Davoli, L., Ferrari, G., Adorni, P., Di Nocera, F., Dall’Olio, A., Pellegrini, C., Mordacci, M., Bertolotti, E.: Iot-enabled smart sustainable cities: Challenges and approaches. Smart Cities 3(3), 1039–1071 (Sep 2020). https:// doi.org/10.3390/smartcities3030052 3. Bhatti, A.R., Tamoor, M., Liaqat, R., Rasool, A., Salam, Z., Ali, A., Sherefa, A.: Electric vehicle charging stations and the employed energy management schemes: A classification based comparative survey. Discov Appl Sci 6(10), 503 (Sep 2024). https://doi.org/10.1007/s42452-024-06190-9 4. Chifu, V.R., Cioara, T., Pop, C.B., Rusu, H.G., Anghel, I.: Deep q-learning-based smart scheduling of evs for demand response in smart grids. Applied Sciences 14(4), 1421 (Feb 2024). https://doi.org/10.3390/app14041421 5. Cui, F., Lin, X., Zhang, R., Yang, Q.: Multi-objective optimal scheduling of charging stations based on deep reinforcement learning. Front. Energy Res. 10, 1042882 (Jan 2023). https://doi.org/10.3389/fenrg.2022.1042882 6. Da Silva, F.L., Nishida, C.E.H., Roijers, D.M., Reali Costa, A.H.: Coordination of electric vehicle charging through multiagent reinforcement learning. IEEE Transactions on Smart Grid 11(3), 2347–2356 (2020). https://doi.org/10.1109/TSG. 2019.2952331 7. Farhani, R., El Hillali, Y., Rivenq, A., Boughaleb, Y., Hajjaji, A.: Assignment approach for electric vehicle charging using traffic data collected by SUMO. World Electric Vehicle Journal 14(2), 40 (2023). https://doi.org/10.3390/ wevj14020040

16

Moumni et al.

8. Han, Y., Li, T., Wang, Q.: A dqn based approach for large-scale evs charging scheduling. Complex Intell. Syst. 10(6), 8319–8339 (Dec 2024). https://doi.org/ 10.1007/s40747-024-01587-w 9. International Transport Forum: Decarbonising Morocco’s transport system. Tech. rep., OECD Publishing, Paris (2021). https://doi.org/10.1787/d9fb9340-en 10. Lee, K.B., Ahmed, M.A., Kang, D.K., Kim, Y.C.: Deep reinforcement learning based optimal route and charging station selection. Energies 13(23), 6255 (2020). https://doi.org/10.3390/en13236255 11. Lee, S., Choi, D.H.: Dynamic pricing and energy management for profit maximization in multiple smart electric vehicle charging stations: A privacy-preserving deep reinforcement learning approach. Applied Energy 304, 117754 (2021). https: //doi.org/10.1016/j.apenergy.2021.117754 12. Li, H., Dai, X., Goldrick, S., Kotter, R., Aslam, N., Ali, S.: Reinforcement learning for ev fleet smart charging with on-site renewable energy sources. Energies 17(21), 5442 (Oct 2024). https://doi.org/10.3390/en17215442 13. Liu, L., Huang, Z., Xu, J.: Multi-agent deep reinforcement learning based scheduling approach for mobile charging in internet of electric vehicles. IEEE Trans. on Mobile Comput. 23(10), 10130–10145 (Oct 2024). https://doi.org/10.1109/ TMC.2024.3373410 14. Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flötteröd, Y.P., Hilbrich, R., Lücken, L., Rummel, J., Wagner, P., Wießner, E.: Microscopic traffic simulation using SUMO. In: Proceedings of the 21st IEEE International Conference on Intelligent Transportation Systems (ITSC). pp. 2575–2582 (2018). https: //doi.org/10.1109/ITSC.2018.8569938 15. Ma, J., Zhang, Y., Duan, Z., Tang, L.: Prolific: Deep reinforcement learning for efficient ev fleet scheduling and charging. Sustainability 15(18), 13553 (Sep 2023). https://doi.org/10.3390/su151813553 16. Michailidis, P., Michailidis, I., Kosmatopoulos, E.: Reinforcement learning for electric vehicle charging management: Theory and applications. Energies 18(19), 5225 (Oct 2025). https://doi.org/10.3390/en18195225 17. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236 18. Moumni, N.E., Alaoui, R., Kiouach, D.: Human-centric cooperative marl for reliable ev charging coordination in smart cities: A controlled multi-seed comparison of centralised and decentralised training. Research Square preprint (2026). https://doi.org/10.21203/rs.3.rs-10139360/v1, posted 25 June 2026; submitted to Human-Centric Intelligent Systems 19. Moumni, N.E., Alaoui, R., Kiouach, D.: A real-time agent-based architecture for smart electric vehicle charging management via IoEV communication. The Open Transportation Journal (2026), under review, Bentham Open 20. Moumni, N.E., Alaoui, R., Kiouach, D., El-Fedany, I.: The smart city of tomorrow: A simulation platform for optimizing EV routes to charging points — a Moroccan perspective. In: Proceedings of the 14th International Conference on Intelligent Systems: Theories and Applications (SITA). pp. 1–7 (2023). https://doi.org/ 10.1109/SITA60746.2023.10373710 21. Moumni, N.E., Alaoui, R., Kiouach, D., El-Fedany, I.: Smart EV Routing to Charging Stations for Traffic Optimization in Smart Cities, pp. 261–276. CRC Press, Boca Raton, first edn. (May 2025). https://doi.org/10.1201/9781032664293-20

Title Suppressed Due to Excessive Length

17

22. Nasreddin, D., El Hafdaoui, H., Jelti, F., Boumelha, A., Khallaayoun, A.: Inhibitors of battery electric vehicle adoption in Morocco. World Electric Vehicle Journal 15(1), 6 (2023). https://doi.org/10.3390/wevj15010006 23. Pang, A., Wang, M., Chen, Y., Pun, M.O., Lepech, M.: Scalable reinforcement learning framework for traffic signal control under communication delays. IEEE Open J. Veh. Technol. pp. 1–13 (2024). https://doi.org/10.1109/OJVT.2024. 3368693 24. Rimal, B., Kong, C., Poudel, B., Wang, Y., Shahi, P.: Smart electric vehicle charging in the era of internet of vehicles, emerging trends, and open issues. Energies 15(5), 1908 (Mar 2022). https://doi.org/10.3390/en15051908 25. Sarda, J., Patel, N., Patel, H., Vaghela, R., Brahma, B., Bhoi, A.K., Barsocchi, P.: A review of the electric vehicle charging technology, impact on grid integration, policy consequences, challenges and future trends. Energy Reports 12, 5671–5692 (Dec 2024). https://doi.org/10.1016/j.egyr.2024.11.047 26. Savio Abraham, D., Verma, R., Kanagaraj, L., Giri Thulasi Raman, S.R., Rajamanickam, N., Chokkalingam, B., Marimuthu Sekar, K., Mihet-Popa, L.: Electric vehicles charging stations’ architectures, criteria, power converters, and control strategies in microgrids. Electronics 10(16), 1895 (Aug 2021). https://doi.org/ 10.3390/electronics10161895 27. Suanpang, P., Jamjuntr, P.: Optimizing electric vehicle charging recommendation in smart cities: A multi-agent reinforcement learning approach. World Electr. Veh. J. 15(2), 67 (Feb 2024). https://doi.org/10.3390/wevj15020067 28. Urooj, S., Alrowais, F., Teekaraman, Y., Manoharan, H., Kuppusamy, R.: Iot based electric vehicle application using boosting algorithm for smart cities. Energies 14(4), 1072 (Feb 2021). https://doi.org/10.3390/en14041072 29. Usman, M., Tareen, W.U.K., Amin, A., Ali, H., Bari, I., Sajid, M., Seyedmahmoudian, M., Stojcevski, A., Mahmood, A., Mekhilef, S.: A coordinated charging scheduling of electric vehicles considering optimal charging time for network power loss minimization. Energies 14(17), 5336 (Aug 2021). https://doi.org/10.3390/ en14175336 30. Wan, Z., Li, H., He, H., Prokhorov, D.: Model-free real-time ev charging scheduling based on deep reinforcement learning. IEEE Trans. Smart Grid 10(5), 5246–5257 (2019). https://doi.org/10.1109/TSG.2018.2879572, doi: 10.1109/TSG.2018.2879572 31. Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992). https://doi.org/10.1007/BF00992698 32. Wilcoxon, F.: Individual comparisons by ranking methods. Biom. Bull. 1(6), 80–83 (1945). https://doi.org/10.2307/3001968, doi: 10.2307/3001968 33. Zhang, W., Liu, H., Wang, F., Xu, T., Xin, H., Dou, D., Xiong, H.: Intelligent electric vehicle charging recommendation based on multi-agent reinforcement learning. In: Proceedings of the Web Conference 2021. pp. 1856–1867 (2021). https://doi.org/10.1145/3442381.3449934

Related documents

• Filter-Augmented Reinforcement Learning for Battery Fast Charging with State of Health and Plating-Aware Reward Design#1386756
• Cloud-Based AI Framework for EV Charging Forecasting and Infrastructure Optimization#1081866
• A Mobile Agent-Based Hierarchical Reinforcement Learning Framework for Energy-Balanced Data Collection and Wireless Charging in WSN#1154887
• IndusAgent: Agentic Reinforcement Learning for Cross-Category Industrial Anomaly Detection#1389295
• Learning-Guided Planning in Large Dynamic Action Spaces: Budgeted Tree Search for One-to-Many Mobile Charging#1123793