1
Securing Autonomous Vehicle Systems via Twin-Aware Federated Reinforcement Learning
arXiv:2607.08137v1 [cs.CR] 9 Jul 2026
Zifan Zhang∗ , Minghong Fang† , Dianwei Chen‡ , Zhuqing Liu§ , Prashant Khanduri¶ , Xianfeng Yang∗ , Anupam Das∗ , Yuchen Liu∗ ∗ North Carolina State University, USA, † University of Louisville, USA, ‡ University of Maryland, USA, § University of North Texas, USA, ¶ Wayne State University, USA
Abstract—Federated reinforcement learning (FRL) is crucial for enabling collaborative learning across multiple agents without sharing raw data, thereby enhancing privacy and scalability in the decision-making process within dynamic vehicular environments. However, poisoning attacks pose a significant threat to the security and reliability of FRL-based systems, particularly in safety-critical autonomous driving, where this vulnerability remains largely unexplored. These attacks can compromise the global control model by subtly injecting malicious system parameters, leading to potential hazards. To counter these challenges, we present SecApp (Secure Aggregation with poisoning-prevention and historical reinforcement) as a defensive framework aimed at enhancing the robustness of FRL systems designed for safety-critical driving scenarios. SecApp strategically integrates digital twins for rehearsal-based learning and leverages historical aggregated model parameters along with a selected central gradient to ensure that only benign data is aggregated, effectively mitigating the influence of malicious agents. Theoretical guarantees are provided for the convergence performance of SecApp in the presence of poisoning attacks. We also validate the effectiveness of SecApp using developed digital twins that model realistic highway environments to evaluate the control of autonomous vehicles under adversarial conditions.
I. I NTRODUCTION Advancements in computing power have significantly improved decision-making and problem-solving. Among various approaches, reinforcement learning (RL) has proven effective in real-world mobile applications, including robotics [1], autonomous driving (AD) [2], [3], GPT-4 [4], wireless networks [5], [6], and healthcare [7]. In essence, agents on mobile entities or controllers receive observations to perceive the dynamics of their environments and train RL models to perform actions that maximize long-term cumulative rewards. However, single-agent RL suffers from low sampling efficiency and limited observability in mobile environments [8]. In safety-critical scenarios like AD, such agents often struggle to find optimal solutions for a wide range of operational conditions, especially in handling accident cases. Overlooking these sparsely distributed corner cases can lead to severe consequences, including threats to human safety. As a remedy, federated reinforcement learning (FRL) is employed by combining the strengths of federated learning (FL) and RL, enabling multiple distributed agents to collaboratively learn optimal policies from a holistic view while preserving the privacy of their individual data. FRL has been widely used in various domains [9]–[12], which enhances efficiency and generalization of decision-making models by leveraging
diverse data sources from different perspectives of agents in dynamic environments. While FRL offers significant advantages, it presents several risks when applied in safety-critical mobile systems [13]. The most critical challenge is the vulnerability to poisoning attacks, where malicious agents deliberately provide incorrect or misleading update signals, compromising the integrity of the global control for decision-making [14]–[17]. These attacks can significantly degrade the performance of FRL-based systems and potentially lead to failures in completing critical tasks. Although prior research has studied several defensive strategies to mitigate the impact of malicious agents [15], [18]– [24], these defense mechanisms are primarily designed for pure FL training and do not account for the unique attributes of RL agents. The primary goal in those FL systems is to aggregate models trained on labeled datasets with the aim of minimizing a predefined loss function. In contrast, FRL systems focus on learning optimal policies through interactions with networked environments, involving sequential decisionmaking and delayed rewards. This introduces additional complexities, such as non-stationarity and the need to balance exploration and exploitation, which are absent in traditional FL scenarios. Besides, RL agents operate in diverse and evolving environments, resulting in more significant variations in local gradients (i.e. the learning updates based on their respective experiences before aggregation) compared to the relatively stable updates in pure FL systems. Our experiments showed that applying FL security mechanisms to the FRL framework yields unsatisfactory performance in AD decisionmaking tasks (detailed in Sec. II). More importantly, there is a lack of comprehensive theoretical analysis regarding the convergence of secure FRL optimization [25], which makes their application in safety-critical systems a long-standing matter requiring prudent consideration. These issues underscore the need to develop new, specialized solutions that integrate both environmental security awareness and theoretical guarantees against potential attacks in FRL-enabled mobile systems. Main Contribution: To bridge this gap between secure computing and robust machine learning in safety-critical mobile systems, we propose a novel robustness-aware framework called SecApp– Secure Aggregation with poisoningprevention and historical reinforcement. The core objective is to protect AD system from poisoning attacks, where malicious entities attempt to corrupt data-driven control model in each
2
mobile entity by sending misleading or harmful information pieces. In essence, SecApp employs a multi-step strategy to mitigate the effects of poisoning attacks with varying levels of severity. Initially, central server filters out the gradients from each mobile agent (e.g., deployed on vehicles) that differ from the majority, ensuring only consistent and harmless gradients are included in the aggregation set. The server then identifies a central gradient to represent the benign mobile agents and aggregates only those gradients that are close to it. This process is repeated across multiple rounds, allowing the system to iteratively refine the global model for decision-making while minimizing impacts of malicious agents. In addition, we develop a digital twin (DT) of the system environment by incorporating historical sensor inputs and operational parameters into a cohesive data-driven model. With the DT in the loop, the system can generate diverse environments that reflect a wide range of operating conditions and support rehearsalbased learning. This diversity enables robust validation of decision-making algorithms, allowing the system to adapt to various scenarios without the risks of physical deployment, i.e. from normal operations to corner cases. Theory Achievement: Theoretically, we establish that under certain mild assumptions, such as the smoothness and nonconvexity of the loss function, the decision-making model learned by our SecApp is proven to converge to a firstorder stationary point of the learning objective, which is the strongest guarantee attainable for the highly non-convex RL loss. This outcome underscores the robustness of SecApp, ensuring that it preserves the reliable performance of the FRL-enabled mobile systems. This property remains valid even in the presence of Byzantine settings, demonstrating that SecApp can effectively handle malicious or faulty updates without compromising performance. Evaluation Remarks: To illustrate the effectiveness of SecApp, we apply it to real-world safety-critical systems, including AD and edge caching scenarios1 . For the AD task, conducting experiments on actual highways is both impractical and unsafe. To address this, we exploit the developed DT, named HighwayDT [26], which accurately simulates realistic safety-critical scenarios, including rare and complex cases, by replicating the dynamic conditions of highway AD environments. HighwayDT receives real-time data from the physical vehicular network system, continuously updating its control model to accurately reflect the evolving attributes of highway settings. It integrates functionalities such as predictive vehicle control, scenario generation, and decision-making, enabling a comprehensive validation process within a synchronized virtual environment. For instance, SecApp is deployed on autonomous vehicles to learn optimal solutions for tasks such as vehicular longitudinal control and collision avoidance. To assess its robustness, SecApp is evaluated under thirteen poisoning attacks, including strong adaptive attack, and demonstrates superior performance compared to state-of-theart defense strategies. The results also align closely with the 1 Additional experiments on mobile edge caching settings can be found in the supplementary materials (Appendix E).
theoretical analysis of SecApp’s convergence, highlighting its consistency and reliability in challenging adversarial scenarios. Our contributions can be summarized as follows: • We introduce SecApp, the first robustness-aware framework specifically designed to safeguard FRL-based safetycritical systems against the impact of poisoning attacks. • We provide the first-of-its-kind comprehensive theoretical analysis that demonstrates the convergence performance of secure FRL in the presence of poisoning attacks. This analysis confirms that SecApp can maintain its stability and effectiveness even when facing adversarial conditions. • We construct HighwayDT to precisely map physical safetycritical mobile environments into digital FRL systems, which can provide risk-free and highly heterogeneous validation scenarios for security tests and analysis. • We conduct extensive experiments in safety-critical scenarios, including AD on highways and edge caching in high-stakes network infrastructures. The evaluation results demonstrate the effectiveness of SecApp against existing state-of-the-art targeted and untargeted attacks. Notations: In this work, notation [K] denotes the set {1, 2, . . . , K} for any positive integer K. The ℓ2 -norm is indicated by ∥·∥, and the size of a set S is written as |S|. II. M OTIVATION AND O BSERVATION Through preliminary experiments, we observe several limitations in existing learning-based control approaches for mobile autonomous vehicle (AV) systems. These observations motivate the need for SecApp frameworks that better account for system heterogeneity, interaction dynamics, and security constraints. Observation 1. Single-agent RL framework struggles to find effective control solutions for AV systems. RL has been widely adopted in the research area of AD [27], [28], and used by several enterprises as well for practical implementation, such as Google [29] and Waymo [30]. In this paper, we use AV system as an example, but the fundamentals of agents and environments can be easily extended to other safetycritical systems, as demonstrated in Appendix E. Modeled as a Markov Decision Process (MDP) [31], the vehicular agent takes actions a in observed states s, transitions to new states s′ , and receives rewards r based on these transitions. The vehicle’s goal is to maximize cumulative rewards by learning an optimal policy π(s), which specifies the best driving decisions in the long run. RL framework used in AD involves trial and error, balancing exploration of new actions and exploitation of known rewarding actions, and utilizes techniques such as policy gradients [25], [32], [33], where the gradient—representing the derivative of the expected reward with respect to policy parameters—guides updates to improve driving performance in ever-changing environments. However, traditional single-agent RL often struggles with low sampling efficiency, particularly in AV systems, where the agent may fail to adequately explore rare but critical corner events, such as accident scenarios. This limited exploration leads to high variance in gradient-based learning, hindering stable and effective policy optimization. To tackle
3
this problem, a variance reduction (VR) method is often used, where a roadside edge server additionally samples a few trajectories to update the global model to reduce the variance of stochastic gradients. Meanwhile, extending from a single agent to multiple distributed agents training on AVs improves sampling efficiency, as agents collectively explore a wider state space, capture diverse experiences, and accelerate learning convergence. Table I presents our preliminary results that an edge server uses different aggregation rules in multiagent distributed training (such as FedAvg [23], Median [34], Trimmed-mean (Trim) [34], and Krum [24]) to combine agents’ local gradients. “w/o VR” indicates that the variance reduction method is not used, where the server directly updates the global model using the aggregated gradient (the full procedure is provided in the supplementary material), while “w/ VR” applies variance reduction to mitigate gradient variance. “Single” represents training on a single agent without shared observation and data aggregation. Table I presents nocollision rates as the performance metric for an AV system2 . The results show that single-agent RL struggles to achieve the desired no-collision rates due to limited sampling efficiency, while the multi-agent solution with VR significantly enhances vehicle control performance, even achieving perfect solutions regardless of aggregation method. TABLE I: Comparison of single- and multi-agent training w/o and w/ variance reduction (VR). w/o VR w/ VR
FedAvg 82.62% 100.0%
Median 84.40% 100.0%
Trim 85.08% 100.0%
Krum 81.41% 100.0%
Single 73.45% 89.46%
Observation 2. Multi-agent FRL framework carries serious risks when deployed in practical mobile systems. FRL [18], [35], [36] overcomes the limitations in singleagent RL by allowing decentralized agents to work together to capture full observations of complex environments and learn optimal strategies while maintaining the privacy of their individual data. Consider an FRL system with K agents. In each training round t, a generic process involves three steps: Step I (Global model synchronization). The central server distributes the current global model w̃t−1 to all participating agents. This corresponds to the operation wt0 ← w̃t−1 in Line 3 of Algorithm 2. • Step II (Local training). Each agent k ∈ [K] executes Algorithm 1 to refine its local gradient. This involves sampling ξt trajectories based on the trajectory distribution d(·|wt0 ), and computing the local gradient µkt based on the distributed global model wt0 , where d(·|wt0 ) denotes the trajectory distribution induced by model wt0 and this distribution changes over time. The local gradient µkt is then uploaded to the central server. Note that g(·) in Line 4 of Algorithm 1 represents the gradient estimator. • Step III (Aggregation and global model updating). The central server aggregates the received local gradients µkt from all participating agents using a particular aggregation rule to produce the aggregated gradient. For instance, if the server employs the FedAvg aggregation method to combine •
2 The detailed evaluation and experimental setup can be found in Section VI.
the local gradients from all agents,Pthe aggregated gradient K 1 k µt can be calculated as µt ← K k=1 µt , as outlined in Line 7 of Algorithm 2. The server then uses VR methods to update the global model and corresponding policy. The updated global model is then distributed to the agents for follow-up interactions and local control. Such FRL framework shows considerable promise in safetycritical systems, yet it carries serious risks when deployed in environments where safety is non-negotiable. A major concern is its susceptibility to model poisoning, where rogue participants intentionally inject erroneous or deceptive updates into the aforementioned Step II. This malicious interference undermines the overall reliability of the collective model in the system, often resulting in sharply diminished control performance and even critical failures. Moreover, while various protective measures have been explored for standard FL, these defenses generally fall short when applied to the more complex and dynamic context of RL agents. Table II presents the preliminary results of different training rules in a multi-agent system under FTI, MinSum, and Adaptive attacks, as studied in [16], [17]. The results indicate that FRL cannot maintain the 100% no-collision rate in validation environments due to its vulnerability to model poisoning attacks. This highlights the need to augment robustness-aware strategies for FRL-based systems to prevent unexpected consequences. TABLE II: Comparison of multi-agent training under poisoning attacks with standard FL-based defensive strategies. Attack
FedAvg
Median
Trim
Krum
No attack
100.0%
100.0%
100.0%
100.0%
FTI attack
12.47%
14.01%
8.36%
11.02%
MinSum attack
10.29%
13.77%
100.0%
9.49%
Adaptive attack
14.49%
11.49%
7.11%
13.68%
Observation 3. Homogeneous testing environments fail to ensure the stability and robustness required for safety-critical mobile systems. Taming AD systems with safety awareness demands extensive testing and validation across a wide spectrum of operational conditions. Relying solely on homogeneous testing environments can overlook rare events that may be encountered in real-world scenarios, thereby compromising system safety. Heterogeneous testing environments are crucial as they expose the AD system to varying environmental conditions, road configurations, and unexpected interactions with surrounding AVs, ensuring that the system’s performance is robust under diverse circumstances. DTs are high-fidelity virtual replicas of physical systems that are continuously updated with real-world data to reflect current conditions and behaviors [37]. They enable detailed what-if analyses, allowing simulation of various scenarios, such as sudden changes in environmental conditions or vehicle component failures, to predict how the AD system might respond without the risks and costs of real-world experiments. In the context of AD, FRL benefits significantly from the integration of DTs. By providing a controlled yet diverse set of virtual environments, DTs enrich the training data with a wide range of operational scenarios, which enhances the robustness
4
Vehicle 1
Vehicle 2
Vehicle 3
Vehicle Positions Over Time
175 150 100 75 50 25 0
Vehicle 5
Vehicle Velocities Over Time
37.5 35.0 32.5 30.0 27.5 25.0 22.5 20.0 17.5
0
50
100 150 200 250 300 350 400
Time Step
Algorithm 1 LocalTraining (t, k, wt0 , ξt , d(·|wt0 )). 1: Output: µk t 2: for each local training round do k ξt 3: Sample ξt trajectories {υt,i }i=1 from d(·|wt0 )
Velocity (m/s)
Position (m)
125
Vehicle 4
0
50
100 150 200 250 300 350 400
Time Step
4: µkt = ξ1t 5: end for
Pξt
k 0 i=1 g(υt,i |wt )
Fig. 1: Vehicles fail to respond due to training on a homogeneous environment.
model, leading to higher collision rates and loss of individual vehicle control through model propagation.
and generalization of the learned policies. Unlike single-agent offline training, distributed training leverages multiple agents interacting with different instances from the DT in parallel, thereby capturing a broader spectrum of reaction variability. A roadside server is essential in this setup to coordinate knowledge updates and ensure consistency across agents, mitigating the risk of divergence or conflicting strategies that could occur in a fully decentralized system. A preliminary result of RLbased vehicles trained without diverse scenarios is shown in Fig. 1. It is observed that the RL agent fails to provide proper maneuvers due to sparse rewards, eventually leading to crashes. For instance, vehicles 2, 3, and 4, the three vehicles in the middle, continue to accelerate even as the leading vehicle begins to decelerate after approximately 100 steps, ultimately causing crashes as indicated by the dashed lines in Fig. 1. This result demonstrates that a policy trained in a single, homogeneous environment lacks stability and robustness when evaluated across diverse validation scenarios. Based on these observations and preliminary validation, we are motivated to design a robustness-aware framework that integrates defensive policies and DTs to safeguard FRL-based mobile systems, as discussed in following sections.
Attacker’s knowledge: We assume a worst-case scenario where attackers have comprehensive knowledge of FRL-based safety-critical system. This includes details about local training trajectories from DTs, aggregation rules, gradients of all agents, and global models. Environment instances generated by DTs can also be accessed and manipulated by attackers. Such extensive knowledge allows the attacker to craft sophisticated attacks, ensuring that our proposed security mechanisms are evaluated under the most challenging conditions.
III. P ROBLEM S TATEMENT AND S YSTEM A RCHITECTURE Global Gradient
Validate
Aggregation Updated Gradient
Roadside Server
Distribute
Upload
Upload
Validation Environment
Defender’s knowledge and goal: The defensive mechanism is designed to operate without any prior information about the attacker’s strategy with assistance from DTs. Each benign agent is unaware of the location, identity, or number of malicious agents in the system and their corresponding DTs. Our objective is to develop a robustness-aware FRL framework that ensures safe and reliable operations under the aforementioned challenging conditions. Specifically, the system aims for high control and learning performance, maintaining its effectiveness in non-adversarial mobile environments with assistance from DTs. The policy learned by each benign agent should perform at a level comparable to averaging-based aggregation, which is recognized for its superior performance in non-adversarial settings. Second, the framework must exhibit both theoretical and practical resilience to model poisoning attacks, ensuring robust operations even in the presence of multiple compromised agents. Lastly, the proposed security component is designed to maintain communication and computation efficiency, introducing minimal or no additional overhead compared to safety-oblivious FRL approaches.
Gradient 2 Environment 2 Agent 2
Distribute
IV. S EC A PP S OLUTION
Policy Update
Agent 1 Action
Local Gradient 1 Digital Twin Environment 1
Gradient 3 Environment 3 Agent 3
Reward State
Malicious Poisoned Attacker Gradient Environment
Fig. 2: System overview under threats. Threat model: In FRL-based mobile systems, we consider a scenario where attackers compromise a subset of agents (e.g., RL-assisted AVs), rendering them malicious. We use a sample system with three benign agents and one malicious agent to illustrate the problem in Fig. 2. Multiple DTs are used to generate diverse AD environment instances for parallel local training. Specifically, the malicious agent manipulates its local model updates (e.g., gradients) and transmits corrupted information to roadside server. Such an adversarial behavior can compromise the integrity of the globally aggregated control
Technically, our proposed SecApp is designed to safeguard the global decision-making model w̃T under the threats through distributed learning across multiple local RL agents over T training rounds. Poisoning attacks occur when some agents (e.g., AVs) provide noisy or malicious gradients to corrupt the global model that gathers holistic observations for system control, and the framework must be able to defend against such “bad-news-travels-fast” behavior. The algorithm achieves this by first constructing a majority-consistent set of computed gradients, then selecting a central gradient from that set, and eventually ensuring that only gradients consistent with the central gradient are aggregated by average. This multi-step approach minimizes the influence of malicious agents, ensuring the aggregated gradient is representative of honest gradients and protects the learning process from being corrupted by any adversarial behavior.
5
Algorithm 2 FRL training process. 1: Input: w̃0 ∈ Re , batch size ξt , mini-batch size bt , step
size ηt 2: for t = 1 to T do 3: wt0 ← w̃t−1 4: for k = 1 to K in parallel do 5: µkt = LocalTraining(t, k, wt0 , ξt , d(·|wt0 )) 6: end for
PK k 1 ▷ FedAvg [23] µt ← K k=1 µt t 8: Sample steps Nt ∼ Geom( ξtξ+b ) t 9: for n = 0 to Nt − 1 do n bt 10: Sample bt trajectories {υt,j }j=1 from d(·|wtn ) P b t 1 n |wtn )− 11: ζtn = bt j=1 [g(υt,j n n 0 n δ(υt,j |wt , wt )g(υt,j |wt0 )] + µt n+1 n n 12: wt = wt + ηt ζt 13: end for 14: w̃t ← wtNt 15: end for 16: Output: w̃a uniformly randomly picked from {w̃t }T t=1 7:
We construct data-driven DTs by first replicating the physical system’s dynamics in a virtual environment using historical data and known system parameters. To generate heterogeneous environments, we introduce diverse variations in system parameters into the DTs, such as noise levels, vehicle maneuvers, and starting points. This controlled perturbation process is automated via parameterized simulation scripts, allowing for the rapid production of a wide range of scenarios that mirror both normal and rare cases. These diverse environment instances enable extensive training and testing of driving behaviors under varying conditions, including dangerous events that cannot be safely replicated in the physical world, thus broadening the agent’s experience and expertise to handle increasingly challenging scenarios. On the other hand, malicious agents in AD systems carry out poisoning attacks by altering either the direction or magnitude of their gradients before transmitting them to the roadside server. To mitigate these attacks, the server first employs a majority-based filtering mechanism to retain only gradients aligned with the majority consensus. In particular, our SecApp constructs a set, denoted as S, by filtering out outliers and including only local gradients that are sufficiently close to the majority of others. Specifically, a gradient µkt is included in the set S if following condition is satisfied: n o ′ K (1) k ′ ∈ [K] : µkt − µkt ≤ ψ > , 2 where ψ is a threshold that defines the maximum allowable norm difference between two gradients for them to be considered “close”. The rule includes a gradient µkt in the set S only if the number of gradients close to it (within the distance ψ) constitutes a majority, i.e., greater than K 2 . This ensures the exclusion of potentially malicious gradients that deviate significantly from the consensus of most gradients. The server further enhances robustness by selecting a center gradient µSt that is closest to the center of S. This technique is designed to minimize the impact of any remaining outliers or malicious gradients that are far from other gradients in S. This
method ensures that the selected gradient is relatively robust to any adversarial behavior that may still be present, but it needs to be refined to achieve a global model that is completely resilient to poisoning attacks. The server then defines the set Wt , which contains the indices of the gradients µkt that are close to the previously selected center gradient µSt with: µkt − µt−1 ≤ λ µSt − µt−1 ,
(2)
where λ is a scaling factor. Specifically, an gradient is included in Wt if its distance from the previous aggregated gradient µt−1 is within a scaled distance, determined by the factor λ, of the distance between µSt and µt−1 . Note that λ can be interpreted as a measure of the allowable rate of change or “smoothness” in the updates, ensuring that agents’ gradients do not deviate too far from the central gradient µSt , which helps in identifying benign gradients. This step ensures that only gradients that are consistent with the robustly selected central gradient µSt are included in the final aggregation. Gradients that are fairly far away from µSt are excluded, as they are more likely to be influenced by malicious agents. The aggregated gradient µt is computed by the average of all gradients in the benign set Wt . This iterative process, which repeats for T rounds, ensures that the global model w̃T incorporates reliable gradients from honest participants, while progressively filtering out any erroneous or malicious gradients. If the set S is empty, we can increase the value of ψ to ensure that it becomes non-empty. Subsequently, the server employs a stochastic variance reduced gradient (SVRG)-like [38], [39] to update the global model for Nt iterations, where Nt is an integer drawn from a t . During each Geometric distribution with a parameter of ξtξ+b t iteration, the server first samples a mini-batch of bt trajectories from the trajectory distribution d(·|wtn ) induced by the current global model wtn . It then computes a semi-stochastic gradient ζtn (Line 11 in Algorithm 2), which is subsequently used to update the current global model (Line 12 of Algorithm 2). Notably, to reduce the learning variance, the control term d(υ n |w0 ) n n δ(υt,j |wtn , wt0 ) is calculated as δ(υt,j |wtn , wt0 ) = d(υnt,j|wnt ) . t,j
t
Remark: According to Eq. (1), our SecApp requires computing pairwise distances between gradients, which can lead to significant computational overhead, particularly when the number of agents K is large. This overhead grows quadratically, with a complexity of O(K 2 ), making it impractical in many scenarios. To mitigate this issue, approximate nearest neighbor (ANN) search [40], [41] is employed. This approach leverages efficient data structures such as KD-Trees [42], Ball Trees [43], or locality-sensitive hashing (LSH) [44] to identify gradients within a distance ψ of a given gradient without explicitly calculating all pairwise distances. While ANN search operates with sub-linear time complexity for neighbor identification, it yields approximate rather than exact results. However, these approximations are typically adequate for consensus-based methods like the proposed SecApp.
6
Algorithm 3 SecApp 1: Input: w̃0 ∈ Re , batch size ξt , mini-batch size bt , step
size ηt , threshold ψ, scaling factor λ 2: for t = 1 to T do 3: wt0 ← w̃t−1 4: for k = 1 to K in parallel do 5: µkt = LocalTraining(wt0 , ξt , d(·|wt0 )) 6: end for 7: S = {µnkt } where k ∈ [K] s.t. o ′ k ′ ∈ [K] : µkt − µkt ≤ ψ > K 2 µSt ← argminµk̃ ∥µk̃t − mean(S)∥ where k̃ ∈ S t 9: Wt = k ∈P[K] : µkt − µt−1 ≤ λ µSt − µt−1 10: µt = |W1 t | k∈Wt µkt t 11: Sample steps Nt ∼ Geom( ξtξ+b ) t 12: for n = 0 to Nt − 1 do n bt 13: Sample bt trajectories {υt,j }j=1 from d(·|wtn ) Pbt 1 n n n 14: ζt = bt j=1 [g(υt,j |wt )− n n |wtn , wt0 )g(υt,j |wt0 )] + µt δ(υt,j n+1 15: wt = wtn + ηt ζtn 16: end for 17: w̃t ← wtNt 18: end for 19: Output: w̃a uniformly randomly picked from {w̃t }T t=1 8:
V. T HEORETICAL A NALYSIS OF ROBUSTNESS G UARANTEES To ensure SecApp’s applicability for safety-critical scenarios, we present the first-of-its-kind theoretical foundations underlying the convergence and sample complexity of FRL-based robust aggregation in SecApp. We use πw to denote the policy parameterized by w. Let H be the trajectory horizon, i.e., the length of the trajectory. Our objective is to maximize the cumulative discounted reward for a trajectory υ. The action space and state space are denoted by A and Ω, respectively. The cumulative PH−1 discounted reward for υ is represented by R(υ) = h=0 γh R (sh , ah ), where R (sh , ah ) : Ω × A 7→ [0, R] is the reward the agent receives after being in state sh and taking action ah , and γh is the discount factor. In RL, the loss function L(w) = Eυ∼d(·|w) [R(υ)|Ξ] is used to evaluate the performance of model w, given the MDP Ξ. It is important to highlight that the loss function L(w) exhibits significant non-convexity.
s ∈ Ω, log-gradient and Hessian of policy satisfy the bounds: ∥∇2w log πw (a|s)∥ ≤ M.
|∇w log πw (a|s)| ≤ G,
Remark. Assumption 1 is commonly employed in various existing works [39], [45]–[47], which bounds the variance of the gradient estimator with a constant σ. This ensures that the gradient estimates remain controlled across different agents, which is critical given the stochastic nature of policy gradients in federated settings. Additionally, we note that Assumption 1 always hold true for complex, real-world problems involving continuous and high-dimensional controls, particularly when the MDP satisfies a Lipschitz continuity condition, as detailed in [48]. Assumption 2 is standard in the literature of RL system (see, e.g., [25], [32]), which introduces a finite bound Q on the variance of importance weights. This constraint helps prevent large fluctuations in importance sampling corrections, thereby stabilizing the learning process. Lastly, Assumption 3 sets specific bounds, denoted by G and M , on the log-gradient and Hessian of the policy. These bounds ensure that policy gradients remain smooth and controlled, where gradients are aggregated from multiple agents. This assumption is widely used in the existing literature [25], [32], [47], [49]. Theorem 1 (Convergence of SecApp). Given the conditions specified in Assumptions 1, 2, and 3, assume that the objective 2 ) , and function L(w) is Lℓ -smooth, where Lℓ = HM (R+HG 1−γ 1 that the fraction of malicious agents satisfies α < 2 , so that benign gradients form the majority required by the filtering 1 rule in Eq. (1). If the step size ηt satisfies ηt ≤ 2 , 2τ ξt3
with bt = 1 and ξt ≥ 4Φ , where Φ = Lg + Zg2 Zw and L2 ℓ
b |) τ = (Lℓ Φ) 3 , the parameters are given by Lg = HM (R+|Z , 1−γ HG(R+|Zb |) , Zb denotes the baseline reward, and Zg = 1−γ Zw = H(Q+1)(2HG2 +M ) withq Q defined in Assumption 2. 2K Let V = 2 log θ and ψ = 2σ ξVt , where the confidence 1
parameter θ ∈ (0, 1) is chosen as θ = θξt 2(1−2θ)
V 4ξt
such that
ξt 2
e ≤ 2K θ ≤ e . The scaling factor is chosen by ψ−∥µS t −µt−1 ∥ λ ≤ ∥µS −µ . At the end of T training rounds, the model t−1 ∥ t w̃a , selected uniformly at random from {w̃t }Tt=1 , satisfies the convergence bound: E[∥∇L(w̃a )∥2 ] ≤
8τ [L(w̃∗ ) − L(w̃0 )] 1 3
+ 16ψ 2 +
T ξt
512σ 2 V , ξt
∗
Assumption 1 (Bounded variance of the gradient estimator). It is assumed that there exists a constant σ such that for any trajectory υ ∼ d(υ|w), the gradient estimator g(υ|w) satisfies ∥g(υ|w) − ∇L(w)∥ ≤ σ for every policy πw . Assumption 2 (Variance of importance weights). It is assumed that there exists a finite constant Q such that for any pair of policies, the variance of the importance weights δ(υ|w1 , w2 ) is bounded by Q, for all w1 , w2 ∈ Re and υ ∼ d(·|w1 ), where e is model dimension. Assumption 3 (Policy gradient properties). Let πw (a|s) denote the policy of an agent in state s. There exist positive constants G and M such that, for any action a ∈ A and state
where w̃ represents the global maximizer of L. Proof. Due to the page limit, we only provide a proof sketch. The full proof can be found in Appendix A. Starting the proof of Theorem 1 from the update equation wtn+1 = wtn + ηt ζtn , we have: Eυtn ∥wtn+1 − wt0 ∥2 = Eυtn ∥wtn − wt0 + ηt ζtn ∥2 ≤ ηt2 [(2Lg + 2Zg2 Zw )∥wtn − wt0 ∥2 + 2∥∇L(wtn )∥2 + 2∥µt − ∇L wt0 ∥2 ] + 2ηt ⟨µt − ∇L wt0 , wtn − wt0 ⟩ + 2ηt ⟨∇L(wtn ), wtn − wt0 ⟩ + ∥wtn − wt0 ∥2
(3)
7
Next, let Et denote the expectation with respect to all trajectories {υt1 , υt2 , . . . }, given Nt . Since the trajectories are independent of Nt , Et can be viewed as the expectation over {υt1 , υt2 , . . . }. Therefore, we have:
We want to choose ηt such that 1 − 2ηt2 (Lg + Zg2 Zw )ξt − 2Lℓ ηt3 (Lg + Zg2 Zw )ξt2 > 0. Denoting Φ = Lg + Zg2 Zw , we have the following equation: 1 > 1 − 2ηt2 Φξt − 2Lℓ ηt3 Φξt2 > 0 1 1 > 2 3 2 2(1 − 2ηt Φξt − 2Lℓ ηt Φξt ) 2
Et ∥wtn+1 − wt0 ∥2 ≤ [(2Zg2 Zw + 2Lg )ηt2 + 1]Et ∥wtn − wt0 ∥2 + 2ηt Et ⟨∇L(wtn ), wtn − wt0 ⟩ + 2ηt Et ⟨µt − ∇L wt0 , wtn − wt0 ⟩ + 2ηt2 Et ∥∇L(wtn )∥2 + 2ηt2 ∥µt − ∇L
Thus, we can select ηt such that: wt0
2 ∥ .
(4)
Taking the expectation over Nt with n = Nt , following Fubini’s theorem and replacing replace wtNt with w̃t , and wt0 with w̃t−1 , taking the expectation over the equation, we have − 2ηt E⟨µt − ∇L wt0 , w̃t − w̃t−1 ⟩ 1 E∥w̃t − w̃t−1 ∥2 ≤ ηt2 (2Lg + 2Zg2 Zw ) − ξt + 2ηt E⟨∇L(w̃t ), w̃t − w̃t−1 ⟩ + 2ηt2 E∥∇L(w̃t )∥2 + 2ηt2 E∥µt − ∇L wt0 ∥2 . (5) Rearrange the terms, it follows E [DN − DN +1 ] = 1 − Γ1 (E[DN ] − D0 ) with Fubini’s theorem. Note that t and w̃t−1 = w0t . If we take expectation over w̃t = wN t all randomness and denote it by E, we get: ηt (1 − Lℓ ηt )E∥∇L(w̃t )∥2 1 ≤ E [L(w̃t ) − L(w̃t−1 )] ξt 1 1 2 2 − + ηt (2Lg + 2Zg Zw ) E∥w̃t − w̃t−1 ∥2 + 2ηt ξt ξt 1 + E ⟨∇L(w̃t ), w̃t − w̃t−1 ⟩ ξt 2 ηt ηt + E∥∇L(w̃t )∥2 + E µt − ∇L wt0 ξt ξt + Lℓ ηt2 (Lg + Zg2 Zw )E∥w̃t − w̃t−1 ∥2 2 , (6) + ηt (1 + Lℓ ηt )E µt − ∇L wt0 We apply Young’s inequality, that For any real numbers x and 2 y, and for any ρ > 0, the following holds: xy ≤ x2ρ + ρ2 y 2 , on E ⟨∇L(w̃t ), w̃t − w̃t−1 ⟩ using x = w̃t − w̃t−1 , y = ∇L(w̃t ), 1−2ηt2 (Lg +Zg2 Zw )ξt −2Lℓ ηt3 (Lg +Zg2 Zw )ξt2 and ρ = to get: ηt ξt 1 1 ρ E ⟨y, x⟩ ≤ E∥x∥2 + E∥y∥2 . ξt 2ρ 2
1 2[1 − 2ηt2 (Lg + Zg2 Zw )ξt − 2Lℓ ηt3 (Lg + Zg2 Zw )ξt2 ] ! 1 E∥∇L(w̃t )∥2 + 1 − Lℓ ηt − ξt
≤
−
1 E [L(w̃t ) − L(w̃t−1 )] ξt 1 2 2V + ηt (1 + Lℓ ηt + ) 2ψ + 64σ , ξt ξt
(9)
Next, we aim to satisfy the condition: 1 1 5 < ≤ 2 3 2 2 2[1 − 2ηt Φξt − 2Lℓ ηt Φξt ] 8 1 1 1 (ii)Lℓ ηt ≤ (iii) ≤ 16 ξt 16 (i)
To ensure the conditions are satisfied, we choose ηt = 12/3 2τ ξt when ξt ≥ 16 which satisfies (9). We can derive the following from (8) and (9). Replacing ηt = 12/3 and telescoping over 2τ ξt t = 1, 2, ..., T with a constant batch size ξt , we have for w̃a uniformly sampled from {w̃t }Tt=1 : E∥∇L(w̃a )∥2 ≤
8τ [L(w̃∗ ) − L(w̃0 )] 1
T ξt3 + 16ψ 2 + 512σ 2
V , ξt
which completes the proof. Remark. Theorem 1 demonstrates that, with appropriately chosen parameters and fewer than half of the agents being malicious, the learned model converges to a first-order stationary point of L even in Byzantine environments. In RLbased systems, where the loss function is highly non-convex, convergence to a stationary point represents the best possible guarantee. We also note that the analysis is carried out for the mini-batch size bt = 1 following [39]; the same argument extends to a general mini-batch size bt , with the corresponding variance terms scaled by b1t . VI. P ERFORMANCE E VALUATION A. Experimental Setup
(7)
Combining (6) and (7) and rearranging, we have: ηt
1 1 1 − ≥ ξt 2[1 − 2ηt2 Φξt − 2Lℓ ηt3 Φξt2 ] 4 1 1 3 Lℓ ηt + + ≤ ξt 2[1 − 2ηt2 Φξt − 2Lℓ ηt3 Φξt2 ] 4
1 − L ℓ ηt −
(8)
1) Scenario Configurations: FRL-based AV environments and DT setup: In our experiments, we gather real-world vehicular data at the I-695 highway segment in Baltimore, Maryland, using drones. Given the dangers and restrictions associated with testing on actual highways, we first develop the HighwayDT based on the collected data, tailored for evaluating FRL-based AV systems in controlled, realistic settings. This DT is a virtual replica of a physical autonomous driving system in highway scenarios, modeling its dynamics and behavior in real time to enable continuous monitoring, simulation, and optimization. HighwayDT builds upon the static environment by incorporating additional functionalities to enhance its capabilities. Figure 4 illustrates the minimal
Distance related DDPG soft update factor
8
𝑛 ∗ Environment Generated
Highway Car-Following Edge-Case Environment Highway Car-Following Edge-Case Environment Highway Car-Following Edge-Case Environment Twin-aware Highway Case Environment Rear Vehicle Ego Vehicle (Sedan) Front Vehicle Rear Vehicle(Truck) (Truck) Ego VehicleEdge (Sedan) Front Vehicle(Sedan) (Sedan) Rear Ego Front RearVehicle Vehicle(Truck) (Truck) EgoVehicle Vehicle(Sedan) (Sedan) FrontVehicle Vehicle(Sedan) (Sedan)
Deep Deterministic Policy Gradient (DDPG) (2)
Critic Network 𝑑𝑑 𝑑𝑑
𝑑𝑑 𝑑𝑑
(1) Distance Gap: Distance Gap:𝑑𝑑𝑑 , ,𝑑𝑑, 𝑑 Velocity: Velocity:𝑣𝑣𝑣, ,𝑣𝑣, 𝑣, ,𝑣𝑣, 𝑣Acceleration: Acceleration:𝑎𝑎𝑎, ,𝑎𝑎, 𝑎, ,𝑎𝑎, 𝑎 Distance DistanceGap: Gap: 𝑑 , 𝑑 Velocity: Velocity: 𝑣 , 𝑣 , 𝑣 Acceleration: Acceleration: 𝑎 , 𝑎 , 𝑎
Input layer
… Scenario 2
Scenario 3
… Scenario 4
Scenario 𝑛
Scenario 5
𝑑 𝑑 𝑣 𝑣 𝑣 𝑎 𝑎 𝑎
1 245 38 2798 … 129 … 1832
3 Hidden layers
Actor Network
Digital Twins Verification
Scenario 1
Output layer
Reward
(1): TD Error (2): Actor Critic Network
Action Input layer
Output layer 3 Hidden layers
𝑎
𝑎: 𝐸𝑔𝑜 𝑉𝑒ℎ𝑖𝑐𝑙𝑒 𝐴𝑐𝑐𝑒𝑙𝑎𝑟𝑡𝑖𝑜𝑛
Fig. 3: Framework of vehicular RL training and validation in HighwayDT. sim-to-reality gap achieved by our DT system. To emulate realistic highway scenarios, we incorporate the Intelligent Driver Model (IDM) for vehicle-following model calibration, real-world vehicle trajectories, and additional physical constraints into a SUMO-CARLA co-simulator. Specifically, it constructs highly heterogeneous autonomous driving scenarios, focusing on critical aspects such as longitudinal control and collision avoidance, as depicted in Figs. 3 and 5. The virtual development environment supports a closed-loop process, encompassing safety-critical scenario generation, highfidelity simulations, and predictive analytics. It is also capable of generating numerous parallel scenarios that allow RL agents to learn and make reliable decisions under complex and everchanging driving conditions. More importantly, the predictive analytics capabilities allow for the simulation of future driving scenarios based on a real-time data stream, helping identify potential risks and test the RL agents’ responses to unforeseen events, as shown in the bottom-left part of Fig. 3. It is worth noting that the agents trained within such emulators eventually perform better than the current integration of Adaptive Cruise Control (ACC) and Automatic Emergency Braking (AEB) systems because of enriched features as evidenced by existing works [50], [51]. The rigorous validation processes ensure that only algorithms such as SecApp meeting high safety and performance standards are considered for deployment in open road tests. This includes testing performance under varied vehicle behaviors and rare crash scenarios. Fig. 5 visualizes two scenarios generated by HighwayDT, where Fig. 5a and Fig. 5c demonstrate a successful vehicle
Distance (m)
11
Real Distance HighwayDT Distance
10 9 8 7 6
0
2
4
6
Time (seconds)
8
10
12
Fig. 4: Fidelity test samples of inter-vehicle following distances in our HighwayDT and the real-world tests.
control scenario, and Fig. 5b and Fig. 5d demonstrate the failed cases. The first six subfigures of Fig. 5a or Fig. 5c display three perspectives (Top, Side, and 3rd Person view) for both the leading and following vehicles. In the successful avoidance scenario, Fig. 5a shows the initial step of the RL agent, while Fig. 5c shows the vehicle fully stopped with no collision. The red vehicle represents the leading vehicle, the yellow ones are controlled by an RL agent, and the blue is the follower. In the unsuccessful scenario, Fig. 5b shows the initial step of an attacked RL agent, and Fig. 5d depicts a collision between the leading and the 1st vehicle. The red-dashed vehicle highlights the collision, while yellow and blue vehicles represent the RLassisted vehicles and the following vehicles, respectively. FRL training framework: As shown in Fig. 3, all RL agents are responsible for controlling the middle ego vehicles in their respective DTs. Note that in our experiments, the FRL-based system controls only the three middle ego vehicles, which are depicted as the three yellow vehicles in Fig. 5c. The entire process operates within a continuous state space represented by an eight-dimensional vector (distance gaps between the leading and RL vehicles (df m ) and between the RL vehicles and the following vehicles (dmr ), along with the velocities (vf , vm , vr ) and accelerations (af , am , ar ) of the leading, RL, and following vehicles. The state space is continuous and entirely sensor-based, derived from actual measurements such as position, velocity, and acceleration rather than discrete time components, which allows the FRL algorithm to align seamlessly with the environmental modeling. The action space which ranges from 3m/s2 to −12m/s2 continuously is defined to control the acceleration of the middle ego RL vehicle agents through a 5-layer neural network (detailed in the supplementary material) that aims to maximize the expected return. Each RL vehicle receives a reward for each successful action (representing no collision) and receives a large penalty if a collision occurs. We exploit the policy gradient [33] to perform local RL training for each vehicle within the DTs. Parameter settings: We assume that there are a total of 10 agents that are assigned to different scenarios. By default, 20% of these agents are considered malicious, following the standard settings as in [14], [15], [18], [19], [21], [22], [34],
9
Time: Start
Top view (Following vehicle)
Time: Start
Top view (Leading vehicle)
Side view (Following vehicle)
3 Person view (Following vehicle)
Side view (Leading vehicle)
3 Person view (Leading vehicle)
Top view (Following vehicle)
Top view (Leading vehicle)
Side view (Following vehicle)
3 Person view (Following vehicle)
Side view (Leading vehicle)
3 Person view (Leading vehicle)
(a) Vehicle positions at the start of the (b) Vehicle positions at the start of the successful scenario. failed scenario. Time: Collision Step Time: Fully Stop
Side view (Following vehicle)
3 Person view (Following vehicle)
Side view (Leading vehicle)
3 Person view (Leading vehicle)
Top view (Following vehicle) Top view (Following vehicle)
Top view (Leading vehicle) Leading Vehicle 1st RL Vehicle 2nd RL Vehicle
Top view (Leading vehicle) Leading Vehicle Leading Vehicle 1st RL Vehicle 1st RL Vehicle nd Vehicle 2nd 2RLRL Vehicle
Side view (Following vehicle)
3 Person view (Following vehicle)
Side view (Leading vehicle)
3 Person view (Leading vehicle)
Crash into Sky Crash into Sky
3rd RL Vehicle 3rd RL Vehicle
3rd RL Vehicle Following Vehicle Following Vehicle
Following Vehicle
Vehicle type third person view diagram Vehicle type third person view diagram
(d) Vehicle positions when a collision (c) Vehicle positions when fully stopped. happens. Fig. 5: Visualization of two scenarios generated by HighwayDT. Fig. 5a and Fig. 5c depict instances of successful actions taken by the RL-assisted vehicle, while Fig. 5b and Fig. 5d illustrate instances of failed actions that eventually lead to crashes. [52], [53]. In the AD scenario, the evaluation advances in discrete time steps, updating vehicle positions and velocities at each interval, starting from an initial time step of 0.01s. A safety threshold of 3 meters – equivalent to the length of the RL vehicles – is maintained to prevent collisions by ensuring a minimum distance between vehicles. The following vehicle’s braking behavior is modeled with a normally distributed deceleration of around −6m/s2 . Parameters related to environment configurations, agent training, and those in our method are provided in the supplementary material. 2) Poisoning Attacks: By default, we employ the following eight untargeted attacks to evaluate the effectiveness of our proposed SecApp in practical FRL scenarios: Trim attack [14], Random attack [54], History attack [54], MPAF attack [54], FTI attack [16], MinMax attack [17], MinSum attack [17], and Adaptive attack [17]. It is worth noting that the Adaptive attack represents the worst-case scenario, where the attacker has full knowledge of the FRL system. Detailed descriptions of these eight attacks can be found in the supplementary materials. Additionally, we evaluate three backdoor attacks specifically designed for RL-based systems in Appendix D, with attack details provided in Appendix F.
3) Baseline Comparison.: We compare SecApp with eleven security rules, including one non-robust aggregation rule (FedAvg [23]) and ten robust aggregation rules (Median [34], Trimmed-mean (Trim) [34], Krum [24], FoolsGold [19], FABA [22], FLTrust [15], FLAIR [21], FedPGBR (FedPG) [18], FLAME [53], and DeepSight [52]). Note that among the ten robust aggregation rules, only FedPG is specifically designed for the FRL system to date, while the other nine methods were originally developed for FL systems but can be adapted to the FRL setting. Comprehensive descriptions of these aggregation rules are provided in the supplementary materials (Appendix C). These baseline schemes are implemented by replacing Lines 7 to 10 in Alg. 3. All other modules stay the same for a fair comparison and stable model training process. B. Evaluation Metrics Similar to the practical AD scenarios, both the leading and following vehicles decelerate from an initial speed to a full stop. The distance covered during this deceleration for the 2 leading vehicle can be calculated by the formula 2aVleading , where V is the starting velocity and aleading is the deceleration rate of the leading vehicle. Likewise, the following vehicle decelerates
10
2
V from its initial speed to a stop over a distance given by 2afollowing , where afollowing represents the deceleration rate of the following vehicle. To ensure that the middle vehicle can safely maintain its position between the leading and following vehicles without causing a collision, the following condition must be satisfied:
dcollision ≥
Vr2init − Vr2final V 2 − Vf2final − f init , 2afollowing 2aleading
(10)
where dcollision is the length of the middle vehicle; Vr init and Vf init are respectively the initial velocities of following and leading vehicles; Vr final and Vf final are respectively the final velocities of following and leading vehicles; and afollowing and aleading are respectively the deceleration rates of following and leading vehicles. Any cases that violate this condition will lead to collisions. The no-collision rate β can be defined as: nno-collision , (11) β= ncollision + nno-collision where nno-collision and ncollision mean the number of successful control cases (i.e., no-collision) and collisions, respectively. We measure the no-collision rate across over 50K scenarios as our primary performance metric. However, some scenarios are inherently collision-prone regardless of the middle vehicle’s actions. Scenarios that violate natural laws are excluded from our evaluation. All parameters are directly obtained from HighwayDT rather than being defined by humans. The larger the no-collision rate β, the better the performance of the implemented scheme. Note that in typical real-world AD scenarios, at least a no-collision rate of 99.95% during the validation process is required for an RL-based approach to be considered successful and ready to implement [55], [56]. C. Experimental Results SecApp is effective: The results presented in Table III demonstrate the superior performance of our proposed SecApp over other baseline schemes in mitigating the impact of various poisoning attacks within the vehicular driving environment. “No attack” indicates that all RL agents are benign, with no malicious agents present in the system. Across all attack scenarios, SecApp consistently achieves a 100% no-collision rate, outperforming all other baseline defense mechanisms, which struggle to maintain similar performance under the same attacks. For instance, under the Random attack, SecApp attains a perfect 100% no-collision rate, while other methods such as FABA and FLAIR only achieve 14.54% and 11.20%, respectively. The success of SecApp in this scenario can be attributed to its ability to detect and filter out anomalous gradients based on historical aggregated gradients, which are randomly introduced by the attackers, ensuring that only benign gradients are aggregated. One step further, Table IV shows the false positive rate (FPR) and false negative rate (FNR) of our SecApp under various attacks, where FPR denotes the fraction of benign agents incorrectly classified as malicious, and FNR is the fraction of malicious agents incorrectly classified as benign. As shown in Table IV, we observe that both the FPR and FNR of our SecApp fall below 5%, indicating an effective filtering performance. Detailed analysis indicates that SecApp excels particularly in scenarios
where other defenses exhibit vulnerabilities. For instance, during the History attack, defenses such as FLAME show significant drops in performance with no-collision rates of only 10.24%. The great performance of SecApp can be explained by its ability to recognize and disregard historical patterns that are exploited by attackers to introduce subtle but harmful changes into the aggregated gradient. Note that our SecApp may misclassify few malicious agents as benign, as shown by a low FNR, but these misclassified malicious agents minimally affect the overall system. Impact of fraction of malicious agents: Fig. 6 explores the impact of fraction of malicious agents on the no-collision rate. As the proportion of malicious agents rises, most conventional methods such as FedAvg and Trim exhibit a steep decline in their ability to maintain a high no-collision rate. This declining trend is consistent across other attack scenarios, as shown in Figs. 6b-h, where methods like FoolsGold, FLTrust, and Krum also display significant vulnerabilities to larger fraction of malicious agents, resulting in frequent collisions. Conversely, SecApp consistently sustains a 100% no-collision rate even when the fraction of malicious agents is 40%. This outstanding performance is due to SecApp’s ability to detect the impact of malicious gradients and augmented experiences from DTs, thereby preserving the integrity of the decision-making. With over 50% malicious agents, SecApp experiences a significant drop in the no-collision rate. However, such a high proportion of malicious agents is unlikely in real-world scenarios. Impact of scaling factor λ: The selection of the scaling factor λ from Eq. (2) in SecApp is critical for optimal defensive performance, as it controls the allowable rate of change in model updates, preventing excessive deviations and aiding in the identification of benign gradients. From Table V, it is observed that when λ is set too high, such as 1,000, the system tends to include too many system parameters in the aggregation process. This over-inclusion can allow malicious gradients to be included, leading to a significant drop in performance. For example, under the MinSum attack, the nocollision rate drops to 18.49% when λ = 1, 000, compared to 100.0% when λ is set to 10. This decline indicates that a large λ compromises the system’s ability to filter out harmful system parameters. However, when λ is set fairly low, such as 0.1, the system becomes overly conservative, i.e. unconsciously filtering out a substantial number of benign gradients. This leads to inadequate gradient aggregation and poor decisionmaking processes. As a result, an appropriately chosen λ is critical to run SecApp, aiming to balance the inclusion of enough benign gradients to make accurate decisions while effectively filtering out potential threats. Performance against more complicated attacks: The “a little is enough” (LIE) attack [57] and Krum attack [14] are more adaptive and complicated poisoning attack strategies designed to exploit vulnerabilities in FL systems by subtly manipulating gradients to degrade overall performance without being detected by conventional anomaly detection mechanisms. The LIE attack injects minor, seemingly harmless changes into gradients that accumulate over time, significantly distorting the final aggregated model. This attack is
11
TABLE III: No-collision rate of different methods under different attacks. Attack
FedAvg
Median
Trim
Krum
FoolsGold FABA
FLTrust
FLAIR
FedPG
FLAME
Deepsight SecApp
No attack
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
Trim attack
14.97%
7.05%
7.05%
100.0%
8.63%
11.79%
100.0%
9.12%
100.0%
100.0%
100.0%
100.0%
Random attack
14.25%
100.0%
100.0%
100.0%
100.0%
14.54%
100.0%
11.20%
100.0%
100.0%
100.0%
100.0%
History attack
8.43%
100.0%
14.88%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
10.24%
100.0%
100.0%
MPAF attack
10.91%
100.0%
100.0%
13.26%
100.0%
14.05%
8.55%
100.0%
100.0%
100.0%
100.0%
100.0%
FTI attack
12.47%
14.01%
8.36%
11.02%
100.0%
9.51%
12.89%
100.0%
10.14%
100.0%
14.22%
100.0%
MinMax attack
12.73%
100.0%
100.0%
1.69%
8.09%
9.87%
10.24%
14.68%
13.76%
11.95%
14.34%
100.0%
MinSum attack
10.29%
13.77%
100.0%
9.49%
11.58%
100.0%
8.35%
12.14%
14.19%
100.0%
100.0%
100.0%
Adaptive attack
14.49%
11.49%
7.11%
13.68%
9.84%
100.0%
13.94%
11.19%
8.97%
8.92%
12.33%
100.0%
(e) FTI attack
FoolsGold Deepsight
FABA SecApp
100 80 60 40 20 0 0 10 20 30 40 50 60
100 80 60 40 20 0 0 10 20 30 40 50 60
(b) Random attack
(c) History attack
(d) MPAF attack 100 80 60 40 20 0 0 10 20 30 40 50 60
No-collision rate (%)
100 80 60 40 20 0 0 10 20 30 40 50 60
Fraction of malicious agents (%)
100 80 60 40 20 0 0 10 20 30 40 50 60
100 80 60 40 20 0 0 10 20 30 40 50 60
(f) MinMax attack
(g) MinSum attack
Fraction of malicious agents (%)
No-collision rate (%)
No-collision rate (%)
No-collision rate (%)
Fraction of malicious agents (%)
Trim FLAME
Fraction of malicious agents (%)
(a) Trim attack
100 80 60 40 20 0 0 10 20 30 40 50 60
Krum FedPG
No-collision rate (%)
Fraction of malicious agents (%)
Median FLAIR
No-collision rate (%)
100 80 60 40 20 0 0 10 20 30 40 50 60
No-collision rate (%)
No-collision rate (%)
FedAvg FLTrust
Fraction of malicious agents (%)
Fraction of malicious agents (%)
Fraction of malicious agents (%)
(h) Adaptive attack
Fig. 6: Impact of fraction of malicious agents.
TABLE IV: FPR and FNR (%) of SecApp under various attacks. Attack
FPR
FNR
No attack Random attack History attack Trim attack FTI attack MPAF attack MinMax attack MinSum attack Adaptive attack
2.5 1.2 0.7 3.2 3.8 2.1 1.9 2.3 0.4
– 2.0 0.3 1.5 0.8 1.3 1.0 1.1 0.1
TABLE V: Performance on different scaling parameter λ. Attack No attack Trim attack Random attack History attack MPAF attack FTI attack MinMax attack MinSum attack Adaptive attack
0.1
1
10
100
1000
100.0% 11.23% 19.54% 12.47% 23.45% 15.67% 13.89% 18.76% 20.32%
100.0% 17.89% 22.13% 100.0% 12.34% 100.0% 100.0% 15.43% 100.0%
100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%
100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 22.54% 34.72%
100.0% 33.26% 100.0% 100.0% 19.42% 100.0% 100.0% 18.49% 23.29%
particularly challenging to be detected because the individual gradients closely resemble normal ones, making it difficult for basic statistical methods to identify them as malicious. Conversely, the Krum attack targets Byzantine-robust aggregation methods by crafting malicious gradients that are sufficiently similar to the majority to be included in the aggregation process, yet are strategically designed to degrade the model’s performance. Table VI illustrates the resilience of SecApp against LIE and Krum attacks. These attacks are specifically engineered to circumvent traditional defense mechanisms by subtly altering gradients in ways that evade detection by basic anomaly detection methods. For example, under the LIE attack, SecApp retains a perfect 100.0% no-collision rate, whereas methods like FoolsGold and FLAIR, which employ less sophisticated detection algorithms, see their no-collision rates drop to 20.02% and 18.71%, respectively. This sharp contrast underscores SecApp’s effectiveness in identifying and neutralizing even well-camouflaged malicious gradients. More experiments: Additional experiments and ablation study can be found in Appendix D, including how SecApp reacts to backdoor attacks, an investigation of the impact of the total number of vehicle agents, the impact of threshold ψ, and the effectiveness of adaptive scaling factor λ. The application of our SecApp in an edge caching scenario is presented in Appendix E. We also discuss the limitation of SecApp in Appendix G.
12
TABLE VI: Performance on advanced attacks. Attack
FedAvg
Median
Trim
Krum
FoolsGold FABA
FLTrust
FLAIR
FedPG
FLAME
Deepsight SecApp
No attack LIE attack Krum attack
100.0% 11.48% 28.57%
100.0% 10.17% 23.97%
100.0% 23.17% 12.68%
100.0% 27.29% 22.86%
100.0% 20.02% 100.0%
100.0% 100.0% 100.0%
100.0% 18.71% 10.99%
100.0% 100.0% 17.06%
100.0% 29.24% 100.0%
100.0% 100.0% 100.0%
100.0% 100.0% 100.0%
Algorithm 4 FRL training process without SVRG procedure.
5: µkt = LocalTraining(t, k, wt0 , ξt , d(·|wt0 )) 6: end for P K 1 k ▷ FedAvg [23] 7: µt ← K k=1 µt 0 8: w̃t = wt + ηt µt 9: end for 10: Output: w̃T
Time(s)
60 50 40 30 20 10 0
Fed A Me vg dia n Tri m K Fo rum ols Go ld FA BA FLT rus FL t AI Fed R P F G De LAME ep sig Se ht cA pp
100.0% 100.0% 10.27%
1: Input: w̃0 ∈ Re , batch size ξt , step size ηt 2: for t = 1 to T do 3: wt0 ← w̃t−1 4: for k = 1 to K in parallel do
Fig. 7: Computational cost of different schemes. VII. C OMPUTATIONAL E FFICIENCY Fig. 7 illustrates the computational costs associated with various defensive schemes, where the computational cost refers to the time required to aggregate gradients over 2,000 rounds. Note that the training process is conducted on a server equipped with NVIDIA RTX 4090 and INTEL 13900K. As illustrated in Fig. 7, SecApp incurs a lower cost of 10 seconds, which is significantly lower than more complex techniques such as FABA and FLTrust, which are 46 and 48 seconds, respectively. Overall, SecApp incurs minimal overhead compared to generic approaches like FedAvg. This demonstrates its ability to balance efficiency and robustness, making it well-suited for secure, large-scale FRL systems. VIII. R ELATED W ORKS Model Poisoning Attacks: Model poisoning attacks significantly compromise the robustness and reliability of FL systems, as documented in several studies [14], [16], [17], [54], [57]–[60]. For instance, the research in [14] reveals how attackers can leverage Byzantine fault tolerance mechanisms in FL to manipulate model updates. Further, the study in [54] demonstrates the escalation of this threat by introducing fake clients that exacerbate the damage through scaled attacks. Additionally, the work in [17] focuses on optimizing these poisoning strategies to maximize their disruptive impact on FL systems. However, these sophisticated attack techniques remain under-explored in safety-critical FRL mobile systems. Byzantine-robust Aggregation Rules: Secure aggregation rules [15], [18], [19], [21], [22], [24], [34], [52], [53] in FL are essential for maintaining the integrity and robustness of the learning process, especially in the presence of poisoning attacks. However, the standard approach FedAvg [23] assumes honest participants, making it vulnerable to Byzantine attacks. The authors in [24] addressed this by proposing a method named Krum that enhances the resilience of FL systems against malicious participants. These methods, while effective in standard FL scenarios, struggle in FRL-based systems where dynamic and continuous learning processes can be exploited by sophisticated adversaries. In such systems, the impact of erroneous gradients can be more pronounced, potentially leading
to unsafe behaviors in real-time applications. To address this gap, our work focuses on developing an innovative defense strategy that effectively mitigates security challenges in FRL systems, with both theoretical guarantee and experimental validation in safety-critical scenarios. FL versus FRL: Although FL and FRL both employ distributed training frameworks, their differences are pronounced in the context of mobile computing and systems [13], [36], [61], [62]. In FL, each client processes locally collected data—often static or semi-static—to learn predictive models with or without labels. These clients typically operate under the assumption that their training data are independent and identically distributed (IID), and aggregation schemes assume a convex loss function [15], [34]. In contrast, FRL is inherently more dynamic, as each mobile agent interacts with its environment across multiple time steps, adapting its policy through trial and error while receiving rewards. This means agents are continuously adjusting to varying computational conditions, device mobility, and environmental constraints, leading to highly heterogeneous trajectories. The resulting non-IID and non-convex nature of FRL training makes it significantly more challenging to establish theoretical convergence guarantees, especially in dynamic environments where network topology, latency, and resource availability fluctuate.
IX. C ONCLUSION In this work, we introduced SecApp, a robustness-aware mechanism designed to mitigate the effects of poisoning attacks, particularly applied in safety-critical scenarios such as AD. By employing a multi-step strategy, SecApp effectively identifies and excludes malicious information from distributed agents, thereby enhancing the integrity of the decision-making system. Our theoretical analysis, supported by extensive evaluations using DTs, demonstrates that SecApp is robust against various attacks in safety-critical systems.
13
Eυtn [g(υtn | wtn ) − δ(υtn | wtn , wt0 )g(υtn | wt0 )] and E∥X∥2 = E∥X − EX∥2 + ∥EX∥2 , we further derive that:
A PPENDIX A P ROOF OF T HEOREM 1 Theorem 1 is the main result to illustrate the convergence performance of our proposed SecApp. In the following proof, we adopt a similar setting from [39] to set the mini-batch size bt = 1. In this section, we first present the proofs of several key lemmas, followed by the proof of Theorem 1. Lemma 1 (Function Smoothness). Under Assumption 3, the function L(w) is smooth with a smoothness constant Lℓ . Let g(υ|w) denote the gradient estimators. Then, for any w, w1 , w2 ∈ Re , the following inequalities hold: ∥g(υ|w)∥ ≤ Zg ,
= Eυtn ∥g(υtn | wtn ) − δ(υtn | wtn , wt0 )g(υtn | wt0 ) + µt − (∇L(wtn ) − ∇L(wt0 ) + µt )∥2 + Eυtn [ζtn ] = Eυtn g(υtn | wtn ) − δ(υtn | wtn , wt0 )g(υtn | wt0 ) − (∇L(wtn ) − ∇L(wt0 ))∥2 2 + ∇L(wtn ) − ∇L wt0 + µt . We derive the following equations using inequalities of 2 2∥ . E∥X∥2 ≥ E∥X − EX∥2 and ∥X1 ∥2 + ∥X2 ∥2 ≥ ∥X1 +X 2 2 − (∇L(wtn ) − ∇L(wt0 ))∥2 + ∇L(wtn ) − ∇L wt0 + µt
where Lℓ , Lg , Zg , and Zb are defined in Section V. Lemma 1 is key to obtaining a fast convergence rate, and the detailed proof is provided in [25]. Lemma 2 (Bounded variance of the Algorithm 3’s update with importance sampling). The variance of the Algorithm 3’s update is bounded as follows: 2
2
Eυtn [∥ζtn ∥2 ] ≤ 2Φ wtn − wt0 + 2 ∥∇L(wtn )∥ 1 X k + 2∥ µt − ∇L wt0 ∥2 , |Wt | k∈Wt
where Φ = Lg + Zg2 Zw and γ are defined in Section V and n ∈ Nt .
≤ Eυtn g(υtn | wtn ) − δ(υtn | wtn , wt0 )g(υtn | wt0 ) 2 2 + 2 ∥∇L(wtn )∥ + 2 µt − ∇L wt0 .
Proof. According to Algorithm 3, we have: b
t 1 X n n n [g(υt,j |wnt ) − δ(υt,j |wtn , wt0 )g(υt,j |wt0 )] + µt . bt j=1
k k∈Wt µt ,
P
following the suggestion from [39], to obtain more precise and theoretically sound outcomes. Thus, we have ζtn = g(υtn | wtn ) − δ(υtn | wtn , wt0 )g(υtn | wt0 ) 1 X k + µt . |Wt | k∈Wt
Next, the unbiasedness of the importance sampling can be shown as follows: g(υn |w0 ) =Eυ∼d(·|w0 ) [∇w0 d(w0 )r(υ)] Z d(·|w0 ) = d(·|wn )∇w0 d(w0 )r(υ)dυ d(·|wn ) d(·|wn ) =Eυ∼d(·|w0 ) g(υ|w0 ) . d(·|w0 ) Then we have,
0
Further, we have Eυtn [ζtn ] = ∇L(wtn ) − ∇L wt + µt Given that ∇L(wtn ) − ∇L(wt0 ) =
(12)
Eυtn g(υtn | wtn ) − δ(υtn | wtn , wt0 )g(υtn | wt0 )
2
= Eυtn ∥g(υtn | wtn ) + g(υtn | wt0 ) − g(υtn | wt0 ) − δ(υtn | wtn , wt0 )g(υtn | wt0 )∥2 = Eυtn ∥g(υtn | wtn ) − g(υtn | wt0 ) + (1 − δ(υtn | wtn , wt0 ))g(υtn | wt0 )∥2 ≤ 2Eυtn g(υtn | wtn ) − g(υtn | wt0 )
2 2
.
(13)
Assuming assumptions 2 and 3 hold true and δ(υ | w1 , w2 ) = d(υ|w1 ) 2 d(υ|w2 ) , Zw = H(Q + 1)(2HG + M ). Then, we have Var(δ(υ | w1 , w2 )) ≤ Zw ∥w1 − w2 ∥2 , Varwtn ,wt0 (δ(υtn | wtn , wt0 )) ≤ Zw ∥wtn − wt0 ∥2 .
(14)
The detailed proof of (14) is provided in [25]. By combining (12) and (13), we obtain: Eυtn [∥ζtn ∥2 ] ≤ 2Eυtn g(υtn | wtn ) − g(υtn | wt0 ) + 2Eυtn (1 − δ(υtn | wtn , wt0 ))g(υtn | wt0 ) 2 2 + 2 ∥∇L(wtn )∥ + 2 µt − ∇L wt0 ≤ 2Lg wtn − wt0
2
≤ 2Lg wtn − wt0
2
2
2
+ 2Zg2 Eυtn ∥(1 − δ(υtn | wtn , wt0 ))∥2 2 2 + 2 ∥∇L(wtn )∥ + 2 µt − ∇L wt0 2
+ 2Zg2 Zw wtn − wt0 1 X k 2 µt − ∇L wt0 ∥2 + 2 ∥∇L(wtn )∥ + 2∥ |Wt |
= (2Lg + 2Zg2 Zw ) 2
δ(υ|wn , w0 )g(υn |w0 ) = g(υ0 |w0 ).
2
Additionally, we have
+ 2Eυtn (1 − δ(υtn | wtn , wt0 ))g(υtn | wt0 )
We start the proof by setting bt = 1 and µt = |W1 t |
2
Eυtn g(υtn | wtn ) − δ(υtn | wtn , wt0 )g(υtn | wt0 )
∥g (υ | w1 ) − g (υ | w2 )∥ ≤ Lg ∥w1 − w2 ∥ ,
ζtn =
Eυtn [∥ζtn ∥2 ] = Eυtn ∥ζtn − Eυtn [ζtn ]∥2 + ∥Eυtn [ζtn ]∥2
k∈Wt n 0 2 wt − w t
+ 2 ∥∇L(wtn )∥ + 2∥
1 X k µt − ∇L wt0 ∥2 . |Wt | k∈Wt
We conclude the proof by noting that 2Lg +2Zg2 Zw = 2Φ.
14
Lemma 3 (Gradient alignment analysis). We have the following equation hold, ηt E µt − ∇L wt0 , E∇L(w̃t ) 1 = E µt − ∇L wt0 , w̃t − w̃t−1 ξt 2 . − ηt E µt − ∇L wt0 Proof. Let Htn = ⟨µt − ∇L wt0 , wtn − wt0 ⟩. We then have: Htn+1 − Htn = ηt ⟨µt − ∇L wt0 , ζtn ⟩ Taking the expectation with respect to υtn , it follows that: Eυtn Htn+1 − Htn = ηt µt − ∇L wt0 , Eυtn [ζtn ] 2 = ηt µt − ∇L wt0 , ∇L(wtn ) + ηt µt − ∇L wt0 . When taking the expectation with respect to the random variable, we replace the random gradient in the inner product with its average. Define Et as the expectation over all trajectories {υt1 , υt2 , . . . }, given Nt . Since the trajectories are independent of Nt , Et effectively represents the expectation over {υt1 , υt2 , . . . }. Thus, we obtain: Et [Htn+1 − Htn ] =ηt µt − ∇L wt0 , Et ∇L(wtn ) 2 + ηt µt − ∇L wt0 .
Proof. We aim to bound the error term µt − ∇L wt0 . We begin with: E[∥µt − ∇L wt0 ∥2 ] = E∥µt − ∇f (xt )∥2 2 1 X k (a) µt − ∇f (xt ) = E |Wt | k∈Wt 2 1 X k =E µt − µSt + µSt − ∇f (xt ) |Wt | k∈Wt
(b)
2 1 X k µt − µSt +2 ∥µSt − ∇f (xt )∥2 . | {z } |Wt | k∈Wt E2 {z } |
≤ 2E
E1
Here, step (a) follows from the definition of µtP in Algorithm n 3,P and step (b) follows from the inequality ∥ i=1 ai ∥2 ≤ n 2 n i=1 ∥ai ∥ . Next, we analyze Terms E1 and E2 separately. For Term E1, we have: 2 1 X k E1 = E µt − µSt |Wt | k∈Wt
(c)
1 X E∥µkt − µSt ∥2 ≤ |Wt | k∈Wt
(d)
≤ ψ2 .
again follows from the inequality Setting n = Nt and taking the expectation with respect to Nt , Here, Pn step2 (c) P n 2 ∥ a ∥ ≤ n we have: i=1 i i=1 ∥ai ∥ , and step (d) follows from D Eour proposed Byzantine filtering rule and the choice of S t −µt−1 ∥ ENt Et (HtNt +1 − HtNt ) =ηt µt − ∇L wt0 , ENt Et ∇L(wtNt ) the filtering constant λ ≤ ψ−∥µ . Given inequalities ∥µS t −µt−1 ∥ 2 S k 0 − µ µ − µ ≤ λ µ t−1 t−1 t t + ηt µt − ∇L w . t
By applying Fubini’s theorem and E [DN − DN +1 ] = 1 − Γ1 (E[DN ] − D0 ), where {Dn }n≥0 is a sequence that E∥DN ∥ < ∞ [39], we proceed as follows: Nt
E Et (HtNt +1 − HtNt ) = − Et ENt (HtNt − HtNt +1 ) = 1−
1 ξt ξt +1
!
Ht0 − ENt Et HtNt
1 Nt E Et µt − ∇L wt0 , w̃t − w̃t−1 ξt D E =ηt µt − ∇L wt0 , ENt Et ∇L(wtNt ) 2 + ηt µt − ∇L wt0 .
Taking the expectation over the entire history concludes the lemma. Lemma 4 (Bound on the Error Term). ConsiderAlgorithm 3. The following bound holds for E∥µt − ∇L wt0 ∥2 : E[∥µt − ∇L
∥
∥µkt − µt−1 ∥ ≤ ψ − ∥µSt − µt−1 ∥ ∥µkt − µt−1 ∥ + ∥µSt − µt−1 ∥ ≤ ψ ∥µSt − µkt ∥ ≤ ψ.
=
wt0
ψ−∥µS −µ
t−1 t ≤ ∥µS −µ × µSt − µt−1 = ψ − ∥µSt − µt−1 ∥, and t−1 ∥ t using the triangle inequality, we obtain:
2 V ∥ ] ≤ 2ψ 2 + 64σ 2 . ξt
Here, ξtqand θ are used to determine the filtering threshold ψ = 2σ ξVt , where V = 2 log 2K and θ ∈ (0, 1). θ
We assume that ψ > ∥µSt − µt−1 ∥. Now, consider Term E2. Note that with high probability, ∥µSt −∇f (xt )∥ is bounded by 2ψ, and it is bounded by 8σ almost surely (see [46, Lemma 8]). To bound E2, we define the following events: Event A: We define ∥µSt − ∇f (xt )∥ ≤ 2ψ as Event A, and its complement, Event A!, is defined as ∥µSt − ∇f (xt )∥ > 2ψ. We have P[Event A] ≥ 1 − θ and P[Event A!] ≤ θ. Using these events, we can express E2 as: E2 = E ∥µSt − ∇f (xt )∥2 = P[Event A]E ∥µSt − ∇f (xt )∥2 Event A + P[Event A!]E ∥µSt − ∇f (xt )∥2 Event A! S ≤ E ∥µt − ∇f (xt )∥2 Event A + θ E ∥µSt − ∇f (xt )∥2 Event A! ≤ 4ψ 2 + 64θσ 2 . q V Using the definition ψ = 2σ ξVt and choosing θ = 4ξ , we t get: V E2 = E ∥µSt − ∇f (xt )∥2 ≤ 32σ 2 . ξt
15
After replacing replace wtNt with w̃t , and wt0 with w̃t−1 , taking the expectation over the entire equation, we have
Combining E1 and E2 through step (b), we conclude: V E[∥µt − ∇L wt0 ∥2 ] ≤ 2ψ 2 + 64σ 2 . ξt
Proof. Starting the proof of Theorem 1 from the update equation wtn+1 = wtn + ηt ζtn , we have:
− 2ηt E⟨µt − ∇L wt0 , w̃t − w̃t−1 ⟩ 1 2 2 ≤ ηt (2Lg + 2Zg Zw ) − E∥w̃t − w̃t−1 ∥2 ξt + 2ηt E⟨∇L(w̃t ), w̃t − w̃t−1 ⟩ + 2ηt2 E∥∇L(w̃t )∥2 + 2ηt2 E∥µt − ∇L wt0 ∥2 . (20)
Eυtn ∥wtn+1 − wt0 ∥2 = Eυtn ∥wtn − wt0 + ηt ζtn ∥2 = 2ηt ⟨E [ζtn ], wtn − wt0 ⟩ + ∥wtn − wt0 ∥2 + ηt2 Eυtn ∥ζtn ∥2 ≤ ηt2 [(2Lg + 2Zg2 Zw )∥wtn − wt0 ∥2 + 2∥∇L(wtn )∥2 + 2∥µt − ∇L wt0 ∥2 ] + 2ηt ⟨µt − ∇L wt0 , wtn − wt0 ⟩ (18) + 2ηt ⟨∇L(wtn ), wtn − wt0 ⟩ + ∥wtn − wt0 ∥2 2 2 n 0 2 = [1 + ηt (2Lg + 2Zg Zw )]∥wt − wt ∥ + 2ηt ⟨∇L(wtn ), wtn − wt0 ⟩ + 2ηt ⟨µt − ∇L wt0 , wtn − wt0 ⟩ + 2ηt2 ∥∇L(wtn )∥2 + 2ηt2 ∥µt − ∇L wt0 ∥2 υtn
where inequality (18) follows from the bound on Eυtn ∥ζtn ∥2 derived earlier. Next, let Et denote the expectation with respect to all trajectories {υt1 , υt2 , . . . }, given Nt . Since the trajectories are independent of Nt , Et can be viewed as the expectation over {υt1 , υt2 , . . . }. Therefore, we have:
From the Lℓ -smoothness of the objective function L(w), we have Eυtn [L(wn+1 )] t Lℓ n+1 n+1 n n n n 2 ≥Eυtn L(wt ) + ⟨∇L(wt ), wt − wt ⟩ − ∥wt − wt ∥ 2 2 L η ℓ t Eυtn [∥ζtn ∥2 ] =L(wnt ) + ηt ⟨Eυtn [ζtn ], ∇L(wnt )⟩ − 2 ≥L(wnt ) + ηt ⟨∇L(wnt ) − ∇L wt0 + µt , ∇L(wnt )⟩ Lηt2 [(2Lg + 2Zg2 Zw )∥wnt − w0t ∥2 − 2 + 2∥∇L(wnt )∥2 + 2∥µt − ∇L wt0 ∥2 ] (21) =L(wnt ) + ηt (1 − Lℓ ηt )∥∇L(wnt )∥2 + ηt ⟨µt − ∇L wt0 , ∇L(wnt )⟩ − Lηt2 (Lg + Zg2 Zw )∥wnt − w0t ∥2 − Lηt2 ∥µt − ∇L wt0 ∥2 .
Et ∥wtn+1 − wt0 ∥2 ≤ [(2Zg2 Zw + 2Lg )ηt2 + 1]Et ∥wtn − wt0 ∥2 + 2ηt Et ⟨∇L(wtn ), wtn − wt0 ⟩ + 2ηt Et ⟨µt − ∇L wt0 , wtn − wt0 ⟩ + 2ηt2 Et ∥∇L(wtn )∥2 + 2ηt2 ∥µt − ∇L
where (21) follows from Lemma 2. Use Et to denote the expectation with respect to all trajectories {υt1 , υt2 , ...}, given Nt . Since {υt1 , υt2 , ...} are independent of Nt , Et is equivalently the expectation with respect to {υt1 , υt2 , ...}. The above inequality gives
wt0 ∥2 .
Taking the expectation over Nt with n = Nt and following Fubini’s theorem, we obtain:
+ ηt (1 − Lℓ ηt )Et ∥∇L(wnt )∥2 + ηt Et ⟨µt − ∇L wt0 , ∇L(wnt )⟩
D E − 2ηt ENt Et µt − ∇L wt0 , wtNt − wt0 2 ≤ 1 + ηt2 (2Lg + 2Zg2 Zw ) ENt Et wtNt − wt0 D E + 2ηt ENt Et ∇L(wtNt ), wtNt − wt0 + 2ηt2 ENt Et ∇L(wtNt )
2
+ 2ηt2 µt − ∇L wt0
− Lηt2 (Lg + Zg2 Zw )Et ∥wnt − w0t ∥2 − Lηt2 ∥µt − ∇L wt0 ∥2 . 2
2
− ENt Et wtNt +1 − wt0 2 1 2 2 ENt Et wtNt − wt0 = (2Zg Zw + 2Lg )ηt − ξt D E + 2ηt ENt Et ∇L(wtNt ), wtNt − wt0 + 2ηt2 ENt Et ∇L(wtNt )
2
+ 2ηt2 µt − ∇L wt0
Et [L(wn+1 )] ≥ Et [L(wnt )] t
Taking n = Nt and using ENt to denote the expectation w.r.t. Nt , we have from the above: t ENt Et [L(wtNt +1 )] ≥ ENt Et [L(wN t )]
2
. (19)
2 t + ηt (1 − Lℓ ηt )ENt Et ∥∇L(wN t )∥ t + ηt ENt Et ⟨µt − ∇L wt0 , ∇L(wN t )⟩ 0 2 t − Lηt2 (Lg + Zg2 Zw )ENt Et ∥wN t − wt ∥ − Lηt2 ∥µt − ∇L wt0 ∥2 .
16
1−2η 2 (Lg +Z 2 Zw )ξt −2Lℓ η 3 (Lg +Z 2 Zw )ξ 2
t g t g t y = ∇L(w̃t ), and ρ = ηt ξt to get: 2 t ηt (1 − Lℓ ηt )ENt Et ∥∇L(wN t )∥ 1 1 ρ 0 2 E ⟨y, x⟩ ≤ E∥x∥2 + E∥y∥2 . (26) t t +1 )] + Lℓ ηt2 (Lg + Zg2 Zw )ENt Et ∥wN ≤ENt Et [L(wN t − wt ∥ t ξt 2ρ 2 t − ηt ENt Et ⟨µt − ∇L wt0 , ∇L(wN Combining (25) and (26) and rearranging, we have: t )⟩ Nt 2 0 2 Nt + Lℓ ηt ∥µt − ∇L wt ∥ − E Et [L(wt )] 1 ηt − 2 1 2 0 2[1 − 2ηt (Lg + Zg Zw )ξt − 2Lℓ ηt3 (Lg + Zg2 Zw )ξt2 ] t = (Et ENt [L(wN t )] − L(wt )) ! ξt 1 Nt 0 − ηt ENt Et ⟨µt − ∇L wt , ∇L(wt )⟩ + 1 − Lℓ ηt − E∥∇L(w̃t )∥2 ξ t 2 0 t + Lℓ ηt2 (Lg + Zg2 Zw )ENt Et wN t − wt 1 ≤ E [L(w̃t ) − L(w̃t−1 )] 2 0 2 ξ t . (22) + Lℓ ηt µt − ∇L wt 1 + ηt (1 + Lℓ ηt + )E∥µt − ∇L wt0 ∥2 (22) follows E [DN − DN +1 ] = 1 − Γ1 (E[DN ] − D0 ) with ξt t 1 Fubini’s theorem. Note that w̃t = wN and w̃t−1 = w0t . If t ≤ E [L(w̃t ) − L(w̃t−1 )] we take expectation over all the randomness and denote it by ξt E, we get V 1 , (27) + ηt (1 + Lℓ ηt + ) 2ψ 2 + 64σ 2 ξt ξt 2 ηt (1 − Lℓ ηt )E∥∇L(w̃t )∥ where (27) follows from Lemma 4. We want to choose ηt 1 = E [L(w̃t ) − L(w̃t−1 )] − ηt E µt − ∇L wt0 , ∇L(w̃t ) such that 1 − 2ηt2 (Lg + Zg2 Zw )ξt − 2Lℓ ηt3 (Lg + Zg2 Zw )ξt2 > 0. ξt Denoting Φ = Lg + Zg2 Zw , we have + Lℓ ηt2 (Lg + Zg2 Zw )E∥w̃t − w̃t−1 ∥2 1 > 1 − 2ηt2 Φξt − 2Lℓ ηt3 Φξt2 > 0 + Lℓ ηt2 E∥µt − ∇L wt0 ∥2 1 1 1 > 2 3 2 = E [L(w̃t ) − L(w̃t−1 )] 2(1 − 2ηt Φξt − 2Lℓ ηt Φξt ) 2 ξt Thus, we can select η such that: t 1 − E µt − ∇L wt0 , w̃t − w̃t−1 1 1 1 ξt 1 − L ℓ ηt − − (28) 2 Φξ − 2L η 3 Φξ 2 ] ≥ 4 2 2 2 ξ 2[1 − 2η t t ℓ t t t + Lℓ ηt (Lg + Zg Zw )E∥w̃t − w̃t−1 ∥ 1 1 3 2 Lℓ ηt + + ≤ + ηt (1 + Lℓ ηt )E µt − ∇L wt0 (23) ξt 2[1 − 2ηt2 Φξt − 2Lℓ ηt3 Φξt2 ] 4 1 Next, we aim to satisfy the condition: ≤ E [L(w̃t ) − L(w̃t−1 )] ξt 1 5 1 1 1 ≤ (i) < 2 2 2 2 3 2 − + ηt (2Lg + 2Zg Zw ) E∥w̃t − w̃t−1 ∥ + 2 2[1 − 2ηt Φξt − 2Lℓ ηt Φξt ] 8 2ηt ξt ξt 1 1 1 1 (ii)Lℓ ηt ≤ (iii) ≤ + E ⟨∇L(w̃t ), w̃t − w̃t−1 ⟩ 16 ξt 16 ξt From condition (i), we deduce: 2 ηt ηt + E∥∇L(w̃t )∥2 + E µt − ∇L wt0 1 ξt ξt ηt2 Φξt + Lℓ ηt3 Φξt2 ≤ 2 2 2 10 + Lℓ ηt (Lg + Zg Zw )E∥w̃t − w̃t−1 ∥ 1 1 2 → ηt2 Φξt ≤ & Lℓ ηt3 Φξt2 ≤ + ηt (1 + Lℓ ηt )E µt − ∇L wt0 , (24) 20 20 1 1 ⇒ ηt ≤ & ηt ≤ where (23) follows from Lemma 3 and (24) follows from (20). 1/2 2/3 1/3 201/2 Φ1/2 ξt 201/3 Φ1/3 ξt Lℓ By rearranging all the terms, Using conditions (ii) and (iii), we get: 1 − 2ηt2 (Lg + Zg2 Zw )ξt − 2Lℓ ηt3 (Lg + Zg2 Zw )ξt2 1 E∥w̃t − w̃t−1 ∥2 ηt ≤ & ξt ≥ 16 2ηt ξt2 16Lℓ 1 + ηt (1 − Lℓ ηt − )E∥∇L(w̃t )∥2 ξt 1 1/3 We can then choose ηt ≤ = 2/3 , where τ = (Lℓ Φ) 1 1 2τ ξt ≤ E [L(w̃t ) − L(w̃t−1 )] + E ⟨∇L(w̃t ), w̃t − w̃t−1 ⟩ 2 1/3 ξt ξt (Lℓ (Lg + Zg Zw )) , s.t. 1 1 + ηt (1 + Lℓ ηt + )E∥µt − ∇L wt0 ∥2 . (25) ξt 2/3 2(Lℓ Φ)1/3 ξt ( ) Now we can apply Young’s inequality, that For any real 1 1 1 ≤ min , , . numbers x and y, and for any ρ > 0, the following holds: xy ≤ 16 201/2 Φ1/2 ξt1/2 201/3 (ΦLℓ )1/3 ξt2/3 ρ 2 x2 + y , on E ⟨∇L( w̃ ), w̃ − w̃ ⟩ using x = w̃ − w̃ , t t t−1 t t−1 2ρ 2
Rearrange the terms,
17
To ensure the conditions are satisfied, we choose ηt = 12/3 2τ ξt when ξt ≥ 16 which satisfies (28). We can obtain the following from (27) and (28): 1 1 ηt E∥∇L(w̃t )∥2 ≤ E [L(w̃t ) − L(w̃t−1 )] 4 ξt V . + 2ηt 2ψ 2 + 64σ 2 ξt Replacing ηt =
1 2/3 2τ ξt
and rearranging, we have " 1 2 E∥∇L(w̃t )∥ ≤ 4 E [L(w̃t ) − L(w̃t−1 )] ξt η t # V + 2 2ψ 2 + 64σ 2 ξt " 2τ E [L(w̃t ) − L(w̃t−1 )] ≤4 1 ξt3 # 2 2V . + 4ψ + 128σ ξt
Telescoping over t = 1, 2, ..., T with a constant batch size ξt , we have for w̃a uniformly sampled from {w̃t }Tt=1 : " 2τ E [L(w̃T ) − L(w̃0 )] E∥∇L(w̃a )∥2 ≤ 4 1 T ξt3 # 2 2V + 4ψ + 128σ ξt ≤
8τ [L(w̃∗ ) − L(w̃0 )] 1
T ξt3 + 16ψ 2 + 512σ 2
V , ξt
which completes the proof. TABLE VII: Experiment parameters. Parameters Total number of agents K Fraction of malicious agents α Training round T Steps Vehicle length Number of hidden layers Batch size ξt Discount factor γ Step size ηt Mini-batch size bt Scaling factor λ Threshold ψ
Values 10 20% 2000 1500 3 meters 3 512 0.9995 0.001 32 10 1
A PPENDIX B D ETAILS OF P OISONING ATTACKS Trim attack [14]: This attack is specifically designed for the Trimmed-mean [34] and Median [34] aggregation rules. In the Trim attack, the attacker meticulously designs the
TABLE VIII: Neural network architecture. Layer Input Fully Connected + Sigmoid Fully Connected + Sigmoid Fully Connected + Sigmoid Fully Connected + Sigmoid Output
Shape 8 8 × 256 256 × 256 256 × 256 256 × 1 1
local gradients on the malicious agents so that the aggregated gradient after the attack significantly diverges from the one before the attack. Random attack [54]: Each malicious agent samples a Gaussian vector from a Gaussian distribution with a mean of 0 and a variance of 10000, then transmits this vector to the server. History attack [54]: In this attack, the attacker replaces the local gradients on the malicious agents with scaled versions of their previous values, thereby skewing the aggregated gradient to disrupt the aggregation process. MPAF attack [54]: Each malicious agent calculates and amplifies the difference between a selected base gradient and the current aggregated gradient using a large scaling factor. FTI attack [16]: The FTI attack operates similarly to the MPAF attack. Each malicious agent calculates the difference between the attacker-selected base gradient and the current aggregated gradient but applies different weights to the base gradient and current aggregated gradient to avoid detection. MinMax attack [17]: This is an attack that is agnostic to the aggregation rule. In this attack, the attacker crafts the malicious local gradients so that the maximum distance between the malicious and benign local gradients is no greater than the maximum distance between any two benign local gradients. MinSum attack [17]: Similar to the MinMax attack, the MinSum attack is also aggregation rule-independent. The attacker precisely designs the malicious local gradients so that the total distance between the malicious and benign local gradients does not exceed the maximum sum of distances between any two benign local gradients. Adaptive attack [17]: The adaptive attack represents the worst-case scenario, where it is assumed that the attacker is aware of the aggregation rule employed by the server (i.e., SecApp in our paper) and has knowledge of the local gradients on all agents. In this attack, the attacker crafts the local gradients on the malicious agents to maximize the distance between aggregated gradient with and without the attacks. A PPENDIX C D ETAILS OF C OMPARISON AGGREGATION RULES FedAvg [23]: The server calculates the aggregated gradient by averaging the local gradients from all agents. Median [34]: The Median rule computes the median of gradients across each dimension coordinate-wise. Trimmed-mean (Trim) [34]: Trim is another coordinatewise robust aggregation rule. For each dimension, the server
18
removes the largest b and smallest b values, then averages the remaining elements, where b is the trim parameter. Krum [24]: In the Krum method, the server selects the local gradient with the smallest sum of distances to its nearest neighbors. FoolsGold [19]: In FoolsGold, the server computes a similarity matrix using cosine similarity among all agents’ gradients and adjusts the weights according to these similarities. The weighted local gradients are then aggregated to create the aggregated gradient. FABA [22]: In FABA, the server employs an iterative method to eliminate outlier local gradients from the system. Specifically, in each iteration, the server removes the local gradients with the largest distance from the current average, repeating this process until m gradients are deleted, where m represents the number of malicious gradients, assumed to be known by the server. FLTrust [15]: In this aggregation rule, it is assumed that the server can compute a gradient. The server will accept local gradients from agents that are positively correlated with this reference gradient. FLAIR [21]: In FLAIR, the server assigns reputation scores to agents based on their historical behavior. These scores are then used to adjust the weight of each agent’s contribution to the aggregated gradient. FedPG-BR (FedPG) [18]: The FedPG uses the mean of median technique to choose a gradient that is in the center of the selected set. After performing a comparison between the mean of median gradient, the local gradients that fall within a threshold are considered benign. FLAME [53]: The FLAME rule uses a clustering algorithm to identify and remove highly impactful gradients. Additionally, it applies a dynamic weight-clipping approach and noise injection technique to further minimize the influence of malicious gradients. DeepSight [52]: DeepSight utilizes a combined approach of deep model analysis and clipping in its filtering technique to detect possible poisoning attacks. A PPENDIX D E XTENSIVE E XPERIMENTS AND A BLATION S TUDY In this section, we provide further evaluations on our SecApp. Different variants of SecApp: We explore several variants of SecApp used in the AD systems, as summarized in Table IX. • Variant I (Var. I): In Line 8 of Algorithm 3, instead of calculating µSt as the model closest to the mean of the set S, we compute it using the median [34] of S. That is, µSt = (k̃) argminµ(k̃) ∥µt − median(S)∥, where k̃ ∈ S. t
•
Variant II (Var. II): In this variant, we remove Lines 8 and 9 from Algorithm 3 entirely. Instead, we use the set S computed in Line 7 directly as the benign model set Wt . By doing so, we ensure that the number of selected models
remains greater than half of the total participating agents, without further adjustments. • Variant III (Var. III): Here, the server uses the model (k̃) calculated in Line 8, µSt = argminµ(k̃) ∥µt − mean(S)∥, t as the aggregated gradient. This version removes Line 9 and uses the computed µSt for the SVRG process. • Variant IV (Var. IV): In Line 7, gradients are not filtered; instead, all gradients are incorporated into the set S. TABLE IX: Performance on different variants of SecApp. Attack
Var. I
Var. II
Var. III
Var. IV
SecApp
No attack Trim attack Random attack History attack MPAF attack FTI attack MinMax attack MinSum attack Adaptive attack
100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 92.45% 90.81%
100.0% 23.17% 100.0% 29.38% 29.77% 28.69% 26.53% 15.10% 20.96%
100.0% 100.0% 100.0% 100.0% 22.29% 28.03% 19.72% 17.61% 14.63%
100.0% 40.15% 100.0% 100.0% 100.0% 100.0% 100.0% 23.19% 31.93%
100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%
Each of the four variants exhibits varying levels of resilience to attacks. As expected, SecApp clearly outperforms the others, maintaining the highest no-collision rates and demonstrating strong resistance to all attacks. Interestingly, Var. I performs similarly to the original SecApp but remains vulnerable in specific cases like the MinSum and Adaptive attacks, where its performance drops to around 90%. Var. II and Var. III fail across most attacks, showing a sharp decline in performance with rates under 30%, leaving them unable to defend against adversarial interference. Var. IV performs better than Var. II and Var. III but still fails in some cases, such as achieving 31.93% under the Adaptive attack. From this ablation study, we confirm that Lines 7, 8, and 9 in Algorithm 3 are crucial to the robustness of SecApp against poisoning attacks. Line 7 ensures the reliability of the set S by filtering out malicious gradients. Line 8 establishes a reliable benchmark by selecting a center gradient from the benign set S, where the choice of mean(S) leads to slightly better performance than Var. I with median(S). Line 9 is the most critical one, as it applies strict filtering to each local model based on µSt and µt−1 . Missing any of these design philosophies in SecApp results in a significant vulnerability to attacks. Impact of total number of agents: Fig. 8 provides a comparative analysis of various aggregation methods in defending against poisoning attacks as the number of participating RL agents increases, the fraction of malicious agents is set to 20%. As depicted in Fig. 8a, traditional aggregation methods such as Median and Trim fail to defend the system across the entire range of agent counts, with no-collision rates remaining around or below 20% as the number of agents scales from 10 to 40, indicating frequent failures in collision avoidance under attack conditions. In contrast, our SecApp demonstrates exceptional robustness, maintaining a 100% no-collision rate regardless of the number of agents used in the system. This trend of superior performance is consistent across all other attack scenarios, as illustrated in Figs. 8b-h. Such desired resilience of SecApp
19
can be attributed to its advanced filtering mechanisms, which effectively identify and neutralize the influence of malicious gradients, ensuring the reliability of the aggregated gradient even in large-scale agent systems. TABLE X: Performance on different threshold ψ. Attack No attack Trim attack Random attack History attack MPAF attack FTI attack MinMax attack MinSum attack Adaptive attack
0.1 82.14% 24.31% 23.62% 43.85% 34.95% 46.30% 43.01% 32.13% 36.34%
1 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%
5 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 34.60% 100.0%
10 100.0% 46.97% 100.0% 100.0% 100.0% 100.0% 100.0% 28.92% 35.88%
Impact of different threshold ψ: Table X evaluates the performance of SecApp under various attack scenarios with different values of the filtering threshold ψ in line 7 of Algorithm 3. A lower ψ enforces more stringent filtering, which can degrade performance by excluding too many benign gradients, as seen with attacks like Trim and Random attacks. For instance, when ψ = 0.1, SecApp achieves only 82.14% in attack-free scenarios and fails under all attack conditions. With ψ = 1, there is a marked improvement, with SecApp reaching 100.0% performance in most cases. This demonstrates that a moderate value of ψ strikes an optimal balance between filtering and retaining sufficient benign gradients for an effective aggregation. At ψ = 5, the algorithm still performs well overall, but specific attacks, such as MinSum Attack, cause performance drops. This suggests that overly lenient filtering allows malicious gradients to infiltrate the gradient aggregation process. While at ψ = 10, performance declines significantly in scenarios such as 46.97% and 35.88% under Trim and Adaptive attacks, respectively, highlighting that excessive leniency compromises the defense mechanism. Therefore, selecting an appropriate filtering threshold ψ is another critical factor in ensuring that SecApp maintains robustness against various attacks. Performance against backdoor attacks: In this section, we demonstrate that our proposed SecApp effectively protects the AD system against targeted attacks, including backdoor attacks. The three backdoor attacks, BACKDOORL [63], BadRL [64], and MARNet [65], are particularly effective in RL-based scenarios due to their adaptive strategies that exploit the decentralized nature of learning. Additional implementation details on these three backdoor attacks can be found in Appendix F. Note that the three backdoor attacks were initially designed for traditional RL. To apply them to AD tasks, each malicious agent trains its local model using deceptive states and rewards, enabling it to execute malicious actions when specific triggers are detected. Table XI shows the effectiveness of various defense mechanisms against three distinct backdoor attacks. These attacks are engineered to subtly manipulate the model’s performance on specific tasks while remaining undetected on others, making them particularly challenging to counteract. The values in
TABLE XI: Performance of SecApp across various backdoor attacks. The results are in the form of “no-collision rate / attack success rate”. Defense
BACKDOORL attack
BadRL attack
MARNet attack
FedAvg
23.68% / 89.45%
27.42% / 81.67%
13.50% / 90.12%
Median
100.0% / 5.62%
100.0% / 9.89 %
100.0% / 6.17%
Trim
15.67% / 56.21%
100.0% / 8.47%
100.0% / 9.28%
Krum
28.45% / 77.84 %
100.0% / 3.11%
21.57% / 62.95%
FoolsGold
100.0% / 4.23%
100.0% / 5.56%
100.0% / 7.34%
FABA
100.0% / 8.56%
14.68% / 73.29%
100.0% / 5.89%
FLTrust
100.0% / 9.78%
100.0% / 6.43 %
100.0% / 7.16%
FLAIR
22.89% / 66.54%
100.0% / 8.99%
100.0% / 6.78%
FedPG
100.0% / 7.12%
100.0% / 8.76%
100.0% / 5.33%
FLAME
100.0% / 6.89%
12.42% / 69.84%
100.0% / 9.47%
Deepsight
100.0% / 8.23%
100.0% / 9.05%
100.0% / 6.78%
SecApp
100.0% / 3.45%
100.0% / 3.12%
100.0% / 2.89%
Table XI are shown in the form of “no-collision rate / attack success rate”. A larger no-collision rate and a smaller attack success rate indicate a more effective defense. Table XI shows that during the BACKDOORL attack, the no-collision rate of defenses such as FLAIR significantly decreases. In stark contrast, SecApp maintains a flawless 100.0% no-collision rate across all evaluated metrics, demonstrating its exceptional proficiency in detecting and neutralizing these stealthy threats. Similarly, under the BadRL attack, many defense mechanisms suffer a significant decline in no-collision rate, with some, such as FLAME and FABA, plummeting to 12.42% and 14.68%, respectively. The attack success rate remains below 5% for SecApp against all backdoor attacks. This indicates that SecApp not only mitigates the effects of backdoor attacks but also maintains the model’s performance on its main control tasks. Adaptive computation of scaling factor λ: In our experiments, we default to using a fixed scaling factor λ as shown in Eq. (2). Considering the mobile vehicular environment, here we investigate a moving average approach to dynamically compute λ, allowing it to adjust to the variability in the gradients. Specifically, let λt denote the scaling factor at training round t. Then λt can be calculated as λt = α · λt−1 · µSt − µt−1 + (1 − α) · λ0 , where the initial scaling factor λ0 = 10 and weighing parameter α = 0.2. The historical component λt−1 ensures that the adjustment of scaling factor is not too abrupt. The no-collision rates of SecApp under both Trim and Adaptive attacks are both 100.0% when the adaptive computation of the scaling factor is applied. Impact on decision-making sequences: Fig. 9 illustrates the impact of poisoning attacks on vehicle behavior in a fivecar scenario within the FRL framework, where the FedAvg aggregation rule is used. “Delta Distance” refers to the distance between two vehicles. When all agents are benign, as shown in Fig. 9a, the FRL system operates without any collision incidents, the vehicles can maintain smooth trajectories with safe inter-vehicle distances, reflecting the effectiveness of the secure aggregation method in ensuring safety. Specifically,
20
30
40
20
30
Total number of agents
20
Total number of agents
40
40
100 80 60 40 20 0 10
(e) FTI attack
20
30
Total number of agents
FoolsGold Deepsight
FABA SecApp
30
100 80 60 40 20 0 10
No-collision rate (%)
30
100 80 60 40 20 0 10
(b) Random attack
No-collision rate (%)
No-collision rate (%)
(a) Trim attack
100 80 60 40 20 0 10
Trim FLAME No-collision rate (%)
100 80 60 40 20 0 10
Krum FedPG
20
Total number of agents
40
(c) History attack
40
(f) MinMax attack
100 80 60 40 20 0 10
20
30
Total number of agents
(g) MinSum attack
20
30
Total number of agents
40
(d) MPAF attack
No-collision rate (%)
20
Total number of agents
Median FLAIR
No-collision rate (%)
100 80 60 40 20 0 10
No-collision rate (%)
No-collision rate (%)
FedAvg FLTrust
40
100 80 60 40 20 0 10
20
30
Total number of agents
40
(h) Adaptive attack
Fig. 8: Impact of total number of agents. TABLE XII: Performance on different model sizes. Attack
3 layers
4 layers
5 layers
No attack Trim attack Random attack History attack MPAF attack FTI attack MinMax attack MinSum attack Adaptive attack
100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%
100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%
100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%
FedAvg rule ensures that the distance between consecutive vehicles remains within a safe range of 4.8 to 20 meters throughout the evaluation, with vehicle velocities decreasing consistently as they approach their destinations, ultimately coming to a safe halt. However, when the FRL model is compromised by the Adaptive attack, as demonstrated in Fig. 9b, the integrity of the system is significantly degraded. When vehicle 5, the leading vehicle, begins its abrupt deceleration, the compromised model causes the RL-controlled vehicle 4 to continue accelerating instead of braking, leading to a collision. The distance between vehicle 4 and vehicle 5 rapidly diminishes, even crossing below zero, indicating an overlap and hence a collision. The failure of vehicle 4 to decelerate, in contrast to the leading vehicle’s abrupt stop, underscores the disruption caused by the malicious gradients. This scenario validation highlights the catastrophic impact of poisoning attacks, where the compromised model fails to maintain effective coordination among vehicles, leading to unsafe behaviors and potential accidents. Impact on numbers of neural network layers: Table XII examines the impact of varying the number of neural network hidden layers on the performance of SecApp. Interestingly, the performance remains remarkably consistent, with SecApp achieving 100.0% accuracy across models with 3, 4, and 5
hidden layers, regardless of the attack type. This indicates that SecApp is highly scalable and can effectively handle models of different complexities without a loss in defensive capability. The consistent performance across different model sizes suggests that SecApp’s defense mechanisms are robust and not overly sensitive to the underlying model architecture, making it a versatile solution for a wide range of FRL applications. A PPENDIX E S EC A PP U SED IN E DGE C ACHING S CENARIOS In the context of emerging next-generation cellular networks, the dense deployment of small-cell base stations (BSs) connected to data centers via low-bandwidth, high-latency backhaul links is anticipated. Each BS is equipped with a cache unit that stores frequently requested content, with the aim of enhancing network performance by minimizing latency and reducing communication costs. The primary objective in such an environment is to maximize the cache hit rate, ensuring that content is readily available in the BS cache, and to minimize peak traffic loads on the backhaul links, which occur when content must be fetched from the data center. Cache management strategies are designed to dynamically replace less frequently accessed content with data that is more likely to be requested in the future, based on predictive models such as the Zipf distribution, which describes the frequency of content requests. We leverage this digital twin environment to assess our proposed defense strategy. The cache hit rate is the percentage of requests that are successfully served from the cache, reducing the need to retrieve data from a slower, more distant source. It is used as a key performance metric, where a higher hit rate signifies better defense performance; a random cache strategy typically achieves a hit rate of around 40-50%.
21
Vehicle 1 Vehicle 2 Vehicle 3 Vehicle 4 Vehicle 5 Delta distance between vehicle 1 and 2 Delta distance between vehicle 2 and 3 Delta distance between vehicle 3 and 4 Delta distance between vehicle 4 and 5 Vehicle Positions Over Time
20
Velocity (m/s)
150
15
100
10
50 0
0
5
0 Vehicle Vehicle Vehicle 4 0 200Vehicle 5 800 1000 1200 1400 0 2200 400 600 800 31000 1200 1400 400 600 Time Step Delta distance: V1-2 Time Step Delta distance: V2-3 distance: V3-4 Delta distance: V4-5under no attack setting. (a) Vehicle behaviors when Delta the FedAvg aggregation rule is used,
Vehicle 200 400 600 800 1000 120011400 Time Step
Vehicle Positions Over Time
175 150 125 100 75 50 25 0
Vehicle Velocities Over Time
35
Velocity (m/s)
Position (m)
30 25 20
0
Delta Distance Over Time
20.0 17.5 15.0 12.5 10.0 7.5 5.0 2.5
Delta Distance (m)
Position (m)
Vehicle Velocities Over Time
25
Delta Distance (m)
200
50
100 150 200 250 300 350 400
Time Step
15
0
50
100 150 200 250 300 350 400
Time Step
40 30 20 10 0 10 20
Delta Distance Over Time
0
50
100 150 200 250 300 350 400
Time Step
(b) Vehicle behaviors when the FedAvg aggregation rule is used, under Adaptive attack setting. Fig. 9: Attacked RL algorithm implementation and verification. TABLE XIII: Cache hit rate in edge caching scenario. Attack
FedAvg
Median
Trim
Krum
FoolsGold FABA
FLTrust
FLAIR
FedPG
FLAME
Deepsight SecApp
No attack
77.29%
77.68%
77.89%
77.17%
76.55%
76.94%
77.83%
77.64%
77.41%
77.12%
76.92%
77.84%
Trim attack
53.98%
76.99%
77.78%
76.13%
44.69%
58.74%
77.39%
50.29%
76.66%
77.23%
76.81%
77.17%
Random attack
48.37%
76.92%
77.11%
52.47%
74.86%
56.92%
77.54%
54.11%
78.79%
78.93%
79.36%
77.66%
History attack
47.12%
77.47%
58.42%
77.39%
77.16%
46.88%
77.02%
76.95%
77.48%
54.62%
77.89%
76.98%
MPAF attack
44.67%
59.18%
77.29%
76.82%
76.84%
57.13%
50.41%
77.36%
76.97%
77.18%
77.08%
77.63%
FTI attack
49.12%
54.78%
49.89%
58.92%
77.11%
54.62%
48.79%
76.92%
59.21%
77.39%
55.44%
77.14%
MinMax attack
51.02%
77.58%
77.33%
47.98%
44.52%
49.69%
55.17%
56.12%
58.23%
54.21%
55.78%
77.44%
MinSum attack
49.62%
56.11%
77.62%
48.49%
53.77%
76.89%
45.23%
52.34%
56.77%
77.48%
76.98%
77.16%
Adaptive attack
58.42%
50.17%
76.77%
55.12%
48.32%
77.38%
51.93%
58.27%
46.71%
77.14%
49.13%
77.29%
The edge caching strategy in cellular networks can be modeled as a MDP, defined by the tuple {S, A, R, T , s0 }. State Space (S): Each state s ∈ S is represented as a tuple (ℓ, tℓ , fℓ ), where ℓ denotes the base station (BS) index, tℓ is the time elapsed since the last cache update, and fℓ indicates the frequency of content requests. • Action Space (A): The action space A defines the possible caching operations. The server can either skip a request (a = 0) or allocate a specific cache slot for the requested content dt . • Reward Function (R(s, a)): The immediate reward reflects the improvement in cache performance, quantified by the increase in cache hits between consecutive requests. • State Transition Probabilities (T ): The state transition probabilities T define the likelihood of moving between states based on the selected actions.
•
Initial State (s0 ): The initial state s0 represents the system’s configuration at the start of the process, where all cache units are assumed to be empty.
•
We consider a network consisting of 10 BSs, each equipped with a local cache capable of storing 150 items. Note that in the edge caching setting, a single RL agent controls one BS. Each BS serves 8 edge devices, and the service areas of edge devices may overlap, allowing an edge device to connect to up to 2 BSs simultaneously. Cache decisions are made dynamically, taking into account the load of the BSs and the current cache status of the involved BSs. The request frequencies fℓ follow a Zipf distribution with parameter p = 0.8, which reflects real-world request patterns where a small subset of content is requested much more frequently than the rest. In our experiments, we set the values of λ and ψ to 10 and 5, respectively. 20% of the agents are compromised. In edge caching scenarios, our proposed SecApp exhibits
22
superior performance across various types of attacks, consistently maintaining high cache hit rate that outperforms other defense mechanisms. This is consistent with its performance in autonomous vehicle scenarios. For instance, under the Random attack, SecApp achieves a cache hit rate of 77.66%, outperforming FoolsGold, which only reaches 74.86%. This suggests that SecApp is particularly effective at countering the effects of randomized adversarial behavior, likely due to its robust aggregation mechanism that filters out malicious gradients. Similarly, when confronted with the Adaptive attack—which dynamically adjusts its strategy to circumvent defenses—SecApp still maintains a high cache hit rate of 77.29%, outperforming Krum and Trim, which drop to 55.12% and 76.77%, respectively. A similar trend is observed in the autonomous driving scenario, where SecApp consistently shows better performance across comparable attack types, underscoring its generalizability and effectiveness in diverse FRL contexts. A PPENDIX F D ETAILS OF BACKDOOR ATTACKS BACKDOORL [63]: BACKDOORL attack embeds hidden policies within the local models of individual agents, which are activated under specific conditions to trigger undesirable behaviors. Despite these triggers, the overall model maintains strong performance on non-targeted tasks, making the attack difficult to detect during the aggregation process. BadRL [64]: BadRL attack targets key decision-making pathways by selectively poisoning critical model components. This targeted approach ensures that only a few gradients are compromised, allowing the attack to evade detection while still achieving its malicious goals by subtly influencing the collective decision-making process. MARNet [65]: This attack exploits the cooperative dynamics of multi-agent RL by coordinating malicious behaviors among agents. Although each agent may appear benign individually, their coordinated actions significantly degrade overall system performance, making the attack particularly challenging to identify and mitigate. A PPENDIX G L IMITATION AND F UTURE W ORKS One limitation of our SecApp is that its robustness has only been demonstrated in two representative safety-critical systems: autonomous driving and edge caching. The performance of SecApp in other critical systems, such as unmanned aerial vehicle (UAV) assisted public safety [66], [67], remains unknown. Additionally, SecApp is currently constrained to server-assisted FRL, where a central server helps manage the global training process. In many real-world serverless safety-critical systems, such as employing fully-decentralized RL [68], [69], there is no central server for coordination, and each agent exchanges information with its neighbors. Therefore, the robustness of SecApp in such fully-decentralized scenarios is yet to be determined. Addressing these limitations opens several promising research directions. Future extensions
of SecApp could incorporate graph-based policy alignment and local-area aggregation to enable reliable learning without centralized coordination. In addition, integrating environmentaware abstractions such as twin–derived surrogate models may help transfer robustness guarantees across domains. R EFERENCES [1] T. Haarnoja, B. Moran, G. Lever, S. H. Huang, D. Tirumala, J. Humplik, M. Wulfmeier, S. Tunyasuvunakool, N. Y. Siegel, R. Hafner et al., “Learning agile soccer skills for a bipedal robot with deep reinforcement learning,” in Science Robotics, 2024. [2] X. He, J. Wu, Z. Huang, Z. Hu, J. Wang, A. Sangiovanni-Vincentelli, and C. Lv, “Fear-neuro-inspired reinforcement learning for safe autonomous driving,” in IEEE TPAMI, 2023. [3] D. Chen, Y. Gong, and X. Yang, “Deep reinforcement learning for advanced longitudinal control and collision avoidance in high-risk driving scenarios,” in arXiv, 2024. [4] OpenAI, “Gpt-4 technical report,” 2023. [5] Z. Zhang, Y. Liu, Z. Peng, M. Chen, D. Xu, and S. Cui, “Digital twinassisted data-driven optimization for reliable edge caching in wireless networks,” in IEEE JSAC, 2024. [6] N. Naderializadeh, J. J. Sydir, M. Simsek, and H. Nikopour, “Resource management in wireless networks via multi-agent deep reinforcement learning,” in IEEE TWC, 2021. [7] J. Yang, A. A. Soltan, D. W. Eyre, and D. A. Clifton, “Algorithmic fairness and bias mitigation for clinical machine learning with deep reinforcement learning,” in Nature Machine Intelligence, 2023. [8] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv, 2020. [9] X. Zhou, X. Zheng, X. Cui, J. Shi, W. Liang, Z. Yan, L. T. Yang, S. Shimizu, I. Kevin, and K. Wang, “Digital twin enhanced federated reinforcement learning with lightweight knowledge distillation in mobile networks,” in IEEE JSAC, 2023. [10] Y. Fu, C. Li, F. R. Yu, T. H. Luan, and Y. Zhang, “A selective federated reinforcement learning strategy for autonomous driving,” in IEEE TITS, 2023. [11] C. Jin, X. Feng, and H. Yu, “Embracing multiheterogeneity and privacy security simultaneously: A dynamic privacy-aware federated reinforcement learning approach,” in IEEE TNNLS, 2024. [12] S. Lee and D.-H. Choi, “Federated reinforcement learning for energy management of multiple smart homes with distributed energy resources,” in IEEE TII, 2020. [13] J. Qi, Q. Zhou, L. Lei, and K. Zheng, “Federated reinforcement learning: Techniques, applications, and open challenges,” Intelligence & Robotics, 2021. [14] M. Fang, X. Cao, J. Jia, and N. Gong, “Local model poisoning attacks to byzantine-robust federated learning,” in USENIX Security, 2020. [15] X. Cao, M. Fang, J. Liu, and N. Z. Gong, “Fltrust: Byzantine-robust federated learning via trust bootstrapping,” in NDSS, 2021. [16] Z. Zhang, M. Fang, J. Huang, and Y. Liu, “Poisoning attacks on federated learning-based wireless traffic prediction,” in IFIP Networking, 2024. [17] V. Shejwalkar and A. Houmansadr, “Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning,” in NDSS, 2021. [18] X. Fan, Y. Ma, Z. Dai, W. Jing, C. Tan, and B. K. H. Low, “Faulttolerant federated reinforcement learning with theoretical guarantee,” in NeurIPS, 2021. [19] C. Fung, C. J. Yoon, and I. Beschastnikh, “Mitigating sybils in federated learning poisoning,” arXiv, 2018. [20] S. Li and Y. Dai, “Backdoorindicator: Leveraging ood data for proactive backdoor detection in federated learning,” in USENIX Security, 2024. [21] A. Sharma, W. Chen, J. Zhao, Q. Qiu, S. Bagchi, and S. Chaterji, “Flair: Defense against model poisoning attack in federated learning,” in ASIACCS, 2023. [22] Q. Xia, Z. Tao, Z. Hao, and Q. Li, “Faba: an algorithm for fast aggregation against byzantine attacks in distributed neural networks,” in IJCAI, 2019. [23] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in AISTATS, 2017.
23
[24] P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” in NeurIPS, 2017. [25] P. Xu, F. Gao, and Q. Gu, “An improved convergence analysis of stochastic variance-reduced policy gradient,” in UAI, 2019. [26] Authors, “HighwayDT will be open-sourced upon the paper acceptance,” 2024. [27] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, “Deep reinforcement learning for autonomous driving: A survey,” in IEEE TITS, 2021. [28] J. Wu, Z. Huang, and C. Lv, “Uncertainty-aware model-based reinforcement learning: Methodology and application in autonomous driving,” in IEEE TIV, 2022. [29] Y. Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, R. Roelofs, B. Sapp, B. White, A. Faust, S. Whiteson et al., “Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios,” in IROS, 2023. [30] A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J.-M. Allen, V.-D. Lam, A. Bewley, and A. Shah, “Learning to drive in a day,” in ICRA, 2019. [31] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction, 2018. [32] M. Papini, D. Binaghi, G. Canonaco, M. Pirotta, and M. Restelli, “Stochastic variance-reduced policy gradient,” in ICML, 2018. [33] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” in Machine Learning, 1992. [34] D. Yin, Y. Chen, K. Ramchandran, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” in ICML, 2018. [35] S. Khodadadian, P. Sharma, G. Joshi, and S. T. Maguluri, “Federated reinforcement learning: Linear speedup under markovian sampling,” in ICML, 2022. [36] H. Jin, Y. Peng, W. Yang, S. Wang, and Z. Zhang, “Federated reinforcement learning with environment heterogeneity,” in AISTATS, 2022. [37] Z. Zhang, Z. Peng, H. Yu, M. Chen, and Y. Liu, “Digital network twins for next-generation wireless: Creation, optimization, and challenges,” arXiv, 2024. [38] R. Johnson and T. Zhang, “Accelerating stochastic gradient descent using predictive variance reduction,” in NeurIPS, 2013. [39] L. Lei, C. Ju, J. Chen, and M. I. Jordan, “Non-convex finite-sum optimization via scsg methods,” in NeurIPS, 2017. [40] P. Indyk and R. Motwani, “Approximate nearest neighbors: towards removing the curse of dimensionality,” in STOC, 1998. [41] S. Arya, D. M. Mount, N. S. Netanyahu, R. Silverman, and A. Y. Wu, “An optimal algorithm for approximate nearest neighbor searching fixed dimensions,” in Journal of the ACM, 1998. [42] J. L. Bentley, “K-d trees for semidynamic point sets,” in SCG, 1990. [43] L. Cayton, “Fast nearest neighbor retrieval for bregman divergences,” in ICML, 2008. [44] M. Datar, N. Immorlica, P. Indyk, and V. S. Mirrokni, “Locality-sensitive hashing scheme based on p-stable distributions,” in SCG, 2004. [45] D. Alistarh, Z. Allen-Zhu, and J. Li, “Byzantine stochastic gradient descent,” in NeurIPS, 2018. [46] S. Bulusu, P. Khanduri, S. Kafle, P. Sharma, and P. K. Varshney, “Byzantine resilient non-convex scsg with distributed batch gradient computations,” in IEEE TSIPN, 2021. [47] Z. Allen-Zhu and E. Hazan, “Variance reduction for faster non-convex optimization,” in ICML, 2016. [48] M. Pirotta, M. Restelli, and L. Bascetta, “Policy gradient in lipschitz markov decision processes,” in Machine Learning, 2015. [49] S. J. Reddi, A. Hefny, S. Sra, B. Poczos, and A. Smola, “Stochastic variance reduction for nonconvex optimization,” in ICML, 2016. [50] M. Galvani, “History and future of driver assistance,” in IEEE IMM, 2019. [51] D. Burton, A. Delaney, S. Newstead, D. Logan, and B. Fildes, “Evaluation of anti-lock braking systems effectiveness,” in Monash Univ. Accident Research Centre, 2004. [52] P. Rieger, T. D. Nguyen, M. Miettinen, and A.-R. Sadeghi, “Deepsight: Mitigating backdoor attacks in federated learning through deep model inspection,” in NDSS, 2022. [53] T. D. Nguyen, P. Rieger, H. Chen, H. Yalame, H. Möllering, H. Fereidooni, S. Marchal, M. Miettinen et al., “Flame: Taming backdoors in federated learning,” in USENIX Security, 2022. [54] X. Cao and N. Z. Gong, “MPAF: Model poisoning attacks to federated learning based on fake clients,” in CVPR Workshops, 2022.
[55] Mercedes-Benz, ““safety first for automated driving” (safad): Mercedesbenz group,” July 2019. [Online]. Available: https://group.mercedesbenz.com/innovation/case/autonomous/safety-first-for-automateddriving-2.html [56] NHTSA, “Automated driving systems 2.0 – a vision for safety,” 2017. [Online]. Available: https://www.nhtsa.gov/ [57] G. Baruch, M. Baruch, and Y. Goldberg, “A little is enough: Circumventing defenses for distributed learning,” in NeurIPS, 2019. [58] V. Shejwalkar, A. Houmansadr, P. Kairouz, and D. Ramage, “Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning,” in IEEE S&P, 2022. [59] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in AISTATS, 2020. [60] C. Xie, K. Huang, P.-Y. Chen, and B. Li, “Dba: Distributed backdoor attacks against federated learning,” in ICLR, 2020. [61] H. H. Zhuo, W. Feng, Y. Lin, Q. Xu, and Q. Yang, “Federated deep reinforcement learning,” arXiv, 2019. [62] Y. Liao, Y. Xu, H. Xu, Z. Yao, L. Huang, and C. Qiao, “Parallelsfl: A novel split federated learning framework tackling heterogeneity issues,” in MobiCom, 2024. [63] L. Wang, Z. Javed, X. Wu, W. Guo, X. Xing, and D. Song, “Backdoorl: Backdoor attack against competitive reinforcement learning,” in IJCAI, 2021. [64] J. Cui, Y. Han, Y. Ma, J. Jiao, and J. Zhang, “Badrl: Sparse targeted backdoor attack against reinforcement learning,” in AAAI, 2024. [65] Y. Chen, Z. Zheng, and X. Gong, “Marnet: Backdoor attacks against cooperative multi-agent reinforcement learning,” in IEEE TDSC, 2022. [66] Q. Kuang, J. Wu, J. Pan, and B. Zhou, “Real-time uav path planning for autonomous urban scene reconstruction,” in ICRA, 2020. [67] J. Tisdale, Z. Kim, and J. K. Hedrick, “Autonomous uav path planning and estimation,” in IEEE Robotics & Automation Magazine, 2009. [68] L. Busoniu, R. Babuska, and B. De Schutter, “A comprehensive survey of multiagent reinforcement learning,” in IEEE TSMC, 2008. [69] M. Tan, “Multi-agent reinforcement learning: Independent vs. cooperative agents,” in ICML, 1993.