Agentic-SecPBFT: Agentic AI-Driven Proactive Security Framework for Wireless PBFT Consensus in Mobile Ad-Hoc Networks

1

Agentic-SecPBFT: Agentic AI-Driven Proactive Security Framework for Wireless PBFT Consensus in Mobile Ad-Hoc Networks

arXiv:2607.03269v1 [cs.NI] 3 Jul 2026

Haoxiang Luo, Yinqiu Liu, Ruichen Zhang, Guangyuan Liu, Gang Sun, Senior Member, IEEE, Hongfang Yu, Senior Member, IEEE, Zhu Han, Fellow, IEEE, and Dong In Kim, Life Fellow, IEEE

Abstract—The standard Practical Byzantine Fault Tolerance (PBFT) protocol, designed for stable, wired environments, exhibits critical vulnerabilities when deployed in settings like mobile ad-hoc networks, thus making it susceptible to sophisticated threats such as Sybil attacks, Byzantine collusion, and message manipulation. Existing static defense mechanisms are illequipped to handle the intelligent and coordinated nature of these attacks. To address this challenge, this paper leverages the Agentic AI paradigm to build a distributed multi-agent system in which each consensus node is equipped with an intelligent agent. These agents employ a hierarchical Multi-Agent Deep QNetwork (MADQN) algorithm to learn and execute proactive security policies in real-time. By observing local network behavior, message consistency, and dynamically maintained reputation scores, the agents collaboratively identify suspicious behavior and recommend defensive actions under standard PBFT quorum and membership rules, thereby improving the integrity of the consensus process. We refer to the resulting framework as Agentic-SecPBFT. Then, we formally model key attack vectors and conduct extensive simulations. The results demonstrate that Agentic-SecPBFT reaches a 95.0% attack detection rate with a 1.8% false positive rate. Compared with mainstream PBFT variants, it achieves 3.1× higher throughput with 56% lower latency on average under 33% malicious nodes, offering a robust and adaptive security solution for decentralized wireless systems. Index Terms—Wireless consensus, PBFT, network security, agentic AI, multi-agent deep reinforcement learning (MADRL).

I. I NTRODUCTION A. Background

T

HE rapid proliferation of the Internet of Things (IoT) and the evolution of autonomous systems have catalyzed a fundamental architectural shift from centralized cloud computing to decentralized edge intelligence [1]. In this decentralized landscape, ensuring data integrity, traceability, and trust among H. Luo is with the WeBank-NTU Joint Research Institute on Fintech, Nanyang Technological University, Singapore 639798, and also with the College of Computing and Data Science, Nanyang Technological University, Singapore 639798 (e-mail:[email protected]). Y. Liu, R. Zhang, and G. Liu are with the College of Computing and Data Science, Nanyang Technological University, Singapore 639798 (e-mail: [email protected]; [email protected]; [email protected].) G. Sun (corresponding author) and H. Yu are with the School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China (e-mail: {gangsun, yuhf}@uestc.edu.cn). Z. Han is with the Electrical and Computer Engineering Department, University of Houston, Houston, TX 77004, USA (email: [email protected]). D. I. Kim is with the Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, South Korea (e-mail: [email protected]).

inherently trustless entities is paramount [2]. Blockchain technology, with its immutable ledger and distributed consensus mechanisms, has emerged as a foundational layer for these next-generation networks. Specifically, in wireless ad hoc networks such as Vehicular Ad Hoc Networks (VANETs) and Industrial IoT (IIoT) [3], [4], blockchain enables secure Peerto-Peer (P2P) transactions and trusted data sharing without reliance on a vulnerable central authority [5]. However, the consensus mechanisms underpinning these blockchain systems, primarily the Practical Byzantine Fault Tolerance (PBFT) protocol, were originally conceived for static, high-bandwidth, and reliable wired networks, [6]. The direct transposition of PBFT to wireless mobile environments introduces critical performance bottlenecks and security vulnerabilities. Wireless links are inherently unreliable, subject to path loss, shadowing, and multipath fading phenomena such as Rician or Rayleigh fading [7]. In a wired network, a timeout typically indicates a node failure or a network partition. In a wireless network, a timeout is frequently caused by deep fading or temporary interference. While standard PBFT interprets this as a malicious leader’s silence, triggering expensive and often unnecessary view-change procedures that halt consensus progress and degrade throughput. Furthermore, the threat landscape in decentralized wireless networks has evolved significantly. Adversaries no longer rely solely on static, easily detectable malicious behaviors such as persistent packet dropping. Instead, they employ intelligent, adaptive strategies designed to exploit the limitations of traditional security protocols. For instance, selective dropping attacks [8], where malicious nodes oscillate between honest and dishonest behaviors to manipulate reputation scores. And Sybil attacks [9], where adversaries spawn multiple illegitimate identities to overwhelm voting processes, are particularly devastating in resource-constrained wireless networks. Existing solutions often rely on reactive reputation models or static thresholding. These methods fail to adapt to these complex, time-varying attack vectors, leading to a reactive security posture that is insufficient for critical infrastructure [10]. B. Research Challenges Deploying a robust, secure, and efficient consensus mechanism in a wireless mobile environment presents a triad of primary technical challenges that this research aims to address: • The Wireless Scalability-Reliability Trade-off: Standard PBFT requires an O(n2 ) message complexity to

2

reach consensus [11], where n represents the number of network nodes. In a wireless channel subject to interference, contention, and fading, the probability of successful message delivery decreases exponentially with network size and message volume [12]. High packet loss rates trigger timeout mechanisms indistinguishable from malicious behavior, leading to view changes. Optimizing this tradeoff requires a mechanism that can intelligently reduce message complexity without compromising Byzantine fault tolerance. • Adaptive and Intelligent Adversarial Behavior: Intelligent adversaries can exploit the forgetting factor of traditional trust models [13]. By behaving honestly to build a high reputation and then launching short-burst attacks, these nodes evade detection by static trust filters. Furthermore, in Sybil attacks, physically proximate nodes or a single node mimicking many can dominate the consensus group [14]. Distinguishing these attacks from legitimate behavior requires a system that learns temporal patterns and physical layer signatures. • Convergence in Dynamic Topologies: In Mobile Ad Hoc Networks (MANETs), the set of available validators changes rapidly due to node mobility. A consensus must distinguish between a node that has moved out of range. It belongs to a connectivity issue requiring a topology update. And a node that is withholding votes, which is a security issue requiring isolation [15]. Standard PBFT lacks the cross-layer awareness to make this distinction. These challenges have been exacerbated by the rise of intelligent, coordinated attacks that bypass static defense mechanisms. For example, reputation-based PBFT variants [16] can be deceived by attackers who build trust through honest behavior before launching attacks. Cryptographic solutions, such as Verifiable Random Functions (VRF) [17], enhance primary node election security but fail to counter message manipulation attacks [18]. The root of this vulnerability lies in a semantic gap within the PBFT security model. Its security is built on a mathematical model that counts signed messages from n distinct nodes to tolerate f faulty ones. Here, f represents the number of Byzantine nodes. In this model, a node is merely a cryptographic identity (i.e., a public key). However, in a wireless ad-hoc network, the real actors are physical devices. These devices are vulnerable to physical capture or software compromise and can easily generate multiple cryptographic identities, i.e., launch a Sybil attack [19]. Thus, an attacker controlling a single physical device can masquerade as multiple logical nodes, easily breaking the fundamental n ≥ 3f + 1 security assumption at the protocol level, unbeknownst to honest nodes. Static defense mechanisms, even reputation-based ones [17], [20], still operate at this flawed logical level, tracking the reputation of cryptographic keys rather than the trustworthiness of the underlying physical entities. Therefore, an effective defense system should reduce this semantic gap without treating a wireless fingerprint as an unforgeable credential. Our solution uses CSI/SINR observations and temporal behavior as probabilistic auxiliary evidence for associating logical identities

with physical transmitters, while cryptographic signatures and PBFT quorum rules remain the basis of consensus correctness. C. Our Contributions To bridge this gap, this paper introduces Agentic-SecPBFT. A holistic security framework leverages the emerging Agentic AI paradigm [21], [22] to transform consensus nodes from passive protocol executors into autonomous defenders. Unlike traditional designs, our framework employs a hierarchical Multi-Agent Deep Reinforcement Learning (MADRL) architecture. It comprises Local Consensus Agents (LCA) for realtime, device-level threat perception and a Cluster Optimization Agent (COA) for global strategy coordination. By utilizing a hierarchical Multi-Agent Deep Q-Network (MADQN) [23], these agents construct dynamic behaviors of potential adversaries, enabling them to predict and preemptively isolate threats based on cross-layer behavioral patterns. To the best of our knowledge, this is the first attempt that uses agentic AI to enhance the wireless consensus security. The main contributions can be summarized as follows: • Comprehensive Wireless and Attack Modeling: We formulate a rigorous mathematical model that integrates wireless channel characteristics into the consensus failure probability. Furthermore, we formally define three distinct attack models, including Sybil, On-Off, and Oscillating Byzantine. It helps us to analyze their impact on network liveness and safety, establishing a baseline for evaluating defense mechanisms. • Agentic AI-Enabled Security Framework: We propose a hierarchical multi-agent architecture. LCA operates on individual nodes to evaluate peer behavior and channel states in real-time, while COA functions at a higher level to manage topology and global learning parameters. The COA is outside the PBFT commit-critical path. Its authenticated outputs guide learning and reputation estimation but cannot commit blocks, change quorum thresholds, or independently revoke membership. • MADQN-Based Dynamic Consensus: We develop a hierarchical MADQN algorithm that optimizes the selection of the primary node and the consensus group. The algorithm utilizes a novel reward function that balances throughput, latency, and security penalties. It allows the system to anticipate suspicious behavior and recommend proactive actions, while safety-critical exclusion and view changes remain subject to standard PBFT validation rules. D. Paper Structure The remainder of this paper is organized as follows. Section II provides a comprehensive review of related work in wireless consensus optimization and AI-driven security. Section III details the system model, including the network topology, channel models, and formal definitions of the attack vectors. Section IV presents the Agentic AI framework, detailing the LCA and COA collaboration, the dynamic reputation mechanism, and the MADQN algorithm. Section V provides a security analysis of the framework against typical attacks. Section VI presents the extensive simulation results and discussion. Finally, Section VII concludes the paper.

3

TABLE I: Related Works Comparison Consensus GRBFT [16] DRDST [18] RoUBC [24] WBFT [25] TRCO [26] DVRC [27] DRC [28] Xiong et al. [29] ML BFT [30] DA-PBFT [31] Agentic-SecPBFT

Intelligent Paradigm % % % % % % % % % ⃝ !

Reputation Management ! ! % ! ! ! ! % % ⃝

Wireless Adaptation % % ! ⃝ ⃝ ⃝

!

!

% ⃝ % ⃝

Communication Complexity control ! ! ! ! ! ! ! ! ! ! !

Active Defense % % % % % % % % % % !

Semantic Gap Fill % % % % % % % % % % !

⃝: Partially mentioned; !: Fully supported; %: Not supported.

II. R ELATED W ORKS The optimization of consensus mechanisms for wireless and mobile environments is a rapidly evolving research area. Existing works can be broadly categorized into three streams, including communication complexity optimization, reputationbased trust mechanisms, and AI-driven security. In Table I, we summarize the related works to highlight the innovations and differences of Agentic-SecPBFT. A. Communication Complexity Optimization in Wireless PBFT To mitigate the O(n2 ) message complexity in PBFT, which is prohibitive in bandwidth-constrained wireless networks, researchers have primarily focused on grouping strategies. The authors in [18], [32] discuss various grouping methods where the network is divided into clusters, and consensus is reached hierarchically. This grouping method, called sharding, can reduce communication costs by enabling multiple shards to reach consensus in parallel. However, it also introduces centralization risks at the shard heads [33]. Specifically, in high-mobility scenarios such as VANETs or Low-Altitude Wireless Networks (LAWNets) [24], the overhead associated with frequent re-clustering and leader election often negates the benefits of reduced consensus traffic. Then it also leads to periods of instability where no consensus can be reached. Additionally, recent work by models the wireless PBFT network using Stochastic Geometry and Poisson Point Processes (PPP) [34]. They derive the end-to-end success probability considering slotted ALOHA as the multiple access technique [35]. Their framework allows for the calculation of optimal transmission intervals and viable coverage areas. While these provide a strong theoretical basis for physical layer optimization, it assumes honest nodes and does not account for Byzantine faults or intelligent jamming. Our work extends this by integrating the physical layer reliability model with an adversarial upper layer, bridging the gap between communication theory and consensus security. B. Reputation-Based Consensus Mechanisms Reputation systems aim to weigh votes or select leaders based on historical behavior, thereby excluding malicious

nodes from the consensus process [18], [25]. EigenTrust and similar models have been adapted for PBFT [36]. For instance, Deng et al. [26]designed Trusted and Robust Consensus Optimization (TRCO) for VANETs, which can optimize the robustness and complexity of consensus. Meanwhile, Zhang et al. introduced a Dynamic Vehicle Reputation Consensus (DVRC) to enhance the data security and communication efficiency of the VANETs [27]. These methods ensure that nodes with lower reputations will be excluded from the consensus group, or their voting weight will be reduced accordingly. However, most existing reputation models use static decay factors. If a node behaves honestly for time t, its reputation R increases linearly or exponentially [18], [25], [28]. Selective dropping attackers exploit this by calculating the exact time required to restore their reputation before attacking again. Furthermore, these systems typically fail to distinguish between packet loss due to poor channel conditions, such as fading, and malicious packet dropping, leading to high False Positive Rates (FPRs) in mobile environments. Therefore, we employ a dynamic, AI-driven reputation management. The Agentic AI analyzes the variance and volatility of a node’s behavior. C. AI-driven Method in Consensus Security The application of AI, e.g., Reinforcement Learning (RL), to consensus security is a nascent but promising field. For instance, in [37], the authors employed an AI-based attack detection method to verify that there were specific security vulnerabilities in the consensus, such as Denial of Service (DoS) attacks. Meanwhile, some work also proposes using RL to detect attacks in IoT networks by modeling the interaction as a Markov Decision Process (MDP) [38]. Deep Q-Networks (DQN) [39] and Proximal Policy Optimization (PPO) [29] have been used to select validators to minimize latency and optimize throughput. Current approaches largely utilize single-agent RL or centralized training with centralized execution. They suffer from scalability bottlenecks and create a single point of failure at the learning agent [40]. Furthermore, few studies explicitly model the wireless channel state as part of the RL state space. A node might have low throughput due to deep fading, not malice;

4

TABLE II: Summary of Notations Symbol dij f f′ hij k KRice N N0 Prx,ij Ptx t Ri,j SIN Rij Vavg γth µ Ψphy σf2 ωpenalty ωreward

Definition & Physical Meaning Euclidean distance between node i and node j Number of Byzantine faulty nodes in the wireless network Perceived fault tolerance threshold in the network under Sybil attack Small-scale fading coefficient modeling multipath effects Number of physical nodes controlled by the adversary in the network Rician K-factor, ratio of LoS to scattered power Total number of physical nodes in the MANET network Additive white Gaussian noise power spectral density Received signal power at node j from node i Transmission power of a wireless consensus node Reputation score of node j from node i at time t Signal-to-Interference-plus-Noise Ratio for link i → j Average movement velocity of the mobile nodes in MANET Minimum SINR threshold for successful packet decoding Forgetting factor in dynamic reputation update Physical certainty factor, weighting reputation penalty by channel quality Power of the scattered multipath components in Rician fading Penalty weight for suspicious node behavior Reward weight for honest node behavior

punishing such a node hurts network resilience and fairness. Thus, we utilize a MADRL approach with cooperative agents. We explicitly include Signal-to-Interference-plus-Noise Ratio (SINR) and Channel State Information (CSI) in the state space. This enables the agent to distinguish between channel errors and Byzantine faults, a critical distinction for mobile networks. Although there are a few works involving the MADRL and wireless consensus [41], they remain at their applications rather than focusing on the consensus security optimization. III. N ETWORK AND ATTACK M ODELS This section establishes the rigorous mathematical foundation for the proposed Agentic-SecPBFT framework. We explicitly model the stochastic nature of the wireless communications and formally define the adversarial capabilities that exploit the semantic gap between cryptographic identities and physical entities. We summary key notations in Table II A. Wireless Consensus Network Model We consider a decentralized MANET where nodes communicate over an open, shared wireless medium. The network model incorporates both the mobility of physical devices and the stochastic characteristics of wireless channels, which are critical for distinguishing between benign failures and malicious behavior. 1) Node Mobility and Topology: The network consists of N mobile nodes moving within a three-dimensional space R3 . The Random Waypoint Model [42] governs the mobility of node i. In this model, a node selects a random destination within the simulation boundary and moves towards it with a velocity vi uniformly distributed in [vmin , vmax ]. Upon reaching the destination, the node pauses for a duration Tpause

before selecting a new destination. This mobility creates a dynamic topology where the set of neighbors Ni (t) for any node i changes continuously over time t. 2) Wireless Channel and Fading Model: Unlike wired networks, where link failures are binary, wireless links suffer from signal degradation [6]. We model the channel using a composite fading model that accounts for both large-scale path loss and small-scale multipath fading. First, the received power Prx,ij is primarily determined by the distance dij . We employ a log-distance path loss mode:  dij +Xσ , d0 (1) where d0 is the reference distance, η is the path loss exponent, and Xσ represents shadowing effects modeled as a zero-mean Gaussian random variable with standard deviation σ. Then, given the potential for LoS links in modern wireless applications, e.g., drone swarms or vehicular convoys, we utilize the Rician fading model [43]. The channel coefficient hij follows a Rician distribution, where the received signal amplitude r has the Probability Density Function (PDF): ! ! r2 + ν 2 rν r I0 , (2) f (r) = 2 exp − σf 2σf2 σf2 

Prx,ij (dBm) = Ptx (dBm)−P L(d0 )−10η log10

where ν 2 is the power of the LoS component, σf2 is the power of the scattered multipath components, and I0 (·) is the modified Bessel function of the first kind with order zero. The fading severity is characterized by the Rician K-factor, namely ν2 . (3) 2σ 2 A higher KRice indicates a dominant LoS path and a more stable link, while KRice = 0 reduces the model to Rayleigh fading, that is, non-LoS. Next, the success of message reception is probabilistic. The instantaneous Signal-to-Interference-plus-Noise Ratio (SINR) at receiver j is calculated as: KRice =

SIN Rij =

N0 +

Prx,ij |hij |2 , 2 k∈T \{i} Prx,kj |hkj |

P

(4)

where T is the set of simultaneously transmitting nodes. A consensus message m is considered successfully received if and only if SIN Rij ≥ γth for the duration of the packet transmission. Here, γth is the minimum required SINR at the receiver. B. Adversary Model and The Semantic Gap We define a robust Byzantine Adversary with the following capabilities and constraints: Physical Control: The adversary fully controls k physical nodes, where k ≤ f . • Cryptographic Agility: The adversary can generate unlimited cryptographic key pairs (pk, sk) but cannot forge signatures of honest nodes. •

5

Collusion: All k controlled nodes share a private out-ofband communication channel, allowing them to coordinate attacks instantly and share internal state. • Awareness: The adversary is aware of the network topology and can estimate the reputation scores assigned to it by honest nodes. • Radio Frequency (RF) Manipulation: The adversary may vary transmission power, mobility, and antenna behavior to influence channel observations, but cannot forge an honest node’s digital signature or predict a fresh signed challenge before it is issued. CSI is not assumed to be a cryptographic or inherently unforgeable identity. It is used only as probabilistic evidence together with fresh signed challenge–response results and temporally consistent observations from multiple LCAs. Let Ephy denote the event that active voting identities are correctly associated with distinct physical transmitters, and let •

Pr[Ephy ] ≥ 1 − ϵphy ,

(5)

where ϵphy captures false splitting or false merging caused by fading, mobility, or deliberate RF manipulation. The security analysis explicitly conditions on Ephy and reports the corresponding residual probability rather than assuming perfect physical identification. Moreover, the fundamental vulnerability we address is the semantic gap in classic consensus. In PBFT, it assumes a bijection between a consensus node and a physical entity (maybe a server). However, in MANETs, this bijection breaks down. The protocol sees n public keys, while the physical reality contains only N devices. If an attacker maps one physical device to S public keys, the protocol operates under the illusion of high redundancy (n ≫ N ), while the physical resilience remains unchanged. C. Attack Models The aforementioned semantic gap will give rise to the following three types of typical attacks, as shown in Fig. 1. 1) Sybil Attack Model: The Sybil attack is one of the most devastating attacks in wireless ad-hoc networks. • Model: Assume an adversary controls k physical nodes (k ≤ f ). By exploiting vulnerabilities in the identity creation process, the adversary generates S Sybil identities for each controlled node. This changes the perceived total number of nodes in the network to N ′ = (N − k) + k · S. • Mechanism: This proliferation of identities directly undermines PBFT’s n ≥ 3f + 1 assumption. The condition for a successful attack can be formalized as N′ − 1 ⌋. (6) k · S > f ′ , where f ′ = ⌊ 3 If the total number of Sybil identities controlled by the attacker k · S is greater than the new perceived fault tolerance threshold f ′ , the attacker can independently forge a “prepared certificate” for a malicious request. Thus, it can deceive honest nodes into believing the request has been endorsed by sufficient nodes. • Impact: In MANETs, the dynamic topology of nodes further increases the difficulty in distinguishing Sybil identi-

Fig. 1: Attack models for wireless consensus networks. (a) Sybil attack; (b) Byzantine collusion; (c) Message delay and dropping. ties from legitimate new nodes. The forged messages not only consume limited bandwidth and cause conflicts, but also make it difficult for honest nodes to collect sufficient valid votes. As a result, it triggers frequent view switches and paralyzes the consensus process [44]. 2) Byzantine Collusion Model (Selective Forwarding): This attack is more subtle than direct message forgery and uses node collusion to disrupt network liveness. • Model: Assume f nodes are colluding, for instance, a malicious primary and f − 1 malicious replicas. • Mechanism: The malicious primary first broadcasts a valid pre-prepare message. In the subsequent prepare phase, the f − 1 malicious replicas adopt a selective forwarding strategy: they send prepare messages to a subset of the honest nodes while remaining silent to the other half. Concurrently, these malicious nodes do not forward any messages from honest nodes, creating an information barrier between them. • Impact: This leads to a “split-brain” scenario. Some honest nodes may receive enough prepare messages and enter the ‘prepared‘ state, while others fail to do so due to insufficient messages. Ultimately, the network cannot gather 2f + 1 commit messages, causing a consensus timeout and triggering a costly view change process. This attack specifically targets system liveness, paralyzing the network by continuously creating deadlocks [45]. 3) Message Delay and Dropping Model: The most dangerous attacks in wireless networks are those that blur the line between inherent network instability (e.g., packet loss due to signal fading) and deliberate malicious behavior (e.g., selective dropping). A successful defense system must be able to probabilistically distinguish between these two conditions. • Model: A malicious primary aims to degrade system performance and liveness in an unnoticeable way. • Mechanism: The primary can use the unreliability of the wireless channel as cover. It can selectively delay sending pre-prepare messages to specific honest nodes or

6

drop them entirely. Since occasional packet loss is normal in a wireless environment, this behavior is difficult to immediately identify as malicious. • Impact: The targeted nodes will time out and mistakenly assume the primary has failed, initiating a view change. The attacker can exploit this ambiguity to continuously trigger unnecessary leader elections, leading to a state of chronic disruption and inefficiency, severely reducing effective throughput and increasing transaction latency. A simple, threshold-based rule system is ineffective in this dynamic environment. It may misidentify honest nodes with poor link quality as malicious (false positives) or fail to identify attackers who cleverly manage their drop rates (false negatives) [46]. IV. AGENTIC AI-E NABLED S ECURITY F RAMEWORK To counter these attacks, we employ Agentic AI to transform the consensus nodes from passive protocol executors to active and autonomous defenders. In this mode, Agentic AI’s function is defined as follows: • Contextual Perception: Fusing upper-layer protocol states with physical-layer CSI to construct a holistic view of the adversarial environment. • Strategic Autonomy: Executing decentralized security policies that optimize long-term network liveness and integrity, rather than merely reacting to immediate protocol violations. • Hierarchical Collaboration: Leveraging a two-tiered multi-agent architecture to balance low-latency responses with strategic adaptation. A. Agentic Hierarchical Structure The complexity of securing a wireless consensus necessitates a separation of concerns. We define a hierarchical agent structure comprising LCAs and the COA, as shown in Fig. 2. The framework employs a Centralized Training, Decentralized Execution (CTDE) paradigm, dividing responsibilities between two classes of agents. 1) LCA Layer: The LCA is an embedded intelligent agent residing on every physical consensus node i ∈ N . It is characterized by real-time execution and local observability. (i) The LCA perceives the node’s local environment state st , which includes SINR, CSI, etc. Also, the LCA is responsible for instantaneous threat detection and response. It executes the learned policy πlocal to make micro-decisions, such as verifying a suspect signature, challenging a potential Sybil node, or filtering messages from a low-reputation peer. Notably, operations must be completed within the PBFT phase timeout windows. 2) COA Layer: Its primary role is not to make real-time security decisions but to guide the learning process of the individual LCAs. In a decentralized setting, the COA function is either hosted on a high-availability super-node, e.g., a base station in VANETs [18], or distributed via a Secure MultiParty Computation (SMPC) committee [47]. The COA aggregates telemetry from multiple LCAs to construct a quasi-global state representation Sglobal . This allows it to detect patterns

Fig. 2: Hierarchical agent structure comprising LCAs and the COA. It consists of telemetry accumulation, uplink transmission, global optimization, and downlink broadcast. invisible to individual nodes, such as network-wide partition attacks or coordinated “split-brain” voting strategies. Also, the COA is responsible for training the global policy network θglobal based on aggregated experiences and disseminating updated model parameters and global reputation scores back to the LCAs. Meanwhile, decisions occur on an epochal basis, e.g., every 100 blocks, allowing for computationally intensive optimization without stalling consensus. 3) Interaction between COA and LCA: To enable robust collaboration between the LCA and COA, we define the Agentic Interaction Protocol (AIP). The specific steps are as follows: • Step 1. Telemetry Accumulation: During a consensus epoch E, the LCA buffers significant experience tuples, namely transitions where the reward prediction error δ > ϵ, into a local Experience Replay Buffer. Where δ = |Q(ht , a) − Qtarget (ht , a), denoting the absolute error between the current Q-value and the target Q-value • Step 2. Uplink Transmission: At the end of epoch E, the LCA compiles an AIP_TELEMETRY packet and unicasts it to the COA. • Step 3. Global Optimization: The COA aggregates received packets, performs a training step on the global MADQN, and computes the global reputation vector. • Step 4. Downlink Broadcast: The COA broadcasts an AIP_POLICY_UPDATE packet containing the new model weights and trust scores. The packet also carries the epoch number, monotonically increasing version, model digest, and COA authentication tag, enabling LCAs to verify integrity and reject replayed updates. All AIP updates are signed, epoch-bound, and versioned. LCAs reject unauthenticated, stale, or rollback updates. Even a faulty COA cannot generate PBFT commit certificates, alter

7

quorum thresholds, or permanently remove validators. If it is unavailable or detected as inconsistent, LCAs retain the latest authenticated policy and continue standard PBFT execution. B. Dynamic Trust and Reputation Management In wireless networks, a packet timeout can result from either deep fading (benign) or selective forwarding (malicious). Traditional models penalize both equally, leading to high false positives. We should define the expected delivery probability Psuccess (i, j) based on the outage event of link ij as follows: Psuccess (i, j) = 1 − Pout (i, j),

(7)

where Pout (i, j) = P[SIN Rij < γth ] denotes the link outage probability. This should be evaluated by taking the expectations with respect to both the shadowing Xσ in Eq. (1) and the fading |hij | = r in Eq. (2). t Then, we model the reputation Ri,j ∈ [0, 1] as node i’s belief in node j’s honesty. To keep the score within its declared range while preserving the original incremental update, we use t+1 t Ri,j = Π[0,1](Ri,j + (1 − µ)[Ihonest ωreward −

Isuspect ωpenalty Ψphy ]),

(8)

where Π[0,1] (x) = min{1, max{0, x}} is the projection operator, µ is the forgetting factor, and Ihonest and Isuspect indicate validated and suspicious behavior, respectively. The physical certainty factor is Ψphy = Psuccess (i, j)(1 − ξv ),

(9)

where ξv = min{1, vi /Vmax } attenuates confidence under high mobility. Thus, a timeout under a poor link causes only a small penalty, whereas a missing message under a reliable link causes a larger penalty. The reputation value is used for monitoring and node-selection recommendations; it cannot by itself revoke PBFT voting rights. C. Hierarchical MADQN for Proactive Defense The core decision engine of the LCA is the MADQN. Beyond proactive security defense, this framework also integrates agentic interference management to ensure the SINR in Eq. (4) meets the minimum rate requirement Rmin = B log2 (1 + γth ) for consensus message transmission, where B denotes the channel bandwidth. We formulate the security problem as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), defined by the tuple ⟨S, A, P, O, R⟩. The goal is to learn a policyP π that maximizes the expected discounted ∞ return J(π) = Eπ [ t=0 γ t rt ]. 1) State Space S: The state space S is designed to capture the fingerprints of the three specific attack vectors. The obser(i) vation vector st for agent i is a concatenation of consensus, network, and reputation features. (i)

st = [Fcon , Fnet , Frep ] ,

(10)

where Fcon denotes the consensus features, including current PBFT phase P haset , view number V iewt , the vector of recent voting behaviors of neighbors Vhistory , and the shannon entropy of the voting distribution Entropyvote , as shwon

below. Fnet shows the network features, including the vector of SINR values Γvec for neighbors SIN Rij , the matrix of spatial correlation between CSI of different neighbors CSIcorr , the variance in message arrival times ∆tarrival , the average interference intensity of the current slot Iavg , and the number of concurrent transmitters in the neighborhood Ntx . Frep is the reputation features, including current local reputation scores 2 Rlocal ,variance of reputation over time σR . X Entropyvote = − p(v) log2 p(v), (11) v∈{0,1}

where p(v) represents the probability of a neighbor voting in favor 1 or against 0. The higher the entropy value, the more chaotic the voting is, and the more likely there is a Byzantine collusion. 2) Action Space A: The LCA has a discrete action space designed to execute specific security countermeasures. The (i) action vector at for agent i includes: • Participate Normally: The default action when no threat is perceived. • Flag Suspicious Node: A preliminary measure to increase monitoring of a specific node without immediate punitive action. • Decrease Reputation Score: Locally adjust a neighbor’s reputation based on evidence of misbehavior; a global reputation update must be delivered through an authenticated AIP update. • Request Challenge-Response: Issue a cryptographic challenge to a suspicious node to verify its liveness and authenticity. This helps counter Sybil nodes that may be computationally constrained by forging multiple identities. • Vote for View Change: Proactively initiate a view change when judging the primary’s behavior to be malicious or inefficient. • Temporarily Reject Messages: Temporarily deprioritize non-critical traffic from a node suspected of a severe attack (e.g., DoS flooding); valid signed PBFT votes remain governed by standard quorum and membership rules. • Adjust Transmission Power: Dynamically tune transmission power within Upper and lower bounds of transmission power, namely [Ptx,min , Ptx,max ] to compensate for channel fading and mitigate interference. • Request Priority Slot: Submit a reservation request to the COA for critical consensus messages to avoid collision with concurrent transmissions. 3) Transition Probability P: Let st = (ct , pt , rt ) denote the wireless-channel, PBFT-protocol, and reputation states, and let at = (aphy , asec t t ) contain communication and security actions. Their coupled transition is factorized as P(st+1 |st , at ) =P (ct+1 |ct , aphy ) t · P (pt+1 |pt , ct+1 , asec t )

(12)

· P (rt+1 |rt , pt+1 , ct+1 , asec t ). This factorization captures the dependence of consensus and reputation evolution on the realized wireless state, rather than

8

treating channel and protocol dynamics as mutually exclusive alternatives. 4) Observation Space O: Each LCA obtains a noisy local observation oi,t = Oi (st , at−1 ) + εi,t , (13) where the CSI/SINR components of εi,t follow channelestimation noise calibrated by link quality, while cryptographically verified protocol fields are treated as discrete observations. This avoids assuming one Gaussian model for all heterogeneous features. 5) Reward Function R: The design of the reward function is critical for training effective defense agents. It is designed (i) as a multi-objective function rt for agent i, calculated at each time step: (i)

rt = w1 · rconsensus + w2 · rsecurity + w3 · rliveness .

(14)

rconsensus : A large positive reward for successfully committing a valid block; a large negative reward if consensus fails or a fraudulent block is committed. • rsecurity : During simulation training, ground-truth attack labels provide positive rewards for confirmed detection and negative rewards for false alarms. During deployment or online adaptation, a security reward is generated only after verifiable evidence, such as an invalid signature, a failed fresh challenge–response, or a PBFT-certified view change; an unconfirmed local suspicion receives zero reward. • rliveness : A small negative reward for each time step that consensus stalls; a larger negative penalty for triggering a view change; a small positive reward for successfully maintaining SIN Rij ≥ γth through power adjustment or reserving a priority slot for critical messages. This incentivizes the agent to maintain network progress, avoid unnecessary disruptive actions, and proactively optimize transmission conditions. The weights w1 , w2 , and w3 are tuned to prioritize security and correctness over raw performance. •

D. Algorithm Steps The algorithm on the LCA is shown as Fig. 3 and follows: • Step 1. Perception and Pre-processing: At time step t, the LCA collects raw physical layer metrics (Ht ) and protocol messages (Mt ). It computes the spatial correlation of CSI to detect potential Sybil clusters and calculates the physical certainty factor Ψphy . • Step 2. State Construction and Inference: The processed features form state st . This is fed into the local Qnetwork. To handle partial observability, we use an LSTM layer to aggregate historical states ht = LSTM(st , ht−1 ). The Dueling head then estimates the value: ! 1 X ′ A(ht , a ) , Q(ht , a) = V (ht ) + A(ht , a) − |A| ′ a (15) where V (ht ) represents the global state value. A(ht , a) is the action advantage value, which is output by an

Algorithm 1: Centralized Training by COA for Agentic-SecPBFT Initialization: 1. Initialize global Q-network parameters θglobal and ′ target Q-network θglobal 2. Initialize global replay buffer Dglobal and global reputation matrix Rglobal = 0 3. Broadcast initial θglobal and Rglobal to all LCAs for epoch = 1 to Emax do Telemetry Aggregation: (i) 1. Receive local experience buffers Dlocal and (i) reputation matrices Rlocal from all LCAs i 2. Aggregate local experiences into Dglobal 3. Compute global reputation Rglobal via weighted (i) averaging of Rlocal Global Model Update: 1. Sample a batch of transitions from Dglobal if Dglobal ≥ B 2. Calculate target Q-values ′ Qtarget = r + γ · maxa′ Qθglobal (s′ , a′ ) 3. Update θglobal by minimizing MSE loss between Qtarget and Qθglobal (s, a) 4. Soft-update target network: ′ ′ θglobal ← τ · θglobal + (1 − τ ) · θglobal (every epochs) Parameter Broadcast: 1. Broadcast the signed and versioned update containing θglobal and Rglobal to all LCAs 2. Clear Dglobal (or retain partial historical experiences)

independent branch of the Dueling head. Both have the same dimension as the action space A. The action at is selected using an ϵ-greedy policy. • Step 3. Action Execution and Defense: If at = Challenge(j), the LCA sends a fresh signed challenge. If j fails or times out, Ri,j is penalized according to the channel-aware rule. If at = Isolate(j), only non-critical traffic is locally deprioritized; valid signed PBFT messages remain processable. Permanent exclusion requires a membership update approved by the underlying PBFT protocol. Otherwise, standard PBFT voting proceeds. • Step 4. Reward Calculation and Storage: After the consensus round, the LCA evaluates the outcome (Block Committed vs. View Change). It calculates reward rt and stores the transition ⟨st , at , rt , st+1 ⟩ in the local replay buffer. • Step 5. CTDE Synchronization: Periodically, the LCA uploads samples to the COA via AIP. The COA trains the global model and returns updated weights, which the LCA integrates. E. Complexity Analysis The computational complexity of the MADQN-based Agentic-SecPBFT framework is split into two phases: offline

9

Fig. 3: The algorithm steps of hierarchical MADQN, combining centralized training by COA and distributed execution by LCA. Algorithm 2: Decentralized Execution by LCA for Agentic-SecPBFT Initialization: 1. Load global parameters θlocal ← θglobal and Rlocal ← Rglobal from COA 2. Initialize local replay buffer Dlocal , ϵ-greedy factor ϵ = ϵinit , and LSTM hidden state h0 = 0 for time step = 1 to Ttotal do State Perception: 1. Extract consensus Fcon , network Fnet , and reputation Frep features 2. Construct local state st = [Fcon , Fnet , Frep ] and aggregate via LSTM: ht = LSTM(st , ht−1 ) Defense Action Selection: 1. Select action at via ϵ-greedy policy (random if rand() < ϵ, else arg maxa Qθlocal (ht , a)) 2. Execute at under the PBFT safety guardrail (e.g., flag suspicious nodes, locally update reputation, or request a challenge) Reputation and Reward Update: 1. Update Rlocal based on neighbor behavior and channel quality (Ψphy ) 2. Calculate rt , using ground-truth labels only in simulation and verified evidence during online adaptation 3. Store transition (st , at , rt , st+1 ) in Dlocal 4. Decay ϵ: ϵ ← max(ϵmin , ϵ · ϵdecay ) COA Synchronization: 1. Upload Dlocal and Rlocal to COA (every Tepoch steps) 2. Receive and update θlocal ← θglobal and Rlocal ← Rglobal

training (COA) with pre-deployment cost, and online execution (LCA) for real-time edge performance. 1) Offline Training Complexity (COA): The offline phase shown in Algorithm 1) is computationally intensive, with complexity dependent on max training epochs Emax , consensus rounds per epoch Tepoch , LCA number N , batch size B, and global forward pass complexity of global Q-networks CQ and local local Q-networks CQ . Let Eupdate denote parameter update steps per epoch. 1) Telemetry Aggregation: All N LCAs upload preprocessed local data to COA per Tepoch round, with comlocal plexity O(Tepoch · N · CQ ). 2) Global Model Update: COA samples B transitions and updates the global Q-network per Eupdate step: global Target Q-value calculation: O(B · CQ ); • Network parameter update (MSE loss minimizaglobal tion): O(B · CQ ); • Target network soft-update (every C epochs): global O(CQ ), which is negligible. •

Thus, the total COA training complexity is O(Emax ·(Tepoch · global local N · CQ + Eupdate · B · CQ )). 2) Online Execution Complexity (LCA): The online phase shown in Algorithm 2) is lightweight, with complexity dependent on the total consensus rounds Ttotal , LCA number N , and neighbors per LCA Ni (t). 1) State Perception & Action Selection: Each LCA con(i) structs state st , runs LSTM aggregation, and selects aclocal tions via ϵ-greedy policy per round: O(Ttotal ·|N |·CQ ). 2) Reputation Update & Synchronization: Reputation matrix update: O(Ni (t)|), it is negligible, due to Ni (t) ≪ N ); • Reward calculation: O(1); • COA synchronization (every Tepoch rounds): negligible.

•

10

Thus, the total LCA execution complexity is O(Ttotal · N · local CQ ). V. S ECURITY A NALYSIS A. Informal Analysis Against Attacks 1) Sybil Attack Defense: Agentic-SecPBFT mitigates Sybil attacks by reducing the semantic gap between cryptographic identities and physical transmitters. Each LCA uses CSI spatial correlation and the Rician K-factor as probabilistic evidence rather than as an identity credential. Suspected co-located identities are further checked through fresh signed challenge– response exchanges and temporally consistent observations from multiple LCAs. The COA aggregates authenticated telemetry to identify network-wide Sybil patterns, and its result adjusts monitoring and reputation recommendations. Consequently, the framework reduces identity-proliferation risk with residual error ϵphy . 2) Byzantine Collusion Defense: Against selective forwarding collusion, LCAs monitor consensus-layer features, such as Entropyvote , Vhistory , to detect “split-brain” anomalies. The colluding nodes create inconsistent voting states, driving entropy beyond legitimate ranges. Then, LCAs execute defensive actions, e.g., reputation penalty, flagging, while the COA identifies coordinated patterns via global telemetry, updating Rglobal to reflect systemic threats. Meanwhile, it reduces deadlocks caused by insufficient 2f + 1 commit messages through proactive monitoring, non-critical traffic deprioritization, and challenge–response, without discarding valid PBFT votes or changing the active membership directly. This can avoid reducing the system’s activity due to passive view changes. Additionally, the hierarchical MADQN reward function strengthens early detection by rewarding threat mitigation and punishing consensus stalls. 3) Message Delay/Dropping Defense: The framework distinguishes benign fading from malicious message manipulation via the channel-aware metric Psuccess . LCAs also weight reputation penalties by Ψphy = Psuccess , including low SINR (SIN Rij < γth ) attributes loss to fading (minimal penalty), while high SINR (SIN Rij ≫ γth ) infers malicious behavior (maximal penalty). Meanwhile, hierarchical MADQN’s state space integrates SINR vectors and message arrival variance, enabling LCAs to learn stealthy delay patterns. Additionally, the reward function minimizes false positives/negatives, ensuring honest nodes with poor channel conditions are not penalized. Attackers exploiting channel unreliability are flagged for additional verification and protocol-governed mitigation. B. Formal Security Analysis We separate PBFT correctness from the effectiveness of the learned defense policy. The following analysis does not assume that MADQN converges to an optimal policy, that LCA observations are error-free, or that the COA is always available. We use the following premises: • The active PBFT committee is configured with N = 3f + 1 voting identities, and at most f corresponding physical validators are Byzantine.

Digital signatures of honest validators cannot be forged, and an honest validator does not sign conflicting blocks at the same height and view. • Learned actions cannot create commit certificates, change the 2f + 1 quorum, or permanently modify committee membership. • The physical-association event Ephy occurs with probability at least 1 − ϵphy ; CSI is only auxiliary evidence for this event. • AIP messages are authenticated and versioned. A faulty or unavailable COA may reduce detection quality, but its output remains advisory and is not required to complete PBFT. •

Theorem 1 (Policy-Independent Safety) Conditioned on Ephy , no two honest validators commit distinct blocks at the same height and view, regardless of the convergence state or detection errors of MADQN. Proof : Assume that two honest validators commit conflicting blocks B ̸= B ′ . Each block must contain a valid commit certificate with at least 2f + 1 distinct signatures. Let Q and Q′ denote the two signer sets. Since N = 3f + 1, |Q ∩ Q′ | ≥ (2f + 1) + (2f + 1) − (3f + 1) = f + 1. (16) At most f physical validators are Byzantine under Ephy ; hence, the intersection contains at least one honest validator. This validator would have signed both B and B ′ , contradicting the honest signing rule. MADQN and the COA cannot forge this signature or alter the quorum threshold. Therefore, safety is independent of the learned policy. Moreover, Pr[Safety] ≥ Pr[Ephy ] ≥ 1 − ϵphy .

(17)

■ Property 1 (COA Fault Containment) A faulty COA can distribute a poor policy or biased reputation recommendation, but cannot by itself cause conflicting PBFT commits. Reasoning: LCAs reject unauthenticated or stale AIP messages, while a valid COA update still contains no validator signatures for a block. Block commitment and permanent membership changes remain controlled by PBFT certificates. Thus, COA compromise affects detection efficiency and may increase latency, but does not bypass the safety quorum. Theorem 2 (Conditional Liveness) Assume partial synchrony, eventual selection of an honest primary, and at least 2f + 1 responsive active validators after the network stabilization time. Then every valid request submitted to an honest validator is committed in finite time, independently of MADQN convergence and temporary COA unavailability. Proof : After the network stabilization time, messages among responsive honest validators are delivered within a bounded delay. Once an honest primary is selected, it broadcasts a valid proposal and at least 2f + 1 responsive validators can complete the prepare and commit phases. Learned actions cannot lower the quorum, permanently exclude a validator, or suppress valid signed PBFT messages outside the standard protocol rules. If the COA is unavailable, LCAs continue with the latest authenticated policy and standard PBFT view change. Therefore, the system eventually forms a valid commit

11

TABLE III: Simulation Parameters Parameter

Value Network Parameters Simulation Area 500 × 500 × 200 m Number of Nodes (N ) 10-50 Average Speed (Vavg ) 10-30 m/s Pause Time (Tpause ) 1-5 s Steepness Factor (β) 2 Wireless Channel Carrier Frequency 2.4 GHz Transmit Power (Ptx ) 1W Noise Power Density (N0 ) -174 dBm/Hz SINR Threshold (γth ) 10 dB Rician K-factor (KRice ) 2-10 dB Transmission Power Range 0.1W, 2W (Ptx,min , Ptx,max ) Channel Bandwidth (B) 20 MHz Minimum Rate Requirement (Rmin ) 1 Mbps Consensus & PBFT Block Size 1024 bytes Transaction Arrival Rate (TPS) 10-30 Malicious Node Ratio (f /N ) 0% - 33% Dynamic Reputation Model Forgetting Factor µ 0.85 Reward Weight ωreward 0.1 Penalty Weight ωpenalty 0.5 MARL Parameters Learning Rate (α) 0.001 Discount Factor (γ) 0.99 Epsilon (ϵ-greedy) Initial 1.0, decaying to 0.01 ω1 , ω 2 , ω 3 0.3, 0.5, 0.2 Experience Replay Buffer Size 100,000 Neural Network Architecture 3-layer MLP (128, 256, 128)

certificate. MADQN improves the probability of satisfying the reachability and responsiveness conditions by reducing false alarms and unnecessary view changes, but liveness does not rely on optimal learning convergence. ■ VI. P ERFORMANCE E VALUATION A. Simulation Setup and Baseline Schemes We build the simulation environment based on a timesynchronized joint framework of NS-3 and Python 3.9. Specifically, NS-3 performs high-fidelity modeling of wireless physical layer characteristics and MAC-layer transmission collisions, which provides realistic wireless channel states for the upper-layer consensus. The Python module runs the consensus, dynamic reputation management, and hierarchical MADQN algorithm. It runs on a server equipped with three 96-core Intel(R) Xeon(R) Gold 5220R CPUs, 1 TB of memory, and 8 NVIDIA GeForce RTX 3090 GPUs. All parameters are listed in Table III. To assess the convergence performance of the proposed Agentic AI-enabled hierarchical MADQN, we compare it with the advanced Double Deep Q-Network (DDQN) [48] and MADQN (without the COA in our hierarchical framework). Furthermore, to verify the optimization effect of this Agentic-SecPBFT, we also perform a performance comparison with the original PBFT, Gaussian ReputationBased BFT (GRBFT) [16], and Dynamic Adaptive PBFT (DA-PBFT) [31]. Finally, we conduct a series of ablation experiments. B. Convergence Performance We compare the convergence performance in four combinations of low mobility (Vavg = 10 m/s) and high mobility

(Vavg = 30 m/s), high interference (Tpause = 1 s) and low interference (Tpause = 5 s). The interference here refers to the communication instability caused by node mobility, dynamic topology changes, and wireless channel fading. Across all four scenarios, hierarchical MADQN achieves the best convergence performance, followed by MADQN, while DDQN consistently lags in both convergence speed and stability. In Fig. 4 (a), hierarchical MADQN converges rapidly, with final average rewards maintained in the 160–180 range and slight fluctuations. MADQN, lacking global COA coordination, converges at a moderate pace and stabilizes around 130–150, outperforming DDQN but falling short of the hierarchical design. DDQN converges more slowly and has fluctuations approximately twice that of hierarchical MADQN. In Fig. 4 (b), hierarchical MADQN still maintains stable convergence, with rewards remaining in the 140–160 range. MADQN shows more noticeable oscillations under high mobility but still retains a clear performance advantage over DDQN. In contrast, DDQN exhibits more obvious oscillations and occasional reward drops due to high node mobility. For Fig. 4 (c) and Fig. 4 (d), hierarchical MADQN’s advantages are also prominent. It can quickly adapt to the unstable communications, with final average rewards remaining at 130-150 and 110-130, respectively, as the environment becomes more challenging. MADQN follows with moderate performance degradation, while DDQN struggles to cope with the combined impact of interference and mobility, showing slow convergence speed and severe reward oscillations in the high mobility and high interference scenario. The reason is that in hierarchical MADQN, LCAs can perceive channel states and node mobility changes in real time, while the COA optimizes global strategies based on aggregated telemetry. It enables the system to proactively adapt to environmental changes. C. Consensus Security We randomly run 50-round simulations based on Table III, to compare the consensus security under attack conditions. Specifically, we simulate under different simulation steps, malicious node ratios, and interference intensity. The three Attack Detection Rate (ADR) comparison results shown in Fig. 5 highlight Agentic-SecPBFT’s unique proactive defense capability. Fig. 5 (a) shows that its ADR climbs to 95.0% after 500 steps, benefiting from the hierarchical MADQN algorithm’s continuous learning of adversarial behavioral patterns. In contrast, PBFT’s ADR remains below 11% without a detection mechanism. GRBFT and DAPBFT’s rates stall at 40.2% and 65.0% respectively. Fig. 5 (b) reveals that Agentic-SecPBFT’s ADR increases with the attack intensity, reaching 95.8% at 33% malicious nodes. This is because more attack samples enhance the hierarchical MADQN’s pattern recognition accuracy. Fig. 5 (c) demonstrates its anti-interference advantage. Even at the highest interference, Agentic-SecPBFT’s ADR remains 92.5%, while GRBFT and DA-PBFT’s rates drop to 38.2% and 60.5%, respectively. Agentic-SecPBFT’s excellent attack detection performance is attributed to its Dec-POMDP formulation. It integrates consensus-layer features, network-layer features,

12

Fig. 4: Convergence performance comparison. (a) Low mobility & low interference; (b) High mobility & low interference; (c) Low mobility & high interference; (d) High mobility & high interference. and reputation-layer features into a unified state space, making the hierarchical MADQN algorithm capable of capturing the multi-dimensional fingerprints of attacks. Also, the COA aggregates global telemetry data from all LCAs, enabling the identification of network-wide coordinated attack patterns that are invisible to a single node. Additionally, it fuses physicallayer CSI/SINR information to distinguish malicious behavior from channel fading caused by interference, thus maintaining high detection accuracy in complex wireless environments. Moreover, Agentic-SecPBFT’s high detection precision is reflected in its consistently low False Positive Rate (FPR) across all scenarios. Fig. 6 (a) shows its FPR stabilizes at 1.8% after 500 steps, far below PBFT’s 10.2%, GRBFT’s 4.0%, and DA-PBFT’s 3.5%. Fig. 6 (b) indicates that even with 33% malicious nodes, Agentic-SecPBFT’s FPR remains at 2.2%, as its dynamic reputation mechanism filters out false alarms. Fig. 6 (c) confirms its ability to avoid misjudging interference as malicious. when Tpause decreases from 5s to 1s, its FPR only rises from 1.7% to 2.2%, while PBFT’s FPR surges from 8.8% to 12.5%. The higher FPR of others is due to their lack of physical layer context awareness, leading to indiscriminate penalties for packet loss or node unresponsiveness. The key to Agentic-SecPBFT’s ultra-low FPR is its channel-aware reliability metric Psuccess . It quantifies the message delivery probability based on real-time channel conditions. And the physical certainty factor Ψphy weights the reputation penalty for suspicious behavior. It weakens the penalty when the channel quality is poor to avoid misjudging honest nodes affected by fading, and imposes maximum penalties only when the channel is reliable. In addition, its dynamic reputation update rule combines a forgetting factor with channel quality. It can effectively filter environmental noise and ensure that honest nodes with poor channel conditions are not misclassified. D. Consensus Performance Then, we compare their consensus success rate, latency, and throughput under different conditions, such as the malicious node ratio, the number of nodes, and the speed of nodes. For the malicious node ratio, Fig. 7 (a) shows AgenticSecPBFT maintains a high success rate, dropping slightly from 99.0% to 80.0%. Standard PBFT plummets drastically from 98.5% to only 10.2%, and GRBFT and DA-PBFT fall to 40.5% and 55.2%. In Fig. 7 (b), when the number of nodes scales from 10 to 50, Agentic-SecPBFT remains stable with

a success rate ranging from 98.5% to 99.0%. PBFT’s success rate drops to 78.3% due to excessive communications. As shown in Fig. 7 (c), with the increase in node speed from 10 m/s to 30 m/s, Agentic-SecPBFT’s success rate only slightly decreases from 99.2% to 97.0%. While PBFT experiences a sharp drop from 98.5% to 79.2%, and GRBFT and DA-PBFT also show obvious declines. The reasons for this superior robustness lie in three core designs of Agentic-SecPBFT. It reduces the semantic gap by using CSI spatial correlation and Rician K-factor analysis as auxiliary evidence for Sybil detection, while retaining cryptographic authentication and PBFT quorum validation. The hierarchical MADQN enables proactive identification and isolation of malicious nodes, preventing them from disrupting the consensus process. Its crosslayer perception capability fuses SINR and CSI features to accurately distinguish benign node mobility from malicious behavior, avoiding unnecessary view changes that damage consensus stability. As shown in Fig. 8 (a), when the malicious node ratio increases from 0% to 33%, Agentic-SecPBFT’s consensus latency increases moderately from 0.20 s to 0.48 s. While PBFT’s latency surges by 507% from 0.28 s to 1.70 s due to frequent view changes triggered by malicious nodes. GRBFT and DA-PBFT’s latency reach 1.10 s and 0.80 s respectively at 33% malicious nodes. In Fig. 8 (b), as the number of nodes scales from 10 to 50, Agentic-SecPBFT’s latency rises gently from 0.18 s to 0.32 s, whereas PBFT’s latency skyrockets to 1.10 s due to the exponential growth of redundant communication. With the node moving speed increasing from 10 m/s to 30 m/s, as shown in Fig. 8 (c), Agentic-SecPBFT’s latency only increases from 0.20 s to 0.33 s, significantly outperforming the three comparison schemes. Because they will wrongly interpret the connectivity disruptions caused by mobility as malicious silences and trigger unnecessary view changes. Agentic-SecPBFT’s low latency is mainly due to the following aspects. The reward function imposes a large negative penalty on unnecessary view changes, incentivizing the model to minimize such latency-consuming operations. Also, LCAs can perform real-time threat flagging and non-critical traffic deprioritization, while permanent removal remains subject to a PBFT-approved membership update. Moreover, its crosslayer awareness capability accurately distinguishes between connectivity issues caused by node mobility and malicious silence, avoiding view changes triggered by misjudgment.

13

Fig. 5: Attack detection rate comparison. (a) Under different simulation steps; (b) Under different malicious node ratios; (c) Under different interference intensities.

Fig. 6: False positive rate comparison. (a) Under different simulation steps; (b) Under different malicious node ratios; (c) Under different interference intensities.

Fig. 7: Consensus success rate comparison. (a) Under different malicious node ratios; (b) Under different numbers of nodes; (c) Under different node moving speed.

Fig. 8: Consensus latency comparison. (a) Under different malicious node ratios; (b) Under different numbers of nodes; (c) Under different node moving speed. Under different malicious node ratios, Fig. 9 (a) shows that Agentic-SecPBFT maintains stable throughput, dropping slightly from 24.1 TPS to 18.9 TPS. While PBFT’s throughput collapses to only 3.2 TPS, and GRBFT and DA-PBFT drop to 9.0 TPS and 13.2 TPS. As shown in Fig. 9 (b), when the number of nodes increases, Agentic-SecPBFT’s throughput first rises and then stabilizes, peaking at 23.9 TPS. It will be thanks to COA-driven load balancing that fully utilizes network resources, and it has significantly outperformed the

comparison scheme. As shown in Fig. 9 (c), with the node moving speed increasing from 10 m/s to 30 m/s, AgenticSecPBFT still retains a high throughput of 21.8 TPS at 30 m/s, and also better than the other three. The stable and high throughput of Agentic-SecPBFT stems from its effective protection of network liveness. The hierarchical MADQN algorithm optimizes the composition of the consensus group, enabling honest nodes to conduct efficient consensus without being overwhelmed by malicious nodes. And the COA realizes

14

Fig. 9: Consensus throughput comparison. (a) Under different malicious node ratios; (b) Under different numbers of nodes; (c) Under different node moving speed. global load balancing, improving the utilization efficiency of wireless bandwidth in large-scale networks. In addition, its cross-layer perception of CSI/SINR enables adaptive adjustment to dynamic network topologies, ensuring continuous and efficient communication between honest nodes. Moreover, the dynamic reputation mechanism also avoids punishing honest nodes that have poor channel conditions. It retains its effective data processing capabilities during the consensus. E. Ablation Study To isolate the contributions of three core components, we evaluate the complete framework and three variants: without CSI/SINR context (w/o PHY), without dynamic reputation (w/o Rep.), and without COA coordination (w/o COA), under both low mobility & low interference and high mobility & high interference scenarios (they represent a relatively stable and extreme communication environment), as summarized in Table IV. Removing physical-layer context causes the sharpest performance degradation in both settings. In the benign scenario, CSR falls from 98.2% to 89.4%, ADR from 95.0% to 87.6%, and FPR rises to 5.9%. Under high mobility and high interference, this gap widens notably. CSR drops to 78.3%, ADR to 76.5%, and FPR surges to 11.2%. Without CSI/SINR awareness, the system cannot distinguish channel-induced packet loss from malicious dropping, triggering excessive unnecessary view changes and misjudgments that grow worse in dynamic environments. Removing dynamic reputation leads to moderate performance loss. CSR, ADR, and FPR shift to 93.7%, 91.3%, and 3.8% in the low-interference case, and deteriorate to 86.4%, 84.7%, and 6.8% under harsh conditions. This demonstrates that temporal trust accumulation suppresses transient misclassification from channel fluctuations, and its stabilizing effect becomes more critical as network instability increases. Removing COA coordination causes the mildest decline. In the benign scenario, CSR remains at 96.1%, ADR at 92.0%, and FPR at 2.7%. Even under high interference, the metrics stay at 89.8%, 87.1%, and 4.9%. Since LCAs retain full local perception and reputation capabilities, they can independently sustain basic defense and consensus. This verifies that the COA acts as a performance enhancement layer rather than a single point of failure. VII. C ONCLUSION In this paper, we demonstrate that standard PBFT and its static security enhancements are highly susceptible to intelli-

TABLE IV: Ablation Study Under Two Typical Scenarios Low Mobility & Low Interference Variant CSR (%) ADR (%) FPR (%) Agentic-SecPBFT 98.2 ± 0.3 95.0 ± 0.4 1.8 ± 0.2 w/o PHY 89.4 ± 0.7 87.6 ± 0.6 5.9 ± 0.4 w/o Rep. 93.7 ± 0.5 91.3 ± 0.5 3.8 ± 0.3 w/o COA 96.1 ± 0.4 92.0 ± 0.5 2.7 ± 0.2 High Mobility & High Interference Variant CSR (%) ADR (%) FPR (%) Agentic-SecPBFT 92.5 ± 0.6 90.2 ± 0.7 3.5 ± 0.3 w/o PHY 78.3 ± 1.2 76.5 ± 1.1 11.2 ± 0.8 w/o Rep. 86.4 ± 0.9 84.7 ± 0.8 6.8 ± 0.5 w/o COA 89.8 ± 0.7 87.1 ± 0.8 4.9 ± 0.4

gent, coordinated threats in wireless networks, such as Sybil and collusion attacks. To address these challenges, we propose a security framework, Agentic-SecPBFT, leveraging an Agentic AI-empowered multi-agent system. It employs a distributed intelligent model, including COA and LCA. The LCA detects suspicious behavior by observing local network conditions and recommends guarded security actions in real time. The COA supports cross-node learning through authenticated, versioned updates; its failure can reduce adaptation quality but cannot directly violate the PBFT commit quorum. The results show that this solution achieves substantial improvements in attack resilience, maintaining a high consensus success rate, low latency, and stable throughput even under severe attacks. Meanwhile, the ablation experiments confirmed the positive benefits of each module. This framework is expected to provide a new secure path for wireless blockchain consensus. R EFERENCES [1] H. Luo et al., “Toward edge general intelligence with multiple-large language model (multi-llm): architecture, trust, and orchestration,” IEEE Transactions on Cognitive Communications and Networking, 2025. [2] ——, “Wireless blockchain meets 6g: The future trustworthy and ubiquitous connectivity,” IEEE Communications Surveys & Tutorials, 2025. [3] W. Li et al., “A blockchain-assisted hierarchical data aggregation framework for iiot with computing first networks,” IEEE Transactions on Network and Service Management, 2025. [4] W. Chen et al., “Blockchain-assisted secure announcement sharing scheme with controllable verifiable distributed security for vanets,” IEEE Transactions on Mobile Computing, 2025. [5] H. Luo et al., “Symbiotic blockchain consensus: Cognitive backscatter communications-enabled wireless blockchain consensus,” IEEE/ACM Transactions on Networking, vol. 32, no. 6, pp. 5372–5387, 2024.

15

[6] M. Xu et al., “Blown: A blockchain protocol for single-hop wireless networks under adversarial sinr,” IEEE Transactions on Mobile computing, vol. 22, no. 8, pp. 4530–4547, 2022. [7] H. Luo et al., “Convergence of symbiotic communications and blockchain for sustainable and trustworthy 6g wireless networks,” IEEE Wireless Communications, vol. 32, no. 2, pp. 18–25, 2025. [8] C. Gu et al., “A dependable and efficient decentralized trust management system based on consortium blockchain for intelligent transportation systems,” IEEE Transactions on Intelligent Transportation Systems, 2024. [9] A. Agrawal et al., “Sybil-resilient publisher selection mechanism in blockchain-based mcs systems,” IEEE Open Journal of the Computer Society, 2025. [10] W. Li et al., “Surveying trust-based collaborative intrusion detection: State-of-the-art, challenges and future directions,” IEEE Communications Surveys & Tutorials, vol. 24, no. 1, pp. 280–305, 2021. [11] ——, “A scalable multi-layer pbft consensus for blockchain,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 5, pp. 1146–1160, 2020. [12] D. Yu et al., “Distributed covert leader election for private and robust communications in open wireless networks,” IEEE Journal on Selected Areas in Communications, 2025. [13] C. Cheong et al., “Multidimensional trust evidence fusion and pathbacktracking mechanism for trust management in vanets,” IEEE Internet of Things Journal, vol. 11, no. 10, pp. 18 619–18 634, 2024. [14] W. Rafique et al., “Trustworthy iot services with blockchain and information-centric networking: A survey,” IEEE Communications Surveys & Tutorials, 2025. [15] J. Xu et al., “Puffchain: A dynamic scaling blockchain system with optimal effective throughput,” IEEE Transactions on Network Science and Engineering, vol. 11, no. 3, pp. 3199–3212, 2024. [16] N. Yang et al., “A gaussian reputation-based hybrid bft consensus with a formal security framework,” IEEE Transactions on Dependable and Secure Computing, 2025. [17] J. Kang et al., “When metaverses meet vehicle road cooperation: Multiagent drl-based stackelberg game for vehicular twins migration,” IEEE Internet of Things Journal, vol. 11, no. 22, pp. 35 928–35 941, 2024. [18] R. Chen et al., “Drdst: Low-latency dag consensus through robust dynamic sharding and tree-broadcasting for iov,” IEEE Transactions on Mobile Computing, 2025. [19] S. Benadla et al., “Detecting sybil attacks in vehicular fog networks using rssi and blockchain,” IEEE Transactions on Network and Service Management, vol. 19, no. 4, pp. 3919–3935, 2022. [20] Y. Li et al., “Reputation-based stable blockchain sharding scheme for smart cities with iot consumer electronics: a deep reinforcement learning approach,” IEEE Transactions on Consumer Electronics, vol. 70, no. 3, pp. 5737–5746, 2024. [21] R. Zhang et al., “Toward agentic ai: generative information retrieval inspired intelligent communications and networking,” IEEE Communications Magazine, 2026. [22] H. Luo, G. Sun, Y. Liu, D. Niyato, H. Yu, M. Atiquzzaman, and S. Dustdar, “A trustworthy agentic multi-llm network: Challenges, solutions, and a use case,” IEEE Wireless Communications, 2026. [23] S. Gronauer and K. Diepold, “Multi-agent deep reinforcement learning: a survey,” Artificial Intelligence Review, vol. 55, no. 2, pp. 895–943, 2022. [24] Z. Wang et al., “Robust permissioned blockchain consensus for unstable communication in fanet,” IEEE/ACM Transactions on Networking, vol. 32, no. 1, pp. 699–712, 2023. [25] H. Luo et al., “A weighted byzantine fault tolerance consensus driven trusted multiple large language models network,” IEEE Transactions on Cognitive Communications and Networking, 2026. [26] Z. Deng et al., “Permissioned blockchain-based trusted and robust consensus optimization orienting intelligent transportation systems,” IEEE Transactions on Intelligent Transportation Systems, vol. 26, no. 2, pp. 2011–2026, 2025. [27] L. Zhang et al., “Dynamic vehicle reputation consensus: enhancing iov communication with a blockchain algorithm,” IEEE Transactions on Vehicular Technology, vol. 74, no. 3, pp. 4788–4806, 2025.

[28] M. M. Rashid et al., “Trustworthy and fair federated learning via reputation-based consensus and adaptive incentives,” IEEE Transactions on Information Forensics and Security, 2025. [29] X. Xiong et al., “Cloud-edge-end collaborative computing-enabled intelligent sharding blockchain for industrial iot based on ppo approach,” IEEE Transactions on Mobile Computing, vol. 24, no. 9, pp. 8011–8024, 2025. [30] Y. Liu et al., “Multi-leader byzantine fault tolerance in blockchain: Performance and security,” IEEE Transactions on Information Forensics and Security, 2026. [31] C. Li et al., “A dynamic adaptive framework for practical byzantine fault tolerance consensus protocol in the internet of things,” IEEE Transactions on Computers, vol. 73, no. 7, pp. 1669–1682, 2024. [32] H. Luo et al., “An energy-efficient wireless blockchain sharding scheme for pbft consensus,” IEEE Transactions on Network Science and Engineering, vol. 11, no. 3, pp. 3015–3027, 2024. [33] Y. Fan et al., “Shard-dag: A scalable and secure block-dag sharding scheme for ai-driven 6g networks,” Digital Communications and Networks, 2025. [34] S. Liu et al., “Partially synchronous bft consensus made practical in wireless networks,” in IEEE Conference on Computer Communications. IEEE, 2025, pp. 1–10. [35] L. Li et al., “Timeliness of slotted aloha-based wireless broadcasting and flooding,” IEEE Transactions on Networking, 2025. [36] S. Gao et al., “T-pbft: An eigentrust-based practical byzantine fault tolerance consensus algorithm,” China Communications, vol. 16, no. 12, pp. 111–123, 2019. [37] A. Altarawneh and J. Owusu-Tweneboah, “Assessing the resilience of bft-raft consensus against insider dos attacks in blockchain,” in IEEE Annual Computing and Communication Workshop and Conference. IEEE, 2025, pp. 58–65. [38] Z. Ren et al., “Blockchain sharding over wireless channels: Dynamic resource allocation with sparse reward,” IEEE Wireless Communications Letters, vol. 14, no. 1, pp. 63–67, 2025. [39] S. Kumar et al., “Reinforcement learning framework for power optimization in 5g ran systems with blockchain-enhanced security,” in IEEE Global Communications Conference Workshops. IEEE, 2024, pp. 1–6. [40] S. M. S. Mohammadabadi et al., “Communication-efficient training workload balancing for decentralized multi-agent learning,” in IEEE International Conference on Distributed Computing Systems. IEEE, 2024, pp. 680–691. [41] H. N. Abishu et al., “Multi-agent drl-based consensus mechanism for blockchain-based collaborative computing in uav-assisted 6g networks,” IEEE Internet of Things Journal, vol. 12, no. 4, pp. 4331–4348, 2025. [42] A. Şahin et al., “Reliable majority vote computation with complementary sequences for uav waypoint flight control,” IEEE Transactions on Wireless Communications, vol. 23, no. 10, pp. 12 608–12 621, 2024. [43] K. Tamura et al., “Joint snr and rician k-factor estimation using multimodal network over mobile fading channels,” IEEE Transactions on Machine Learning in Communications and Networking, vol. 2, pp. 766–779, 2024. [44] Y. Wang et al., “Scraft: Achieving fast and stable elections in raft consensus algorithm,” in IEEE International Conference on High Performance Computing and Communications. IEEE, 2024, pp. 561–568. [45] J. Zheng and Y. Zhang, “Trbft: An efficient blockchain consensus for edge computing-enabled iot systems,” IEEE Internet of Things Journal, 2025. [46] D. Chao et al., “A systematic survey on security in anonymity networks: Vulnerabilities, attacks, defenses, and formalization,” IEEE Communications Surveys & Tutorials, vol. 26, no. 3, pp. 1775–1829, 2024. [47] Y. Li et al., “Secure multi-party learning: Fundamentals, frameworks, state of the art, trends, and challenges,” IEEE Transactions on Network Science and Engineering, 2025. [48] S. Shang et al., “Joint trajectory and beamforming optimization for uavris-empowered multiuser communication networks: A double deep qnetwork approach,” IEEE Transactions on Vehicular Technology, 2025.