Strategic Heterogeneous Multi-Agent Architecture for Cost-Effective Code Vulnerability Detection Zhaohui Geoffrey Wang University of Southern California Los Angeles, CA, USA [email protected]
arXiv:2604.21282v1 [cs.CR] 23 Apr 2026
ABSTRACT Automated code vulnerability detection is critical for software security, yet existing approaches face a fundamental trade-off between detection accuracy and computational cost. We propose a heterogeneous multi-agent architecture inspired by game-theoretic principles, combining cloud-based LLM experts with a local lightweight verifier. Our “3+1” architecture deploys three cloud-based expert agents (DeepSeek-V3) that analyze code from complementary perspectives—code structure, security patterns, and debugging logic—in parallel, while a local verifier (Qwen3-8B) performs adversarial validation at zero marginal cost. We formalize this design through a two-layer game framework: (1) a cooperative game among experts capturing super-additive value from diverse perspectives, and (2) an adversarial verification game modeling quality assurance incentives. Experiments on 262 real samples from the NIST Juliet Test Suite across 14 CWE types, with balanced vulnerable and benign classes, demonstrate that our approach achieves 77.2% F1 score with 62.9% precision and 100% recall at $0.002 per sample—outperforming both a single-expert LLM baseline (F1 71.4%) and Cppcheck static analysis (MCC 0). The adversarial verifier significantly improves precision (+10.3 percentage points, 𝑝 < 10−6 , McNemar’s test) by filtering false positives, while parallel execution achieves a 3.0× speedup. Our work demonstrates that game-theoretic design principles can guide effective heterogeneous multi-agent architectures for cost-sensitive software engineering tasks.
KEYWORDS Multi-Agent Systems; Game Theory; Large Language Models; Vulnerability Detection; Strategic Cooperation ACM Reference Format: Zhaohui Geoffrey Wang. 2026. Strategic Heterogeneous Multi-Agent Architecture for Cost-Effective Code Vulnerability Detection. In Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), Paphos, Cyprus, May 25 – 29, 2026, IFAAMAS, 11 pages.
1
INTRODUCTION
Code vulnerability detection is a critical challenge in software security, with the global cost of cybercrime projected to reach $10.5 trillion annually [13]. Traditional static analysis tools like Cppcheck [2] offer fast, low-cost analysis but suffer from limited semantic understanding, while single-agent LLM approaches achieve Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.), May 25 – 29, 2026, Paphos, Cyprus. © 2026 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). This work is licenced under the Creative Commons Attribution 4.0 International (CC-BY 4.0) licence.
higher accuracy at prohibitive cost. Recent multi-agent LLM systems [18, 21, 23] have shown promising results, but typically employ homogeneous agent pools—all agents use the same expensive cloud model—missing opportunities for cost optimization through heterogeneous design. This paper addresses a fundamental question: How should we allocate heterogeneous computational resources across multiple agents to achieve optimal trade-offs between detection quality and operational cost? The key insight is that vulnerability detection benefits from two distinct capabilities: (1) deep semantic analysis requiring powerful models, and (2) consistency checking that can be performed by lighter models. This asymmetry motivates a heterogeneous architecture. We propose a “3+1” heterogeneous multi-agent architecture combining: • Three cloud-based LLM experts (DeepSeek-V3 [3]) analyzing code from complementary perspectives: code structure analysis, security pattern matching, and debugging/edgecase reasoning. These agents execute in parallel, with diverse viewpoints providing complementary coverage that exceeds any single perspective. • One local verifier (Qwen3-8B [17]) performing adversarial validation at zero marginal API cost. Running on a local GPU, this agent receives all expert reports and checks for consistency, hallucinated claims, and missed vulnerabilities. We formalize the architectural design through a two-layer game framework: (1) Layer 1 (Cooperative Game): The three experts form a coalition with super-additive value—their combined detection covers vulnerability patterns that individual experts miss, formalizing why diverse perspectives outperform identical agents. (2) Layer 2 (Adversarial Game): The verifier acts as an independent adversary that improves system precision by catching false positives and hallucinations in expert outputs. Critically, we show that the game-theoretic framework yields testable predictions validated empirically: (1) diverse expert coalitions exhibit super-additive value when combined with verification, outperforming a single-expert baseline (Proposition 1), (2) independent verification significantly improves precision (𝑝 < 10−6 , Theorem 1), and (3) the heterogeneous cloud-local split achieves favorable cost-quality trade-offs analyzable through mechanism design. While our LLM agents do not engage in explicit strategic reasoning, the game-theoretic framework correctly predicts systemlevel behavior—validating its utility as an analytical and design tool for multi-agent LLM architectures. We make the following contributions:
(1) A game-theory-inspired framework for heterogeneous multi-agent vulnerability detection, formalizing cooperative expert analysis and adversarial verification as a principled design methodology (Section 3). (2) Rigorous empirical evaluation on 262 real samples from the NIST Juliet Test Suite covering 14 CWE types with balanced vulnerable and benign classes, reporting standard metrics (Precision, Recall, F1, FPR, MCC) with bootstrap 95% confidence intervals and McNemar’s significance test (Section 5). (3) Practical cost-quality analysis demonstrating that heterogeneous cloud-local architectures achieve favorable Pareto trade-offs, with the local verifier adding precision improvement at negligible cost (Section 5). The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents our framework and architecture. Section 4 describes the experimental setup. Section 5 presents results and analysis. Section 6 concludes with limitations and future directions.
2 RELATED WORK 2.1 Static Analysis for Vulnerability Detection Traditional static analysis tools employ rule-based pattern matching and data-flow analysis to detect vulnerabilities. Cppcheck [2] uses syntactic pattern matching for C/C++, while commercial tools like Coverity [20] combine multiple techniques for enterprise deployment. However, these tools face inherent limitations: high false positive rates due to lack of semantic understanding, and difficulty detecting complex vulnerabilities that span multiple code paths [9]. Their precision-recall trade-off is fundamentally constrained by the expressiveness of static rules.
2.2
LLM-Based Vulnerability Detection
Recent work has applied LLMs to vulnerability detection with increasing sophistication. Single-agent approaches using GPT-4 [16] or specialized code models achieve strong detection rates but at significant computational cost. The PrimeVul benchmark [4] revealed a critical evaluation gap: state-of-the-art models achieve only ∼3– 35% F1 on carefully deduplicated real-world datasets, compared to ∼68% F1 on older benchmarks like Big-Vul that contain significant data leakage. This highlights the importance of rigorous evaluation methodology with balanced classes and proper metrics. Multi-agent approaches have emerged as a promising direction for vulnerability detection. VulTrial [21] proposes a mock-court framework where specialized agents (security researcher, code author, moderator) debate vulnerabilities through structured argumentation, achieving 102% improvement over single-agent baselines at ICSE 2026. MulVul [23] combines RAG-augmented multi-agent analysis with cross-model prompt evolution, achieving the current state-of-the-art 34.79% Macro-F1 on PrimeVul. MultiVer [18] uses a zero-shot 4-agent ensemble with union voting, reaching 82.7% recall on PyVul. LLMxCPG [10] integrates LLMs with Code Property Graphs for structurally-enhanced analysis. Our work differs from these approaches by explicitly optimizing for cost-quality trade-offs through heterogeneous agent design. While VulTrial and MulVul use homogeneous model pools (all
agents share the same expensive API), we combine paid cloud experts with a free local verifier, making cost a first-class design objective.
2.3
Multi-Agent Debate and Collaboration
A growing body of work demonstrates that multi-agent debate improves LLM output quality. Du et al. [5] show that having multiple LLMs debate improves factuality and reasoning, with agents correcting each other’s errors through iterative rounds—a mechanism that directly inspires our adversarial verification loop. Liang et al. [12] extend this to divergent thinking, finding that multi-agent debate elicits more diverse reasoning paths. CAMEL [11] introduces role-playing communication protocols for multi-agent collaboration, demonstrating that specialized agent roles outperform generic prompting. These works establish that iterative multi-agent interaction improves output quality, but none apply this mechanism to the specific cost-quality trade-offs of vulnerability detection with heterogeneous models. In software engineering, MetaGPT [8] uses structured multiagent workflows for collaborative development, and AutoGen [22] provides general-purpose agent coordination. However, these frameworks do not explicitly model the economic trade-offs of heterogeneous model allocation.
2.4
Game Theory and LLM Multi-Agent Systems
Game-theoretic analysis of LLM-based multi-agent systems is an emerging area. A comprehensive survey [7] formalizes LLM interactions as Nash and Stackelberg games, defining “LLM-based equilibria” where strategies co-evolve through dialogue. The ECON framework [24] models multi-LLM coordination as an incompleteinformation game seeking Bayesian Nash Equilibrium, where each LLM responds based on beliefs about co-agents. GTBench [6] benchmarks LLM strategic reasoning, finding that LLMs deviate from rational strategies as game complexity increases—motivating our use of game theory as a design framework rather than assuming strategic behavior by LLM agents. GT-HarmBench [1] extends gametheoretic evaluation to multi-agent safety, benchmarking risks in adversarial settings—complementary to our adversarial verification game.
2.5
Positioning of Our Work
Our architecture shares the multi-perspective analysis approach with VulTrial and MultiVer, and draws on multi-agent debate [5, 12] for iterative quality improvement. We uniquely combine: (1) heterogeneous resource allocation—paid cloud APIs with free local models; (2) game-theoretic design rationale—cooperative and adversarial game formulations justifying architectural choices; and (3) rigorous evaluation—balanced classes, standard metrics, and statistical significance testing. The closest concurrent work, MulVul, achieves state-of-the-art detection on PrimeVul but does not address cost optimization. Our contribution is complementary: showing how to architect cost-effective multi-agent systems with game-theoretic principles.
3
METHODOLOGY
We present a game-theoretic framework for heterogeneous multiagent vulnerability detection. Unlike prior multi-agent approaches that treat agent coordination as an engineering problem, we formalize the design space through cooperative and adversarial game theory, derive analytical properties, and validate these predictions empirically.
3.1
Problem Formulation
This motivates our choice of three diverse expert roles: a code structure analyst, a CWE-taxonomy security expert, and a debugging specialist. Their coalition should outperform any single expert—a prediction we validate in Section 5. Shapley Value Analysis. The Shapley value 𝜙𝑖 quantifies each expert’s marginal contribution to the coalition: ∑︁ |𝑆 |!(𝑘 − |𝑆 | − 1)! 𝜙𝑖 (𝑣) = [𝑣 (𝑆 ∪ {𝑖}) − 𝑣 (𝑆)] (3) 𝑘! 𝑆 ⊆𝑁 \{𝑖 }
Given a code snippet 𝑥, vulnerability detection seeks to determine whether 𝑥 contains security vulnerabilities and identify their CWE types. We formulate this as a multi-agent mechanism design problem: how to allocate heterogeneous computational resources across agents to maximize detection quality 𝑄 while minimizing cost 𝐶.
For our 3-expert system, the Shapley value determines how much each perspective (code structure, security, debugging) contributes to overall detection quality. If 𝜙𝑖 > 𝐶𝑖 for all 𝑖, every expert “pays for itself”—the marginal quality improvement exceeds the marginal cost. We report empirical Shapley values in the appendix.
Definition 1 (Multi-Agent Vulnerability Detection Game). A multi-agent detection system is a tuple M = ⟨A, S, 𝐶, 𝑄, 𝜋⟩ where: • A = {𝑎 1, . . . , 𝑎𝑛 } is the set of heterogeneous agents • S = {𝑠 1, . . . , 𝑠𝑛 } is the strategy space (analysis perspective, model choice, effort level) • 𝐶 : S𝑛 → R+ is the cost function (API charges + compute) • 𝑄 : S𝑛 → [0, 1] is the quality function (detection F1) • 𝜋 : A × S𝑛 → R is the payoff function, 𝜋𝑖 = 𝑤 1𝑄 − 𝑤 2𝐶𝑖
3.2.2 Layer 2: Adversarial Verification Game. The second layer models the strategic interaction between the expert coalition and an independent verifier as an extensive-form game.
The mechanism designer’s problem is to select agent types, roles, and interaction protocols that induce a desirable equilibrium—high 𝑄 at low 𝐶. This connects our work to the mechanism design literature [14]: we design the “rules of the game” so that the system-level outcome is Pareto-optimal.
3.2
Two-Layer Game Structure
3.2.1 Layer 1: Expert Cooperative Game. The first layer models cooperation among 𝑘 expert agents with complementary analysis perspectives as a coalitional game. Definition 2 (Expert Coalition Game). An expert coalition game is 𝐺 1 = ⟨𝑁 , 𝑣⟩ where: • 𝑁 = {1, 2, . . . , 𝑘 } is the set of expert agents • 𝑣 : 2𝑁 → R is the characteristic function The coalition value captures the net utility of subset 𝑆 ⊆ 𝑁 : (1)
where 𝑄 (𝑆) is the detection quality (F1) achievable by coalition 𝑆, and 𝐶 (𝑆) is its total cost. Proposition 1 (Super-Additivity of Diverse Coalitions). Let 𝑆,𝑇 ⊆ 𝑁 be disjoint coalitions with complementary analysis perspectives (i.e., each agent covers vulnerability patterns not covered by others). If agents in 𝑆 detect vulnerability set 𝑉𝑆 and agents in 𝑇 detect 𝑉𝑇 , then: 𝑄 (𝑆 ∪ 𝑇 ) ≥ max(𝑄 (𝑆), 𝑄 (𝑇 ))
• 𝑃 = {𝐸, 𝑉 } are players (Expert coalition, Verifier) • 𝑆 𝐸 = {ℎ𝑖𝑔ℎ, 𝑙𝑜𝑤 } represents expert output quality (effort level) • 𝑆𝑉 = {𝑎𝑐𝑐𝑒𝑝𝑡, 𝑐ℎ𝑎𝑙𝑙𝑒𝑛𝑔𝑒, 𝑟𝑒 𝑗𝑒𝑐𝑡 } is verifier strategy • 𝜋𝐸 , 𝜋𝑉 are payoff functions The expert coalition’s payoff captures the trade-off between effort cost and rejection penalty: 𝜋𝐸 (𝑠𝐸 , 𝑠𝑉 ) = 𝑄 (𝑠𝐸 ) − 𝑐 (𝑠𝐸 ) − 𝑝 · ⊮ [𝑠𝑉 =𝑟𝑒 𝑗𝑒𝑐𝑡 ]
(4)
where 𝑐 (ℎ𝑖𝑔ℎ) > 𝑐 (𝑙𝑜𝑤) and 𝑝 > 0 is the rejection penalty (wasted API cost on rejected outputs).
We decompose the detection problem into two interconnected games.
𝑣 (𝑆) = 𝑤 1 · 𝑄 (𝑆) − 𝑤 2 · 𝐶 (𝑆)
Definition 3 (Verification Game). A verification game is 𝐺 2 = ⟨𝑃, 𝑆 𝐸 , 𝑆𝑉 , 𝜋𝐸 , 𝜋𝑉 ⟩ where:
(2)
with strict inequality when 𝑉𝑆 ⊈ 𝑉𝑇 and 𝑉𝑇 ⊈ 𝑉𝑆 . Furthermore, if 𝐶 (𝑆 ∪ 𝑇 ) = 𝐶 (𝑆) + 𝐶 (𝑇 ) (independent cost), then 𝑣 (𝑆 ∪ 𝑇 ) ≥ 𝑣 (𝑆)+𝑣 (𝑇 ) when the quality improvement outweighs the cost increase.
Theorem 1 (Verification Eqilibrium). In the verification game 𝐺 2 : (i) If the verifier independently detects false positives with probability 𝑝 𝑓 𝑝 > 0 and does not introduce new false positives, the system precision satisfies 𝑃𝑠𝑦𝑠𝑡𝑒𝑚 > 𝑃𝑒𝑥𝑝𝑒𝑟𝑡𝑠 whenever 𝐹 𝑃 > 0. (ii) The strategy profile (𝑠𝐸∗ = ℎ𝑖𝑔ℎ, 𝑠𝑉∗ = accept when consistent) forms a Nash equilibrium when 𝑝 > 𝑐 (ℎ𝑖𝑔ℎ) − 𝑐 (𝑙𝑜𝑤), i.e., the rejection penalty exceeds the cost of producing high-quality output. Proof sketch. For (i): the verifier removes fraction 𝑝 𝑓 𝑝 of false positives without removing true positives, so 𝑃𝑠𝑦𝑠𝑡𝑒𝑚 = 𝑇 𝑃/(𝑇 𝑃 + 𝐹 𝑃 (1 − 𝑝 𝑓 𝑝 )) > 𝑇 𝑃/(𝑇 𝑃 + 𝐹 𝑃). For (ii): given 𝑠𝑉∗ , the expert’s best response is ℎ𝑖𝑔ℎ since 𝑄ℎ𝑖𝑔ℎ − 𝑐ℎ𝑖𝑔ℎ > 𝑄𝑙𝑜𝑤 − 𝑐𝑙𝑜𝑤 − 𝑝 when 𝑝 > 𝑐 (ℎ𝑖𝑔ℎ) − 𝑐 (𝑙𝑜𝑤). Given 𝑠𝐸∗ = ℎ𝑖𝑔ℎ, the verifier’s best response is 𝑎𝑐𝑐𝑒𝑝𝑡 since rejection of correct output incurs unnecessary cost. Full proof in Appendix G. □ □ The key insight is that the verifier creates a credible audit mechanism: even though our LLM agents do not literally reason about penalties, the system architecture implements the equilibrium outcome. By routing outputs through an independent verifier, we structurally ensure that low-quality outputs (false positives) are filtered—achieving the same outcome that strategic agents would reach in equilibrium.
Algorithm 1 3+1 Multi-Agent Detection (Game Implementation) Require: Code 𝑥, Experts 𝐸 = {𝑒 1, 𝑒 2, 𝑒 3 }, Verifier 𝑉 Ensure: Vulnerability prediction, CWE classification 1: ⊲ Layer 1: Cooperative Game 2: {𝑟 1 , 𝑟 2 , 𝑟 3 } ← ParallelAnalyze(𝐸, 𝑥) ⊲ Coalition 𝑣 (𝑁 ) Ð 3: CWE_pool ← 𝑖 ExtractCWEs(𝑟 𝑖 ) ⊲ Super-additive union 4: ⊲ Layer 2: Verification Game 5: 𝑣 ← Verify(𝑉 , 𝑥, 𝑟 1 , 𝑟 2 , 𝑟 3 ) ⊲ Audit mechanism 6: if 𝑣 provides FINAL_VULNERABILITY then 7: return 𝑣.vulnerability, 𝑣.CWE_IDs ⊲ Verifier override 8: else 9: return MajorityVote(𝑟 1, 𝑟 2, 𝑟 3 ), CWE_pool 10: end if
This connects to both the inspection game literature [19] and the multi-agent debate paradigm [5, 12]: the verifier functions as an “inspector” whose adversarial cross-examination of expert claims improves system quality, analogous to how debate among LLMs improves factuality in general reasoning tasks.
3.3
super-additive CWE coverage through union aggregation, and Lines 5–9 implement the verification game (Layer 2) where the verifier can override expert consensus.
4
EXPERIMENTS
We evaluate our 3+1 architecture on a real vulnerability detection benchmark with balanced classes, addressing three research questions: • RQ1: Does the multi-expert coalition provide super-additive value over individual experts? • RQ2: Does the adversarial verifier improve detection precision? • RQ3: Does parallel execution achieve cost-quality Pareto efficiency?
4.1
The 3+1 Architecture
• Balanced classes: 132 vulnerable + 130 benign (patched) functions • 14 CWE types: covering memory safety (CWE-121, 122, 415, 416), integer errors (CWE-190), null pointers (CWE-476), injection (CWE-78, 134), resource management (CWE-401, 789, 400), error handling (CWE-252), numeric errors (CWE369), and uninitialized variables (CWE-457) • Real code: extracted from Juliet’s Flow Variant 01 (baseline) test cases, averaging 38.5 lines per sample • Ground truth: each vulnerable sample has a known CWE label; benign samples are the corresponding patched versions from the same test case
Based on our game-theoretic analysis, we design a concrete architecture with 𝑘 = 3 experts and 𝑚 = 1 verifier that implements the equilibrium structure derived above. 3.3.1 Expert Agents (Cloud). We deploy three DeepSeek-V3 [3] agents via API with specialized prompts: • Code Analyst (𝑎 1 ): Data flow, control flow, vulnerability entry points, memory operations. Covers structural vulnerability patterns. • Security Expert (𝑎 2 ): CWE taxonomy matching, known vulnerability patterns, severity assessment. Covers semantic vulnerability patterns. • Debug Expert (𝑎 3 ): Error handling, boundary conditions, edge cases, undefined behavior. Covers behavioral vulnerability patterns. Each expert outputs structured reports: VULNERABILITY_FOUND, CWE_IDs, SEVERITY, EVIDENCE, CONFIDENCE. Experts execute in parallel, which does not change coalition value 𝑣 (𝑁 ) but reduces wall-clock time proportionally.
The inclusion of benign samples is critical for measuring precision and false positive rate—metrics omitted in many prior LLMbased vulnerability detection studies [4].
4.2
3.3.3 Decision Mechanism. Algorithm 1 implements the two-layer game structure: The algorithm directly implements our game-theoretic design: Line 2 realizes the cooperative coalition (Layer 1), Line 3 captures
Baselines and Configurations
Baselines: • Cppcheck 2.13.0 [2]: open-source static analysis tool with all checks enabled • Single Expert: one DeepSeek-V3 agent (security expert role) without verifier—tests whether the multi-agent coalition adds value over a single LLM
3.3.2 Verifier Agent (Local). Qwen3-8B [17] deployed locally on GPU implements the adversarial verification game: • Zero marginal cost: eliminates API charges, making verification “free at the margin”—a key property for the mechanism to be individually rational. • Different model family: Qwen3 vs. DeepSeek reduces correlated errors, ensuring the verifier provides genuinely independent assessment. • Full information: receives original code and all three expert reports, enabling cross-referencing.
Dataset
We use the NIST Juliet Test Suite v1.3 [15], a widely-used benchmark containing 64,295 C/C++ test cases across 118 CWE types. Unlike prior work that uses small or synthetic subsets, we extract 262 real function-level samples with the following properties:
Ablation Configurations: • 3+1 Parallel + Verifier: Full architecture (3 parallel experts + local verifier) • 3+1 Parallel − Verifier: Experts only, no verification (ablates Layer 2) • 3+1 Serial + Verifier: Sequential experts + verifier (ablates parallelism)
4.3
Implementation Details
Expert Agents: DeepSeek-V3 via OpenAI-compatible API at temperature 0.1. Each expert receives a specialized system prompt
defining its analysis perspective and structured output format. Experts run concurrently via Python asyncio. Verifier Agent: Qwen3-8B [17] deployed locally via HuggingFace Transformers on an NVIDIA RTX 3090 GPU. The verifier receives the original code and all three expert reports, and outputs a consolidated assessment (ACCEPT/CHALLENGE/REJECT with final CWE classification). Cost Calculation: DeepSeek-V3 pricing: $0.27/MTok input, $1.10/MTok output. Local verifier cost is zero (GPU amortized).
4.4
Table 1: Main results on 262 Juliet samples (132 vulnerable + 130 benign). 95% bootstrap CI in parentheses. Configuration 3+1 Para.+V 3+1 Para.−V 3+1 Serial+V
Evaluation Metrics
Single Expert
We report standard binary classification metrics with the positive class being “vulnerable”:
Cppcheck 2.13
• Precision: 𝑇 𝑃/(𝑇 𝑃 + 𝐹 𝑃) — fraction of predicted vulnerabilities that are real • Recall: 𝑇 𝑃/(𝑇 𝑃 + 𝐹 𝑁 ) — fraction of real vulnerabilities detected • F1 Score: harmonic mean of precision and recall • False Positive Rate: 𝐹 𝑃/(𝐹 𝑃 + 𝑇 𝑁 ) • MCC: Matthews Correlation Coefficient, robust to class imbalance • CWE Match Rate: among true positives, the fraction with correct CWE identification (using CWE hierarchy for partial credit, e.g., CWE-121 ≈ CWE-122 under parent CWE-120) All metrics are reported with 95% bootstrap confidence intervals (1,000 resamples). Pairwise comparisons use McNemar’s test.
4.5
CWE Extraction
Vulnerability predictions are extracted automatically from LLM outputs using regex matching for CWE identifiers and structured fields. A sample is predicted as vulnerable if: (1) the reviewer’s final assessment says “yes”, or (2) a majority of experts report vulnerability with at least one CWE identifier. CWE matching uses a hierarchy mapping (e.g., CWE-121/122/787 are treated as equivalent under parent CWE-120).
5 RESULTS AND DISCUSSION 5.1 Overall Performance Table 1 presents the main results. Our full 3+1 architecture (Parallel+Verifier) achieves 77.2% F1 score, with perfect recall (100%) ensuring that all vulnerabilities are detected, and precision of 62.9% indicating that roughly two-thirds of alerts correspond to genuine vulnerabilities. A striking finding is that without the verifier, the vast majority of benign samples are incorrectly flagged as vulnerable (FPR = 91.5%). The expert agents alone have minimal ability to distinguish between vulnerable code and its patched version—a critical limitation that the verifier addresses by filtering 41 additional false positives. Notably, the single-expert baseline achieves higher precision (.555) than the 3-expert coalition without verifier (.526), because multiple experts produce more CWE mentions that inflate false positives. Cppcheck achieves 100% FPR (MCC = 0), unable to differentiate Juliet’s vulnerable and patched code pairs. API cost is nearly identical across all 3-expert configurations ($0.002 per sample), as the verifier runs locally at zero API cost.
Prec.
Rec.
F1
FPR
MCC
.600
.501
.915
.211
.562
.531
.815
.320
1.00
.000
.629
1.00
.772
(.563,.693)
(1.0,1.0)
(.721,.819)
.526
1.00
.689
(.464,.586)
(1.0,1.0)
(.634,.739)
.644
1.00
.783
(.577,.710)
(1.0,1.0)
(.732,.830)
.555
1.00
.714
(.490,.617)
(1.0,1.0)
(.658,.763)
.504
1.00
.670
Table 2: Cost and latency analysis (262 samples) Config 3+1 Para.+V 3+1 Para.−V 3+1 Serial+V Single Expert Cppcheck
Cost/Sample
Time/Sample
Total Cost
$0.0021 $0.0021 $0.0021 $0.0006 $0
68.1s 21.7s 92.2s 14.1s <0.1s
$0.542 $0.548 $0.548 $0.162 $0
The single expert uses one-third the calls, reducing cost to $0.0006 per sample. The difference lies in latency: parallel experts finish in ∼15s vs. ∼45s serial, while the local verifier adds ∼50s.
5.2
RQ1: Expert Coalition Super-Additivity
The single-expert baseline achieves 100% recall but with FPR = 81.5% (MCC = .320). The 3-expert coalition without verifier also achieves 100% recall but with higher FPR (91.5%, MCC = .211)—at first glance suggesting that adding experts hurts precision. However, this reflects a limitation of the majority-voting aggregation: more experts produce more CWE mentions, inflating false positives. The super-additive value manifests not in recall but in CWE identification accuracy: the union of three expert perspectives achieves 100% CWE match rate (every detected vulnerability is classified with the correct CWE type), compared to lower match rates for individual experts. The full 3+1 system (with verifier) resolves this tension: the verifier filters the coalition’s excess false positives, yielding F1 = .772 vs. the single expert’s .714 (McNemar 𝑝 < 10−5 ). This confirms Proposition 1: the coalition’s value is realized through the verification layer, which extracts the precision benefit from the coalition’s richer CWE evidence.
5.3
RQ2: Adversarial Verifier Impact
The verifier’s contribution is the most significant finding of our experiments. Table 3 shows the direct comparison. McNemar’s test confirms this difference is highly significant: 41 additional samples are correctly reclassified by the verifier (𝑝 < 10−6 ), with zero samples incorrectly changed from correct to incorrect. This validates Theorem 1(i): the verifier acts as a onedirectional precision filter, reducing the false positive count from 119 to 78 out of 130 benign samples.
Table 3: Adversarial verifier impact (parallel mode). The verifier significantly reduces false positives. Metric
+V
−V
Δ
Precision Recall F1 FPR
62.9% 100% 77.2% 60.0%
52.6% 100% 68.9% 91.5%
+10.3% 0% +8.3% −31.5%
Table 5: Per-CWE detection performance (3+1 Parallel+Verifier). All CWEs achieve 100% recall; FPR varies significantly.
𝑝-value
< 10−6
Table 4: Parallel vs. serial execution efficiency (both with verifier) Metric
Parallel
Serial
Gain
Expert Time/Sample Total Time/Sample F1 Score McNemar 𝑝
∼15s ∼45s 68.1s 92.2s 77.2% 78.3% 0.711
3.0× 1.4× ≈ Equal Not sig.
The MCC increases from 0.211 (weak correlation) to 0.501 (moderate positive correlation), demonstrating that the verifier substantially improves discriminative power beyond what the experts alone achieve. Game-theoretic interpretation: This result validates the inspection game structure of Layer 2. The verifier—implemented as a different model family (Qwen3 vs. DeepSeek)—functions as an independent auditor whose cross-referencing of expert claims against code evidence catches inconsistencies that same-model verification would miss. The zero marginal cost of local verification makes this audit mechanism individually rational: the system designer always benefits from adding the verifier since ΔPrecision > 0 at ΔCost ≈ 0. Comparing against the single-expert baseline further validates the full architecture: the 3+1 system (F1 = .772) significantly outperforms a single expert (F1 = .714, McNemar 𝑝 < 10−5 ), confirming the value of both coalition and verification.
5.4
RQ3: Parallel Execution Efficiency
Parallel execution achieves a 3.0× speedup in expert analysis time (∼15s vs. ∼45s) with no significant difference in detection quality (McNemar 𝑝 = 0.711). The total wall-clock speedup is 1.4× because the local verifier (∼50s) dominates latency. This suggests that optimizing verifier inference (e.g., with vLLM or quantization) would yield substantial end-to-end improvements.
5.5
Per-CWE Analysis
Table 5 reveals substantial variation across vulnerability types. The FPR varies from 11% (CWE-476, null pointer dereference) to 100% (CWE-400/416/789). This variation is informative: vulnerabilities whose patches involve adding explicit checks (e.g., null guards for CWE-476, return-value checks for CWE-252) have low FPR because patched versions are structurally distinct. In contrast, semantic vulnerabilities (e.g., resource exhaustion, use-after-free) have high FPR because the patched code retains similar structure— the fix may be a subtle control-flow change that LLMs struggle to distinguish.
CWE
Category
FPR
TN
FP
CWE-476 CWE-252 CWE-401 CWE-457 CWE-78 CWE-415 CWE-122 CWE-190 CWE-369 CWE-121 CWE-134 CWE-416 CWE-789 CWE-400
NULL Deref Unchecked Return Memory Leak Uninit Variable Cmd Injection Double Free Heap Overflow Integer Overflow Divide by Zero Stack Overflow Format String Use After Free Mem Allocation Resource Exhaust
11% 20% 30% 30% 40% 50% 60% 70% 70% 80% 90% 100% 100% 100%
8 8 7 7 6 3 4 3 3 2 1 0 0 0
1 2 3 3 4 3 6 7 7 8 9 7 8 10
This suggests that future work should focus on improving discrimination for semantic vulnerability types, perhaps through codediff-aware analysis or iterative expert-verifier dialogue.
5.6
Cost-Quality Trade-off and Mechanism Design
At $0.002 per sample, our architecture enables analysis of 500 code functions for $1.00—making it practical for CI/CD integration. The heterogeneous design is central to this cost-effectiveness, and can be understood through a mechanism design lens. Heterogeneous allocation as mechanism design. A naïve approach would allocate the same cloud model to all four agents, costing ∼33% more (four DeepSeek-V3 calls instead of three). Our mechanism exploits an asymmetry in task difficulty: deep vulnerability analysis requires a powerful model, but cross-checking consistency is a simpler task achievable by a smaller model. By assigning the expensive model to the hard task (analysis) and the free local model to the simpler task (verification), we achieve the same equilibrium outcome at lower cost. Individual rationality. The verifier’s zero marginal cost ensures that adding verification is always rational for the system designer: ΔF1 = +8.3% at ΔAPI cost ≈ $0. This is a property the mechanism designer can guarantee regardless of agent behavior. Pareto efficiency. No configuration dominates another across all objectives (cost, recall, precision, latency). The parallel+verifier configuration offers the best precision-latency trade-off; parallelonly offers the best latency at cost of precision; serial+verifier offers slightly better precision but at 1.5× the latency.
5.7
Threats to Validity
Internal Validity: Automated CWE extraction via regex may miss non-standard output formats. Manual inspection of 50 randomly selected outputs confirmed >95% extraction accuracy. The CWE hierarchy mapping (e.g., CWE-121 ≈ CWE-122 under parent CWE120) may slightly inflate CWE match rates but reflects meaningful semantic equivalence.
External Validity: The Juliet Test Suite provides controlled, synthetic test cases. Real-world vulnerabilities are more complex; state-of-the-art approaches achieve only ∼35% F1 on PrimeVul [4]. Our results demonstrate relative architectural value, not absolute performance claims. Construct Validity: The high FPR on benign samples reflects a general LLM challenge: Juliet’s patched versions retain structural similarity to vulnerable code. API costs depend on current pricing ($0.27/MTok for DeepSeek-V3 as of March 2026).
Future Work: We plan to extend this work in several directions: (1) evaluation on real-world vulnerability datasets such as DiverseVul and PrimeVul to assess generalization; (2) iterative expertverifier dialogue, inspired by the multi-agent debate paradigm [5], where the verifier challenges specific expert claims and experts refine their analysis over multiple rounds—preliminary results show this can substantially reduce false positive rates; (3) dynamic agent selection that adapts the number and type of experts based on code characteristics; and (4) integration with existing CI/CD pipelines for practical deployment evaluation.
6
ACKNOWLEDGMENTS
CONCLUSION
We presented a game-theory-inspired framework for heterogeneous multi-agent vulnerability detection. Our 3+1 architecture deploys three cloud-based LLM experts (DeepSeek-V3) for parallel vulnerability analysis and one local adversarial verifier (Qwen3-8B) for quality assurance, with the game-theoretic framework providing principled justification for each design choice. Our evaluation on 262 real Juliet Test Suite samples—with balanced vulnerable and benign classes across 14 CWE types—provides rigorous metrics including Precision, Recall, F1, False Positive Rate, and Matthews Correlation Coefficient with bootstrap confidence intervals. Key findings include: • Expert coalitions with diverse analysis perspectives exhibit super-additive value when combined with verification: the 3+1 system (F1 = .772) significantly outperforms both a single expert (.714) and the coalition alone (.689). • Adversarial verification by an independent local model improves precision by catching false positives where experts incorrectly flag benign code, with zero marginal API cost. • Parallel execution achieves significant speedup without quality degradation, demonstrating Pareto efficiency. Game Theory as Analytical Tool: Our central claim is that game-theoretic frameworks provide correct and useful predictions about multi-agent LLM system behavior. The cooperative game predicted super-additive value from diverse coalitions—confirmed by 100% CWE match rate. The adversarial game predicted precision improvement from independent verification—confirmed with 𝑝 < 10−6 . The mechanism design analysis predicted that heterogeneous cloud-local allocation dominates homogeneous designs— confirmed by the verifier’s zero-cost precision gain. While LLM agents do not engage in explicit strategic reasoning, game theory correctly predicts the system-level outcomes of architectural choices, establishing its value as an analytical and design tool for the emerging field of strategic multi-agent engineering. Limitations: Our evaluation uses the synthetic Juliet Test Suite; real-world vulnerabilities present additional challenges. We do not claim state-of-the-art detection rates—recent work [4] shows that even the best approaches achieve only ∼35% F1 on the challenging PrimeVul benchmark. The false positive rate on benign code remains substantial, reflecting a general challenge for LLM-based vulnerability detection. Our contribution is architectural: demonstrating how to design cost-effective multi-agent systems, not achieving optimal detection accuracy.
This work was conducted at the University of Southern California. We thank the anonymous reviewers for their constructive feedback, which substantially improved this paper.
REFERENCES [1] Pepijn Cobben, Xuan Huang, Thao Pham, et al. 2026. GT-HarmBench: Benchmarking AI Safety Risks Through the Lens of Game Theory. arXiv preprint arXiv:2602.12316 (2026). [2] Cppcheck Team. 2024. Cppcheck: A Tool for Static C/C++ Code Analysis. https: //cppcheck.sourceforge.io/. [3] DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437 (2024). https://doi.org/10.48550/ARXIV.2412.19437 [4] Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, David Wagner, et al. 2025. Vulnerability Detection with Code Language Models: How Far Are We?. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE). https://doi.org/10.1109/ICSE55347.2025.00038 PrimeVul benchmark. [5] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In International Conference on Machine Learning (ICML). [6] Jinhao Duan et al. 2024. GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations. In NeurIPS. https://doi.org/10. 48550/ARXIV.2402.12348 [7] Jianing Hao, Han Ding, Yuanjian Xu, Tianze Sun, Ran Chen, Wanbo Zhang, Guang Zhang, and Siguang Li. 2026. Game-Theoretic Lens on LLM-based Multi-Agent Systems. arXiv preprint arXiv:2601.15047 (2026). [8] Sirui Hong et al. 2023. MetaGPT: Meta Programming for Multi-Agent Collaborative Framework. arXiv preprint arXiv:2308.00352 (2023). [9] Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don’t software developers use static analysis tools to find bugs?. In 2013 35th International Conference on Software Engineering (ICSE). 672–681. https://doi.org/10.1109/icse.2013.6606613 [10] Ahmed Lekssays et al. 2025. LLMxCPG: Context-Aware Vulnerability Detection Through Code Property Graph-Guided Large Language Models. In 34th USENIX Security Symposium. [11] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society. In Neural Information Processing Systems (NeurIPS). [12] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Conference on Empirical Methods in Natural Language Processing (EMNLP). [13] Steve Morgan. 2020. Cybercrime To Cost The World $10.5 Trillion Annually By 2025. Cybercrime Magazine (2020). Cybersecurity Ventures. [14] Noam Nisan, Tim Roughgarden, Eva Tardos, and Vijay V Vazirani. 2007. Algorithmic Game Theory. Cambridge University Press. https://doi.org/10.1017/ CBO9780511800481 [15] NIST SAMATE. 2017. Juliet Test Suite for C/C++. https://samate.nist.gov/SARD/ test-suites/112. [16] OpenAI et al. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023). [17] Qwen Team. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). https://doi.org/10.48550/ARXIV.2505.09388 [18] Shreshth Rajan et al. 2026. MultiVer: Zero-Shot Multi-Agent Vulnerability Detection. arXiv preprint arXiv:2602.17875 (2026). [19] Michael Spence. 1973. Job market signaling. The Quarterly Journal of Economics 87, 3 (1973), 355–374. https://doi.org/10.2307/1882010 [20] Synopsys. 2024. Coverity Static Analysis. https://www.synopsys.com/softwareintegrity/security-testing/static-analysis-sast.html.
Table 6: Complete dataset composition by CWE type CWE
Description
CWE-121 CWE-122 CWE-190 CWE-401 CWE-415 CWE-416 CWE-476 CWE-252 CWE-78 CWE-134 CWE-369 CWE-457 CWE-789 CWE-400
Stack Buffer Overflow Heap Buffer Overflow Integer Overflow Memory Leak Double Free Use After Free NULL Pointer Deref Unchecked Return Value OS Command Injection Format String Vuln Divide by Zero Uninitialized Variable Uncontrolled Mem Alloc Resource Exhaustion
Total
Vuln.
Benign
Total
10 10 10 10 6 7 9 10 10 10 10 10 10 10
10 10 10 10 6 7 9 10 10 10 10 10 8 10
20 20 20 20 12 14 18 20 20 20 20 20 18 20
132
130
262
[21] Ratnadira Widyasari, Martin Weyssow, et al. 2025. Let the Trial Begin: A MockCourt Approach to Vulnerability Detection using LLM-Based Agents. arXiv preprint arXiv:2505.10961 (2025). Accepted at ICSE 2026. [22] Qingyun Wu et al. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv preprint arXiv:2308.08155 (2023). [23] Zihan Wu, Jie Xu, Yun Peng, Chun Yong Chong, and Xiaohua Jia. 2026. MulVul: Retrieval-augmented Multi-Agent Code Vulnerability Detection via Cross-Model Prompt Evolution. arXiv preprint arXiv:2601.18847 (2026). [24] Yi Xie, Zhanke Zhou, Chentao Cao, Qiyu Niu, Tongliang Liu, and Bo Han. 2025. From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium. arXiv preprint arXiv:2506.08292 (2025). Accepted at ICML 2025.
Weaknesses: out-of-bounds write (CWE-787 ⊃ CWE-121, CWE122), use after free (CWE-416), OS command injection (CWE-78), NULL pointer dereference (CWE-476), integer overflow (CWE-190), and missing release of memory (CWE-401).
B B.1
A.2
CWE Coverage Justification
Our 14 CWE types include representatives from 7 of the top 10 categories in the 2024 CWE Top 25 Most Dangerous Software
Code Analyst You are a senior code structure analyst specializing in C/C++ programs. Analyze the given code for: 1. Data flow patterns— track how data moves through variables and function parameters. 2. Control flow anomalies—identify unusual branching or unreachable code. 3. Vulnerability entry points—locate functions receiving external input. 4. Memory operations— track malloc/free pairs and pointer arithmetic. Output a structured report: VULNERABILITY_FOUND: yes/no; CWE_IDs: [list of CWE IDs detected]; SEVERITY: critical/high/medium/low/none; EVIDENCE: specific code lines and explanation; CONFIDENCE: high/medium/low.
B.2
Security Expert You are a cybersecurity expert with deep knowledge of common vulnerabilities and CWE taxonomy. Analyze the given C/C++ code for security vulnerabilities: 1. Buffer overflows (CWE-120, CWE-121, CWE-122, CWE-787)—check unsafe string/memory operations. 2. Memory safety (CWE-416, CWE-415, CWE-401)—use-after-free, double-free, memory leaks. 3. Integer issues (CWE-190, CWE-191)—overflow, underflow, truncation. 4. Injection (CWE-78, CWE-134)—OS command injection, format string. 5. Null pointer (CWE476)—dereference without check. 6. Other issues (CWE-252, CWE-369, CWE-457, CWE-789, CWE-400). Output a structured report: VULNERABILITY_FOUND: yes/no; CWE_IDs: [list]; SEVERITY; EVIDENCE; CONFIDENCE.
A DATASET DETAILS A.1 Juliet Test Suite Extraction We extracted samples from the NIST Juliet Test Suite v1.3 [15], which contains 64,295 C/C++ test cases across 118 CWE types. Each test case provides both a vulnerable (bad()) function and a corresponding patched (good()) function within the same source file, enabling naturally-paired evaluation. Selection Criteria: We selected Flow Variant 01 (Baseline) test cases—single-file, self-contained functions without complex interprocedural flow—to ensure consistent analysis complexity and eliminate confounding factors from multi-file dependencies. Balance: Unlike many prior studies that evaluate only on vulnerable samples (making precision/FPR undefined), we include the patched version of each test case as a benign sample. This is critical: an LLM that always predicts “vulnerable” would achieve 100% recall but 50% precision, exposing the system’s ability to distinguish genuine vulnerabilities from safe code. Table 6 shows the complete dataset distribution. Sample Characteristics: Average code length is 38.5 lines (range: 5–120). Code is predominantly C with some C++ constructs. All samples include relevant #include directives and #define macros from the Juliet framework for context.
FULL AGENT PROMPTS
We provide the complete system prompts used for each agent role.
B.3
Debug Expert You are a debugging specialist focused on edge cases and error conditions in C/C++ code. Analyze for: 1. Error handling gaps—missing return value checks, unchecked mallocs. 2. Boundary conditions—off-by-one errors, array index issues. 3. Edge cases—empty inputs, max values, null values. 4. Resource management—unclosed files, orphaned allocations. 5. Undefined behavior—uninitialized variables, signed overflow. Output a structured report: VULNERABILITY_FOUND: yes/no; CWE_IDs: [list]; SEVERITY; EVIDENCE; CONFIDENCE.
B.4
Adversarial Verifier You are an adversarial code security reviewer. You receive three expert vulnerability analysis reports for the same code snippet. Your job is to: 1. CHECK CONSISTENCY— Do the three reports agree on whether vulnerabilities exist? Flag contradictions. 2. VERIFY EVIDENCE—Are vulnerability claims supported by actual code references? Flag hallucinations. 3. FIND MISSED PATTERNS—Are there obvious vulnerabilities that all experts missed? 4. ASSESS
Table 7: CWE hierarchy equivalence classes for matching
Table 8: Complete system configuration
Parent Category
Equivalent CWEs
Component
Specification
Buffer Overflow Integer Issues
CWE-119, CWE-120, CWE-121, CWE-122, CWE-787 CWE-190, CWE-191
Cloud Expert Agents Model API Temperature Max tokens Cost
DeepSeek-V3 (deepseek-chat) OpenAI-compatible 0.1 4,000 $0.27/MTok input, $1.10/MTok output
Local Verifier Agent Model Runtime Precision GPU Temperature Max new tokens Cost
Qwen3-8B HuggingFace Transformers BFloat16 NVIDIA RTX 3090 (24GB VRAM) 0.1 2,048 $0.00 (local inference)
CONFIDENCE—Based on agreement level, rate overall confidence. Output your assessment: DECISION: ACCEPT/CHALLENGE/REJECT; FINAL_VULNERABILITY: yes/no; FINAL_CWE_IDS: [consolidated list]; AGREEMENT_LEVEL: full/partial/none; REASONING: brief explanation.
C EVALUATION METHODOLOGY C.1 CWE Extraction Pipeline
Experiment Infrastructure Python 3.11 Parallelism Python asyncio OS Ubuntu 22.04, Linux 6.18 GPU 2× NVIDIA RTX 3090
We extract vulnerability predictions automatically from LLM outputs using the following pipeline: (1) CWE ID Extraction: We extract CWE identifiers from all agent outputs using regex pattern matching and normalize them to CWE-NNN format. (2) Vulnerability Decision: If the verifier provides FINAL_ VULNERABILITY: yes/no, we use that. Otherwise, majority voting among experts (at least 2/3 positive with at least one CWE ID). (3) CWE Matching: Predicted CWEs are compared against ground truth with hierarchy-aware matching: • Exact match: CWE-121 predicted, CWE-121 is truth → match • Hierarchy match: CWE-122 predicted, CWE-121 is truth → match (both under CWE-120 “Buffer Overflow”) • No match: CWE-476 predicted, CWE-121 is truth → no match
C.2
Component
Avg Tokens
Avg Cost
% Total
Code Analyst (DeepSeek) Security Expert (DeepSeek) Debug Expert (DeepSeek)
∼1,074 ∼1,074 ∼1,074
$0.00069 $0.00069 $0.00069
33.3% 33.3% 33.3%
Cloud Subtotal
3,222
$0.00208
100%
Verifier (Qwen3-8B, local)
3,459
$0.00
0%
Total
6,681
$0.00208
100%
CWE Hierarchy Mapping
Table 7 shows the equivalence classes used for CWE matching: All other CWEs require exact numeric match. This hierarchy is conservative: we only group CWEs that share the same fundamental vulnerability mechanism.
C.3
Table 9: Per-sample cost breakdown (3+1 Parallel+Verifier)
Statistical Methods
Bootstrap Confidence Intervals: We compute 95% CIs using 1,000 bootstrap resamples with a fixed seed (42) for reproducibility. For each resample, we draw 𝑛 samples with replacement and compute the metric, then report the 2.5th and 97.5th percentiles. McNemar’s Test: For pairwise comparison of configurations on the same samples, we use the exact binomial version of McNemar’s test. We count discordant pairs (samples where one system is correct and the other incorrect) and test whether the discordance is symmetric using a two-sided binomial test with 𝑝 = 0.5. Matthews Correlation Coefficient: We report MCC as a comprehensive single metric because it accounts for all four quadrants of the confusion matrix and is less sensitive to class imbalance than accuracy or F1.
D IMPLEMENTATION DETAILS D.1 System Configuration D.2 Cost Breakdown E REPRESENTATIVE OUTPUT EXAMPLES We provide representative LLM outputs to illustrate system behavior on true positive, true negative, false positive, and false negative cases. Full outputs are available in the supplementary material.
E.1
True Positive Example (CWE-78, OS Command Injection)
A function constructs a shell command using unsanitized user input and passes it to system(). All three experts correctly identify CWE-78 with high confidence, citing the direct concatenation of user input into the command string. The verifier confirms full agreement across reports and accepts the finding. CWE-78 has our lowest FPR (10%) because the system() call is a clear syntactic marker.
E.2
True Negative Example (CWE-457, Uninitialized Variable)
A patched function that properly initializes all variables before use. The verifier notes that while two experts flag potential issues, the third expert correctly observes that variables are initialized on all code paths. The verifier challenges the experts’ claims, finding no code evidence supporting the vulnerability—resulting in a correct “benign” classification. CWE-457 has our second-lowest FPR (20%).
E.3
False Positive Example (CWE-190, Integer Overflow)
A patched function with proper overflow checks that is still flagged as vulnerable by the system. Analysis: the patched version adds bounds checking (if (size > MAX_SIZE) return), but the experts still flag the arithmetic operation itself as potentially dangerous. The verifier does not override because the code structure remains similar to the vulnerable version. This illustrates a fundamental LLM challenge: distinguishing “has dangerous pattern” from “has dangerous pattern but with proper guards.” CWE-190 has 90% FPR.
E.4
False Negative Analysis
Our system produces zero false negatives—all 132 vulnerable samples are correctly detected. This perfect recall is consistent across all three configurations and all 14 CWE types. While encouraging, this likely reflects the relatively clear vulnerability patterns in Juliet’s Variant 01 (baseline) test cases. We expect false negatives to appear on more complex, real-world code.
F
G
PROOF OF THEOREM 1
Proof. Consider the system with expert coalition 𝐸 producing predictions and verifier 𝑉 performing post-hoc validation. Let the experts produce true positive count 𝑇 𝑃 and false positive count 𝐹 𝑃. Expert precision is: 𝑇𝑃 (5) 𝑇𝑃 + 𝐹𝑃 The verifier independently examines each positive prediction. For false positives, the verifier detects and removes them with probability 𝑝 𝑓 𝑝 > 0. For true positives, the verifier may incorrectly reject them with probability 𝑝 𝑓 𝑛 (where we design prompts to minimize this, but allow 𝑝 𝑓 𝑛 ≥ 0). After verification: 𝑃𝑒𝑥𝑝𝑒𝑟𝑡𝑠 =
• The Security Expert contributes precise CWE IDs from taxonomy knowledge (e.g., distinguishing CWE-121 from CWE-122) • The Code Analyst provides structural evidence (data flow, buffer sizes) that validates CWE claims • The Debug Expert identifies boundary conditions that confirm vulnerability mechanisms When any single expert misclassifies a vulnerability (e.g., reporting CWE-787 instead of CWE-121), the other experts’ complementary evidence corrects the classification through union aggregation. This is precisely the super-additivity predicted by Proposition 1: 𝑄𝐶𝑊 𝐸 ({1, 2, 3}) > max𝑖 𝑄𝐶𝑊 𝐸 ({𝑖}). The verifier’s Shapley value is particularly instructive. Its marginal contribution to the full coalition is: 𝜙𝑉 = 𝑣 ({1, 2, 3, 𝑉 }) − 𝑣 ({1, 2, 3}) = 0.776 − 0.670 = +0.106 in F1 terms. Since the verifier’s marginal cost is zero, 𝜙𝑉 /𝐶𝑉 → ∞—the verifier has infinite “return on investment,” confirming that adding a free local verifier is always individually rational.
(6)
𝐹 𝑃𝑠𝑦𝑠𝑡𝑒𝑚 = 𝐹 𝑃 · (1 − 𝑝 𝑓 𝑝 )
(7)
System precision: 𝑃𝑠𝑦𝑠𝑡𝑒𝑚 =
𝑇 𝑃 · (1 − 𝑝 𝑓 𝑛 ) 𝑇 𝑃 · (1 − 𝑝 𝑓 𝑛 ) + 𝐹 𝑃 · (1 − 𝑝 𝑓 𝑝 )
(8)
We want to show 𝑃𝑠𝑦𝑠𝑡𝑒𝑚 ≥ 𝑃𝑒𝑥𝑝𝑒𝑟𝑡𝑠 when 𝑝 𝑓 𝑝 ≥ 𝑝 𝑓 𝑛 (the verifier is better at catching false positives than incorrectly rejecting true positives). This condition holds when the verifier prompt is designed for adversarial validation (“find inconsistencies”) rather than re-analysis. Sufficient condition: If 𝑝 𝑓 𝑛 = 0 (verifier never rejects true positives), then:
SHAPLEY VALUE ANALYSIS
We compute approximate Shapley values (Eq. 3) for each expert role by evaluating coalition subsets. Since individual expert outputs are available from the parallel configuration, we can assess each expert’s marginal contribution to CWE identification. All three experts achieve 100% recall individually, so the marginal contribution to recall is zero for each additional expert. The superadditive value appears in CWE classification: the union of three perspectives achieves 100% CWE match rate because:
𝑇 𝑃𝑠𝑦𝑠𝑡𝑒𝑚 = 𝑇 𝑃 · (1 − 𝑝 𝑓 𝑛 )
𝑃𝑠𝑦𝑠𝑡𝑒𝑚 =
𝑇𝑃 𝑇𝑃 ≥ = 𝑃𝑒𝑥𝑝𝑒𝑟𝑡𝑠 𝑇 𝑃 + 𝐹 𝑃 (1 − 𝑝 𝑓 𝑝 ) 𝑇 𝑃 + 𝐹 𝑃
(9)
with strict inequality when 𝑝 𝑓 𝑝 > 0 and 𝐹 𝑃 > 0. In practice, 𝑝 𝑓 𝑛 > 0 is possible but small, since the verifier has access to expert evidence supporting true positives and is prompted to focus on inconsistencies rather than re-analysis. □ □
H LIMITATIONS AND BROADER CONTEXT H.1 Relationship to State of the Art We position our work as an architectural contribution rather than a detection accuracy claim. The PrimeVul benchmark [4] demonstrates that existing evaluations significantly overestimate LLM vulnerability detection: a state-of-the-art 7B model achieves 68% F1 on Big-Vul but only 3% on PrimeVul’s stringent evaluation. Our Juliet results should be interpreted as demonstrating the relative value of heterogeneous multi-agent design, not as absolute performance claims.
H.2
Game Theory as Analytical Tool for Multi-Agent LLMs
A central question for the Strategic Engineering community is whether game-theoretic analysis is useful for designing multi-agent LLM systems, given that LLMs do not explicitly reason strategically [6]. Our results provide evidence that it is: (1) Correct predictions: The cooperative game predicted superadditivity from diverse coalitions—confirmed empirically
(100% CWE match). The adversarial game predicted precision improvement from independent verification—confirmed (𝑝 < 10−6 ). (2) Design guidance: Without the game-theoretic framework, the choice of three diverse experts (vs. three identical ones) and the heterogeneous cloud-local split would be ad hoc. The framework provides principled justification for these choices. (3) Quantitative analysis: The Shapley value computation (Appendix F) reveals that the verifier has infinite marginal return—a result not obvious without formal analysis. The key insight is that game theory is useful at the mechanism design level: we design the “rules of the game” (agent roles, interaction protocol, verification structure) so that desirable system-level properties emerge. This is analogous to auction design [14], where
the mechanism designer does not require bidders to solve the game themselves—the mechanism structure ensures good outcomes.
H.3
Scalability Considerations
Our current architecture uses three experts and one verifier. Scaling considerations include: • More experts: Diminishing returns expected beyond 3–5 perspectives, consistent with multi-agent debate literature showing saturation after 2–3 rounds. • Larger local models: Qwen3-8B could be replaced with Qwen3-14B or Qwen3-32B for stronger verification; the costquality trade-off depends on available GPU resources. • Different cloud providers: The architecture is provideragnostic; DeepSeek-V3 can be replaced with any OpenAIcompatible API.