1
Preprint — Under Review
VCAO: Verifier-Centered Agentic Orchestration for Strategic OS Vulnerability Discovery Suyash Mishra AI Researcher, Basel, Switzerland [email protected]
arXiv:2604.08291v1 [cs.GT] 9 Apr 2026
April 2026
Abstract We formulate operating-system vulnerability discovery as a repeated Bayesian Stackelberg search game in which a Large Reasoning Model (LRM) orchestrator allocates analysis budget across kernel files, functions, and attack paths while external verifiers—static analyzers, fuzzers, and sanitizers—provide evidence. At each round, the orchestrator selects a target component, an analysis method, and a time budget; observes tool outputs; updates Bayesian beliefs over latent vulnerability states; and re-solves the game to minimize the strategic attacker’s expected payoff. We introduce VCAO (Verifier-Centered Agentic Orchestration), a six-layer architecture comprising surface mapping, intra-kernel attack-graph construction, game-theoretic file/function ranking, parallel executor agents, cascaded verification, and a safety governor. Our DOBSS-derived MILP allocates budget optimally across heterogeneous √ analysis tools under resource constraints, with formal Õ( T ) regret bounds from online Stackelberg learning. Experiments on five Linux kernel subsystems—replaying 847 historical CVEs and running live discovery on upstream snapshots—show that VCAO discovers 2.7× more validated vulnerabilities per unit budget than coverage-only fuzzing, 1.9× more than static-analysis-only baselines, and 1.4× more than non-game-theoretic multi-agent pipelines, while reducing false-positive rates reaching human reviewers by 68%. We release our simulation framework, synthetic attack-graph generator, and evaluation harness as open-source artifacts.
Keywords: vulnerability discovery · Bayesian Stackelberg games · large reasoning models · agentic orchestration · kernel security · game-theoretic resource allocation
1
Introduction
The landscape of operating-system vulnerability discovery is undergoing a paradigm shift. Recent demonstrations by Anthropic show that frontier reasoning models can identify thousands of zero-day vulnerabilities across every major OS and browser [Carlini et al., 2026b,a], with exploit development success rates exceeding 72%. Google’s Big Sleep project independently demonstrated LLM-discovered vulnerabilities in production software [Glazunov and Brand, 2024]. These results suggest that the primary bottleneck in vulnerability discovery is shifting from tool capability to orchestration intelligence: deciding where to look, how to look, and when to verify.
Preprint — Under Review
2
Existing kernel security workflows deploy powerful tools—CodeQL for data-flow analysis [GitHub, 2024], Syzkaller for coverage-guided fuzzing [Vyukov, 2015], KASAN for memorysafety detection [Linux Kernel Documentation, 2024]—but coordinate them through ad-hoc heuristics. The 2025 CWE Top 25 [MITRE, 2025] confirms that memory-safety and accesscontrol weaknesses remain dominant, with out-of-bounds writes (CWE-787) and use-after-free (CWE-416) accounting for approximately 35% of all Linux kernel CVEs. The gap is not tool capability but decision-theoretic coordination: no existing system answers the question “which analysis action most reduces the strategic attacker’s advantage?” We address this gap by formulating vulnerability discovery as a repeated Bayesian Stackelberg search game. The defender (LRM orchestrator) commits to a mixed analysis strategy over kernel components; a strategic attacker best-responds by choosing exploit paths that maximize damage. The orchestrator updates beliefs from tool observations and re-solves at each round. This formulation inherits three desirable properties from the Stackelberg security games literature [Tambe, 2011, Sinha et al., 2018]: (i) commitment power yields higher defender utility than simultaneous play; (ii) the DOBSS algorithm [Paruchuri et al., 2008] provides an efficient MILP for computing optimal strategies under Bayesian uncertainty over attacker types; and (iii) online learning extensions [Balcan et al., 2015] guarantee sublinear regret when attacker behavior is initially unknown. Contributions. We make four contributions: 1. A formal game-theoretic formulation of OS vulnerability discovery as a repeated Bayesian Stackelberg game with intra-kernel attack graphs (§3). 2. The VCAO architecture: a six-layer agentic system that operationalizes the game with LRM orchestration, heterogeneous tool integration, and cascaded verification (§4). 3. A budget-allocation MILP adapted from DOBSS with formal regret bounds, and a Bayesian belief-update mechanism for vulnerability state estimation (§5). 4. Comprehensive evaluation on five Linux kernel subsystems showing significant improvements in validated vulnerability yield, false-positive reduction, and attacker-payoff minimization (§6).
2
Related Work
Stackelberg Security Games. The foundational framework of Stackelberg Security Games (SSGs) originated with ARMOR [Tambe, 2011] and was formalized by Conitzer and Sandholm [2006]. Paruchuri et al. [2008] introduced DOBSS, an efficient MILP for Bayesian extensions with multiple attacker types. Deployed systems include IRIS, GUARDS, and PROTECT [Kiekintveld √ et al., 2009]. Online extensions by Balcan et al. [2015] achieve Õ( T ) regret. Zhang and Malacaria [2021] applied BSSGs to cybersecurity portfolio selection but not to vulnerability discovery resource allocation. LLM-Based Vulnerability Discovery. Anthropic’s work with PNNL [Anthropic and Pacific Northwest National Laboratory, 2026] demonstrated agentic attack-chain construction. The
3
Preprint — Under Review
Opus 4.6 evaluation [Carlini et al., 2026a] found 500+ vulnerabilities at $4,000 total cost. Mythos Preview [Carlini et al., 2026b] achieved 72.4% exploit success with a file-ranking scaffold that we formalize game-theoretically. Google’s Naptime/Big Sleep [Glazunov and Brand, 2024] provided tool-use architectures. ChatAFL [Meng et al., 2024] and KernelGPT [Yang et al., 2025] use LLMs for fuzzing guidance. IRIS [Li et al., 2025] combines LLMs with CodeQL achieving 55/120 CVE detection. Our work differs by providing a principled allocation framework atop these capabilities. Game-Theoretic Software Testing. Godefroid and Kinder [2010] first framed fuzzing as a two-player game. EcoFuzz [Yue et al., 2020] models coverage fuzzing as a multi-armed bandit. MEGA-PT [Bland et al., 2024] uses meta-games for penetration testing. Böhme and Félegyházi [2010] formulate pen-testing ROI in a weakest-link game. None combine Stackelberg commitment with multi-tool orchestration for kernel vulnerability discovery. Attack Graph Analysis. MulVAL [Ou et al., 2005, 2006] introduced logic-based attack-graph generation. Bayesian Attack Graphs [Frigault and Wang, 2008, Munoz-González and Lupu, 2017] propagate CVSS-derived probabilities. All prior work targets network-level multi-host graphs. We introduce the first intra-kernel attack-graph model.
3
Problem Formulation
3.1
Intra-Kernel Attack Graph
Definition 1 (Intra-Kernel Attack Graph). An intra-kernel attack graph is a directed acyclic graph G = (V, E, C, ϕ, ψ) where: • V = Ventry ∪ Vfunc ∪ Vpriv ∪ Vgoal partitions vertices into entry points (syscalls, ioctls, parsers), internal functions, privilege boundaries, and attacker goals (root, sandbox escape, data exfiltration, DoS). • E ⊆ V × V represents control-flow, data-flow, or privilege-transition edges. • C = {c1 , . . . , cK } is a set of vulnerability classes (e.g., CWE-787, CWE-416, CWE-362). • ϕ : V × C → [0, 1] maps each vertex-class pair to a prior vulnerability probability, derived from CVSS base scores and historical defect density. • ψ : E → [0, 1] assigns edge exploitability probabilities. The probability that an attacker can traverse path P = (v1 , e1 , v2 , . . . , vn ) to reach a goal g ∈ Vgoal is: Pr[reach g | P ] =
Y (vi ,vi+1 )∈P
ψ(vi , vi+1 ) · 1 −
Y
(1 − ϕ(vi , c))
(1)
c∈C
where the inner term represents the probability that at least one vulnerability class is present at vertex vi .
4
Preprint — Under Review
3.2
Bayesian Stackelberg Vulnerability Discovery Game
Definition 2 (BSVD Game). A Bayesian Stackelberg Vulnerability Discovery game is a tuple Γ = (G, Ad , Aa , L, p, Ud , Ua , B) where: • G is the intra-kernel attack graph. • Ad = {(f, m, τ ) : f ∈ V, m ∈ M, τ ∈ R+ } is the defender’s action space: target f , method m ∈ M = {CodeQL, Fuzz, KASAN, KCSAN, PatchMine, Verify}, budget τ . • Aa = {P : P is an attack path in G} is the attacker’s action space. • L = {ℓ1 , . . . , ℓ|L| } are attacker types (APT, opportunistic, insider) with prior p = (p1 , . . . , p|L| ). • Ud , Ua : Ad × Aa × L → R are utility functions. • B ∈ R+ is the total analysis budget. Defender Utility.
Let c = (c1 , . . . , c|V| ) denote the defender’s coverage vector, where cf is
the fraction of budget allocated to vertex f . Given coverage c and attacker path P of type ℓ: Ud (c, P, ℓ) =
X
cf · Vbug (f ) · ρ(f ) · ηver (f ) − λ · FP(f, m)
f ∈P
− (1 − cf ) · Impactℓ (f )
(2)
where Vbug (f ) is the validated-bug value (product of CVSS severity and reachability), ρ(f ) is the detection probability under method m, ηver (f ) is verifier confidence, FP(f, m) is the false-positive cost, and Impactℓ (f ) is the damage from an undetected vulnerability exploited by type ℓ. Attacker Utility.
For type ℓ attacking path P : Uaℓ (c, P ) =
X
(1 − cf · ρ(f )) · Raℓ (f ) − cf · ρ(f ) · Daℓ (f )
(3)
f ∈P
where Raℓ (f ) is the attacker’s reward from exploiting f and Daℓ (f ) is the deterrence cost if detected.
3.3
Strong Stackelberg Equilibrium
The defender commits to c∗ maximizing expected utility against all attacker types’ best responses: Theorem 1 (BSVD Equilibrium). The optimal defender strategy c∗ satisfies: c∗ = arg max c∈∆B
X
pℓ · Ud c, P ℓ∗ (c), ℓ
(4)
ℓ∈L
where P ℓ∗ (c) = arg maxP ∈Aa Uaℓ (c, P ) is type ℓ’s best-response path, and ∆B = {c ≥ 0 : f wf cf ≤ B} is the budget-feasible simplex with per-target costs wf .
P
5
Preprint — Under Review
3.4
DOBSS-VD: MILP Formulation
We linearize the bilevel optimization in (4) following the DOBSS decomposition [Paruchuri et al., ℓ 2008]. Let qPℓ ∈ {0, 1} indicate whether type ℓ attacks path P , and let zf,P = cf · qPℓ . The MILP
is:
max
X
c,q,z
s.t.
pℓ
X Xh
i
ℓ ℓ zf,P · Udcov (f, ℓ) + (qPℓ − zf,P ) · Udunc (f, ℓ)
(5)
ℓ∈L
P ∈Aa f ∈P
X
wf · c f ≤ B
(6)
qPℓ = 1,
(7)
f
X
∀ℓ ∈ L
P
Xh
i
ℓ ℓ zf,P · Uaℓ,cov (f ) + (qPℓ − zf,P ) · Uaℓ,unc (f )
f ∈P
≥
X h
i
cf · Uaℓ,cov (f ) + (1 − cf ) · Uaℓ,unc (f ) − M (1 − qPℓ ),
f ∈P ′
∀P, P ′ ∈ Aa , ∀ℓ ℓ zf,P ≤ cf ,
ℓ zf,P ≤ qPℓ ,
cf ∈ [0, 1],
qPℓ ∈ {0, 1},
ℓ zf,P ≥ cf + qPℓ − 1
(8) (9)
ℓ zf,P ∈ [0, 1]
Constraint (6) enforces the analysis budget. Constraint (7) ensures each attacker type selects one path. Constraint (8) encodes the attacker’s best-response via big-M linearization. Constraint (9) is the McCormick envelope for bilinear terms.
3.5
Bayesian Belief Update
After executing action at = (ft , mt , τt ) and observing result ot ∈ {alert, clean, crash, timeout}, the orchestrator updates beliefs:
bt+1 (f, c) =
Pr[ot | vuln(f, c), at ] · bt (f, c) Pr[ot | vuln(f, c), at ] · bt (f, c) + Pr[ot | ¬vuln(f, c), at ] · (1 − bt (f, c))
(10)
The observation likelihoods are method-specific: Pr[alert | vuln, CodeQL] = ρCQL (c)
(true positive rate)
(11)
Pr[alert | ¬vuln, CodeQL] = αCQL (c)
(false positive rate)
(12)
Pr[crash | vuln, Fuzz] = 1 − (1 − ρfuzz )τ /τ0
(13)
where (13) models fuzzing crash probability increasing with budget τ at rate ρfuzz per quantum τ0 .
6
Preprint — Under Review
Linux Kernel Source Surface Mapper Agent L1 Syscall Extraction • ioctl Interfaces • Parser Boundaries • Namespace/Capability Paths
Feedback loop: belief update
Intra-Kernel Attack Graph Builder L2 G = (V, E, C, ϕ, ψ)
L3
Privilege Transitions • Data-Flow Dependencies • CVSS Probabilities
Game-Theoretic File/Function Ranker (DOBSS-VD Solver) P ℓ ℓ∗
c∗ = arg maxc∈∆B
ℓ
p Ud (c, P
(c))
•
Bayesian Belief Update bt+1
Parallel Executor Agents
L4 Patch-Diff Miner
CodeQL Agent
Fuzzing Agent
KASAN Agent
KCSAN Agent
Cascaded Verifier / Critic Layer L5 V1 : Reproducibility → V2 : Severity Assessment → V3 : Deduplication → Human Triage
Safety Governor L6
Isolation • Logging • Human Review Gate • Disclosure Controls • Misuse Detection
Figure 1: The six-layer VCAO architecture. The game-theoretic ranker (L3) solves DOBSS-VD to optimally allocate budget across parallel executor agents (L4). Observations feed back through Bayesian belief updates to re-solve the game at each round.
3.6
Online Regret Guarantee
Theorem 2 (Regret Bound). Under the VCAO online learning protocol with T rounds, n = |V| targets, and K vulnerability classes, the expected regret satisfies: Regret(T ) =
T X
∗
[Ud (c , Pt∗ ) − Ud (ct , Pt )] ≤ O
q
T · n2 K · log(nK)
(14)
t=1
where c∗ is the optimal fixed strategy in hindsight. Proof sketch. We adapt Balcan et al. [2015] to our setting. The defender maintains an EXP3based distribution over a discretized coverage space. At each round, the defender samples ct , observes the attacker’s action Pt , and updates weights. The key adaptation is that observations are noisy (tool outputs, not exact attacker behavior), requiring a Thompson sampling layer over beliefs bt . The n2 K factor arises from the joint target-class space, and the logarithmic term from the multiplicative-weights update. Full proof in Appendix A.
4
The VCAO Architecture
Figure 1 presents the six-layer VCAO architecture. L1: Surface Mapper. An LRM agent extracts security-relevant entry points: syscall handlers, ioctl dispatch tables, file-system parsers, credential paths, and namespace/capability boundaries. For each entry point v ∈ Ventry , the agent identifies reachable internal functions via call-graph analysis, constructing Vfunc .
7
Preprint — Under Review
L2: Attack Graph Builder. Given the surface map, L2 constructs the intra-kernel attack graph G. Privilege boundaries (user/kernel, namespace crossings, capability checks) become Vpriv nodes. Edge probabilities ψ(e) are derived from CVSS exploitability metrics of historically similar code regions, following the BAG framework [Munoz-González and Lupu, 2017]. Attacker goals Vgoal include privilege escalation, sandbox escape, data exfiltration, and denial of service. L3: Game-Theoretic Ranker. This is the core computational layer. Given current beliefs bt and attack graph G, L3 solves DOBSS-VD (Equations 5–9) to produce the optimal coverage vector c∗t . The solver output determines: (a) which files/functions receive analysis budget, (b) which methods to apply, and (c) how much budget each receives. L4: Parallel Executor Agents. Five specialized agents execute analysis in parallel, each allocated budget τfm = cf · wm · B for its assigned targets: • Patch-Diff Miner: searches git history for incomplete propagation of prior fixes and identifies sibling patterns. • CodeQL Agent: synthesizes and runs data-flow queries (source→sink taint tracking) for suspected vulnerability classes. • Fuzzing Agent: directs Syzkaller effort toward high-priority targets with customized syzlang descriptions. • KASAN Agent: runs memory-safety-instrumented execution for heap/stack overflow and use-after-free detection. • KCSAN Agent: runs concurrency-sanitized execution for data-race detection in concurrencyheavy subsystems. L5: Cascaded Verifier. Inspired by Anthropic’s verifier layer [Carlini et al., 2026b], findings pass through three verification stages: V1 (reproducibility confirmation), V2 (severity assessment and CVSS scoring), V3 (deduplication against known CVEs and other findings). The cascaded design reduces false-positive escape probability to Pescape =
i=1 αi , following the Swiss Cheese
Q3
model [Dhuliawala et al., 2024]. L6: Safety Governor.
All execution occurs in isolated containers. The governor enforces:
offline-only experimentation, comprehensive audit logging, mandatory human review before any disclosure, and automatic misuse detection. This mirrors Anthropic’s published safety protocols [Carlini et al., 2026a].
5
Algorithms
5.1
Orchestration Loop
Algorithm 1 presents the main VCAO orchestration loop.
8
Preprint — Under Review
Algorithm 1 VCAO Orchestration Loop Require: Attack graph G, budget B, rounds T , priors b0 , p Ensure: Validated vulnerability set Vfound 1: Vfound ← ∅ 2: for t = 1, . . . , T do 3: // L3: Solve game 4: c∗t ← DOBSS-VD(G, bt , p, Bt ) 5: // L4: Execute agents in parallel 6: for (f, m, τ ) ∈ Dispatch(c∗t ) do 7: of,m ← Execute(f, m, τ ) 8: end for 9: // Bayesian update 10: for each observed (f, m, of,m ) do 11: bt+1 (f, c) ← BayesUpdate(bt (f, c), of,m , m) 12: end for 13: // L5: Verify candidates 14: candidates ← {(f, c) : of,m ∈ {alert, crash}} 15: for (f, c) ∈ candidates do 16: if CascadedVerify(f, c) then 17: Vfound ← Vfound ∪ {(f, c, severity)} 18: Update G: remove mitigated edges 19: end if 20: end for P 21: Bt ← Bt − τ 22: Update attacker type posterior p via observed attack patterns 23: end for 24: return Vfound
5.2
▷ Eq. (5) ▷ Tool invocation
▷ Eq. (10)
▷ Remaining budget
Path Enumeration and Pruning
Since enumerating all attack paths is exponential, we prune using belief-weighted expected payoff: Score(P ) =
X
bt (f ) · CVSS(f ) · Reachability(f )
(15)
f ∈P
Paths with Score(P ) < θ are pruned. We maintain the top-K paths using a priority queue, updated incrementally after each belief update.
5.3
Sibling Pattern Search
After discovering a vulnerability at (f ∗ , c∗ ), the orchestrator triggers a sibling search over structurally similar code: S(f ∗ , c∗ ) = {f ′ ∈ V : sim(f ′ , f ∗ ) > σ ∧ f ′ ̸= f ∗ }
(16)
where sim(·) combines code-structure similarity (AST edit distance), shared callers/callees, and historical co-fix patterns. Budget for siblings is drawn from a reserve pool Bsib = β · B.
9
Preprint — Under Review
Table 1: Main results across five kernel subsystems (Replay Mode, 847 CVEs). SVUB = Severityweighted validated findings per unit budget. FPR@Human = false positive rate reaching human reviewers. Best results in bold. Method B1: Uniform B2: Churn-based B3: Fuzz-only B4: Static-only B5: Multi-agent (no GT) B6: VCAO (no sib.) VCAO (full)
6
Evaluation
6.1
Experimental Setup
T2F (hrs) ↓
SVUB ↑
FPR% ↓
Sibling ↑
PR% ↑
14.2 ± 3.1 11.8 ± 2.7 8.4 ± 2.3 9.7 ± 2.9 5.1 ± 1.4 3.8 ± 1.1 3.2 ± 0.9
0.31 ± 0.04 0.39 ± 0.05 0.42 ± 0.06 0.59 ± 0.07 0.81 ± 0.09 1.02 ± 0.08 1.13 ± 0.07
42.7 38.2 31.4 47.3 24.6 15.3 15.1
0.0 0.0 0.0 0.0 1.3 0.0 2.4
18.3 22.1 26.7 31.2 48.5 61.7 67.8
Target Subsystems. We select five Linux kernel subsystems based on attacker relevance and defect diversity: (1) Filesystem (VFS, ext4, overlayfs mount parsing), (2) Networking (TCP/IP stack, netfilter, NFS), (3) Namespace/Capability code, (4) Selected drivers (USB, NVMe, GPU), (5) io_uring and BPF/eBPF. Evaluation Modes.
Replay mode: 847 historical CVEs (2019–2025) replayed on prior kernel
snapshots. Ground truth is the known CVE; metric is time-to-first-discovery. Live mode: current upstream snapshots (6.12–6.14) in isolated sandboxes; discoveries validated through manual reproduction. Baselines.
B1: Uniform allocation (equal budget per file). B2: Churn-based ranking (git
commit frequency). B3: Coverage-only fuzzing (Syzkaller with default configuration). B4: Static-analysis-only (CodeQL with standard query suites). B5: Non-game-theoretic multi-agent (LRM ranking without Stackelberg optimization). B6: VCAO without sibling search (β = 0). Metrics. µ1 : Time to first validated vulnerability. µ2 : Severity-weighted validated findings per unit budget (SVUB =
P
i CVSSi /B). µ3 : False-positive rate at human review. µ4 : Sibling-bug
yield. µ5 : Modeled attacker payoff reduction on G.
6.2
Results
Table 1 shows results in replay mode. VCAO achieves 2.7× higher SVUB than coverage-only fuzzing (B3), 1.9× higher than static-analysis-only (B4), and 1.4× higher than the non-gametheoretic multi-agent baseline (B5). The false-positive rate drops from 31.4–47.3% (tool baselines) to 15.1%, a 68% reduction versus the worst baseline. Figure 2 shows the vulnerability discovery curve. VCAO’s advantage increases with budget as the game-theoretic solver dynamically reallocates away from diminishing-return subsystems.
10
Validated Vulns Found
Preprint — Under Review
VCAO (full) Multi-agent (no GT) Fuzz-only Static-only Uniform
40
20
0
0
10
20
30 40 50 60 70 Analysis Budget (GPU-hours)
80
90
100
Figure 2: Validated vulnerabilities discovered vs. analysis budget. VCAO maintains superior efficiency throughout, with the gap widening at higher budgets as game-theoretic allocation exploits diminishing returns in exhausted subsystems. Table 2: Ablation results (Replay Mode). ∆SVUB is relative to full VCAO. Ablation Full VCAO − Stackelberg (use UCB) − Bayesian update (static) − Cascaded verifier − Attack graph (flat) − Sibling search − KCSAN agent
6.3
SVUB
∆SVUB
1.13 0.89 0.78 0.96 0.85 1.02 1.05
— −21.2% −31.0% −15.0% −24.8% −9.7% −7.1%
Ablation Study
Table 2 confirms that Bayesian belief update (−31%), Stackelberg optimization (−21.2%), and attack-graph structure (−24.8%) are the three most impactful components.
6.4
Per-Subsystem Analysis
7
Discussion
Scalability. The DOBSS-VD MILP scales as O(|V| · |Aa | · |L|) variables. For a subsystem with 500 files, 50 candidate paths, and 3 attacker types, the MILP has ∼75,000 variables and solves in <5 seconds using Gurobi. Path pruning (Eq. 15) keeps |Aa | manageable. Real-time re-solving every 10 minutes is feasible. Safety Considerations. This is dual-use research. We follow established precedent [Carlini et al., 2026a,b]: all experiments run in isolated offline containers, no exploitation of live systems, findings pass mandatory human review, and validated vulnerabilities follow coordinated disclosure. The game-theoretic formulation itself is defensive: it models the attacker to improve the defender’s allocation.
11
Preprint — Under Review
2 VCAO
SVUB
1.5
Multi-agent Fuzz-only
1 0.5 0
FS
Net
NS/Cap
Drivers
io_uring
Figure 3: Per-subsystem SVUB comparison. VCAO’s advantage is largest in Networking (complex attack surfaces) and Namespace/Capability code (authorization logic poorly suited to fuzzing alone). Limitations.
(1) The BSVD game assumes rational attackers; real adversaries may act
irrationally, though SSE is robust to bounded irrationality [Sinha et al., 2018]. (2) Intra-kernel attack graphs require manual validation of privilege boundaries. (3) Tool-specific observation models (Eqs. 11–13) require calibration per kernel version.
8
Conclusion
We have presented VCAO, the first game-theoretic framework for operating-system vulnerability discovery that unifies Bayesian Stackelberg security games, intra-kernel attack graphs, and LRM-orchestrated multi-tool analysis. Our DOBSS-VD formulation provides principled budget allocation with formal regret guarantees, and our six-layer architecture operationalizes this theory into a practical system. Experiments on five Linux kernel subsystems demonstrate significant improvements in validated vulnerability yield, false-positive reduction, and strategic attacker-payoff minimization over both tool-specific and multi-agent baselines. We release our simulation framework and evaluation harness to support reproducible research.
References Anthropic and Pacific Northwest National Laboratory.
Experimenting with AI to de-
fend critical infrastructure. Anthropic Blog, 2026. https://red.anthropic.com/2026/ critical-infrastructure-defense/. Maria-Florina Balcan, Avrim Blum, Nika Haghtalab, and Ariel D. Procaccia. Commitment without regrets: Online learning in Stackelberg security games. In Proc. 16th ACM Conference on Economics and Computation (EC), pages 61–78, 2015. John Bland et al. MEGA-PT: A meta-game framework for agile penetration testing. In Proc. Conference on Decision and Game Theory for Security (GameSec), 2024. Marcel Böhme and Márk Félegyházi. Optimal information security investment with penetration testing. In Proc. Conference on Decision and Game Theory for Security (GameSec), 2010.
Preprint — Under Review
12
Nicholas Carlini, Keane Lucas, Evyatar Ben Asher, Newton Cheng, Hasnain Lakhani, and David Forsythe. Evaluating and mitigating the growing risk of LLM-discovered 0-days. Anthropic Red Team Report, 2026a. https://red.anthropic.com/2026/zero-days/. Nicholas Carlini et al. Assessing Claude Mythos preview’s cybersecurity capabilities. Anthropic Red Team Report, 2026b. https://red.anthropic.com/2026/mythos-preview/. Vincent Conitzer and Tuomas Sandholm. Computing the optimal strategy to commit to. In Proc. 7th ACM Conference on Electronic Commerce (EC), pages 82–90, 2006. Shehzaad Dhuliawala et al. Chain-of-verification reduces hallucination in large language models. In Findings of the Association for Computational Linguistics (ACL), 2024. Marcel Frigault and Lingyu Wang. Measuring network security using Bayesian network-based attack graphs. In Proc. 32nd IEEE International Computer Software and Applications Conference (COMPSAC), 2008. GitHub. Analyzing data flow in C and C++ — CodeQL documentation. https://codeql. github.com/docs/codeql-language-guides/analyzing-data-flow-in-cpp/, 2024. Sergei Glazunov and Mark Brand. From Naptime to Big Sleep: Using large language models to catch vulnerabilities in real-world code. Google Project Zero Blog, 2024. Patrice Godefroid and Johannes Kinder. Improving fuzz testing using game theory. In Proc. IEEE International Conference on Software Testing, Verification and Validation Workshops, 2010. Christopher Kiekintveld, Manish Jain, Jason Tsai, James Pita, Fernando Ordoñez, and Milind Tambe. Computing optimal randomized resource allocations for massive security games. In Proc. 8th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pages 689–696, 2009. Ziyang Li, Saikat Dutta, and Mayur Naik. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In Proc. 13th International Conference on Learning Representations (ICLR), 2025. Linux Kernel Documentation. The Kernel Address Sanitizer (KASAN). https://www.kernel. org/doc/html/latest/dev-tools/kasan.html, 2024. Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. Large language model guided protocol fuzzing. In Proc. Network and Distributed System Security Symposium (NDSS), 2024. MITRE. 2025 CWE top 25 most dangerous software weaknesses. https://cwe.mitre.org/ top25/archive/2025/2025_cwe_top25.html, 2025. Luis Munoz-González and Emil C. Lupu. Efficient attack graph analysis through approximate inference. ACM Transactions on Privacy and Security, 20(3), 2017.
Preprint — Under Review
13
Xinming Ou, Sudhakar Govindavajhala, and Andrew W. Appel. MulVAL: A logic-based network security analyzer. In Proc. 14th USENIX Security Symposium, 2005. Xinming Ou, Wayne F. Boyer, and Miles A. McQueen. A scalable approach to attack graph generation. In Proc. 13th ACM Conference on Computer and Communications Security (CCS), pages 336–345, 2006. Praveen Paruchuri, Jonathan P. Pearce, Janusz Marecki, Milind Tambe, Fernando Ordoñez, and Sarit Kraus. Playing games for security: An efficient exact algorithm for solving Bayesian Stackelberg games. In Proc. 7th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 895–902, 2008. Arunesh Sinha, Fei Fang, Bo An, Christopher Kiekintveld, and Milind Tambe. Stackelberg security games: Looking beyond a decade of success. In Proc. 27th International Joint Conference on Artificial Intelligence (IJCAI), pages 5494–5501, 2018. Milind Tambe. Security and Game Theory: Algorithms, Deployed Systems, Lessons Learned. Cambridge University Press, 2011. Dmitry Vyukov. syzkaller — kernel fuzzer. https://github.com/google/syzkaller, 2015. Chenyuan Yang et al. KernelGPT: Enhanced kernel fuzzing via large language models. In Proc. 30th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2025. Tai Yue, Pengfei Wang, Yong Tang, Enze Wang, Bo Yu, Kai Lu, and Xu Zhou. EcoFuzz: Adaptive energy-saving greybox fuzzing as a variant of the adversarial multi-armed bandit. In Proc. 29th USENIX Security Symposium, 2020. Mengmeng Zhang and Pasquale Malacaria. Bayesian Stackelberg games for cyber-security decision support. Decision Support Systems, 148:113599, 2021.