CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

: Can Long-Horizon Agents Synthesize Static-Analysis Checkers? Hang He∗ , Li Wang∗ , Hao Chen , Yuchen Shao , Yuling Shi , Lisheng Wang , Peiyang Liu , Goose Lin , Zaiyuan Wang , Haiying Sun , Ting Su , Chengcheng Wan # Equal Contribution, # Corresponding Author

East China Normal University, Humanlaya Data, Shanghai Jiao Tong University, Peking University, Shanghai Innovation Institute

Abstract Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection and rarely assess whether an agent can develop a working checker in a repository from start to finish. We introduce CheckerBench, an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems. Each task includes vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. We further introduce CheckerLab, a common evaluation framework that independently rebuilds submitted checkers and measures vulnerable–fixed diagnostic contrast, patch localization, false positives, and tool use. Across 21 model–harness configurations and three independent repeats per configuration, mean Pass@1 is 32.30%, while the best reaches 45.33%. These results show that reliable, reusable checker development remains challenging for current coding agents. GitHub: https://github.com/ahang0712/CheckerBench-Dataset Hugging Face: https://huggingface.co/datasets/Benchanything/checkerbench

50

OpenCode

Claude Code

Hermes Agent

45.33

45 40 35

31.67 32.33 32.33

30 25

21.67 22.33 23.00

34.67 35.00 33.00 33.67 34.00 34.33

37.33 38.00 38.33 38.67

39.67

23.67 24.33 25.00

20 15 10 5

T-5 .6 HY 4-P rev iew Kim i-K Qw 3 en 3.8 -M ax G De PT -5. ep 6 Se ek -V 4-P ro GL M -5. 2 GL M -5. HY 2 4-P rev iew GP Cla T-5 ud .6 eO pu s4 Qw .8 en 3.8 -M Cla ax ud eO pu s4 Qw .8 en 3.8 -M ax K im De i-K ep 3 Se ek -V 4-P ro Kim De i-K ep 3 Se ek -V 4-P Cla ro ud eO pu s4 .8

2 -5.

GP

GL M

rev

iew

0

HY 4-P

Pass@1 (%)

arXiv:2610.07557v1 [cs.SE] 6 Oct 2026

∗

Figure 1 Pass@1 for 21 model–harness pairs on 300 CheckerBench tasks (Table 2).

1

“Let agents turn past flaws into reusable checks, so yesterday’s failures can guard tomorrow’s code.”

Recurring evaluation gaps (a) Not checker-centric

(b) Limited interaction horizon The workflow follows predefined stages.

Tasks target individual bug fixes. Issue

Coding agent

Git patch

Patch example

Reusable checker

Bug pattern

Stop

A patch fixes one observed failure.

Plan

Generate checker

Patch validation

Bounded repairs

A one-off patch is not a reusable checker.

Fixed stages limit interaction.

(c) System-specific agent scaffold

(d) Fragmented correctness signals

The same LLM uses different scaffolds. Shell-only Lower Same Tool-assisted LLM Higher Search

View

Patch tests omit checker-specific signals. Issue & repo

Git patch

Human test suite

Resolved / unresolved

Reported: patch-level success Missing: checker rerun, localization, false-positive evidence Patch success cannot prove checker correctness.

Edit

The scaffold changes the score.

Figure 2 Four recurring evaluation gaps: (a) Not checker-centric [32]; (b) Limited interaction horizon [67]; (c) System-

specific agent scaffold [68]; and (d) Fragmented correctness signals [53].

1

Introduction

Static analysis provides a scalable approach for detecting complex program defects without executing the target programs [11, 28, 36, 59, 67]. However, existing static analyzers typically rely on manually designed checkers, which require substantial expert effort to encode defect patterns and analysis strategies into toolspecific rules [24, 59, 67]. Recent advances in large language models (LLMs) have opened new opportunities for automated code understanding and defect detection. Nevertheless, directly applying LLMs to code snippets remains challenging due to high inference costs and limited access to repository-wide context, which can lead to incomplete reasoning and hallucinated findings [27, 46]. To overcome these limitations, recent coding agents extend LLMs with the ability to interact with software repositories and external tools [9, 10], enabling iterative code modification, execution, and feedback-driven refinement. These capabilities provide a promising foundation for automating static-analysis checker development, which traditionally requires substantial manual engineering effort. Given a defect specification, an agent can navigate analyzer infrastructures, implement checker logic, integrate the checker into the build system, and iteratively refine its behavior based on compiler and analyzer feedback [32, 52, 58, 62, 66]. However, synthesizing effective static-analysis checkers remains a challenging long-horizon task: agents must understand complex analyzer frameworks, make coordinated modifications across multiple files, and ensure that the resulting checker compiles, detects target defects, and maintains low false-positive rates [11, 36, 65]. We refer to this task as long-horizon agentic static-analysis checker synthesis. However, as illustrated in Figure 2, existing benchmarks are insufficient for evaluating long-horizon agentic static-analysis checker synthesis. First, current coding and vulnerability benchmarks are not checker-centric: repository-level benchmarks typically evaluate code completion, patch generation, or feature implementation, while vulnerability benchmarks focus on detection, classification, or repair rather than reusable checker synthesis [32, 39, 61, 64, 69, 73]. Second, existing benchmarks provide limited task interaction. They either evaluate isolated tasks or decompose workflows into predefined stages, preventing agents from performing the iterative planning, implementation, and debugging required for checker development [24, 35, 67]. Third, benchmarks differ substantially in environments, tool interfaces, and evaluation protocols, making results 2

Table 1 Systems and benchmarks for agentic static-analysis checker synthesis. ✓ denotes full support, ✓ partial

support, and ✗ no support. Appendix A defines the operational criteria. System / Benchmark Repository Executable Multi-Harness Long-Horizon Checker Semantic False-Positive Context Environment Support Interaction Synthesis Verification Evaluation RhoSynth [24] KNighter [67] SWE-bench [32] SWE-Gym [53] R2E-Gym [31] ReposVul [61] VulEval [64] JitVul [69]

✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✗ ✗ ✓ ✓ ✓ ✗ ✗ ✓

✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗

CheckerBench

✓

✓

✓

✓

✓

✓

✓

difficult to compare across systems [62, 66, 68]. Finally, current evaluation metrics mainly measure patchlevel correctness or test-suite outcomes, but do not capture checker-specific properties such as vulnerability coverage, false-positive control, and analyzer compatibility [32, 34, 63]. These limitations motivate a unified benchmark that evaluates agents’ ability to autonomously synthesize reliable and reusable static-analysis checkers. To address these gaps, we introduce CheckerBench, the first benchmark for evaluating long-horizon staticanalysis checker synthesis. CheckerBench contains 300 real-world tasks collected from 297 CVEs, 167 repositories, 85 CWEs, and five programming-language ecosystems, covering diverse analyzer environments and defect patterns. Each task provides a reproducible repository, an analyzer-specific environment, and reusable skills, enabling agents to perform iterative development and validation rather than isolated code generation. We further develop CheckerLab, a unified evaluation framework that assesses coding-agent systems under a common task specification and scoring protocol. CheckerLab performs independent repeated generation and frozen-candidate verification: each synthesized checker is rebuilt, executed on vulnerable and fixed revisions, and evaluated based on diagnostic reduction, patch localization, residual reports, and computational cost. Across 21 model–harness configurations, the average Pass@1 score is only 32.30%, with the best-performing configuration reaching 45.33% (Figure 1). These results demonstrate that synthesizing reliable and reusable static-analysis checkers remains a challenging problem for current coding agents. In summary, our contributions are threefold. • To the best of our knowledge, we are the first to build a comprehensive executable benchmark for evaluating repository-level, long-horizon static-analysis checker synthesis by coding agents. CheckerBench contains 300 real-world tasks spanning multiple programming-language ecosystems and analyzer environments, enabling systematic evaluation of agentic checker development. • We develop a scalable benchmark construction pipeline that automatically reconstructs analyzer environments, packages synthesis tasks, and validates checker behavior. This pipeline enables reproducible task creation beyond manually curated checker examples. • We present C HECKER L AB, a unified evaluation framework for coding-agent systems with a shared task specification and scoring protocol. CheckerLab supports independent repeated generation and frozen-candidate verification, providing fine-grained measurements of checker correctness, patch localization, false-positive control, and computational cost.

2

Related Work

2.1

Static Analysis and Checker Synthesis

Static analyzers rely on expert-authored checkers that encode defect patterns and analyzer-specific logic [24, 67]. RhoSynth learns graph-based code-quality rules from examples, while Gordon explores program-specific analyses from behavioral examples [24, 25]. LLM-based systems execute analysis pseudocode or filter bug reports [27, 46]. IRIS infers taint specifications for repository-wide analysis [36]; MoCQ and CQLLM generate vulnerability patterns or CodeQL queries [35, 60]. QLCoder synthesizes CodeQL queries through tool use and iterative feedback [59]. KNighter synthesizes and refines Clang Static Analyzer checkers from Linux kernel patches [67]. These systems target specific analyzers or domains, motivating a common evaluation of 3

(a) Construction pipeline Vulnerability source

1 collection

2

Link CVEs to fixes Recover both revisions Vulnerable and fixed pair

Checker-synthesis

Executable environment reconstruction

Independent verifier

Human-guided

Quality control and

3 task construction

4 construction

5 rollout collection

Pin tools and analyzers Reproduce both builds

Bundle patch and code Add build/scan steps

Rebuild in isolation Check report changes

Review failed attempts Revise and rerun

Executable revision pair

Agent-visible task package

Frozen held-out verifier

First pass or exclude after 3

Verified checkers and traces

CVE-2017-7866

pngdec.c / decode_zbuf()

(b) Real task: FFmpeg PNG buffer overflow Vulnerable revision

C language

Stages 1–3

av_bprint_get_buffer(bp, 1, &buf, &buf_size); if (!buf_size) { ret = AVERROR(ENOMEM); goto fail; } zstream.next_out = buf; zstream.avail_out = buf_size; ret = inflate(&zstream, Z_PARTIAL_FLUSH); // ... bp->str[bp->len] = 0; N-byte buffer: full capacity may be used

Clang Static Analyzer (CSA)

Fixed revision

Stages 1–3

av_bprint_get_buffer(bp, 2, &buf, &buf_size); if (buf_size < 2) { ret = AVERROR(ENOMEM); goto fail; } zstream.next_out = buf; zstream.avail_out = buf_size - 1; ret = inflate(&zstream, Z_PARTIAL_FLUSH); // ... bp->str[bp->len] = 0; N-byte buffer: one byte kept for NUL

0 When full, the NUL write is out of bounds.

Stages 4–6

Generated CSA checker (excerpts) State = State->add<BprintCapSet>(CapSym); // ... if (!State->get<BprintCapSet>() .contains(VSym)) return; // ... C.emitReport(std::move(R));

CapSym: capacity; VSym: avail_out at inflate() Archived verifier findings

Output ≤ N − 1; the terminator fits.

300 tasks

Validate artifacts Deduplicate tasks

Checker and verification results

0

1

Vulnerable

0

Fixed

Archived pass 41 interaction steps Mainline scan: 0 reports / 954 files

300 verified checkers

167 repositories

6 split construction

5 languages

300 pilot trajectories 297 CVEs

85 CWEs

Figure 3 CheckerBench construction.

general-purpose agents developing and validating checkers across analyzer environments.

2.2

Coding Agents for Software Engineering

Coding agents combine LLMs with repository navigation, code editing, tool execution, and feedback. SWEagent and OpenHands support interactive development, while RepoGraph adds repository structure [52, 62, 68]. RepoAudit applies an agent to repository-level bug detection [26]. These systems produce patches or bug findings. Checker synthesis requires analyzer integration and validation through compiler and scan feedback on vulnerable and fixed revisions.

2.3

Benchmarks for Repository-Level Code Agents

Repository-level benchmarks cover issue resolution (SWE-bench and SWE-bench Pro), feature development (FeatureBench), and terminal tasks (Terminal-Bench) [19, 32, 43, 73]. SWE-Gym and R2E-Gym provide executable environments and verifiers for issue-resolution tasks [31, 53]. ReposVul, VulEval, and JitVul target vulnerability detection; SEC-bench evaluates proof-of-concept generation and patching [34, 61, 64, 69]. These works score task completion, patches, or vulnerability detection. CheckerBench evaluates reusable checker construction with held-out verification of builds, vulnerable–fixed diagnostics, patch localization, and false positives (Table 1).

3

The C HECKER B ENCH Benchmark

Problem formulation. Each agent-facing task xi = (pi , ℓi , αi , Ei , Qi ) combines a vulnerability-fix instance + pi = (ri , c− i , ci , ∆i ) (repository, vulnerable/fixed revisions, and patch) with its language, analyzer, reproducible version-pinned environment, and synthesis instruction. All runs share the initial skill library K; agents may edit checker artifacts but must preserve both revisions. (h,θ)

Under an active generation time budget B, policy πθ in harness h yields trajectory τi = (h,θ) (o0 , a0 , . . . , aNi −1 , oNi ) and candidate checker qi . Here Ni counts interactions, ak is an action, and ok+1 (h,θ) (h,θ) (h,θ) its resulting observation [8]. A released record is di = (Ei , Qi , Vi , τi , qi ), with executable verifier Vi hidden during synthesis. The environment, instruction, and verifier are fixed across model–harness configurations; trajectories and checkers vary by rollout.

4

(a) Programming languages

30 (10%)

30 (10%)

(c) CWE distribution Linux

30 (10%)

51 (17.0%)

(b) Repositories

C/C++

159 (53.0%)

Java Python JavaScript

177 (59.0%)

Go

107 (35.7%)

4 (1.3%) 4 (1.3%) 4 (1.3%) 4 (1.3%)

FFmpeg ImageMagick JerryScript QEMU Others

227 (72.3%)

26 (8.3%) 21 (6.7%) 15 (4.8%) 13 (4.1%) 12 (3.8%)

CWE-787: Out-of-bounds write CWE-125: Out-of-bounds read CWE-22: Path traversal CWE-476: NULL pointer dereference CWE-79: Cross-site scripting Others

Figure 4 CheckerBench distributions by (a) language, (b) repository, and (c) CWE.

3.1

Benchmark Construction

Figure 3 summarizes the six-stage pipeline: recover executable vulnerability-fix pairs, package synthesis tasks, freeze independent verifiers, and curate verified checker–trajectory pairs. All candidates start from the same harness-component baseline and shared skill library. Stage I: Vulnerability Source Collection. We link public CVE records from the National Vulnerability Database1 to fixing commits in open-source C/C++, Java, Python, JavaScript, and Go repositories. Each candidate must have a retrievable vulnerable–fixed revision pair and a patch that exposes a reusable staticanalysis property. We exclude unresolved provenance and incompatible redistribution terms, and retain CVE/CWE mappings for auditing and stratified sampling. Stage II: Executable Environment Reconstruction. We reconstruct both revisions in isolated workspaces with pinned toolchains and analyzers: Clang Static Analyzer (CSA) for C/C++ and CodeQL for the other four languages. Revision-compatible build snapshots and compilation databases recover the analyzable scope of CSA tasks; CodeQL tasks use pinned analysis environments. Each pair includes affected functions and reproducible checker compilation and scan commands. We reject pairs that cannot be reconstructed or do not support stable differential analysis. Stage III: Checker-Synthesis Task Construction. Each task exposes the patch, pre- and post-fix functions, build metadata, compile/scan commands, and a minimal checker scaffold. Random workspace identifiers and omitted CVE/CWE labels reduce provenance shortcuts. The shared library K supplies harness-specific workflow skills and read-only analyzer documentation, example checker implementations, and semantic utilities; it excludes task-specific checker solutions. The agent must infer the defect mechanism, implement a reusable checker, and refine it through compilation and analysis feedback, producing checker artifacts and an interaction trace. Stage IV: Independent Verifier Construction. We freeze a held-out verifier before rollout collection. It rebuilds each submitted checker from source in a clean workspace and checks vulnerable–fixed diagnostic contrast and patch relevance under backend-specific rules. CSA requires a disappearing report in a patchmodified function. Passing additionally requires process and semantic quality, mainline false-positive control, and mechanism and anti-hardcoding checks (Appendix F). Stage V: Human-Guided Rollout Collection and Sampling. Calibration uses fixed model, harness, tool, and budget settings. After a failed attempt, a human diagnoses the verification evidence and revises prompt guidance for a fresh rollout; the task, verifier, and thresholds stay fixed. Each task permits one initial attempt and at most two rerolls, stopping at the first pass and excluding tasks with no pass after three attempts. We retain all attempts and interventions for audit. Checker–trace pairs enter sampling only if the checker passes the frozen verifier and the trace contains at least 40 interaction steps (agent decisions). Eligible pairs are sampled jointly over repository, CWE, and horizon. The 40-step threshold is a curation rule; these pilot interventions are separate from autonomous main evaluation. 1 https://nvd.nist.gov/vuln

5

(a) CheckerLab

(b) Archived pilot trajectory TorchServe

Visible during synthesis

Task package 50-minute budget Patch

Versioned code

Analyzer

Skills

Coding agent Harnesses Models

Claude Code

OpenCode

Hermes Agent

Memory

Planner

Tools

Context

Inspect

Edit Build Diagnose Compiler & Analyzer feedback

Long-Horizon trace

Freeze checker

Post-submission only

Independent verifier Build Contrast Mainline FP scan Gate Process CSA ≥80/100 CodeQL

Java / CodeQL

CVE-2023-48299

Task: Detect Zip Slip during archive extraction

Judge model Patch relevance Scope Review up to 10 scan alerts Semantic FP rate ≥7/10 ≤25% ≥8/10

Flag archive-entry paths that escape the extraction directory.

Agent action

Observed feedback

1 Inspect Read patch and trace archive names to Files.newOutputStream(...).

Patch evidence Fix adds canonical paths and a containment check before write

2 Implement & run Reuse ZipSlipFlow and implement and run vulnerability.ql.

Initial validation Compile: OK; Vulnerable: 1 Fixed: 0; ZIP taint path found

3 Diagnose coverage Probe missing TAR flows and inspect extracted calls and types.

Scope limitation Commons types unresolved; only JDK ZIP flow is observable

4 Test the mechanism Remove the sanitizer barrier in a control query and scan fixed code.

Scratch control (barrier off) Fixed: 1 in scratch query only Final query keeps 1 → 0 contrast

5 Finalize Keep query and ZIP-only scope, recompile and rescan both revisions.

Stable final artifact Compile: OK; Vulnerable: 1 Fixed: 0; query and trace frozen

Archived pilot: 5 selected stages, 56 assistant turns, 71 tool calls

Post-submission: Archived verifier passed

Figure 5 CheckerLab workflow and five selected stages from an archived TorchServe pilot.

Stage VI: Quality Control, Deduplication, and Splits. Automated checks establish artifact integrity, reproducible builds and scans, verifier isolation, and package completeness; manual review confirms defect semantics and patch-localized diagnostics. We remove duplicate revision pairs and keep tasks connected by repository, CVE, shared revisions, or similar patches in one split. Task-specific reference checkers, calibration traces, and verifier details remain hidden from evaluation agents. Appendix C details these checks and split safeguards.

3.2

Benchmark Statistics

Projects and tasks. CheckerBench contains 300 tasks from 167 open-source repositories, including Linux [38] (107 tasks), FFmpeg [23] (4), and ImageMagick [29] (4). Tasks cover C/C++ (159), Java (30), Python (51), JavaScript (30), and Go (30). Static analysis environments are CSA (159) and CodeQL (141). Trajectories and interaction horizons. We release 300 accepted calibration trajectories, one per task. Calibration includes human intervention to diagnose failures and revise prompts. All trajectories pass executable verification and the construction-time semantic-score threshold. They are separate from independent main-experiment repeats. Interaction steps reflect workload under the calibration configuration [57]. Appendices D.4 and D.5 report trajectory statistics, including the assistant-turn and tool-call distributions of the admitted pilot trajectories (Figure 11). Defect and task coverage. CheckerBench covers 85 CWEs. The most frequent are CWE-787 (out-ofbounds write), CWE-125 (out-of-bounds read), and CWE-22 (path traversal), as shown in Figure 4(c). Each task also records repository scale, integration scope, verification evidence, and calibration horizon. Appendix D provides complete benchmark statistics.

4

C HECKER L AB: Evaluation Framework

CheckerLab separates interactive checker synthesis from independent evaluation of the frozen checker and its trajectory [41] (Figure 5).

6

Table 2 Performance of seven models across three agent harnesses on CheckerBench. Bold and underlined values

mark the best and second-best models per metric within each harness. Model-row subscripts show numerical changes relative to GPT-5.6 in the same harness: ↑ increases and ↓ decreases (percentage points for percentage metrics). Header arrows indicate the preferred direction. Model-row metrics are means over three independent repeats. Overall Average summarizes the 21 model–harness configurations. Semantic ↑

Avg. Turns ↓

GPT-5.6 31.67 98.33 85.33 Claude Opus 4.8 45.33↑13.66 81.67↓16.66 81.00↓4.33 Qwen3.8-Max 37.33↑5.66 69.33↓29.00 69.00↓16.33 DeepSeek-V4-Pro 39.67↑8.00 92.67↓5.66 90.67↑5.34 GLM-5.2 32.33↑0.66 77.33↓21.00 75.33↓10.00 Kimi-K3 38.67↑7.00 83.33↓15.00 81.67↓3.66 HY4-Preview 33.67↑2.00 72.67↓25.66 71.33↓14.00 Average 36.95 82.19 79.19

OpenCode 0.00 7 80.22 13.79↑13.79 174 86.42↑6.20 5.77↑5.77 52 90.15↑9.93 2.56↑2.56 78 84.89↑4.67 10.34↑10.34 58 86.73↑6.51 1.59↑1.59 63 88.48↑8.26 12.38↑12.38 105 86.90↑6.68 6.63 76.71 86.26

7.68 8.32↑0.64 8.46↑0.78 8.12↑0.44 8.11↑0.43 8.30↑0.62 8.32↑0.64 8.19

29.19 60.13 44.78↑15.59 64.35↑4.22 35.96↑6.77 47.54↓12.59 51.09↑21.90 65.28↑5.15 41.21↑12.02 61.02↑0.89 32.42↑3.23 43.53↓16.60 41.28↑12.09 50.15↓9.98 39.42 56.00

GPT-5.6 34.00 93.67 83.00 Claude Opus 4.8 35.00↑1.00 63.67↓30.00 62.33↓20.67 Qwen3.8-Max 34.67↑0.67 67.33↓26.34 67.00↓16.00 DeepSeek-V4-Pro 38.33↑4.33 91.33↓2.34 89.67↑6.67 GLM-5.2 33.00↓1.00 76.33↓17.34 74.67↓8.33 Kimi-K3 38.00↑4.00 83.67↓10.00 82.33↓0.67 HY4-Preview 23.67↓10.33 49.33↓44.34 47.33↓35.67 Average 33.81 75.05 72.33

Claude Code 2.44 41 81.41 2.33↓0.11 43 86.88↑5.47 4.55↑2.11 44 89.58↑8.17 1.72↓0.72 58 85.25↑3.84 0.00↓2.44 18 89.67↑8.26 2.74↑0.30 73 87.79↑6.38 0.00↓2.44 13 90.83↑9.42 1.97 41.43 87.34

7.69 8.21↑0.52 8.38↑0.69 8.27↑0.58 8.39↑0.70 8.34↑0.65 8.35↑0.66 8.23

35.71 60.58 32.60↓3.11 45.63↓14.95 33.91↓1.80 41.57↓19.01 47.17↑11.46 62.50↑1.92 38.75↑3.04 49.93↓10.65 31.81↓3.90 38.61↓21.97 50.66↑14.95 62.23↑1.65 38.66 51.58

GPT-5.6 23.00 100.00 64.00 Claude Opus 4.8 34.33↑11.33 80.33↓19.67 78.67↑14.67 Qwen3.8-Max 25.00↑2.00 44.00↓56.00 42.33↓21.67 DeepSeek-V4-Pro 32.33↑9.33 94.00↓6.00 90.67↑26.67 GLM-5.2 22.33↓0.67 79.67↓20.33 78.00↑14.00 Kimi-K3 24.33↑1.33 87.00↓13.00 79.67↑15.67 HY4-Preview 21.67↓1.33 57.67↓42.33 52.67↓11.33 Average 26.14 77.52 69.43

Hermes Agent 0.00 6 79.85 5.45↑5.45 55 85.58↑5.73 11.11↑11.11 27 90.92↑11.07 0.00=0.00 12 85.29↑5.44 0.00=0.00 3 82.87↑3.02 4.76↑4.76 63 82.62↑2.77 0.00=0.00 3 88.42↑8.57 3.05 24.14 85.08

7.50 8.32↑0.82 8.53↑1.03 8.03↑0.53 7.81↑0.31 8.08↑0.58 8.44↑0.94 8.10

25.94 46.91 44.84↑18.90 61.89↑14.98 40.55↑14.61 51.95↑5.04 39.82↑13.88 62.37↑15.46 42.85↑16.91 63.22↑16.31 32.73↑6.79 42.12↓4.79 49.69↑23.75 62.06↑15.15 39.49 55.79

8.17

39.19

Overall Average

Pass@1 (%) ↑

32.30

Build (%) ↑

78.25

Diff. SR (%) ↑

FPpass (%) ↓

Process ↑

Model / System

73.65

3.88

#WN

47.43 86.23

Avg. Tool Calls ↓

54.46

Evaluation workflow. Each run receives an isolated workspace with the task instruction, fixing patch, code context, analyzer tools, and enabled skills; the reference checker and held-out verifier remain hidden. Agents use native harness capabilities and compilation and analysis feedback within a common 50-minute generation budget. At termination, we freeze the submitted CSA checker or CodeQL query source and record the trajectory and tool invocations. The verifier then rebuilds the frozen checker from source in a clean environment and checks vulnerable–fixed diagnostic contrast and patch relevance under backend-specific rules (Section 3.1). A broader mainline scan assesses false positives without checker edits. Claude Opus 5 [7] judges process quality, vulnerability semantics, and sampled mainline alerts in separate stages. A run passes only after normal generation completion and satisfaction of all gates in Equation 5; Appendix F gives the full rubrics. Metrics. Our primary metric, Pass@1, is the percentage of tasks whose checker passes the full evaluation after one independent synthesis run. For each of the 21 harness–model pairs, we evaluate the same 300 tasks in three independent repeats and average the three rates equally. Generation failures, refusals, and candidatecode errors count as failures. Build and Diff. SR are the percentages of all tasks passing independent compilation and differential verification, respectively. Process and Semantic are mean P valid scores P on 100point and 10-point scales. Within each repeat, the false-positive percentage FPpass = 100 FP/ (TP+FP) pools fully adjudicated recorded mainline alerts from passed checkers. The acceptance gate uses each checker’s sampled mainline alerts, with a default FP cutoff of 25% (Appendix F). We compute each metric per repeat and average equally (Appendix H). In Table 2, #WN is the arithmetic mean of the counts of fullyP adjudicated recorded mainline warnings from passed checkers across the three repeats. Each repeat’s count is (TP+FP). 7

(b) CWE

23.3 65.4 51.1 50.0 46.7

24.4 15.9 57.8 28.2 41.7 16.7 27.8 40.0

Qwen3.8-Max

11.3 62.7 48.9 54.4 53.3

21.8 22.2 51.1 7.7 61.1 16.7 11.1 36.7

DeepSeek-V4-Pro

24.5 56.9 53.3 42.2 45.6

28.2 22.2 48.9 25.6 38.9 13.9 33.3 36.7

GLM-5.2

14.0 58.8 46.7 41.1 30.0

16.7 17.5 46.7 12.8 36.1 13.9 25.0 40.0

Kimi-K3

20.8 57.5 37.8 47.8 43.3

20.5 27.0 48.9 23.1 27.8 13.9 22.2 46.7

HY4-Preview

7.3 51.0 44.4 47.8 45.6

2.6 20.6 31.1 5.1 52.8 8.3 8.3 46.7

pt cri

aS

100

Jav

0 50 Pass@1 (%)

Jav

(a) (b)

DeepSeek Claude Kimi DeepSeek Kimi Qwen Qwen HY4 Claude GLM Claude GPT DeepSeek Kimi GPT GLM HY4

40 30

GPT

20 10

CW E-7 CW 87 E-1 2 CW 5 E-2 CW 2 E-4 7 CW 6 E-7 CW 9 E-3 CW 62 E-4 CW 16 E-4 00

Claude Opus 4.8

a

17.9 19.0 46.7 17.9 30.6 13.9 25.0 40.0

Go

15.1 55.6 37.8 41.1 42.2

C/ C+ + Py tho n

GPT-5.6

(c) Turns 50

Pass@1 (%)

(a) Language

20

Qwen GLM HY4

OpenCode Claude Code Hermes Agent Model GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview

35 50 Avg. Turns (passed only)

Figure 6 Pass@1 by (a) language and (b) CWE, averaged equally across three harnesses. (c) Pass@1 versus average

turns on successful runs; colors denote harnesses and shapes denote models.

The Average and Overall Average rows report mean counts across seven models and all 21 configurations, respectively. Warning counts are descriptive, with no ranking or change subscripts. Appendix E.6 details alert selection, recorded warning volume, and sensitivity to precision weighting and all-recorded-alert gates. For interaction efficiency, we average turns, tool calls, and explicit skill calls over successful trajectories within each repeat, including subagent activity. We then average these means equally across repeats and report success counts (Appendix E).

5

Experiments

5.1

Experimental Setup

Models and agent harnesses. We evaluate 7 models using 3 agent harnesses, including OpenCode [3], Claude Code [5], and Hermes Agent [48]. The models are GPT-5.6-Sol [49], Claude Opus 4.8 [6], Qwen3.8-Max [2], DeepSeek-V4-Pro-0813 [18], GLM-5.2 [70], Kimi-K3 [33], and HY4-Preview [56]. Task execution. Each CheckerLab run uses an isolated E2B sandbox and workspace containing the task instruction, fixing patch, relevant code context, and analyzer tools. Agents can inspect code, implement checkers, and use compilation and analysis feedback within a 50-minute active generation budget. The submitted source and execution records are frozen at termination for independent verification. Environment and execution details are provided in Appendix E. Evaluation and judge. Each frozen checker is evaluated using backend-specific executable checks and scoring rubrics. We use Claude Opus 5 [7] as the LLM judge for false-positive triage, process quality, and vulnerability semantics. Human validation gives a pooled Cohen’s κ = 0.85 for judge–human agreement. Appendix E.9 reports the sampling, labeling, adjudication, and stage-specific agreement. The scoring rubrics are detailed in Appendix F. For the 21 model–harness configurations, we report mean Pass@1 over three independent repeats on the same 300 tasks, with additional diagnostic metrics defined in Section 4. Efficiency metrics are averaged over successful trajectories only.

5.2

Overall Performance

End-to-end success. Across 21 model–harness configurations, mean Pass@1 is 32.30%; even the best combination, Claude Opus 4.8 with OpenCode, reaches only 45.33% (Figure 1; Table 2). Thus, more than half of tasks remain unsuccessful even for the strongest configuration. Overall Build and Diff. SR are much higher, at 78.25% and 73.65%, respectively. The gap between Diff. SR and Pass@1 is 41.35 percentage points, much larger than the 4.60-point gap between Build and Diff. SR. For example, GPT-5.6 with Hermes Agent achieves 100.00% Build but only 64.00% Diff. SR and 23.00% Pass@1. The larger gap between differential 8

verification and full acceptance highlights the challenge of meeting the additional requirements for semantic fidelity, process quality, false-positive control, and normal completion. Model–harness dependence. OpenCode has the highest mean Pass@1 across models (36.95%), followed by Claude Code (33.81%) and Hermes Agent (26.14%), with a 10.81-percentage-point difference between the highest and lowest averages. The best harness varies by model: GPT-5.6 and GLM-5.2 attain their highest Pass@1 with Claude Code. The leading model also changes across harnesses. Claude Opus 4.8 ranks first with OpenCode and Hermes Agent, whereas DeepSeek-V4-Pro leads with Claude Code (38.33%, compared with 35.00% for Claude Opus 4.8). Kimi-K3 further illustrates the sensitivity to harness choice, declining from 38.67% with OpenCode to 24.33% with Hermes Agent, a 14.34-point gap. These shifts show that harness choice changes both success rates and model rankings, making the complete model–harness configuration the appropriate unit for comparison and selection. Diagnostic quality and false positives. Qwen3.8-Max with Hermes Agent has the highest Process (90.92/100) and Semantic (8.53/10) scores across all configurations, yet reaches only 25.00% Pass@1 and 44.00% Build. OpenCode’s higher mean Pass@1 is also accompanied by a higher FPpass than Claude Code (6.63% vs. 1.97%). Within OpenCode, Claude Opus 4.8 has 13.79% FPpass with #WN= 174, compared with 0.00% and #WN= 7 for GPT-5.6. Both metrics summarize recorded, fully adjudicated alerts from passed checkers. No evaluated configuration leads simultaneously in synthesis success, process and semantic quality, and observed diagnostic precision. Checker reliability should therefore be assessed across these dimensions, with warning volume providing context for false-positive comparisons. Language and defect-family variation. Figure 6(a,b) averages each model’s results equally across the three harnesses. Every model has its lowest language-specific Pass@1 in C/C++, where even the leading model, DeepSeek-V4-Pro, reaches only 24.53%. This stratum contains 159 of the 300 tasks and therefore strongly influences aggregate success. The leaders change in other languages: Claude Opus 4.8 leads Python (65.36%), DeepSeek-V4-Pro leads Go (53.33%), and Qwen3.8-Max leads Java (54.44%) and JavaScript (53.33%). Qwen3.8-Max also outperforms DeepSeek-V4-Pro on CWE-79, with the ordering reversed on CWE-787. C/C++ uses CSA while the other languages use CodeQL, and CWE groups differ in repository and language composition. These results identify C/C++ tasks under CSA as a shared challenge and reveal complementary model strengths across the evaluated task groups, supporting performance breakdowns by language and defect family alongside aggregate Pass@1. Success and interaction efficiency. Kimi-K3 uses the fewest average tool calls on successful runs in all three harnesses. With Claude Code, it achieves nearly the same Pass@1 as DeepSeek-V4-Pro (38.00% vs. 38.33%), while using fewer turns (31.81 vs. 47.17) and tool calls (38.61 vs. 62.50), a 38.2% reduction in tool calls (Figure 6(c)). Kimi-K3 thus combines a competitive success rate with efficient tool use on successful runs, highlighting the value of considering both Pass@1 and interaction efficiency when selecting a model–harness configuration.

5.3

Comparison with Baselines

We compare our skill-guided checker-synthesis method, impleTable 3 Checker validity on KNighter’s 61 mented with OpenCode, against KNighter on the 61 Linux- Linux-kernel tasks. kernel tasks from KNighter’s original benchmark. All methods use DeepSeek-V4-Pro. Our method achieves an 80.3% Method Valid Rate valid-checker rate, compared with 67.2% for KNighter (TaOurs 49/61 80.3% ble 3). Removing the skill library lowers our method’s rate KNighter 41/61 67.2% to 29.5%. KNighter organizes synthesis through stage-specific Ours w/o skills 18/61 29.5% prompts and a checker template, while our method integrates the shared reusable guidance as skills. Its lead on KNighter’s original tasks suggests that this skill-guided workflow transfers well to an independently defined task set. Details in Appendix H.1.

9

GPT-5.6 /

OpenCode

GPT-5.6 /

median = 58

40%

20%

median = 53

20%

0% 50

100

Qwen3.8-Max /

150

200

median = 45

20%

50

100

150

Qwen3.8-Max /

200

median = 37.5

40%

100

GLM-5.2 /

150

200

50

100

150

GLM-5.2 /

median = 58

20%

100

150

200

median = 49

median = 49

100

150

100

150

median = 61

40%

100

150

100

OpenCode

20%

100

150

200

200

median = 58

50

100

150

Claude Code

50

100

Kimi-K3 /

150

median = 59

40%

100

150

200

Hermes Agent median = 60

40%

200

0

Claude Code median = 38

40%

50

100

Kimi-K3 /

150

200

Hermes Agent median = 37

40%

20%

0% 0

HY4-Preview /

50

DeepSeek-V4-Pro /

0% 0

200

0

20%

0% 0

median = 65

Claude Code

20%

50

100

150

200

0

50

100

150

200

Hermes Agent median = 59

40%

20%

0% 50

200

median = 41.5

20%

0%

150

40%

OpenCode

40%

HY4-Preview /

median = 49

40%

0

150

Kimi-K3 /

200

100

DeepSeek-V4-Pro /

0% 50

50

0% 50

Hermes Agent

0% 0

20%

0

Claude Opus 4.8 / 40%

20%

OpenCode

20%

0

HY4-Preview /

200

median = 59

Hermes Agent

GLM-5.2 /

200

150

40%

0% 50

100

DeepSeek-V4-Pro /

200

20%

0

50

0% 50

median = 41

0% 0

20%

0

0% 50

200

Claude Code

40%

20%

Hermes Agent

40%

200

20%

0

150

Qwen3.8-Max /

Claude Code

40%

0%

100

0%

OpenCode

40%

50

20%

0

Claude Opus 4.8 /

median = 64.5

0% 0

0% 50

OpenCode

20%

Claude Code

20%

0

Claude Opus 4.8 / 40%

0% 0

0%

median = 46

20%

OpenCode

40%

Hermes Agent

GPT-5.6 / 40%

0% 0

Share of trajectories

Claude Code

40%

0% 0

50

100

150

200

0

50

100

150

200

Tool calls per trajectory Figure 7 Tool-call distributions over successful trajectories for the 21 model–harness combinations. Histogram bins

have width 10; dashed lines indicate medians. OpenCode

Claude Code

Hermes Agent

50 GPT-5.6

Pass@1 (%)

40

Claude Opus 4.8 Qwen3.8-Max

30

DeepSeek-V4-Pro 20

GLM-5.2 Kimi-K3

10

0

HY4-Preview

10

20

30

40

50

Active time budget (min)

10

20

30

40

Active time budget (min)

50

10

20

30

40

50

Active time budget (min)

Figure 8 Retrospective Pass@1 versus active generation time for three agent harnesses.

5.4

Ablation Study

We compare DeepSeek-V4-Pro on the same 300 tasks, using its original OpenCode run as Full. As summarized by Table 4, Pass@1 is 39.67% for Full, 8.33% without skills and strategy references, 13.33% with execution feedback hidden, and 0.33% for one-shot generation without tools or a harness. These gaps persist after excluding Process and normal-termination requirements: artifact pass is 48.67%, 18.67%, 20.33%, and 1.33%, respectively. One-shot performance also reflects invalid or incomplete submissions in 237/300 tasks. Differences between historical and new run conditions prevent attributing the gaps solely to the ablated components. See details in Appendix H.2.

5.5

Time-Budget Sensitivity

We assess time-budget sensitivity by retrospectively applying 10, 20, 30, 40, and 50-minute active generation cutoffs to the existing trajectories across all 21 model–harness configurations. Only final artifacts completed by each cutoff contribute, with all 300 tasks retained per configuration. Figure 8 shows that aggregate Pass@1 rises from 7.24% at 10 minutes to 26.05% at 30 minutes and 32.30% at 50 minutes, with diminishing gains: the 40-minute cutoff captures 93.02% of the successes observed by 50 minutes, and the final 10 minutes add 2.25 percentage points. These late gains are largest for Qwen3.8-Max and HY4-Preview. The curves 10

characterize completion timing under the original 50-minute protocol; performance under separately imposed shorter deadlines remains untested. Full metrics and timing rules appear in Appendix H.3.

5.6

FP-Threshold Sensitivity

To quantify the influence of the FP threshold on checker acceptance, we evaluate thresholds of 10%, 20%, 25%, 30%, 40%, and 50% across all 21 model–harness configurations (Figure 9). For checkers with adjudicated alerts, the additional acceptance criterion requires 100 FP/(TP + FP) to be no greater than the specified threshold. Adjudication labels and all other eligibility criteria are held fixed, with 300 tasks retained in the denominator for each configuration. The 25% threshold reproduces the acceptance rates for all 21 configurations in Table 2. The aggregate acceptance rate increases from 32.05% at 10% to 32.30% at 25% and 32.83% at 50%, yielding a net increase of 0.78 percentage points over the evaluated range. Over the same range, the largest increase for an individual configuration is 1.67 percentage points, observed for GLM-5.2 with Claude Code. These results suggest that aggregate acceptance is relatively stable across the evaluated FP thresholds within the recorded cohort.

5.7

Analysis of Checker Synthesis Failures

We analyze 4,360 selected failed runs across seven models and three harnesses using tool traces, independent verification logs, and existing judge records. The leading failure categories are budget exhaustion (25.32%), inadequate synthesis process (22.96%), and inadequate semantic modeling (21.15%). These records expose gaps between analysis plans and implementations: for example, a planned path-sensitive analysis becomes AST matching that omits locking and control-flow conditions. Excess false positives account for 15.96% of failures, underscoring the need to evaluate precision beyond the original vulnerable–fixed pair. The remaining failures involve differential detection or localization (8.17%), patch-specific overfitting or anchoring (3.56%), missing or invalid artifacts (1.83%), and compilation or registration (1.05%). Key challenges for checker synthesis include timely completion, faithful implementation of analysis plans, and precise semantic modeling.

5.8

Data Contamination and Prior Exposure

CVE publication postdates release for 3/300 tasks with Claude Opus 4.8, one with GLM-5.2, and none with other models (Figure 10(a,b)); code or patches may have appeared earlier. On 100 language-stratified tasks, we compare original inputs (O), paraphrases retaining identifiers (P), and versions of P with meaningpreserving identifier substitutions (D) [21]. Figure 10(c) shows P–D Pass@1 gaps of +3.00 and +2.00 percentage points for Claude Opus 4.8 and DeepSeek-V4-Pro. Exact CVE recall falls from 10.00% to 1.00% and from 8.00% to 1.00%, respectively. The 7–9-point recall decline accompanies only a 2–3-point Pass@1 drop, suggesting that recognizing familiar vulnerabilities yields only limited gains under this intervention (Appendix C.3).

6

Conclusion

We introduced CheckerBench, an executable benchmark for long-horizon static-analysis checker synthesis with 300 tasks across five programming-language ecosystems. CheckerLab independently rebuilds subTable 4 Performance comparison of DeepSeek-V4-Pro ablations on 300 CheckerBench tasks. Bold and underlined

values mark the best and second-best configurations per metric. Subscripts show numerical changes relative to Full: ↑ increases and ↓ decreases (percentage points for percentage metrics). Header arrows indicate the preferred direction. N/A denotes no alerts.

Configuration Full w/o Skills w/o Execution Feedback w/o Harness

Skill Exec. lib. feedback

Pass@1 (%) ↑

Build (%) ↑

Diff. SR (%) ↑

FPpass (%) ↓

Process ↑

Semantic ↑

Avg. Turns ↓

Avg. Tool Calls ↓

8.12 7.39↓0.73

57.63 59.56↑1.93

72.73 80.53↑7.80

✓ ✗

✓ ✓

39.67 92.67 90.67 8.33↓31.34 79.33↓13.34 52.67↓38.00

2.56 6.25↑3.69

84.89 65.31↓19.58

✓

✗

13.33↓26.34

67.33↓25.34

41.67↓49.00

25.00↑22.44

79.93↓4.96

8.00↓0.12

53.58↓4.05

72.16↓0.57

✗

✗

0.33↓39.34

3.33↓89.34

3.33↓87.34

N/A

73.64↓11.25

7.45↓0.67

1.00↓56.63

0.00↓72.73

11

GPT-5.6 / 33.0%

OpenCode

GPT-5.6 /

at 25% = 31.67%

Claude Code

35.0%

Hermes Agent

GPT-5.6 / 24.0%

at 25% = 34.00%

Claude Opus 4.8 /

OpenCode

at 25% = 23.00%

Claude Opus 4.8 /

at 25% = 45.33%

36.0%

Claude Code at 25% = 35.00%

Claude Opus 4.8 / 35.5%

Hermes Agent at 25% = 34.33%

46.0% 23.5% 32.0%

35.0%

34.0%

35.0% 45.0%

23.0%

31.0%

33.0% 10

20

25

30

40

Qwen3.8-Max /

50

22.5% 10

OpenCode

20

25

30

40

Qwen3.8-Max /

at 25% = 37.33%

36.0%

50

10

Claude Code

20

25

30

40

50

34.0% 10

Hermes Agent

Qwen3.8-Max / 26.0%

at 25% = 34.67%

34.5%

44.0% 20

25

30

40

DeepSeek-V4-Pro /

at 25% = 25.00%

41.0%

34.0%

50

10

OpenCode

20

25

30

40

DeepSeek-V4-Pro /

at 25% = 39.67%

40.0%

50

10

Claude Code at 25% = 38.33%

20

25

30

40

DeepSeek-V4-Pro / 34.0%

50

Hermes Agent at 25% = 32.33%

38.0% 35.0%

Pass@1 (%)

25.0% 37.0%

40.0%

39.0%

33.0%

34.0% 24.0% 10

20

25

30

GLM-5.2 / 33.0%

40

50

10

OpenCode

20

25

30

40

GLM-5.2 /

at 25% = 32.33%

50

39.0% 10

20

Claude Code

30

40

50

38.0% 10

20

Hermes Agent

24.0%

at 25% = 33.00%

35.0%

25

GLM-5.2 /

25

30

Kimi-K3 /

at 25% = 22.33%

40

32.0%

50

10

20

OpenCode

40.0%

25

30

Kimi-K3 /

at 25% = 38.67%

32.5%

40

50

40.0%

23.0%

39.0%

22.0%

38.0%

20

25

30

at 25% = 38.00%

40

50

Hermes Agent

Kimi-K3 / 25.0%

at 25% = 24.33%

24.5%

39.0% 34.0%

10

Claude Code

32.0%

24.0% 38.0%

33.0% 31.5% 10

20

25

30

40

50

10

20

25

30

40

50

10

HY4-Preview / 35.0%

20

25

30

40

OpenCode

50

23.5% 10

HY4-Preview / 25.0%

at 25% = 33.67%

20

25

30

40

Claude Code

50

10

at 25% = 23.67%

20

25

30

40

50

10

20

25

30

40

50

Hermes Agent

HY4-Preview / 23.0%

at 25% = 21.67%

34.0% 24.0%

22.0%

33.0% 23.0% 10

20

25

30

40

50

21.0% 10

20

25

30

40

50

10

20

25

30

40

50

FP threshold (%) Panel-specific y-axis ranges. Dashed line: 25% cutoff.

Figure 9 Pass@1 versus FP threshold for 21 model–harness pairs (300 fixed tasks each). Translucent bands highlight

the curves. Dashed lines mark the 25% cutoff; annotations show Pass@1 at this cutoff. (b) Model releases

Go (30) Java (30)

Latest CVE: 26 Jun 60

60 38

40

37 22

20 0

28

28

30

1

15

May

1

15

Jun

1

15

Jul

1

15

Aug

Pass@1 (P)

Pass@1 (D)

Δ (P−D) +3.00 pp 100

Claude Opus 4.8 · 28 May GLM-5.2 · 16 Jun GPT-5.6 · 9 Jul Kimi-K3 · 16 Jul Qwen3.8-Max · 2 Aug DeepSeek-V4-Pro · 13 Aug HY4-Preview · 28 Aug

57

0 1 2 3 4 5 6 9 01 202 202 202 202 202 202 202

≤2

(c) Prior-exposure sensitivity Pass@1 (O)

JavaScript (30)

40

ID Recall P−D: 9.00 pp

12

10.0

31

0

100

ID Recall P−D: 7.00 pp

80 8 43.0 42.0 39.0 4

20

Claude Opus 4.8

60

0

12

8.0

8

55.0 54.0 52.0

40

4

20

1.0

Score (%)

ID Recall (D)

Δ (P−D) +2.00 pp

80 60

ID Recall (P)

0

ID Recall (%)

(a) CVE publications C/C++ (159) Python (51)

1.0

Score (%)

0

DeepSeek-V4-Pro

Figure 10 Temporal coverage and prior exposure: (a) CVE publication counts; (b) model release dates; (c) Pass@1

(O/P/D; left axes, 0–100%) and exact CVE recall (P/D; right axes, 0–12%).

mitted checkers and assesses diagnostic behavior on vulnerable and fixed revisions, patch localization, and false positives. Across seven models and three agent harnesses, the best configuration achieves 45.33% Pass@1, leaving substantial room for improvement within the evaluated repositories and analyzer configurations. CheckerBench provides a reproducible testbed for agents that turn vulnerability specifications into reliable, reusable checkers.

References [1] AgentRE-Bench. AgentRE-Bench. Project website, 2026. URL https://www.agentre-bench.ai/. [2] Alibaba Cloud. Qwen3.8-Max: Model studio release log. https://www.alibabacloud.com/help/en/model-studio/ newly-released-models, 2026. Release entry dated August 2, 2026; accessed September 19, 2026. [3] Anomaly Contributors. OpenCode. https://github.com/anomalyco/opencode, 2026. Accessed August 25, 2026. [4] Anonymous. BIN-BENCH: Can LLM Agents Reason Through Long-Horizon Binary Analysis? OpenReview, 2026. URL https://openreview.net/pdf?id=KCBDPLjVrV. Anonymous ACL submission. [5] Anthropic. How Claude Code works. https://code.claude.com/docs/en/how-claude-code-works, 2026. Accessed August 2, 2026.

12

[6] Anthropic. Introducing Claude Opus 4.8. https://www.anthropic.com/news/claude-opus-4-8, 2026. Released May 28, 2026; accessed October 4, 2026. [7] Anthropic. Introducing Claude Opus 5. September 20, 2026.

https://www.anthropic.com/news/claude-opus-5, 2026.

Accessed

[8] J. Chai, G. Yin, Z. Xu, C. Yue, Y. Jia, S. Xia, X. Wang, J. Jiang, X. Li, C. Dong, H. He, and W. Lin. RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use, 2025. URL https://arxiv.org/abs/2509.06980. [9] H. Chen, Z. Hu, J. Chai, H. Yang, H. He, X. Wang, W. Lin, L. Wang, G. Yin, and Z. Zhao. ToolForge: A Data Synthesis Pipeline for Multi-Hop Search without Real-World APIs, 2025. URL https://arxiv.org/abs/ 2512.16149. [10] H. Chen, S. Zhou, Z. Hu, J. Chai, H. He, H. Yang, X. Wang, W. Lin, T. Wang, Z. Zhao, and G. Yin. ToolForge: A Data Synthesis Pipeline for Multi-Hop, Multi-Turn, and Self-Reflective Tool-Use Data. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, 2026. URL https://openreview.net/ forum?id=vFFehO66Y0. [11] T. Chen, S. Lu, S. Lu, Y. Gong, C. Yang, X. Li, M. R. H. Misu, H. Yu, N. Duan, P. Cheng, F. Yang, S. K. Lahiri, T. Xie, and L. Zhou. Automated proof generation for Rust code via self-evolution. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/ hash/b2e20d7402c9985eae4ba924c65370a8-Abstract-Conference.html. [12] Cilium eBPF Maintainers. BTF: Fixed panics during parsing of malformed input. https://github.com/cilium/ ebpf/pull/2021, 2026. Pull request 2021; merged May 27, 2026. [13] Clang Project. Clang Static Analyzer. https://clang.llvm.org/docs/ClangStaticAnalyzer.html, 2026. Accessed August 4, 2026. [14] CVE Program. CVE List V5: Official CVE records. https://github.com/CVEProject/cvelistV5, 2026. Per-record JSON snapshots archived September 15, 2026. [15] I. David and A. Gervais. CrackMeBench: Binary Reverse Engineering for Agents, 2026. URL https://arxiv. org/abs/2605.10597. [16] DeepSeek-AI. DeepSeek Harness. https://github.com/deepseek-ai/deepseek-harness, 2026. Accessed August 25, 2026. [17] DeepSeek-AI. DeepSeek-V4-Pro GA release. https://api-docs.deepseek.com/news/news260813/, 2026. DeepSeek-V4-Pro-0813 released August 13, 2026; accessed September 20, 2026. [18] DeepSeek-AI et al. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. URL https: //arxiv.org/abs/2606.19348. [19] X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V. Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?, 2025. URL https://arxiv.org/abs/2509.16941. [20] J. Ding, S. Long, C. Pu, H. Zhou, H. Gao, X. Gao, C. He, Y. Hou, F. Hu, Z. Li, W. Shi, Z. Wang, D. Zan, C. Zhang, X. Zhang, Q. Chen, X. Cheng, B. Deng, Q. Gu, K. Hua, J. Lin, P. Liu, M. Li, X. Pan, Z. Peng, Y. Qin, Y. Shan, Z. Tan, W. Xie, Z. Wang, Y. Yuan, J. Zhang, E. Zhao, Y. Zhao, H. Zhu, L. Zhu, C. Zou, M. Ding, J. Jiao, J. Liu, M. Liu, Q. Liu, C. Tao, J. Yang, T. Yang, Z. Zhang, X. Chen, W. Huang, and G. Zhang. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents, 2025. URL https://arxiv.org/abs/2512.12730. [21] Y. Fang, T. Sun, Y. Shi, M. Wang, and X. Gu. LastingBench: Defend benchmarks against knowledge leakage. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 18304–18317. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-emnlp.993. URL https://aclanthology.org/ 2025.findings-emnlp.993/. [22] Y. Feng, J. Sun, Z. Yang, J. Ai, C. Li, Z. Li, F. Zhang, K. He, R. Ma, J. Lin, J. Sun, Y. Xiao, S. Zhou, W. Wu, Y. Liu, P. Liu, Y. Qiao, S. Zhang, and K. Zhang. LongCLI-Bench: A Preliminary Benchmark and Study for Longhorizon Agentic Programming in Command-Line Interfaces, 2026. URL https://arxiv.org/abs/2602.14337.

13

[23] FFmpeg Project. FFmpeg Source Repository. https://github.com/FFmpeg/FFmpeg, 2026. Accessed August 28, 2026. [24] P. Garg and S. H. Sengamedu. Synthesizing code quality rules from examples. Proceedings of the ACM on Programming Languages, 6(OOPSLA2):1757–1787, 2022. doi: 10.1145/3563350. URL https://doi.org/10.1145/ 3563350. [25] C. S. Gordon. Synthesizing program-specific static analyses, 2018. URL https://arxiv.org/abs/1810.06600. [26] J. Guo, C. Wang, X. Xu, Z. Su, and X. Zhang. RepoAudit: An Autonomous LLM-Agent for RepositoryLevel Code Auditing. In International Conference on Machine Learning, pages 21083–21100, 2025. URL https: //proceedings.mlr.press/v267/guo25n.html. [27] Y. Hao, W. Chen, Z. Zhou, and W. Cui. E&V: Prompting large language models to perform static analysis by pseudo-code execution and verification, 2023. URL https://arxiv.org/abs/2312.08477. [28] H. He, Y. Luo, C. Wan, T. Su, H. Sun, and G. Pu. Automated detection of atomicity violations in large-scale systems, 2025. URL https://arxiv.org/abs/2504.00521. [29] ImageMagick Project. ImageMagick Source Repository. https://github.com/ImageMagick/ImageMagick, 2026. Accessed August 28, 2026. [30] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/ file/94074dd5a072d28ff75a76dabed43767-Paper-Conference.pdf. [31] N. Jain, J. Singh, M. Shetty, T. Zhang, L. Zheng, K. Sen, and I. Stoica. R2E-Gym: Procedural environment generation and hybrid verifiers for scaling open-weights SWE agents. In Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=7evvwwdo3z. [32] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. [33] Kimi Team et al. Kimi K3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653. [34] H. Lee, Z. Zhang, H. Lu, and L. Zhang. SEC-bench: Automated Benchmarking of LLM Agents on RealWorld Software Security Tasks. In Advances in Neural Information Processing Systems, 2025. URL https: //arxiv.org/abs/2506.11791. [35] P. Li, S. Yao, J. S. Korich, C. Luo, J. Yu, Y. Cao, and J. Yang. Neuro-symbolic Static Analysis with LLMgenerated Vulnerability Patterns, 2025. URL https://arxiv.org/abs/2504.16057v5. Version 5, revised April 2026. [36] Z. Li, S. Dutta, and M. Naik. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/ paper/2025/hash/582d4e27fa24168f3af1f4582655034b-Abstract-Conference.html. [37] Z. Li, Z. Li, Y. Shi, R. Wang, J. Yang, Z. Liu, X. Wu, A. Li, Y. Yu, N. Liu, L. Sun, H. Mi, and L. Liang. LongHorizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense RewardBased Grading, 2026. URL https://arxiv.org/abs/2607.08964. [38] Linux Kernel Community. Linux Kernel Source Repository. https://github.com/torvalds/linux, 2026. Accessed August 28, 2026. [39] T. Liu, C. Xu, and J. McAuley. RepoBench: Benchmarking repository-level code auto-completion systems. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/ paper/2024/hash/d191ba4c8923ed8fd8935b7c98658b5f-Abstract-Conference.html. [40] LLVM Project. Clang Static Analyzer Checker Implementations. https://github.com/llvm/llvm-project/tree/ main/clang/lib/StaticAnalyzer/Checkers, 2026. Accessed August 25, 2026. [41] H. He, C. Yue, C. Dong, M. Tian, H. Chen, Z. Liu, J. Chai, X. Wang, Y. Zhang, Q. Liao, G. Yin, W. Lin, C. Wan, H. Sun, and T. Su. LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services. In

14

Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. Association for Computing Machinery, 2026. doi: 10.1145/3770855.3817466. URL https://doi.org/10.1145/3770855.3817466. [42] H. Luo, H. Zhang, X. Zhang, H. Wang, Z. Qin, W. Lu, G. Ma, H. He, Y. Xie, Q. Zhou, Z. Hu, H. Mi, Y. Wang, N. Tan, H. Chen, Y. R. Fung, C. Yuan, and L. Shen. UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios, 2025. URL https://arxiv.org/abs/2509.21766. [43] M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=a7Qa4CcHak. [44] D. Miessler and PAI Contributors. Personal AI Infrastructure. https://github.com/danielmiessler/Personal_ AI_Infrastructure, 2026. Accessed August 25, 2026. [45] MITRE Corporation. Common Weakness Enumeration (CWE). https://cwe.mitre.org/, 2026. Accessed August 25, 2026. [46] M. M. Mohajer, R. Aleithan, N. S. Harzevili, M. Wei, A. B. Belle, H. V. Pham, and S. Wang. SkipAnalyzer: A tool for static code analysis with large language models, 2023. URL https://arxiv.org/abs/2310.18532. [47] Moonshot AI. Kimi K3: Model release in Kimi Code. https://www.kimi.com/code/docs/en/kimi-code/whats-new. html, 2026. Release entry dated July 16, 2026; accessed September 19, 2026. [48] Nous Research. Hermes Agent. https://github.com/NousResearch/hermes-agent, 2026. Accessed August 2, 2026. [49] OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition. https://openai.com/index/gpt-5-6/, 2026. Accessed September 14, 2026. [50] OpenClaw Contributors. OpenClaw. https://github.com/openclaw/openclaw, 2026. Accessed August 2, 2026. [51] G. Orlanski, D. Roy, A. Yun, C. Shin, A. Gu, A. Ge, D. Adila, N. Roberts, F. Sala, and A. Albarghouthi. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks, 2026. URL https://arxiv.org/abs/2603.24755. [52] S. Ouyang, W. Yu, K. Ma, Z. Xiao, Z. Zhang, M. Jia, J. Han, H. Zhang, and D. Yu. RepoGraph: Enhancing AI software engineering with repository-level code graph. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/ 4a4a3c197deac042461c677219efd36c-Abstract-Conference.html. [53] J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang. Training software engineering agents and verifiers with SWE-Gym, 2025. URL https://arxiv.org/abs/2412.21139. [54] W. Peng, Y. Shi, Y. Wang, X. Zhang, B. Shen, and X. Gu. SWE-QA: Can language models answer repositorylevel code questions?, 2026. URL https://arxiv.org/html/2509.14635v2. Version 2, April 26, 2026; accepted to Findings of ACL 2026. [55] J. Su, Z. Zhao, H. Wang, and H. Chen. CoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method, 2026. URL https://arxiv.org/abs/2608.17536. [56] Tencent. Tencent releases and open-sources Tencent Hy4 preview. https://www.tencent.com/ tencent-releases-and-open-sources-tencent-hy4-preview/, 2026. Released August 28, 2026; accessed September 19, 2026. [57] H. He, C. Yue, C. Dong, C. Wan, T. Su, H. Sun, J. Chai, X. Wang, and G. Yin. VistaHop: Benchmarking Long-Horizon Visual DeepSearch, 2026. URL https://arxiv.org/abs/2606.03273. [58] B. Wang, W. Xu, Y. Li, X. Gao, Y. Xie, H. Sun, and D. Chen. Improving code localization with repository memory. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum? id=8yjWLJy2eX. [59] C. Wang, Z. Li, S. Dutta, and M. Naik. QLCoder: A query synthesizer for static analysis of security vulnerabilities. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_ files/paper/2026/hash/2adf01ab15adde8820622f7f24bd516b-Abstract-Conference.html.

15

[60] L. Wang, C. Chen, J. Zhu, R. Zhan, and W. Han. CQLLM: A Framework for Generating CodeQL Security Vulnerability Detection Code Based on Large Language Model. Applied Sciences, 16(1):517, 2026. doi: 10.3390/ app16010517. URL https://www.mdpi.com/2076-3417/16/1/517. [61] X. Wang, R. Hu, C. Gao, X.-C. Wen, Y. Chen, and Q. Liao. ReposVul: A repository-level high-quality vulnerability dataset, 2024. URL https://arxiv.org/abs/2401.13169. [62] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/ hash/a4b6ad6b48850c0c331d1259fc66a69c-Abstract-Conference.html. [63] Z. Wang, T. Shi, J. He, M. Cai, J. Zhang, and D. Song. CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale. In International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=2YvbLQEdYt. [64] X.-C. Wen, X. Wang, Y. Chen, R. Hu, D. Lo, and C. Gao. VulEval: Towards repository-level evaluation of software vulnerability detection, 2024. URL https://arxiv.org/abs/2404.15596. [65] H. Wu, C. Barrett, and N. Narodytska. Lemur: Integrating large language models in automated program verification. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr. cc/paper_files/paper/2024/hash/0c86142265c5e2c900613dd1d031cb90-Abstract-Conference.html. [66] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying LLM-based software engineering agents, 2024. URL https://arxiv.org/abs/2407.01489. [67] C. Yang, Z. Zhao, Z. Xie, H. Li, and L. Zhang. KNighter: Transforming static analysis with LLM-synthesized checkers, 2025. URL https://arxiv.org/abs/2503.09002. [68] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793. [69] A. Yildiz, S. G. Teo, Y. Lou, Y. Feng, C. Wang, and D. M. Divakaran. Benchmarking LLMs and LLM-based agents in practical vulnerability detection for code repositories, 2025. URL https://arxiv.org/abs/2503.03586. [70] Z.ai. GLM-5.2: Built for long-horizon tasks. https://z.ai/blog/glm-5.2, 2026. Accessed August 2, 2026. [71] Z.ai. GLM-5.2: Developer release notes. https://docs.z.ai/release-notes/new-released, 2026. Release entry dated June 16, 2026; accessed September 19, 2026. [72] W. Zeng, J. Zhang, H. Chen, Z. Hu, Y. Liang, J. Chai, D. Liu, Z. Liu, S. Yan, M. Xue, X. Wang, W. Lin, and G. Yin. RecRM-Bench: Benchmarking Multidimensional Reward Modeling for Agentic Recommender Systems, 2026. URL https://arxiv.org/abs/2605.11874. [73] Q. Zhou, J. Zhang, H. Wang, R. Hao, J. Wang, M. Han, Y. Yang, S. Wu, F. Pan, L. Fan, D. Tu, and Z. Zhang. FeatureBench: Benchmarking agentic coding for complex feature development. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=41xrZ3uGuI.

16

Appendix Contents A Extended Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

B

Dataset Construction Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.1 Source Collection and Provenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.2 Build Recovery and Executable Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.3 Task Packaging and Shared References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.4 Human-Guided Pilot Rerolls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18 18 19 19 19

C Quality-Control and Contamination Audits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.1 Temporal Coverage and Prior-Exposure Audit . . . . . . . . . . . . . . . . . . . . . . . . . . C.2 Time-Window Evaluation Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.3 Paired Prior-Exposure Sensitivity Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20 21 22 22

D Dataset Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.1 Repository-Level Task Counts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 Programming Languages and Static Analysis Frameworks . . . . . . . . . . . . . . . . . . . D.3 Complete CWE Label Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.4 Calibration Horizon and Difficulty Distributions . . . . . . . . . . . . . . . . . . . . . . . . . D.5 Pilot Trajectory Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.6 CVE Publication-Time Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24 24 27 27 29 30 31

E

Evaluation and Inference Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.1 System Configurations and Reproducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . E.2 Task Admission and Independent Repeats . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.3 Source and Evidence Freezing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.4 Backend Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.5 Frozen Mainline Scan and Single-Judge Decisions . . . . . . . . . . . . . . . . . . . . . . . . E.6 Mainline Alert Selection and Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.7 Failure Accounting, Recovery, and Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . E.8 Statistical Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.9 Human Validation of the LLM Judge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31 31 32 32 32 32 33 34 34 35

F

Post-Synthesis Verifier Rubric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.1 Independent Replay and Diagnostic Predicates . . . . . . . . . . . . . . . . . . . . . . . . . F.2 CSA Deterministic Objective Gate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.3 CSA Process and Semantic Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.4 CodeQL Quality Gates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.5 Mainline False Positives and Final Verdict . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36 36 36 37 37 37

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

38

H Additional Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H.1 KNighter Comparison Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H.2 Ablation Outcomes and Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H.3 Time-Budget Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H.4 Language, Repository, and CWE Performance . . . . . . . . . . . . . . . . . . . . . . . . . .

39 39 40 41 43

G Representative CSA Checker Patterns

17

A

Extended Related Work

Table 1 applies the following operational criteria. Repository Context requires the evaluated agent or system to receive a complete repository snapshot together with general-purpose navigation capabilities. Isolated functions, patch-only inputs, pre-extracted dependency views, and repositories used solely by an external validation pipeline receive partial credit. Executable Environment requires a reproducible repository or analyzer state with automated build and execution support. Multi-Harness Support is assessed against the Claude Code [5], Hermes six coding-agent harnesses integrated through CheckerBench’s adapters: Agent [48], OpenClaw [50], PAI [44], OpenCode [3], and DS Harness [16]. Full support requires documented, reproducible integrations for all six harnesses under a shared task and scoring contract; support for a subset receives partial credit. A generic patch-submission format, compatibility that depends on an unreleased wrapper, or a system-specific internal agent does not count as released harness support. Long-Horizon Interaction requires a repeated observation–tool–edit–execution loop. Protocols that permit multi-step agent execution or bounded retrieval without specifying the complete loop receive partial credit. Checker Synthesis requires the task output to be an executable, reusable static-analysis rule or checker. Semantic Verification requires an independent rerun of the submitted checker on vulnerable and fixed code together with patch-localized diagnostic evidence; self-reported counts, generic repository tests, and vulnerability labels receive partial credit. False-Positive Evaluation requires residual diagnostic control on the fixed revision, a production scan, or an equivalent procedure that measures overly broad reporting. The audit reflects the task definitions, released artifacts, and evaluation protocols documented by each work, not capabilities that could be supplied by an external agent implementation. Adjacent agent benchmarks. Recent benchmarks extend agentic software engineering beyond short issue resolution tasks. NL2Repo-Bench starts with a natural-language specification and an empty workspace, requiring agents to construct an installable, multi-module Python library [20]. SWE-Bench Pro instead draws complex issues from maintained repositories, where solutions often require coordinated changes across multiple files [19]. LongCLI-Bench covers from-scratch development, feature addition, bug fixing, and refactoring in a command-line environment, using requirement and regression tests together with step-level scoring [22]. SlopCodeBench follows agents as they repeatedly extend their own solutions under evolving requirements, measuring checkpoint success alongside structural erosion and code verbosity [51]. Other evaluations focus on the trajectory itself. Long-Horizon-Terminal-Bench uses graded subtasks across software engineering and other terminal workflows to measure partial progress during extended runs [37]. UltraHorizon tests sustained exploration in partially observable environments, probing planning, memory, and tool use over long interactions [42]. These benchmarks expose failure modes in sustained execution that also matter when an agent must implement, compile, and refine a checker. Binary-analysis benchmarks address a related security workflow. BIN-BENCH evaluates long chains of toolassisted binary analysis and examines reasoning traces in addition to final outcomes [4]. AgentRE-Bench asks agents to investigate unfamiliar binaries without source code and scores their structured behavioral claims against fixed ground truth [1]. CrackMeBench requires agents to recover a binary’s validation logic and submit an input or generator accepted by the executable [15]. Together, these works measure software construction, intermediate progress, and binary reasoning. CheckerBench evaluates a distinct artifact: a reusable analyzer checker, independently rebuilt and tested for vulnerable–fixed diagnostic contrast, patch localization, and false positives.

B

Dataset Construction Details

B.1

Source Collection and Provenance

We collect public CVE records from the National Vulnerability Database and link them to fixing commits in open-source repositories written in C/C++, Java, Python, JavaScript, and Go. A candidate must identify an exact fix, a retrievable parent revision, and a patch that exposes a defect-bearing program change. CVE identifiers provide instance-level provenance; CWE labels support coverage analysis and stratified sampling. We discard candidates with unresolved provenance, unavailable source, incompatible redistribution terms, or 18

no reusable static-analysis property. A stable instance identifier binds each retained repository, revision pair, patch, and downstream artifact.

B.2

Build Recovery and Executable Scope

The following build-anchor and compilation-database details describe the CSA construction adapter. CodeQL tasks retain their own pinned query-analysis environment and reconstruction metadata. The construction catalog links each vulnerability-fixing commit pair to its repository and to one build anchor per revision. Each anchor records a compilation database, a build snapshot, and build-completeness metadata. Construction first extracts the unified diff, commit message, and pre- and post-fix functions. The vulnerable-side compilation database then filters changed files to translation units for which the original include paths, preprocessor definitions, language mode, and generated dependencies can be recovered. Unmatched files remain in the provenance record as skipped inputs; pairs with no analyzable changed translation unit do not enter synthesis. For each retained pair, the builder creates isolated Git worktrees for the two source revisions and overlays revision-compatible generated files from the corresponding snapshots. Paths in compilation-database entries are rewritten from the original repository and build roots to the worktree and snapshot roots. Output directives, warnings-as-errors, disabled system includes, and known incompatible GCC-specific options are removed or normalized before CSA execution. Worktrees permit the two revisions to coexist and allow pairs to be processed concurrently without changing the shared repository mirror. They are removed after scanning, and stale worktrees from interrupted tasks are pruned before subsequent construction runs. After recovering both revisions, we extract the affected pre- and post-fix functions and generate deterministic commands for compiling a checker and scanning the pair. We reject an instance if either revision remains unreconstructable, affected translation units cannot be analyzed after bounded build recovery, or the pair does not support stable differential analysis. The resulting source–build correspondence, build metadata, and scan context remain available for auditing failures.

B.3

Task Packaging and Shared References

Each task is packaged in a randomly named workspace containing the patch, affected functions, a minimal checker template, compile and scan scripts, and read-only reference assets. The visible configuration contains repository, revision, build, and scan-scope metadata but omits CVE and CWE identifiers. Static templates and reference assets are hard-linked when possible and copied otherwise; in either case they are made readonly. A separate provenance ledger maps the random identifier back to the source pair. This design exposes all inputs required for synthesis while preventing vulnerability labels from becoming task shortcuts. The shared library K has two layers. Harness-specific skills support patch interpretation, checker design, build recovery, functional validation, and false-positive triage. A harness-independent, read-only reference layer provides analyzer API documentation, official checker or rule implementations (including CSA resources for C/C++ tasks [13, 40]), reusable language-specific AST, data-flow, and state utilities, CWE mechanism cards [45], and repository-specific conventions when available. These assets provide implementation knowledge but do not encode the task’s defect-specific state-transition logic or diagnostic reporting predicate. The agent must infer defect semantics, select analyzer extension points and semantic or state representations, implement and register the checker, and refine it using compiler and analyzer feedback. A completed run produces checker source and plugin or query artifacts, a structured defect analysis, and a scope statement. The declarations support analysis; only independent execution establishes correctness.

B.4

Human-Guided Pilot Rerolls

Calibration fixes model and harness versions, each task’s analyzer container and language toolchain, workspace schema, enabled tools, shared skills, and time, token, and interaction budgets for each attempt. Pilot runs cover the inspect–implement–compile–scan–repair cycle. We retain observations, edits, tool calls, and responses. One interaction step is one agent decision, independent of token count or low-level tool events. For each candidate task, the construction protocol permits one initial rollout and at most two additional rollouts after human review. Each attempt uses an isolated workspace and the same task, analyzer configuration, verification predicates, and acceptance thresholds. After a failed checker submission, the reviewer identifies 19

the failure from execution logs and verification evidence, revises the prompt guidance, and records the reason for the change before launching the next rollout. A prompt revision guides the agent’s next attempt; every resulting checker is independently checked. The ledger links the attempt index (1–3), prompt version and changes, human diagnosis, source checksum, complete trace, verification evidence, verdict, and termination status. The first passing checker advances to quality control and sampling. A task with no passing checker after attempt 3 is excluded. The released checker–trace pairs therefore all pass the construction-time checker-admission criteria. These pilot retries are recorded separately from independent main-evaluation repeats and their infrastructure-recovery attempts. Human diagnosis can address compiler errors, missed or mislocalized diagnostics, semantic mismatches, or false positives. Verifier internals and task-specific reference checker source remain held out from the calibration agent. Each submitted checker undergoes independent contrastive and patch-relevance checks; accepted executable checkers additionally undergo a subsystem scan on a prebuilt main-branch worktree and sampled LLM warning triage (Appendix C). All applicable executable, semantic, scope, and false-positive gates must pass. Final selection requires reproducible artifacts, complete traces, and independent verification records. We retain passing trajectories with at least 40 interaction steps and sample jointly over repository, CWE category, observed horizon, and difficulty. This threshold selects sustained interaction under the calibration configuration; it is not an intrinsic lower bound on task difficulty. The construction ledger retains checker source and traces for every attempt, independent verification and false-positive evidence, human diagnoses and prompt revisions, attempt counts, and either a passing checker–trace pair eligible for curation or an exclusion record after the cap.

C

Quality-Control and Contamination Audits

Automated admission checks cover revision ancestry, patch and artifact integrity, the presence of build anchors, compilation-database coverage, clean checker compilation, registration and plugin loading, deterministic vulnerable–fixed scans, verifier isolation, and package completeness. The CSA contrastive gate requires Ni− > Ni+ and Ni+ < 50; CodeQL retains its own frozen differential predicate. Passing the CSA gate is insufficient by itself: code verification recomputes diagnostics and requires at least one disappearing vulnerablerevision report whose source location maps to a function modified by the patch. Hence empty scans, unrelated diagnostic reductions, and agent-reported counts cannot satisfy the hard gate. For accepted executable checkers, a broader subsystem scan on a prebuilt main-branch worktree supplies falsepositive evidence. This worktree is neither the vulnerable nor the fixed task revision. During constructiontime calibration, a triage model assesses sampled diagnostics. Failed false-positive checks enter the same human-diagnosis and prompt-revision loop as other admission failures, within the shared three-attempt cap (Appendix B.4). Each new checker is rebuilt and independently rechecked. The main evaluation uses the Claude Opus 5 judge in Section 4, without returning scores for repair. frozen-candidate scan and single Manual audit samples passing and failing submissions across projects, CWE families, and horizon strata, and inspects both defect-mechanism fidelity and the patch-localized true positive. Final removal counts, auditor agreement, and contamination-overlap statistics will be reported with the frozen release. Deduplication and split assignment. Manual review confirms that the patch supports the stated defect and that disappearing reports lie in changed defect-bearing functions. We remove exact duplicate revision pairs; separable units from the same CVE retain a shared provenance link. Connected groups formed by repository identity, CVE identity, shared revisions, and patch similarity are each assigned to one split. Repositorydisjoint and weakness-family-disjoint subsets are additionally marked for generalization analysis. Artifact isolation. Reference states, task-specific checker implementations, calibration trajectories, verifier thresholds, and adjudication records remain outside the agent-visible snapshot. The vulnerability patch is an intentional task input. Potential training exposure to public vulnerability information is distinct from evaluation-time leakage of checker implementations, calibration traces, or verifier artifacts. The following

20

Table 5 Official model release dates and task counts by CVE publication date. The CVE snapshot is September 15,

2026; model release sources were checked September 19–20, 2026, with Opus 4.8 rechecked October 4, 2026. “On date” counts publications on the release day. Later publication alone does not certify an unseen task. Model GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview

Release date Before On date After 2026-07-09 2026-05-28 2026-08-02 2026-08-13 2026-06-16 2026-07-16 2026-08-28

300 295 300 300 299 300 300

0 2 0 0 0 0 0

0 3 0 0 1 0 0

audits examine public dates and model release boundaries; artifact isolation addresses the separate evaluationtime leakage risk.

C.1

Temporal Coverage and Prior-Exposure Audit

Public evidence and date provenance. We archive CVE Program JSON records for the fixed task inventory, recording the source URL, retrieval provenance, and SHA-256 checksum alongside each task [14]. The audit snapshot of September 15, 2026 covers all 300 tasks and 297 canonical CVEs. The original inventory identifier CVE-2024-7773 is a rejected duplicate of CVE-2024-45436; the audit retains the original identifier and uses the canonical record for dating. No task is removed by this normalization. Publication dates range from June 14, 2010, to June 26, 2026. Appendix D.6 gives the complete year distribution. The machine-readable ledger is figures/temporal/task_dates.csv, accompanied by the archived records and a public-evidence index. The ledger separates CVE publication, the CNA’s explicit public-disclosure date, dated public events, and independently checked patch or checker publication evidence. Discovery dates, CVE reservation dates, packaging dates, and CVE identifier years do not establish public availability. Git author and committer timestamps alone also do not establish when a patch became public. For 48 tasks, the CVE record explicitly dates a public event before its own publication. The earliest documented event establishes that information was public by that date; it does not prove the absence of still earlier sources. The full first-public patch and checker audit is unresolved, so the ledger does not certify any task as previously unseen on the basis of publication date alone. Official model release dates. Table 5 uses public release dates documented in provider announcements or release logs, checked on September 19–20, 2026, with the Opus 4.8 source rechecked on October 4, 2026. GPT-5.6 Sol and Claude Opus 4.8 were released on July 9 and May 28, respectively [6, 49]. The official records date Qwen3.8-Max to August 2, DeepSeek-V4-Pro (the evaluated 0813 version) to August 13, GLM-5.2 to June 16, and Kimi-K3 to July 16 [2, 17, 47, 71]. HY4-Preview was released on August 28 [56]. Dates follow the named model version. Public service or API availability counts as release even if weights follow later; named public previews are included and restricted partner previews excluded. The ledger figures/temporal/model_releases.json records each source and date convention. Counts distinguish CVE publication before, on, and after the reported release day. These boundaries describe public availability, not a model’s training-data cutoff. The actual model version must be pinned and matched to its release before temporal scoring. A patch-before-CVE example. CVE-2026-10722, a task in cilium/ebpf, was published on June 3, 2026. Its CVE record references pull request #2021, whose fix was already merged on May 27, 2026 [12]. Thus the patch was public a week before the CVE publication. A CVE-only filter can overstate the evidence for novelty relative to a model release between those dates. We retain both dates and require review of the earlier source; we do not infer whether the model actually trained on it.

21

C.2

Time-Window Evaluation Protocol

Following LiveCodeBench [30], the temporal analysis groups eligible tasks by audited public-availability windows and compares Pass@1 for each harness–model pair using the same three independent generation repeats as the main comparison. Eligibility requires a pinned model version and its official release date together with an audit of public vulnerability descriptions, patches, and task-specific checker sources. Tasks with unresolved public-availability dates remain explicitly unclassified. A common post-release comparison requires the same audited task subset after all included models’ releases. The current inventory has no CVE publications after the latest of the seven release dates, so it supplies no common post-release subset for all seven models. Model-specific comparisons report their own eligible task counts and remain conditional on the earlier-source audit. Windows are fixed from date and task metadata before inspecting outcomes. We begin with half-year windows and merge adjacent windows until each contains at least 20 distinct CVEs, without crossing a model’s release date. A remaining smaller subset is reported with its sample count and interval, without fitting a temporal trend. Each point reports its distinct task count and a 95% confidence interval. For each window, we compute Pass@1 from the final-pass verdicts of the same eligible tasks separately in each repeat and report the arithmetic mean of the three rates, weighting repeats equally. Resampling keeps the three repeat outcomes for each task and all tasks sharing a CVE together. We report language and static analysis framework strata wherever both periods have support, and disclose repository and CWE composition for every window; unsupported strata are not treated as matched comparisons. Construction-time horizon information serves only as a workload proxy. A score decline across time is interpreted jointly with these composition differences, rather than as proof of memorization. The patch remains an intended synthesis input. Reference checker code, calibration trajectories, and verifier artifacts remain held out from the agent-visible package. The audit of generation traces separately records any external retrieval of task-specific solutions or verification material; time-window evidence alone cannot rule out such evaluation-time leakage. The CVE publication counts in Figure 10(a) describe coverage only and do not supply performance estimates. The construction pilot trajectories do not supply independent temporal performance estimates. Performance curves are populated only from completed main-evaluation verdicts and audited task windows.

C.3

Paired Prior-Exposure Sensitivity Protocol

Sampling. The sampling protocol was fixed before collecting supplementary model outcomes. The initial candidate roster contains 100 distinct canonical CVEs: 53 C/C++, 17 Python, and 10 each in Go, Java, and JavaScript. Within each language, task IDs are sorted and placed in a fixed pseudorandom order; canonicalCVE duplicates are skipped across strata. The full candidate order and original metadata hashes are retained. A task that fails environment validation is replaced by the next eligible task in its frozen language order, with the failure evidence and replacement recorded before any generation or recognition probe. The final roster is admitted only after all 100 tasks pass validation. Selection and replacement never use model outcomes or probe recognition. Claude Opus 4.8 and DeepSeek-V4-Pro each run with OpenCode under Conditions and execution. original (O), ordinary-paraphrase (P), and identity-cue-reduced (D) conditions. P rewrites non-executable task prose while preserving requirements, source identifiers, and the complete patch. D starts from P and consistently substitutes safely modifiable internal identifiers with meaningful alternatives, and removes residual provenance cues from visible metadata. Standard-library and required external API names, types, constants, control/data flow, and the vulnerability and fix mechanisms are preserved. Unsafe transformations involving reflection, macros, or dynamic name lookup are rejected. This is a semantics-preserving intervention, distinct from the answer-changing counterfactuals studied by LastingBench [21]. There is one independent generation per task–model–condition combination, for 600 scheduled generations. This supplementary experiment does not reuse historical Full trajectories. All three conditions share the same skill library, executable feedback, 3,000-second active budget, model decoding settings within each Claude Opus 5 as judge. Condition order is randomized within model, and frozen verification rules with

22

task blocks. Workspaces and sessions are independent, and probes never enter generation context. Model API IDs, returned version metadata, access dates, harness version, and environment fingerprints are archived. Mutable API aliases cannot establish that a provider’s underlying weights remained unchanged. Transformation and admission checks. The before, after, and mainline source trees, patch, extracted context, metadata, and tool-visible paths must agree with the same transformation. For CodeQL, all three corresponding databases must be rebuilt from transformed source; changing only prompts, source-archive text, or database display labels does not qualify. Original Git history, source copies, private reverse maps, reference checkers, and verifier material must be inaccessible to generation. All conditions use preinstalled dependencies and local documentation, with network access restricted to the model gateway. Admission requires reverse-mapped syntax checks, human review of defect and fix equivalence, successful builds or database extraction, and independent reference-checker replay covering differential behavior, patch localization, and mainline false positives. Validation logs and human review records bind to the exact variant hashes. Missing evidence blocks admission. Infrastructure failures follow the existing bounded recovery policy and are reported separately; they are not imputed as model failures. Valid model failures and budget exhaustion remain failures, without score-based retries. Claude Opus 4.8 drafts transformation proposals and DeepSeek-V4-Pro provides preliminary critiques. These preparation calls are separate from the 600 synthesis runs and 400 recognition probes and request neither checker solutions nor CVE identification. A human reviewer checks defect and fix equivalence in the complete transformed environments; proposal review does not admit an unbuilt environment. Final semantic approval must reference the exact transformed environment hashes and retain the reviewer identity, review date, supporting evidence, and decision. Independent executable checks remain mandatory. Human review records are retained with the experiment artifacts. Closed-book recognition probes. Each task–model pair receives separate P and D probes, giving 400 scheduled short calls. Each probe contains pre-fix code only: no fixing patch, CVE/CWE label, source URL, or identifying document heading. A fixed extraction rule selects at most 16,000 characters of code at complete line boundaries; P and D use corresponding source spans, with equal clipping boundaries rather than independently selected excerpts. Probe requests use temperature 0 and an output limit of 256 tokens, with no tools or retrieval. The common instruction is: Identify the publicly documented CVE associated with the supplied pre-fix code only if you recognize this specific instance. Do not infer an identifier from the general vulnerability type. Return exactly one JSON object with the key cve_id: a single CVE identifier, or "unknown" if you do not know. Do not use tools or external retrieval.

ID Recall is the fraction of valid completed calls returning the canonical CVE; documented duplicate identifiers are normalized through the frozen ledger. Incorrect, unknown, refused, or malformed model answers do not receive repair attempts; infrastructure-incomplete calls remain unobserved. A correct identifier indicates instance familiarity, not memorization of a checker solution. Failure to recall an identifier does not establish non-exposure. Estimation and interpretation. the primary effect is

The primary outcome is the existing final full_pass verdict. For each model, ∆identity =

N
100 X Yi,P − Yi,D , N i=1

N = 100,

in percentage points. O–P and O–D differences separate general paraphrase sensitivity from the total intervention. The prespecified uncertainty analysis uses 10,000 paired bootstrap draws, resampling canonical-CVE clusters jointly across conditions, to obtain percentile 95% intervals. A supplementary analysis resamples repositories while retaining within-repository task weights. The intervals describe variation across sampled tasks/clusters under one realized generation per condition; they do not separately estimate repeatedgeneration variability. The two models are reported separately.

23

Probe P–D recall differences and descriptive P–D effects among tasks recognized and unrecognized under P provide supporting evidence. A decline in recognition together with a positive P–D effect supports sensitivity to identifying cues; ordinary paraphrase effects and residual difficulty changes remain alternative explanations. We prespecify a practical margin of ±5 percentage points. Only an entire 95% interval inside that margin supports a bounded effect under this intervention. A nonsignificant difference alone does not support equivalence or absence of training contamination. SWE-QA similarly interprets its older-versus-newer source-cohort gap as a possible exposure association [54]. Build and Diff. SR retain the scheduled task denominator. Semantic scores report their own valid-score counts. Incomplete conditions and contrasts remain unavailable in the main comparison rather than silently changing denominators. Figure 10(c) in the main text reports aggregate Pass@1 and exact CVE recall for Claude Opus 4.8 both models. The P–D effects are derived from O/P/D Pass@1 of 43.00/42.00/39.00% for DeepSeek-V4-Pro, yielding +3.00 and +2.00 percentage points, respectively. and 55.00/54.00/52.00% for The corresponding P–D recall declines are 9.00 and 7.00 percentage points.

D

Dataset Composition

D.1

Repository-Level Task Counts

CheckerBench contains 300 tasks from 167 repositories: 159 C/C++ tasks use CSA, while the 141 CodeQL tasks comprise Java (30), Python (51), JavaScript (30), and Go (30). Table 6 lists every repository, its target language, static analysis framework, task count, and share. Projects are counted at the repository level. Dataset prefixes are removed, and repository identifiers are compared without case sensitivity. The historical python-imaging/Pillow identifier is merged with python-pillow/Pillow; ImageMagick/ImageMagick and ImageMagick/ImageMagick6 remain separate repositories. Target language denotes the language analyzed by the task, which can differ from the repository’s dominant implementation language. The counts are taken from the frozen task inventory and its repositorylevel aggregation. Across the full corpus, 151 repositories contribute one task each, 7 contribute two, 4 contribute three, and 4 contribute four; the Linux kernel repository contributes 107 tasks (35.7%). Table 6 Complete repository-level task counts. Shares use all 300 tasks as the denominator and are rounded to one

decimal place.

Repository

Target language

torvalds/linux ffmpeg/ffmpeg imagemagick/imagemagick jerryscript-project/jerryscript qemu/qemu airlift/aircompressor freerdp/freerdp gpac/gpac libredwg/libredwg ikus060/rdiffweb libarchive/libarchive mlflow/mlflow openexr/openexr python-pillow/pillow rweather/noise-java verdammelt/tnef adaptivecomputing/torque alex/rply andialbrecht/sqlparse ansible/ansible apache/incubator-answer apache/netbeans-html4j apache/sling-org-apache-sling-jcr-base apache/thrift

C/C++ C/C++ C/C++ C/C++ C/C++ Java C/C++ C/C++ C/C++ Python C/C++ Python C/C++ Python Java C/C++ C/C++ Python Python Python JavaScript Java Java JavaScript

Static analysis framework Tasks

Share

CSA CSA CSA CSA CSA CodeQL CSA CSA CSA CodeQL CSA CodeQL CSA CodeQL CodeQL CSA CSA CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL

35.7% 1.3% 1.3% 1.3% 1.3% 1.0% 1.0% 1.0% 1.0% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3%

107 4 4 4 4 3 3 3 3 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1

Continued on next page

24

Table 6 continued Repository

Target language

appleboy/gorush argoproj/argo-events argoproj/argo-workflows auth0/nextjs-auth0 avo-hq/avo aws/aws-sdk-go bettercap/bettercap bitly/oauth2_proxy bonigarcia/webdrivermanager bonitasoft/bonita-connector-webservice briancappello/flask-unchained brokercap/bifrost bspkrs/mcpmappingviewer caronc/apprise ccxvii/mujs cesnet/libyang chaosblade-io/chaosblade churchcrm/crm cilium/ebpf codehaus-plexus/plexus-utils containernetworking/plugins corazawaf/coraza cowtowncoder/java-merge-sort craftcms/cms cryptpad/cryptpad darrenofficial/dpaste datadog/guarddog decidim/decidim decolua/9router dedsecinside/torbot deeplook/svglib dhowden/tag dojo/dojo duke-git/lancet dwisiswant0/apkleaks eladnava/mailgen electerm/electerm encode/starlette ethyca/fides fonttools/fonttools foxcpp/maddy gomarkdown/markdown googleapis/google-oauth-java-client goreleaser/goreleaser gphper/ginadmin gruntjs/grunt hedgedoc/hedgedoc horazont/xmpp-http-upload horovod/horovod huggingface/transformers hybridgroup/gobot hyperadev/dragonfly hyperledger/fabric ifmeorg/ifme imagemagick/imagemagick6 jberet/jsr352 jdhwpgmbca/pcapture jedisct1/pure-ftpd jenkinsci/aws-global-configuration-plugin jenkinsci/azure-ad-plugin jenkinsci/codedx-plugin jenkinsci/dependency-track-plugin jenkinsci/environment-manager-tools-plugin jenkinsci/xcode-plugin jupyter-server/jupyter_server keycloak/keycloak keylime/keylime knative/func

Go Go Go JavaScript JavaScript Go Go Go Java Java Python JavaScript Java Python C/C++ C/C++ Go JavaScript Go Java Go Go Java JavaScript JavaScript Python Python JavaScript JavaScript Python Python Go JavaScript Go Python JavaScript JavaScript Python Python Python Go Go Java Go Go JavaScript JavaScript Python Python Python Go Java Go JavaScript C/C++ Java Java C/C++ Java JavaScript Java Java Java Java Python JavaScript Python Go

Static analysis framework Tasks CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CSA CSA CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CSA CodeQL CodeQL CSA CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Share 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3%

Continued on next page

25

Table 6 continued Repository

Target language

kyverno/kyverno langchain-ai/langchain libical/libical libssh2/libssh2 libvnc/libvncserver line/centraldogma mansuf/mangadex-downloader matrix-org/matrix-react-sdk matrix-org/sydent matrix-org/synapse mde/ejs meetecho/janus-gateway mholt/archiver michaelforney/samurai misp/misp-maltego mity/md4c mommyheather/advancedbackups moodle/moodle mrvautin/expresscart mwarning/kadnode nautobot/nautobot-plugin-device-onboarding netplex/json-smart-v2 netty/netty-incubator-codec-quic nltk/nltk nvbn/thefuck nvidia/nvflare nyuccl/psiturk oblac/jodd-http ofirdagan/cross-domain-local-storage ollama/ollama openstack/ceilometer opensuse/libsolv ossrs/srs outray-tunnel/outray overleaf/overleaf pallets/werkzeug panva/jose paramiko/paramiko petl-developers/petl pgadmin-org/pgadmin4 phpmyadmin/phpmyadmin pomerium/pomerium prismjs/prism prometheus/client_golang pytorch/serve pytorchlightning/pytorch-lightning rasahq/rasa redis/redis revel/revel saitoha/libsixel sajari/docconv sass/libsass satori/go.uuid seccomp/libseccomp sehmaschine/django-grappelli skvadrik/re2c smallrye/smallrye-config snowflakedb/snowflake-connector-python spiral-project/ihatemoney stranger6667/pyanyapi strukturag/libde265 strukturag/libheif tankywoo/simiki tdunning/pig-vector tenable/integration-jira-cloud txthinking/brook uclouvain/openjpeg vadz/libtiff

Go Python C/C++ C/C++ C/C++ JavaScript Python JavaScript Python Python JavaScript JavaScript Go C/C++ Python C/C++ Java JavaScript JavaScript C/C++ Python Java Java Python Python Python Python Java JavaScript Go Python C/C++ Go JavaScript JavaScript Python JavaScript Python Python Python JavaScript Go JavaScript Go Java Python Python C/C++ Go C/C++ Go C/C++ Go C/C++ Python C/C++ Java Python Python Python C/C++ C/C++ Python Java Python Go C/C++ C/C++

Static analysis framework Tasks CodeQL CodeQL CSA CSA CSA CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CSA CodeQL CSA CodeQL CodeQL CodeQL CSA CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CSA CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CSA CodeQL CSA CodeQL CSA CodeQL CSA CodeQL CSA CodeQL CodeQL CodeQL CodeQL CSA CSA CodeQL CodeQL CodeQL CodeQL CSA CSA

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Share 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3%

Continued on next page

26

Table 6 continued Static analysis framework Tasks

Repository

Target language

vyperlang/vyper weaveworks/tf-controller wildfly/jboss-ejb-client xsuchy/templated-dictionary xuxueli/xxl-job zopefoundation/products.genericsetup zwczou/weixin-python

Python Go Java Python Java Python Python

CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL CodeQL

Total: 167 repositories

5 ecosystems

CSA / CodeQL

D.2

1 1 1 1 1 1 1

Share 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3%

300 100.0%

Programming Languages and Static Analysis Frameworks

Table 7 gives the complete task distribution by target programming language and static analysis framework. Each task contributes once. Language denotes the code analyzed by the task, using the same definition and frozen inventory as Table 6. Table 7 Complete task counts by target language and static analysis framework. Shares use all 300 tasks. Target language

D.3

CSA

CodeQL

Tasks

Share

C/C++ Java Python JavaScript Go

159 0 0 0 0

0 30 51 30 30

159 30 51 30 30

53.0% 10.0% 17.0% 10.0% 10.0%

Total

159

141

300

100.0%

Complete CWE Label Distribution

Table 8 lists all 85 CWEs and the separate Other group. The source retains original task annotations, supplements three previously unlabeled tasks with explicit CWE labels from saved public CVE records, and assigns Other to the 14 tasks for which no explicit CWE label was found. Labels are deduplicated within each task. Of the 300 tasks, 272 have one CWE label, 14 have two, and 14 have none; 286 tasks therefore have an explicit CWE label. The table’s Other row contains the 14 tasks without an explicit CWE label. Each task contributes once to each of its labels. Table percentages use all 300 tasks, so rows can overlap and their shares sum to 104.7% including Other. Figure 4(c) instead partitions the 314 task–label pairs into pie sectors. Its five leading CWEs are shown with their names. The pie’s Other sector combines 213 pairs from the remaining 80 CWEs with the 14 unlabeled tasks, giving 227 pairs (72.3%). Ties are ordered by ascending CWE number. The CWEs comprise 82 Weakness entries and three upstream Category entries (CWE-189, CWE-264, and CWE-399), as recorded in the supplied CWE v4.20 catalog [45]. Category annotations are retained without remapping them to more specific weaknesses; CWE-264 is marked obsolete in that catalog. CSA and CodeQL columns report task counts for the respective static analysis frameworks. Table 8 Complete distribution of 85 CWEs and Other. Each task counts once per distinct label; shares use all 300

tasks. Multiple-label tasks appear in more than one row. † marks an upstream Category entry. CWE

Name

CWE-787 CWE-125 CWE-22

Out-of-bounds Write Out-of-bounds Read Improper Limitation of a Pathname to a Restricted Directory (’Path Traversal’) NULL Pointer Dereference Improper Neutralization of Input During Web Page Generation (’Cross-site Scripting’) Concurrent Execution using Shared Resource with Improper Synchronization (’Race Condition’)

CWE-476 CWE-79 CWE-362

CSA

CodeQL

Tasks

Share

25 14 2

1 7 13

26 21 15

8.7% 7.0% 5.0%

13 0

0 12

13 12

4.3% 4.0%

9

3

12

4.0%

Continued on next page

27

Table 8 continued CWE

Name

CWE-416 CWE-400 CWE-908 CWE-667 CWE-190 CWE-415 CWE-20 CWE-611 CWE-617 CWE-119

Use After Free Uncontrolled Resource Consumption Use of Uninitialized Resource Improper Locking Integer Overflow or Wraparound Double Free Improper Input Validation Improper Restriction of XML External Entity Reference Reachable Assertion Improper Restriction of Operations within the Bounds of a Memory Buffer Integer Underflow (Wrap or Wraparound) Exposure of Sensitive Information to an Unauthorized Actor Insecure Temporary File Missing Release of Memory after Effective Lifetime URL Redirection to Untrusted Site (’Open Redirect’) Allocation of Resources Without Limits or Throttling Improper Neutralization of Special Elements used in an OS Command (’OS Command Injection’) Improper Validation of Array Index Uncontrolled Recursion Improper Control of Generation of Code (’Code Injection’) Numeric Errors Improper Authentication Inadequate Encryption Strength Inefficient Regular Expression Complexity Relative Path Traversal Improper Link Resolution Before File Access (’Link Following’) Improper Neutralization of Special Elements in Output Used by a Downstream Component (’Injection’) Improper Neutralization of Special Elements used in a Command (’Command Injection’) Improper Access Control Use of Cryptographically Weak Pseudo-Random Number Generator (PRNG) Cross-Site Request Forgery (CSRF) Deserialization of Untrusted Data Insertion of Sensitive Information into Log File Operation on a Resource after Expiration or Release Missing Release of Resource after Effective Lifetime Loop with Unreachable Exit Condition (’Infinite Loop’) Incorrect Authorization Path Traversal: ’\..\filename’ Improper Neutralization of Argument Delimiters in a Command (’Argument Injection’) XML Injection (aka Blind XPath Injection) Buffer Copy without Checking Size of Input (’Classic Buffer Overflow’) Heap-based Buffer Overflow Incorrect Calculation of Buffer Size Off-by-one Error Insertion of Sensitive Information Into Sent Data Observable Discrepancy Generation of Error Message Containing Sensitive Information Execution with Unnecessary Privileges Plaintext Storage of a Password Permissions, Privileges, and Access Controls Incorrect Privilege Assignment Improper Certificate Validation Cleartext Storage of Sensitive Information Use of a Broken or Risky Cryptographic Algorithm Insufficient Entropy Improper Verification of Cryptographic Signature Signal Handler Race Condition Race Condition within a Thread Divide By Zero Resource Management Errors Inefficient Algorithmic Complexity Incomplete Cleanup Improper Cleanup on Thrown Exception

CWE-191 CWE-200 CWE-377 CWE-401 CWE-601 CWE-770 CWE-78 CWE-129 CWE-674 CWE-94 CWE-189† CWE-287 CWE-326 CWE-1333 CWE-23 CWE-59 CWE-74 CWE-77 CWE-284 CWE-338 CWE-352 CWE-502 CWE-532 CWE-672 CWE-772 CWE-835 CWE-863 CWE-29 CWE-88 CWE-91 CWE-120 CWE-122 CWE-131 CWE-193 CWE-201 CWE-203 CWE-209 CWE-250 CWE-256 CWE-264† CWE-266 CWE-295 CWE-312 CWE-327 CWE-331 CWE-347 CWE-364 CWE-366 CWE-369 CWE-399† CWE-407 CWE-459 CWE-460

CSA

CodeQL

Tasks

Share

12 2 9 8 6 7 1 0 6 5

0 8 0 0 1 0 5 6 0 0

12 10 9 8 7 7 6 6 6 5

4.0% 3.3% 3.0% 2.7% 2.3% 2.3% 2.0% 2.0% 2.0% 1.7%

5 0 0 5 0 3 0

0 5 5 0 5 2 4

5 5 5 5 5 5 4

1.7% 1.7% 1.7% 1.7% 1.7% 1.7% 1.3%

3 3 0 1 0 0 0 0 0

1 1 3 2 3 3 3 2 2

4 4 3 3 3 3 3 2 2

1.3% 1.3% 1.0% 1.0% 1.0% 1.0% 1.0% 0.7% 0.7%

0

2

2

0.7%

0

2

2

0.7%

0 0

2 2

2 2

0.7% 0.7%

0 0 0 2 2 2 0 0 0

2 2 2 0 0 0 2 1 1

2 2 2 2 2 2 2 1 1

0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.3% 0.3%

0 1

1 0

1 1

0.3% 0.3%

1 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 0

0 0 0 1 0 1 1 1 1 1 1 1 1 1 1 0 1 0 0 1 0 1

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3%

Continued on next page

28

Table 8 continued CWE

Name

CWE-522 CWE-538

Insufficiently Protected Credentials Insertion of Sensitive Information into Externally-Accessible File or Directory Files or Directories Accessible to External Parties Insufficient Session Expiration Authorization Bypass Through User-Controlled Key Exposure of Resource to Wrong Sphere Always-Incorrect Control Flow Implementation Integer Overflow to Buffer Overflow Incomplete Denylist to Cross-Site Scripting Memory Allocation with Excessive Size Value Premature Release of Resource During Expected Lifetime Access of Resource Using Incompatible Type (’Type Confusion’) Missing Authorization Server-Side Request Forgery (SSRF) Initialization of a Resource with an Insecure Default Improper Neutralization of Special Elements Used in a Template Engine Other (no explicit CWE found)

CWE-552 CWE-613 CWE-639 CWE-668 CWE-670 CWE-680 CWE-692 CWE-789 CWE-826 CWE-843 CWE-862 CWE-918 CWE-1188 CWE-1336 Other

Sum across label rows

D.4

CSA

CodeQL

Tasks

Share

0 0

1 1

1 1

0.3% 0.3%

0 0 0 0 0 1 0 0 1 1

1 1 1 1 1 0 1 1 0 0

1 1 1 1 1 1 1 1 1 1

0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3% 0.3%

0 0 0 0

1 1 1 1

1 1 1 1

0.3% 0.3% 0.3% 0.3%

8

6

14

4.7%

165

149

314

104.7%

Calibration Horizon and Difficulty Distributions

Table 9 expands the horizon and difficulty statistics reported in Section 3.2. The minimum horizon is a curation constraint: only accepted calibration trajectories with at least 40 interaction steps are eligible for the final 300-instance sample. Difficulty scores are used by the dataset-selection pipeline. The median horizon is 60 steps and the maximum is 123. The associated difficulty score has median 0.684 and range 0.414–0.846. These retained construction-time fields use the original calibration metadata. Table 9 Interaction-horizon and difficulty distributions of the calibration trajectories. Interval

Count

Share

Interaction steps [40, 60) 144 [60, 80) 104 [80, ∞) 52

48.0% 34.7% 17.3%

Difficulty score [0.0, 0.5) 32 [0.5, 0.6) 74 [0.6, 0.7) 59 [0.7, 0.8) 130 [0.8, 1.0) 5

10.7% 24.7% 19.7% 43.3% 1.7%

Table 10 reports every one-step bin underlying the horizon intervals above, including zero-count bins. These counts use construction-time interaction-step metadata; the message-level pilot counts follow below.

Table 10 Complete one-step frequency counts in the retained construction-time interaction-step metadata. Read down

each three-column block. All 84 bins, including zero-count bins, are shown; shares use 300 trajectories. Steps

Traces

Share

Steps

Traces

Share

Steps

Traces

Share

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

4 4 4 3 2 6 7 8 6 11 3 12 7 8 11

1.3% 1.3% 1.3% 1.0% 0.7% 2.0% 2.3% 2.7% 2.0% 3.7% 1.0% 4.0% 2.3% 2.7% 3.7%

68 69 70 71 72 73 74 75 76 77 78 79 80 81 82

2 7 8 5 5 6 5 5 5 6 1 4 1 1 9

0.7% 2.3% 2.7% 1.7% 1.7% 2.0% 1.7% 1.7% 1.7% 2.0% 0.3% 1.3% 0.3% 0.3% 3.0%

96 97 98 99 100 101 102 103 104 105 106 107 108 109 110

2 0 1 1 0 1 0 0 0 2 0 1 1 0 0

0.7% 0.0% 0.3% 0.3% 0.0% 0.3% 0.0% 0.0% 0.0% 0.7% 0.0% 0.3% 0.3% 0.0% 0.0%

Continued on next page

29

Table 10 continued

D.5

Steps

Traces

Share

Steps

Traces

Share

Steps

Traces

Share

55 56 57 58 59 60 61 62 63 64 65 66 67

6 13 9 12 8 7 8 4 5 3 5 6 7

2.0% 4.3% 3.0% 4.0% 2.7% 2.3% 2.7% 1.3% 1.7% 1.0% 1.7% 2.0% 2.3%

83 84 85 86 87 88 89 90 91 92 93 94 95

2 3 1 3 3 3 4 4 0 1 1 2 1

0.7% 1.0% 0.3% 1.0% 1.0% 1.0% 1.3% 1.3% 0.0% 0.3% 0.3% 0.7% 0.3%

111 112 113 114 115 116 117 118 119 120 121 122 123

1 0 0 0 0 1 0 0 0 0 1 0 1

0.3% 0.0% 0.0% 0.0% 0.0% 0.3% 0.0% 0.0% 0.0% 0.0% 0.3% 0.0% 0.3%

Pilot Trajectory Statistics

Figure 11 summarizes the original pilot traces for the 300 selected tasks (159 CSA and 141 CodeQL), with one admitted trajectory per task. Selection follows the original checker-admission records, including 13 trajectories whose checkers passed admission while their agent runs ended at the spending limit. These statistics describe the construction pilot; independent main-evaluation runs follow the final-pass criteria in Section 4. (a) Assistant turns

(b) Tool calls Mean = 72.73

60

Number of trajectories

Number of trajectories

Mean = 58.43

40

20

0 20

40

60

80

100

120

75

50

25

0

140

Assistant turns per trajectory

25

50

75

100

125

150

Tool calls per trajectory

Figure 11 Distributions of (a) assistant turns and (b) tool calls across 300 admitted pilot trajectories in Checker-

Bench. Histogram bins have width 10. Dashed red lines mark sample means; solid red curves show Gaussian kernel density estimates with Scott’s bandwidth, scaled to histogram counts.

An assistant turn is a unique assistant message, deduplicated by session and message identifier. A tool call is a unique tool-use invocation, deduplicated by session and invocation identifier; failed calls and calls without a matching result are included. The tool-call total includes explicit skill invocations. Assistant-turn counts are computed from message events and use a different definition from the construction-time interaction-step metadata in Table 9 and the runner’s native turn counter. Table 11 gives all histogram bins for both pilot metrics. The two columns each account for the same 300 admitted pilot trajectories under their respective event-counting definitions.

30

Table 11 Complete bin counts for the pilot distributions in Figure 11. Intervals are left-inclusive and right-exclusive.

Each share uses 300 pilot trajectories. Interval

Assistant turns

Share

Tool calls

Share

[20, 30) [30, 40) [40, 50) [50, 60) [60, 70) [70, 80) [80, 90) [90, 100) [100, 110) [110, 120) [120, 130) [130, 140) [140, 150) [150, 160)

18 38 57 58 48 28 25 17 7 2 1 1 0 0

6.0% 12.7% 19.0% 19.3% 16.0% 9.3% 8.3% 5.7% 2.3% 0.7% 0.3% 0.3% 0.0% 0.0%

1 8 23 51 72 46 37 26 16 12 5 1 1 1

0.3% 2.7% 7.7% 17.0% 24.0% 15.3% 12.3% 8.7% 5.3% 4.0% 1.7% 0.3% 0.3% 0.3%

300

100.0%

300

100.0%

Total

D.6

CVE Publication-Time Distribution

Table 12 gives every publication year, including empty years. Counts use the 300 task records, so separable tasks sharing a CVE contribute separately; uncertainty estimates for temporal performance keep those tasks grouped. The 2026 bin is observed through the September 15 metadata snapshot. The complete dating and boundary policies are in Appendix C.1. Table 12 CVE publication years for the fixed 300 tasks. Dates use the canonical CVE record; the CVE identifier year

is not used. Publication year

CSA

CodeQL

Tasks

Share

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026

0 1 0 1 2 0 2 7 5 5 9 6 4 9 44 43 21

1 0 0 1 2 0 0 3 2 6 13 22 24 28 16 14 9

1 1 0 2 4 0 2 10 7 11 22 28 28 37 60 57 30

0.3% 0.3% 0.0% 0.7% 1.3% 0.0% 0.7% 3.3% 2.3% 3.7% 7.3% 9.3% 9.3% 12.3% 20.0% 19.0% 10.0%

Total

159

141

300

100.0%

E

Evaluation and Inference Protocol

E.1

System Configurations and Reproducibility

The seven evaluated models are GPT-5.6-Sol [49], Claude Opus 4.8 [6], Qwen3.8-Max [2], DeepSeek-V4-Pro-0813 [18], GLM-5.2 [70], Kimi-K3 [33], and HY4-Preview [56]. We record harness and model versions, context and decoding settings, reasoning effort, and enabled tools. All systems share pinned repositories, toolchains, containers, resource limits, and the verifier. Released artifacts record 31

prompts, skills, tool schemas, commands, workspace diffs, diagnostics, termination reasons, and submissions.

E.2

Task Admission and Independent Repeats

The evaluation manifest fixes 300 environments (159 CSA and 141 CodeQL), three harnesses, seven models, and three independent generation repeats per environment–harness–model combination. All three repeats cover the same 300 tasks for each of the 21 harness–model pairs, yielding 6,300 logical runs per repeat and 18,900 in total. The reported results are the arithmetic mean of the three repeat-level estimates, with equal weight for each repeat. Recovery attempts are tracked separately. A held-out reference checker is used to verify environment usability before runs are admitted. Only environment–model combinations that pass preflight are released, and blocked combinations remain visible in the coverage ledger. The reference checker is never part of the synthesis inputs. Each independent repeat uses a separate workspace and a 50-minute generation budget. It receives the task instruction, patch, relevant code context, tools, and enabled skills. Visible compilation and analysis feedback can support multiple inspect–write–compile–repair cycles within this budget. The final source is checker.cpp for CSA or CodeQL query source. Generation completion status is recorded independently of downstream verification.

E.3

Source and Evidence Freezing

At termination, the ledger records the submitted source, SHA-256 digest, complete trajectory, tool-call events, explicit skill invocations, elapsed time, usage, and termination reason. Task, harness, model, container, tool, and scorer versions are retained with independent-repeat and recovery-attempt indices. Subsequent scans and judges consume this exact candidate. Compiled artifacts can be regenerated from it, but verification recovery and a valid low quality score cannot replace its source. Raw evidence is retained alongside any normalized summaries and referenced by path and content hash.

E.4

Backend Execution

The reference execution image pins LLVM/Clang 18 and mounts the task workspace at /work. Checker source is compiled with /opt/llvm/bin/clang++ in C++17 mode as a position-independent shared object using -fPIC, -shared, -fno-rtti, -fno-exceptions, the pinned LLVM include directory, and -Wl,–allow-shlib-undefined. The resulting object is a CSA plugin, not a standalone executable. Each scan invokes /opt/llvm/bin/clang –analyze, loads the plugin through the analyzer, enables its registered checker name, and appends the translated compilation-database flags and source file. Diagnostics are attributed to the checker through its registered diagnostic label. The task-local validator and the held-out verifier call the same generated scripts, but the verifier rebuilds from source in a clean workspace. For CodeQL, the verifier independently compiles and executes the submitted query using the task’s pinned CodeQL environment and revision-specific analysis inputs. It enforces the frozen CodeQL rules for target and patch relevance, vulnerable–fixed differential behavior, query scope, diagnostic quality, and hardcoding. CSA-specific compiler commands, report signatures, count limits, and plugin gates are not reused as CodeQL acceptance rules. The backend and rule version accompany every reported gate result.

E.5

Frozen Mainline Scan and Single-Judge Decisions

After paired-revision verification, the same checker is scanned over a broader mainline scope. The mainline code is distinct from the vulnerable and fixed task revisions. At most ten warnings requiring judgment are sampled; the selected warning IDs and supporting code context are retained. Appendix E.6 specifies the selection rule, recorded warning counts, and the distinction between the sampled gate and aggregate Claude Opus 5 [7] performs false-positive triage and, in separate stages, assesses process quality precision. and vulnerability semantics. All stages use this one judge model and record the exact model configuration and rubric version. Judge decisions cannot override a failed executable or risk gate. Each stage commits its first valid decision, including valid low scores and negative verdicts. Only unavailable or invalid evidence follows recovery; quality-based resampling is prohibited. Mainline “refinement” is an assessment of the frozen candidate, with no follow-up edits by the synthesis model. Empty warning sets 32

Table 13 Recorded warning volume in the historical repeat-1 cohort of passed checkers. Counts precede the ten-alert

triage selection. Backend

Checkers

Zero alerts

> 10 alerts

Recorded alerts

Sampled alerts

CSA CodeQL

550 1,390

348 1,218

11 5

880 455

596 382

Total

1,940

1,566

16

1,335

978

and missing or invalid judgments follow each backend’s frozen rules; an undefined sample statistic is not fabricated.

E.6

Mainline Alert Selection and Sensitivity

Selection unit and ordering. The frozen implementation selects individual alert records from each checker’s saved mainline output. Selection is deterministic conditional on that list; it is not uniform random sampling over alerts, files, or alert types, and the selector does not rank by severity. All records are judged when the list has at most ten entries. For a longer list of length N , the zero-based selected indices are ICSA = {k⌊N/10⌋ : k = 0, . . . , 9}, ICodeQL = {round(k(N − 1)/9) : k = 0, . . . , 9}.

(1)

CSA retains checker-tagged warnings and deduplicates identical path–line–column locations within each compiler invocation. Its collected list follows scan-worker completion order, so a fresh scan need not reproduce the same ordering. In particular, for 11–19 CSA warnings the rule takes the first ten. CodeQL uses the normalized finding order and first persists at most 100 findings; its ten-alert selector operates on that retained list. Neither backend allocates a fixed quota per file or alert type. Replaying a sample therefore requires the saved list and selected warning identities. Warning volume before triage. Table 13 reports the archived audit of 1,940 passed checkers from retained repeat 1, covering all 21 harness–model combinations. This historical cohort predates the subsequent aggregate corrections to Table 2; it is not a three-repeat total. Of its 1,335 recorded alerts, 978 were initially sampled and 357 required supplementary judgment. Only 16 checkers had more than ten recorded alerts; 1,566 had none. These counts cover the passed cohort, not all generated checkers. CSA stops collecting after a compiler invocation brings the collected total to at least 100 warnings, and one scan in this cohort reached that cap. Consequently, recorded counts do not establish the number of alerts an uncapped scan of the entire repository would produce. Alert-weighted and checker-weighted precision. The ten-alert sample supplies a per-checker acceptance gate. The reported FPpass instead pools recorded alerts from passed checkers, including supplementary judgments beyond that sample. For a fixed system and repeat, let C+ contain those checkers with Ni = TPi + FPi > 0. The corresponding alert-weighted and checker-weighted precision percentages are P 100 X TPi i∈C TPi , Pmacro = . (2) Pmicro = 100 P + |C+ | Ni i∈C+ Ni i∈C+

Thus FPpass = 100 − Pmicro ; larger alert sets receive more weight in this metric. Pmacro gives every alertbearing checker equal weight. Zero-alert checkers are reported separately and enter neither precision denominator; a clean scan does not establish perfect precision. Observed sensitivity in the archived cohort. We recompute both estimators using the same post hoc Opus 5 audit of all 1,335 saved alerts. It contains 1,063 TP and 272 FP labels. We average the seven system-level estimates equally within each harness. The precision ordering changes from OpenCode, Claude Code, Hermes Agent under alert weighting to Claude Code, OpenCode, Hermes Agent 33

under checker weighting. Within Claude Code, the precision leader changes from GPT-5.6 (93.33% micro) to HY4-Preview (96.11% macro). These are conditional precision rankings, distinct from Pass@1 rankings. Adding the all-recorded-alert gate removes 81 of the 1,940 original passes. The harness ordering by Pass@1 Claude Code/ Kimi-K3 (122/300 remains unchanged, but the leading harness–model pair changes from originally) to OpenCode/ DeepSeek-V4-Pro (119/300 after the added gate). The archived sampled gates and supplementary alert audit both used Claude Opus 5 judgments. The additional gate expands coverage from the ten-alert sample to all recorded alerts while retaining the same judge. Moreover, this is a passed-only audit of saved output: it does not re-adjudicate originally failed checkers or recover alerts beyond scan caps. A representative-subset comparison would need a common task subset spanning repositories and backends, include checkers reaching the FP stage regardless of their original verdict, and compare the sampled and complete outputs under the same judge and rubric. That analysis is not established by this audit; stability of the current full-evaluation rankings under exhaustive adjudication remains unverified.

E.7

Failure Accounting, Recovery, and Aggregation

Final passed requires normal generation completion and every backend-specific executable, mainline, process, semantic, mechanism, scope, and anti-hardcoding gate. Valid generation failures, refusals, and candidate code errors are failures. Infrastructure failures permit at most two additional attempts under the recovery policy. When source already exists, verification recovery reuses it and preserves prior valid stage decisions. Each additional attempt is attached to its original logical run and does not contribute an additional independent value to the reported estimate. Unresolved infrastructure outcomes are recorded separately from valid model failures, and evaluation coverage is reported with the scheduled manifest. Incomplete repeat groups are reported as incomplete coverage; they are not silently treated as completed failures or removed from the scheduled task set. For each harness–model pair, we compute Pass@1 as the percentage of the 300 tasks whose checker passes the full evaluation in each independent repeat. We report the arithmetic mean of these three rates, giving each repeat equal weight (Section 4). Each independent run contributes its own final verdict; a successful repeat does not replace a failed repeat. Final-success efficiency summaries average interaction turns, tool calls, and explicit skill calls over passed trajectories in that repeat, counting recorded invocation events including subagent activity. The conditional means from the three repeats are averaged equally. Each repeat’s passed-trajectory count is retained. If a retained repeat has no successful trajectory, its conditional efficiency and the aggregate efficiency mean are N/A, not zero. Full failure and recovery records remain archived for audit and resource accounting.

E.8

Statistical Reporting

The reported main comparison, time-budget levels, and task subsets average the same three independent repeats equally. The separate ablation study uses one generation per task for each new variant and reuses the original Full cohort (Appendix H.2). The subgroup figure and tables report descriptive estimates without significance claims. Ablation intervals resample paired task IDs jointly across configurations; they reflect task composition and do not estimate uncertainty over repeated model generations or independent CVE clusters. Valid-score sample sizes and backend-specific breakdowns accompany diagnostic means; passed-trajectory counts accompany efficiency summaries. Build and Diff. SR use all 300 tasks as their denominator. Process and Semantic average available valid scores on their respective scales. For passed checkers, P eachPrecorded mainline alert is labeled TP or FP. Within each repeat, FPpass pools these alerts as 100 FP/ (TP + FP). The reported FPpass and #WN are equal-weighted means of the three repeat-level proportions and warning counts, respectively. Checkers with no alerts contribute no denominator. One retained mainline scan reached its warning cap. Judge labels are not human ground truth. The Average and Overall Average rows in Table 2 are unweighted means of the reported metric values over the seven models within each harness and all 21 harness–model combinations, respectively.

34

E.9

Human Validation of the LLM Judge

We validate Claude Opus 5’s false-positive triage, process-quality assessment, and vulnerability-semantic assessment against independent human annotations under the backend-specific rubrics in Appendix F. The three annotators have backgrounds in computer science graduate study and software development. Sampling and judgment units. We sampled 100 of the 159 CSA tasks and 100 of the 141 CodeQL tasks. The CodeQL allocation comprised 25 tasks each for Java, Python, JavaScript, and Go. The task allocation followed these backend and language strata, with one frozen checker submission per task and no scoreboundary or outcome oversampling. Each of the 200 submissions received one process and one semantic judgment. For each backend, 80 of the 100 sampled tasks contributed 100 individual mainline warnings: 60 tasks contributed one warning each, and 20 contributed two each. The remaining 20 tasks contributed none. This yielded 200 warning judgments, 400 submission-stage judgments, and 600 judge–human pairs. No item was excluded or weighted. Model, harness, repeat, and pass/fail status were retained for audit, without allocation quotas. Human labels and reference. All three annotators labeled every item independently, giving 1,800 initial human labels. Their view contained the warning and source context for false-positive triage; checker source and trajectory evidence for process quality; and the fixing patch plus vulnerable and fixed code for semantic quality. Judge labels and rationales, model and harness identities, and final verdicts were hidden during initial labeling. A majority of the three binary labels defined the human reference. For process and semantics, the binary label records whether the score and applicable discrete gates meet the backend’s rubric: process at least 80/100, CSA semantics at least 7/10, and CodeQL semantics at least 8/10. The table covers these stage decisions; numeric-score error and individual rubric-criterion agreement are outside its scope. Label distribution and agreement. Table 14 gives separate judge and majority-human positive counts. Positive means a false positive at the warning stage and an accepted decision at the process and semantic stages. Each row contains 100 valid pairs. Judge–human agreement uses unweighted Cohen’s κ; human–human agreement uses three-rater Fleiss’ κ on the original independent binary labels. Exact matches range from 89 to 95 per row. The 2 × 2 cells in table order are (21, 3, 2, 74), (23, 2, 5, 70), (67, 3, 5, 25), (76, 3, 2, 19), (59, 7, 4, 30), and (65, 5, 3, 27), with cells ordered as judge-positive/human-positive, judge-positive/humannegative, judge-negative/human-positive, and judge-negative/human-negative. The minority vote is positive on (3, 5, 4, 3, 7, 5) split items in table order; the remaining split items have a negative minority vote. Table 14 Human validation of the three categorical judge stages, with 100 items per row. J+/H+ gives judge and

majority-human positive counts; exact agreement is a count. Split gives items with a nonunanimous initial human vote. Backend Stage

J+/H+ Exact J–H κ H–H κ Split

CSA CodeQL CSA CodeQL CSA CodeQL

FP triage FP triage Process Process Semantic Semantic

24/23 25/28 70/72 79/78 66/63 70/68

Pooled

All stages 334/332

95 93 92 95 89 92

0.861 0.821 0.806 0.852 0.760 0.813

0.867 0.835 0.852 0.883 0.800 0.832

7 10 9 6 14 11

556

0.852

0.872

57

Adjudication. The annotation set has 57 nonunanimous human items. Majority voting resolved all 57; no item was tied or removed from the denominator. The majority label changed only the human audit reference and left benchmark verdicts intact. Fleiss’ κ uses the 1,800 initial labels before majority resolution.

35

CSA versus CodeQL and interpretation of κ = 0.85. Across the six rows, the 556 matching labels among 600 pairs give exact agreement of 0.9267. The judge-positive total is 334 and the human-positive total is 332, so chance agreement is 0.5060 and pooled Cohen’s κ = (0.9267 − 0.5060)/(1 − 0.5060) = 0.852 to three decimal places. These calculations give the pooled κ = 0.85 reported in the main text after rounding to two decimal places. Reproducing the CSA and CodeQL results requires the frozen task IDs, checker and warning IDs, judge labels, all three independent human labels, and the adjudicated references. Task-clustered uncertainty intervals likewise require those records.

F

Post-Synthesis Verifier Rubric

CSA and CodeQL retain separate frozen executable and scoring policies. Their common final requirements are normal generation completion, independent executable evidence, mainline false-positive control, process and semantic quality, and backend-specific mechanism, scope, and anti-hardcoding checks. Figure 5 gives the numerical thresholds. The CSA implementation details below describe the post-synthesis-v4 rubric; CodeQL retains its own backend requirements.

F.1

Independent Replay and Diagnostic Predicates

Before collecting trajectories, we instantiate and freeze each task’s verifier Vi from its recovered revision pair and pinned environment Ei . Given checker source qi , the verifier ignores agent-produced binaries, rebuilds + or compiles the candidate in a clean workspace, and scans the task-specific scope in c− i and ci . CSA uses a loadable checker plugin; CodeQL uses submitted query source in its pinned analysis environment. Versionpinned scripts preserve each backend’s context, while isolation prevents artifact leakage. Replay produces diagnostic multisets Wi− (qi ) and Wi+ (qi ), with Ni± (qi ) = |Wi± (qi )|. Executable validity combines three binary indicators for build success, diagnostic contrast, and patch relevance: Validi (qi ) = Bi (qi ) Ci (qi ) Li (qi ), Bi (qi ) = 1[Buildαi (qi , Ei )], Ci (qi ) = Cαi ,i (Wi− , Wi+ ),

(3)

Li (qi ) = Lαi ,i (Wi− , Wi+ , ∆i ). Here, Cαi ,i is the frozen differential-detection predicate and Lαi ,i is its target/patch-relevance predicate. CSA contrast requires Ni− > Ni+ and Ni+ < Tres , with Tres = 50 fixed before collection. Reports are matched using the line-insensitive signature κi (w) = (checker(w), file(w), func(w), msg(w)). At least one disappearing report must lie in a patch-modified function. CodeQL applies its own frozen query, differential, and relevance rules; CSA’s count and matching predicates are not substituted for them. Replay retains normalized locations, diff-hunk distances, residual diagnostics, and in- and out-of-patch reports as evidence. Let Oi denote the completed objective gate, including required artifacts, independent replay, and the backend’s scope and diagnostic-quality checks. CSA uses the operational G6 gate below; CodeQL uses its own frozen objective gate. In both cases, Oi ⇒ Validi (qi ) = 1. Executable validity, process quality, semantic fidelity, and mainline false-positive control are non-compensating requirements.

F.2

CSA Deterministic Objective Gate

This gate invokes no LLM. The source and rebuilt plugin are checker.cpp and checker.so. The full frozen CSA contrast and localization predicates must hold in addition to the recorded gate evidence. G0. The task work directory and interaction trace exist. G1. The checker source exists and contains at least 50 bytes. G2. Independent compilation produces the loadable analyzer artifact. 36

G3. verify_result.json exists and the vulnerable revision produces at least one diagnostic, nbuggy > 0. G4. Vulnerable and fixed diagnostics satisfy the frozen CSA contrast rule; equal positive counts are rejected. G5. Patch relevance is at least 0.5 and diagnostic sparsity is at least 0.3. G6. All G0–G5 requirements pass. Only G6 makes a run eligible for a nonzero CSA process score. Four additional objective diagnostics— Differential Detection (15 points), Diagnostic Sharpness (15), False-Positive and Scope Control (15), and basic CSA Implementation Quality (4)—are reported for diagnosis. These 49 points are excluded from Sp to avoid counting executability again as process quality.

F.3

CSA Process and Semantic Quality

The process stage examines the patch, vulnerable and fixed functions, state.json, scope statement, frozen source, synthesis summary, and validation evidence. Its raw score Qi comprises Patch Semantic Understanding (12 points), Detection Plan Quality (12), Plan-to-Code Consistency (12), and Validation Quality (6), for Qmax = 42. Repeated reads, redundant commands, excessive edits, and non-progressing build loops incur a waste penalty pwaste ∈ [0, 5]. The process floor is 80/100. i The semantic stage first infers the root cause, trigger and safe conditions, defect class, propagation requirements, and minimum analyzer mechanism from the patch and vulnerable functions. It then compares the frozen checker with those requirements. The ordered mechanism ladder is syntax-only → call-match → path-sensitive → dataflow → cross-procedure. Mechanism adequacy is classified as adequate, partial, inadequate, or overkill; semantic correctness as yes, partial, or no. For ten-point correctness and mechanism-fit subscores sc,i , sm,i , the process and semantic scores are 100 max(0, Qi − pwaste ) i , Ss,i = 0.6sc,i + 0.4sm,i . (4) Sp,i = 1[Oi ] Qmax The CSA semantic floor is 7/10. CSA additionally requires mechanism_adapted=adequate, semantic_correct in {yes,partial}, valid parsing, and no semantic-anchor risk. A patch-adjacent AST match may therefore fail semantic assessment despite satisfying a diagnostic contrast.

F.4

CodeQL Quality Gates

CodeQL follows the layered admission structure described above while retaining backend-specific executable, differential, relevance, scope, and hardcoding predicates. Its query is independently rebuilt and checked on the vulnerable and fixed revisions before scoring, and its refinement scan uses the shared mainline false-positive gate, with a default sampled FP cutoff of 25% in the main comparison. Queries that fail or incompletely satisfy these prerequisites are not scored on process or semantic quality. The CodeQL process-quality score is a 100-point score with a process floor of 80. Its dimensions are vulnerability modeling (18 points), observable-evidence planning (14), mechanism selection (18), plan-to-code consistency (18), validation and recovery (14), efficiency and finalization (10), and generalization discipline (8). The semantic-quality score is a 10-point score in half-point increments with a floor of 8; it assesses the vulnerable sink, trigger and dataflow, safety barrier, CodeQL mechanism, and report-scope generalization. In addition to the numeric floor, automatic admission requires correct vulnerability and safety-barrier modeling, sound dataflow or control-flow support, acceptable scope generalization, an adequate mechanism, and no patch hardcoding or project-specific scope. These CodeQL-specific conditions refine the common final requirements without changing the noncompensating relation between executable validity, false-positive control, process quality, and semantic quality.

F.5

Mainline False Positives and Final Verdict

The frozen candidate’s mainline scan and at most ten judged warnings provide false-positive evidence. In the main comparison, the sampled FP rate must be at most 25% for both CSA and CodeQL. For a nonempty, 37

validly judged sample, FP rate is the fraction classified as false positives. Sample composition, warninglevel decisions, and the backend’s complete refinement outcome are stored. A backend-recognized clean scan follows its frozen empty-sample policy. Let Fi denote normal generation completion; Ji the required valid judge decisions; Ri the mainline falsepositive gate, including the backend’s full refinement rules; Mi adequate mechanism fit; Cisem the required semantic-correctness classification; and Hi , Ai hardcoding and semantic-anchor risks. With tCSA = 7 and tCodeQL = 8, the final verdict is Passi = Fi ∧ Oi ∧ Ji ∧ Ri ∧ [Sp,i ≥ 80] ∧ [Ss,i ≥ tαi ] ∧ Mi ∧ Cisem ∧ ¬(Hi ∨ Ai ).

(5)

Section 5.6 varies only the cutoff within Ri , replacing its default 25% value while retaining the saved warning Claude samples, judge decisions, and all remaining acceptance requirements. The main evaluation uses Opus 5 [7] for staged false-positive, process, and semantic judgments and retains each stage’s first valid decision. A valid low score is a failure, not a reason to ask again. A valid failed gate or low score yields a failed quality outcome; no new checker is requested. An unavailable judge or missing verification evidence cannot grant a pass and follows the bounded infrastructure-recovery rules, without discarding valid earlier decisions. Only a normally completed generation whose candidate satisfies every required gate receives final passed. Objective-only scoring cannot issue that final outcome.

G

Representative CSA Checker Patterns

This appendix summarizes a selected set of implementation patterns from official LLVM 18 Clang Static Analyzer (CSA) checkers [40]. The catalog applies only to the CSA-backed C/C++ subset of CheckerBench; tasks in other language ecosystems use the corresponding analyzer-native rule APIs. We retain patterns that expose distinct synthesis challenges rather than reproducing complete checker implementations. Table 15 Selected CSA checker patterns used as implementation references for C/C++ tasks, from stateless local

checks to path-sensitive lifecycle analyses with persistent program state. Pattern / official checker

Callbacks / state

Synthesis challenge

Constraint violation DivZeroChecker

PreStmt State: none

Split feasible states and report only a definitely zero denominator.

Undefined value UndefResultChecker

PostStmt State: none

Identify the undefined operand and retain useful diagnostic provenance.

Resource lifecycle SimpleStreamChecker

PostCall, PreCall, DeadSymbols, PointerEscape State: symbol-to-state map

Track opened and closed resources, double close, escape, and leaks.

Nullability propagation NullabilityChecker

Binding, calls, returns, locations, assumptions, and events State: two maps and an invariant flag

Propagate annotations and constraints across multiple semantic events.

Memory management MallocChecker

Calls, locations, dead symbols, escapes, and assumptions State: region-state map plus auxiliary maps and sets

Model allocation families, deallocation compatibility, leaks, double free, and use after free.

Stateless constraint checks. DivZeroChecker and UndefResultChecker illustrate the smallest useful checker structures. They register a single statement callback, query the symbolic value already maintained by CSA, and emit a report without introducing persistent state. The former uses a dual feasibility assumption to separate zero and nonzero denominators; the latter inspects an undefined result after expression evaluation and attributes it to the relevant operand. These patterns test whether an agent can select the correct callback phase and translate a local defect predicate into analyzer constraints. 38

Finite-state resource tracking. SimpleStreamChecker represents the minimal lifecycle pattern. A programstate map associates each resource symbol with an opened or closed state. Post-call handling records acquisition, pre-call handling validates and updates release, dead-symbol handling reports leaks, and pointer-escape handling stops tracking resources whose ownership can no longer be established. This pattern requires the synthesized checker to coordinate several callbacks while preserving a compact state machine. Multi-callback semantic propagation. NullabilityChecker and MallocChecker represent the complex end of the catalog. They combine call, statement, location, assumption, liveness, and escape events with multiple state traits. Their purpose here is not to provide templates for direct copying, but to expose the design decisions required for repository-level synthesis: choosing semantic events, defining stable state keys, suppressing infeasible or duplicate reports, and attaching path evidence to diagnostics. Common reporting contract. Across the selected patterns, a checker obtains symbolic values from the current analysis state, creates a sink or non-fatal error node as appropriate, constructs a path-sensitive bug report, marks relevant symbols and source ranges, optionally installs a visitor for path notes, and finally emits the report. CheckerBench evaluates the resulting behavior through independent rebuilding and vulnerable–fixed diagnostic contrast rather than by matching these reference implementations.

H

Additional Experimental Results

H.1

KNighter Comparison Details

KNighter’s Tasks and comparison. The comparison in Section 5.3 uses the 61 Linux-kernel tasks from original benchmark [67]. Each task takes a bug-fixing commit as input and requires a CSA checker that distinguishes buggy code from its patched version. We evaluate our OpenCode-based method, KNighter, and our method without skills, all using DeepSeek-V4-Pro. Table 3 reports valid-checker counts and rates. Time budget. All three configurations use the same 50-minute (3,000-second) active generation budget per task. The budget covers model responses, tool execution, and checker-repair iterations across the complete workflow, including KNighter’s internal agent coordination. Recorded admission queues and maintenance pauses, as well as subsequent independent verification, are excluded under the timing convention in Appendix H.3. The submitted checker is frozen when generation terminates. Accessible documentation. Our method and KNighter receive the same fixing patch, source-code context, repository revisions, and pinned CSA environment. Both can consult the same read-only Linux/CSA documentation, analyzer API references, and official checker examples described in Appendix B.3. These shared resources exclude held-out reference checkers and task-specific checker solutions. Skill library. Both systems have access to the same reusable skill and strategy content for patch interpretation, checker design, build recovery, and validation. The content is exposed through each system’s native interface. The w/o skills variant of our method removes these skill packages and strategy references while retaining the same task inputs, factual API documentation, tools, and execution budget. Task prompts. The task-facing instructions standardize the synthesis objective, supplied patch and code context, available resources, required checker artifact, and validation requirements. System-specific role prompts and internal orchestration retain each harness’s workflow, while the task information and reusable knowledge made available to our method and KNighter are held constant. Validation criteria. All three configurations use the same independent verifier, scan scopes, and acceptance predicates. The verifier rebuilds each frozen checker from source in the pinned CSA environment and applies the same build-success, vulnerable–fixed diagnostic-contrast, and patch-relevance checks (Equation 3). Agentreported success does not determine the verdict. Table 3 reports the number of valid checkers and the corresponding percentage over the same 61 tasks. 39

Table 16 Artifact pass and differences from Full. All rates use the full task denominator; CSA and CodeQL use 159

and 141 tasks. Intervals use 10,000 paired task bootstrap samples. Configuration Full w/o Skills w/o Execution Feedback w/o Harness

H.2

All (%)

∆ pp [95% CI]

CSA (%)

CodeQL (%)

48.67 18.67 20.33 1.33

Reference −30.00 [−36.33, −23.67] −28.33 [−34.67, −22.00] −47.33 [−53.00, −41.33]

44.03 15.72 17.61 0.00

53.90 21.99 23.40 2.84

Ablation Outcomes and Coverage

Each configuration covers the same 300 tasks: 159 CSA and 141 CodeQL tasks. The three new variants have 900 frozen candidates in total; Full reuses the original 300 candidates. We independently evaluated the 21 historical artifacts whose assessment had stopped at the generation deadline. All 21 fail an artifact requirement, including three with no submitted source; Full therefore has 146/300 artifact passes and 119/300 full passes (39.67%). No candidate was regenerated or edited during this completion. Configurations and generation. Each new configuration uses one generation per task. w/o Skills jointly removes skill packages and strategy references while retaining task inputs, repository tools, and factual API documentation. w/o Execution Feedback retains skills but returns neutral receipts for build and analyzer OpenCode with a 3,000-second active calls; actual outputs remain outside the agent workspace. Both use budget. w/o Harness makes one model call without tools or skills, using the full patch and deterministically extracted code context. The new variants share the model route, sampling settings, and 16,384-token output limit per response; one-shot generation has no feedback or repair call. Endpoints and comparability. Artifact pass applies the compilation, differential, semantic, anti-hardcoding, and false-positive gates without requiring Process or normal agent termination. Frozen artifacts are evaluated even when generation reaches the time limit. The new runs retain their frozen Claude Opus 5 judgments for Process, Semantic, and task gates; a separate Opus 5 audit of all saved alerts supplies the reported FPpass without changing the original verdicts. Historical Full differs from the new runs in execution date, client context limit, and workspace/tool isolation, so the comparisons do not isolate causal effects of individual components. The one-shot result also reflects its strict output protocol: 237/300 requests yield invalid or incomplete structured submissions, and another 53 fail compilation. Using the displayed rates in Table 4, the Pass@1 differences from Full are −31.34, −26.34, and −39.34 percentage points, respectively. The artifact-pass intervals in Table 16 describe variation over paired tasks for one generation per task; they do not cover model resampling, shared-CVE dependence, or differences between historical and new execution conditions. Process/Semantic valid sample counts are 239/239 for Full, 119/122 for w/o Skills, 100/100 for w/o Execution Feedback, and 10/10 for w/o Harness. These conditional means are not averages over all 300 tasks. Three unavailable Process judgments are excluded instead of using heuristic fallback scores. Missing evidence is never treated as a pass. One failed candidate has seven false positives among ten sampled alerts, exceeding its 25% threshold. Opus 5 independently adjudicates every saved alert of the 25, 40, and 1 passed new checkers (16, 64, and 0 alerts, respectively). All 80 labels are resolved after source-evidence completion; the Full FP figure reuses the 78-alert Opus 5 audit from the main experiment. These post hoc labels define FPpass , not the original sampled FP gates. The false-positive counts are 2/78 (2.56%) for Full, 1/16 for w/o Skills, and 16/64 for w/o Execution Feedback. One-shot has no saved alerts, so its N/A entry does not denote a zero falsepositive proportion. No-alert checkers add nothing to the alert denominator. The metric is a false-discovery proportion over saved alerts, not a repository-wide false-positive rate. Because the ablation variants have substantially lower pass rates and therefore fewer successful trajectories, Table 4 averages turns and tool calls over all 300 tasks, including failed trajectories. This differs from Table 2, which reports interaction metrics over successful trajectories only. Table 17 additionally restricts 40

Table 17 Interaction cost on tasks passed by both artifacts. Each pair uses its own common task subset; these means

should not be compared across rows as if the subsets were identical. Variant

Tasks

Full turns

Variant turns

Full tools

Variant tools

42 43 3

49.29 45.21 22.00

64.02 63.19 1.00

63.21 60.63 39.33

89.24 84.09 0.00

w/o Skills w/o Execution Feedback w/o Harness

each comparison to tasks whose artifacts pass in both Full and the corresponding variant.

H.3

Time-Budget Sensitivity

Tables 18–20 provide the complete time-budget results for the analysis in Section 5.5. The tables cover all seven models and three harnesses at 10, 20, 30, 40, and 50 minutes of active generation time, using the completed-within-budget policy below. Timing and eligibility. We retrospectively apply the cutoffs to existing trajectories, retaining each task’s current evaluated attempt. Active generation time includes model responses and tool execution, but excludes recorded API and compute admission queues, maintenance pauses, and subsequent external verification and judging. Only final artifacts completed within a cutoff contribute their existing evaluation outcomes; unfinished runs count as unsuccessful for Pass@1, Build, and Diff. SR, with all 300 tasks retained in each denominator. Intermediate artifacts are not evaluated, and agents are not rerun with shorter deadlines. Realized execution time and trajectory length can differ from the budget because agents may terminate early, batch tool operations, or spend time waiting for responses and recovering from build failures. Conditional metrics and model differences. Process and Semantic average available valid scores from eligible completed runs; Avg. Turns and Avg. Tool Calls average eligible passed runs. FPpass pools recorded alerts from those passed runs, using the same adjudicated labels as the main table. These statistics describe changing subsets of completed runs, not within-trajectory quality improvements. Averaged across harnesses, the 40-to50-minute Pass@1 gains are largest for Qwen3.8-Max (3.67 percentage points) and HY4-Preview (3.33), compared with 0.44 for GPT-5.6 and 3.00 for Claude Opus 4.8. These differences describe when the retained trajectories finish successfully. Table 18 Time-budget sensitivity on CheckerBench:

Model

OpenCode.

Budget Avg. Avg. (min) Pass@1 (%) ↑ Build (%) ↑ Diff. SR (%) ↑ FPpass (%) ↓ Process ↑ Semantic ↑ Turns ↓ Tool Calls ↓

GPT-5.6

10 20 30 40 50

12.00 23.33 28.33 31.33 31.67

25.33 64.67 85.00 95.33 98.33

25.00 58.67 75.33 83.00 85.33

0.00 0.00 0.00 0.00 0.00

84.57 82.20 80.96 80.71 80.22

8.26 7.83 7.75 7.68 7.68

23.50 26.89 28.40 29.22 29.19

54.53 57.60 58.85 60.06 60.13

Claude Opus 4.8

10 20 30 40 50

8.33 24.00 36.00 42.33 45.33

9.67 31.67 53.67 70.33 81.67

9.67 31.33 53.00 69.67 81.00

0.00 0.00 5.36 14.04 13.79

88.28 89.42 88.11 87.07 86.42

9.04 8.84 8.51 8.35 8.32

26.20 33.60 39.41 42.78 44.78

39.36 50.25 57.24 62.07 64.35

Qwen3.8-Max

10 20 30 40 50

6.33 22.33 29.33 35.00 37.33

7.67 33.33 50.33 63.00 69.33

7.67 33.00 50.00 62.67 69.00

— 0.00 0.00 6.00 5.77

92.15 91.72 91.16 90.58 90.15

9.22 8.69 8.50 8.49 8.46

24.11 31.15 33.18 34.84 35.96

31.68 41.39 44.00 46.10 47.54

DeepSeek-V4-Pro

10 20 30 40 50

13.33 26.00 34.33 38.67 39.67

22.67 55.67 75.33 87.67 92.67

21.67 54.33 73.67 85.67 90.67

0.00 5.26 5.00 2.82 2.56

89.98 88.05 86.34 85.32 84.89

8.40 8.18 8.19 8.16 8.12

34.25 39.64 44.62 49.52 51.09

47.88 52.94 58.42 63.78 65.28

Continued on next page

41

Table 18 continued Model

Budget Avg. Avg. (min) Pass@1 (%) ↑ Build (%) ↑ Diff. SR (%) ↑ FPpass (%) ↓ Process ↑ Semantic ↑ Turns ↓ Tool Calls ↓

GLM-5.2

10 20 30 40 50

3.67 18.00 27.33 31.33 32.33

4.33 34.33 59.67 73.00 77.33

4.33 33.00 58.33 71.00 75.33

0.00 14.63 10.34 10.34 10.34

91.61 88.84 87.41 86.90 86.73

9.58 8.20 8.15 8.12 8.11

24.82 33.98 38.62 40.23 41.21

43.73 53.26 58.17 59.55 61.02

Kimi-K3

10 20 30 40 50

8.33 22.33 30.67 36.67 38.67

9.67 40.33 62.00 76.00 83.33

9.67 40.00 60.33 74.33 81.67

0.00 0.00 2.56 1.75 1.59

92.15 90.99 89.67 88.90 88.48

9.34 8.53 8.36 8.34 8.30

22.32 26.10 28.93 31.32 32.42

33.28 36.84 39.91 42.35 43.53

HY4-Preview

10 20 30 40 50

5.33 19.00 28.67 32.00 33.67

7.67 34.33 55.67 65.33 72.67

7.00 33.67 55.00 64.67 71.33

0.00 18.18 14.29 13.98 12.38

90.07 89.70 88.20 87.54 86.90

8.76 8.40 8.32 8.31 8.32

25.06 33.04 38.64 40.32 41.28

32.44 41.02 47.22 49.15 50.15

Budgets threshold existing trajectories by active generation time, excluding recorded resource-admission queues and maintenance pauses. External verification and judging are outside the generation clock. Only final artifacts completed by the cutoff contribute; unfinished runs contribute no success to Pass@1, Build, or Diff. SR, whose denominator remains 300. Process and Semantic average available valid scores among eligible completed runs. Turns and Tool Calls average only eligible passed runs. FPpass is the pooled alert-level FP fraction for those passed runs, using the same adjudicated labels as the main table. A dash denotes no eligible score or a zero alert denominator. Per-cell sample sizes accompany the CSV. These are retrospective completion curves, not separately rerun deadline experiments.

Table 19 Time-budget sensitivity on CheckerBench:

Model

Claude Code.

Budget Avg. Avg. (min) Pass@1 (%) ↑ Build (%) ↑ Diff. SR (%) ↑ FPpass (%) ↓ Process ↑ Semantic ↑ Turns ↓ Tool Calls ↓

GPT-5.6

10 20 30 40 50

10.00 23.67 29.67 33.00 34.00

19.33 55.67 76.33 92.00 93.67

19.00 53.33 69.00 81.33 83.00

0.00 0.00 2.63 2.44 2.44

88.09 82.77 81.89 81.38 81.41

8.24 7.73 7.70 7.67 7.69

28.37 31.34 34.12 35.18 35.71

51.77 54.06 57.94 59.87 60.58

Claude Opus 4.8

10 20 30 40 50

9.00 18.33 28.67 32.67 35.00

10.00 25.33 45.33 55.33 63.67

9.00 24.00 44.00 54.00 62.33

0.00 0.00 2.94 2.50 2.33

87.40 88.30 87.77 87.26 86.88

9.28 8.85 8.47 8.30 8.21

17.52 23.36 28.49 30.23 32.60

30.11 35.98 41.53 43.16 45.63

Qwen3.8-Max

10 20 30 40 50

7.00 23.00 27.33 31.33 34.67

8.67 39.33 51.33 61.00 67.33

8.67 39.00 51.00 60.67 67.00

0.00 4.76 5.88 4.88 4.55

92.41 90.96 90.33 89.86 89.58

9.12 8.50 8.41 8.35 8.38

24.00 27.90 30.12 32.17 33.91

29.33 34.01 36.60 39.36 41.57

DeepSeek-V4-Pro

10 20 30 40 50

14.00 23.67 31.00 36.67 38.33

22.67 49.33 73.67 86.00 91.33

22.33 49.00 72.67 84.33 89.67

0.00 0.00 0.00 1.79 1.72

89.43 87.26 85.86 85.61 85.25

8.56 8.29 8.26 8.29 8.27

35.07 39.44 42.24 45.71 47.17

48.45 53.23 56.84 60.81 62.50

GLM-5.2

10 20 30 40 50

1.00 12.67 23.00 28.00 33.00

1.33 23.00 47.00 62.67 76.33

1.33 22.33 45.33 61.00 74.67

— 0.00 0.00 0.00 0.00

95.75 93.37 91.55 90.55 89.67

9.38 8.89 8.64 8.48 8.39

21.67 30.92 34.01 36.69 38.75

40.33 41.89 45.87 47.92 49.93

Kimi-K3

10 20 30 40 50

11.00 22.67 30.33 35.67 38.00

13.33 47.67 66.33 78.33 83.67

13.33 47.00 65.33 77.00 82.33

0.00 0.00 3.28 2.86 2.74

92.51 90.11 88.49 88.04 87.79

9.33 8.29 8.27 8.29 8.34

24.88 27.32 30.00 31.46 31.81

31.21 33.54 36.31 37.83 38.61

Continued on next page

42

Table 19 continued Budget Avg. Avg. (min) Pass@1 (%) ↑ Build (%) ↑ Diff. SR (%) ↑ FPpass (%) ↓ Process ↑ Semantic ↑ Turns ↓ Tool Calls ↓

Model HY4-Preview

10 20 30 40 50

2.00 8.33 16.00 19.67 23.67

2.33 13.33 29.67 41.00 49.33

2.33 13.00 29.00 39.33 47.33

0.00 0.00 0.00 0.00 0.00

91.99 93.03 91.86 91.42 90.83

9.14 8.66 8.63 8.44 8.35

24.00 34.04 43.00 47.02 50.66

32.33 43.00 53.85 58.29 62.23

Budgets threshold existing trajectories by active generation time, excluding recorded resource-admission queues and maintenance pauses. External verification and judging are outside the generation clock. Only final artifacts completed by the cutoff contribute; unfinished runs contribute no success to Pass@1, Build, or Diff. SR, whose denominator remains 300. Process and Semantic average available valid scores among eligible completed runs. Turns and Tool Calls average only eligible passed runs. FPpass is the pooled alert-level FP fraction for those passed runs, using the same adjudicated labels as the main table. A dash denotes no eligible score or a zero alert denominator. Per-cell sample sizes accompany the CSV. These are retrospective completion curves, not separately rerun deadline experiments.

Table 20 Time-budget sensitivity on CheckerBench:

Hermes Agent.

Budget Avg. Avg. (min) Pass@1 (%) ↑ Build (%) ↑ Diff. SR (%) ↑ FPpass (%) ↓ Process ↑ Semantic ↑ Turns ↓ Tool Calls ↓

Model GPT-5.6

10 20 30 40 50

9.67 19.33 22.33 23.00 23.00

27.67 71.67 89.67 94.67 100.00

Claude Opus 4.8

10 20 30 40 50

6.00 16.00 26.33 30.67 34.33

8.00 26.33 47.00 62.67 80.33

Qwen3.8-Max

10 20 30 40 50

3.00 10.00 15.67 19.67 25.00

DeepSeek-V4-Pro

10 20 30 40 50

GLM-5.2

Kimi-K3

Hermes Agent 20.00 48.67 59.33 62.33 64.00

— 0.00 0.00 0.00 0.00

83.85 81.95 80.49 80.22 79.85

7.97 7.67 7.57 7.57 7.50

24.00 25.55 25.90 25.94 25.94

44.21 46.62 47.19 46.91 46.91

8.00 26.00 46.67 62.33 78.67

0.00 0.00 0.00 0.00 5.45

88.23 88.30 88.15 87.23 85.58

9.23 8.59 8.48 8.41 8.32

21.00 28.31 38.24 42.08 44.84

30.50 41.40 53.27 58.26 61.89

3.67 15.00 25.33 34.33 44.00

3.67 14.67 25.00 34.00 42.33

— 0.00 0.00 5.88 11.11

93.73 93.15 92.03 91.35 90.92

9.05 9.07 8.76 8.62 8.53

20.56 29.37 34.09 36.93 40.55

28.67 38.43 44.11 47.54 51.95

9.33 22.00 27.67 30.33 32.33

15.00 49.33 70.67 85.00 94.00

15.00 48.00 69.00 82.33 90.67

0.00 0.00 0.00 0.00 0.00

90.90 88.11 86.73 85.93 85.29

8.78 8.34 8.09 8.07 8.03

29.21 34.56 37.27 38.24 39.82

50.00 56.45 59.41 60.56 62.37

10 20 30 40 50

3.67 12.33 19.00 21.67 22.33

6.00 28.00 53.00 70.00 79.67

6.00 27.33 51.67 68.67 78.00

— 0.00 0.00 0.00 0.00

90.69 87.49 85.54 83.98 82.87

8.59 8.27 7.99 7.90 7.81

25.18 35.41 39.93 41.37 42.85

43.36 55.84 60.49 62.00 63.22

10 20 30 40 50

7.67 16.33 22.67 24.00 24.33

14.67 41.00 67.67 78.67 87.00

14.33 39.33 63.00 72.67 79.67

0.00 3.64 4.76 4.76 4.76

90.14 87.42 84.78 83.07 82.62

8.38 8.30 8.18 8.06 8.08

21.96 25.90 31.13 31.65 32.73

29.43 34.55 40.41 41.04 42.12

HY4-Preview

10 1.33 1.33 1.33 — 94.00 9.00 24.50 35.25 20 7.67 12.00 12.00 — 92.19 8.91 31.39 42.83 30 12.67 25.33 25.00 — 89.98 8.69 37.32 48.32 40 17.33 38.33 37.00 0.00 90.34 8.53 44.44 56.00 50 21.67 57.67 52.67 0.00 88.42 8.44 49.69 62.06 Budgets threshold existing trajectories by active generation time, excluding recorded resource-admission queues and maintenance pauses. External verification and judging are outside the generation clock. Only final artifacts completed by the cutoff contribute; unfinished runs contribute no success to Pass@1, Build, or Diff. SR, whose denominator remains 300. Process and Semantic average available valid scores among eligible completed runs. Turns and Tool Calls average only eligible passed runs. FPpass is the pooled alert-level FP fraction for those passed runs, using the same adjudicated labels as the main table. A dash denotes no eligible score or a zero alert denominator. Per-cell sample sizes accompany the CSV. These are retrospective completion curves, not separately rerun deadline experiments.

H.4

Language, Repository, and CWE Performance

These breakdowns average the same three independent repeats as Table 2 and Figure 6. Each model–harness pair has the same 300 tasks in each repeat. The main heatmaps average the three harness-specific estimates equally; the tables below retain each harness. The abbreviated model names in the CWE tables follow the full names in Table 2. All rates are percentages and are descriptive; small task subsets do not establish reliable model rankings. The downloadable CSVs preserve full precision.

43

Languages and analyzers. C/C++ uses CSA (159 tasks); Python (51), Go (30), Java (30), and JavaScript (30) use CodeQL. Language and analyzer effects cannot be separated. Table 21 Language-specific Pass@1 (%) by harness. All tasks in each language remain in the denominator. Model

C/C++

Python

Go

Java

JavaScript

46.67 60.00 46.67 53.33 53.33 36.67 60.00

46.67 70.00 53.33 40.00 46.67 50.00 53.33

50.00 60.00 66.67 53.33 33.33 40.00 56.67

GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview

12.58 26.42 15.72 28.30 16.35 27.04 13.84

OpenCode 62.75 72.55 72.55 58.82 60.78 68.63 54.90

GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview

22.01 21.38 14.47 28.30 20.13 23.90 5.66

Claude Code 62.75 62.75 54.90 56.86 64.71 60.78 52.94

30.00 53.33 60.00 46.67 43.33 46.67 36.67

46.67 40.00 66.67 43.33 43.33 56.67 43.33

40.00 36.67 50.00 46.67 26.67 46.67 36.67

GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview

10.69 22.01 3.77 16.98 5.66 11.32 2.52

Hermes Agent 41.18 60.78 60.78 54.90 50.98 43.14 45.10

36.67 40.00 40.00 60.00 43.33 30.00 36.67

30.00 40.00 43.33 43.33 33.33 36.67 46.67

36.67 43.33 43.33 36.67 30.00 43.33 43.33

Table 22 Repository sensitivity by harness (%). Task equal uses 300 tasks; Repo equal averages 167 repository rates;

No Linux uses 193 tasks in 166 repositories. Model

Task equal

Repo equal

No Linux

46.23 57.32 53.73 49.17 43.63 47.92 48.60

42.49 53.37 49.74 46.11 39.38 44.56 43.52

42.60 44.51 49.44 46.68 42.50 49.01 38.06

39.90 40.41 45.60 43.52 38.86 44.56 33.68

32.88 43.70 40.54 42.64 35.51 34.25 35.99

31.09 40.41 36.79 38.34 31.61 31.09 32.12

OpenCode GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview

31.67 45.33 37.33 39.67 32.33 38.67 33.67

GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview

34.00 35.00 34.67 38.33 33.00 38.00 23.67

GPT-5.6 Claude Opus 4.8 Qwen3.8-Max DeepSeek-V4-Pro GLM-5.2 Kimi-K3 HY4-Preview

23.00 34.33 25.00 32.33 22.33 24.33 21.67

Claude Code

Hermes Agent

Interaction turns. Figure 6(c) compares all 21 combinations. Pass@1 uses all 300 tasks; Avg. Turns uses passed trajectories, including subagents. Different task sets do not establish a causal effect of more turns. Passed-trajectory counts appear in the companion summary.

44

Complete CWE breakdown. The following tables include all 85 explicit CWE labels and the 14 tasks without an explicit label (Other). A task contributes once to each distinct CWE label it carries; 14 tasks have two labels. Consequently the row counts sum to 314, not 300. Other does not pool known low-frequency CWEs. The main figure selects the eight explicit labels with at least ten tasks using counts alone, before examining performance. This display threshold is not a claim of statistical precision. CWE groups also differ in language, analyzer, and repository composition. Table 23 Complete CWE-specific Pass@1 (%) for

OpenCode. n counts distinct tasks, not harnesses or repeats.

CWE

n

GPT

Claude

Qwen

DeepSeek

GLM

Kimi

HY4

CWE-787 CWE-125 CWE-22 CWE-476 CWE-79 CWE-362 CWE-416 CWE-400 CWE-908 CWE-667 CWE-190 CWE-415 CWE-20 CWE-611 CWE-617 CWE-119 CWE-191 CWE-200 CWE-377 CWE-401 CWE-601 CWE-770 CWE-78 CWE-129 CWE-674 CWE-94 CWE-189 CWE-287 CWE-326 CWE-1333 CWE-23 CWE-59 CWE-74 CWE-77 CWE-284 CWE-338 CWE-352 CWE-502 CWE-532 CWE-672 CWE-772 CWE-835 CWE-863 CWE-29 CWE-88 CWE-91 CWE-120 CWE-122 CWE-131 CWE-193 CWE-201 CWE-203 CWE-209 CWE-250 CWE-256 CWE-264 CWE-266 CWE-295 CWE-312 CWE-327 CWE-331

26 21 15 13 12 12 12 10 9 8 7 7 6 6 6 5 5 5 5 5 5 5 4 4 4 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

7.69 23.81 60.00 7.69 33.33 8.33 25.00 40.00 11.11 12.50 14.29 28.57 33.33 83.33 0.00 0.00 0.00 40.00 80.00 20.00 40.00 40.00 25.00 0.00 0.00 100.00 66.67 33.33 100.00 66.67 50.00 50.00 50.00 50.00 50.00 50.00 100.00 100.00 50.00 50.00 0.00 0.00 100.00 100.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00

26.92 28.57 73.33 30.77 66.67 25.00 25.00 50.00 33.33 12.50 14.29 14.29 83.33 83.33 16.67 0.00 40.00 20.00 100.00 80.00 40.00 20.00 25.00 25.00 25.00 100.00 100.00 66.67 100.00 100.00 100.00 100.00 50.00 50.00 50.00 100.00 100.00 100.00 100.00 50.00 50.00 50.00 100.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 100.00 0.00 100.00 0.00 0.00 100.00 100.00 100.00 0.00

19.23 23.81 53.33 15.38 58.33 8.33 8.33 40.00 0.00 12.50 0.00 28.57 33.33 66.67 16.67 0.00 0.00 60.00 60.00 20.00 40.00 40.00 75.00 25.00 50.00 66.67 33.33 66.67 66.67 66.67 100.00 100.00 100.00 100.00 50.00 50.00 100.00 100.00 100.00 50.00 100.00 0.00 100.00 100.00 100.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 100.00 100.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00

30.77 23.81 60.00 30.77 41.67 8.33 50.00 50.00 33.33 0.00 28.57 28.57 33.33 33.33 33.33 0.00 20.00 60.00 100.00 60.00 20.00 0.00 50.00 25.00 25.00 100.00 100.00 66.67 100.00 66.67 50.00 100.00 50.00 100.00 50.00 50.00 50.00 100.00 0.00 100.00 50.00 0.00 50.00 100.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00

15.38 19.05 53.33 15.38 41.67 8.33 25.00 50.00 11.11 37.50 0.00 28.57 16.67 50.00 16.67 0.00 0.00 40.00 100.00 60.00 20.00 20.00 25.00 25.00 0.00 66.67 66.67 0.00 33.33 66.67 50.00 100.00 50.00 100.00 50.00 100.00 0.00 50.00 50.00 50.00 0.00 0.00 100.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 100.00 100.00 0.00 0.00 100.00 100.00 100.00 0.00

26.92 33.33 53.33 23.08 33.33 25.00 25.00 70.00 33.33 12.50 0.00 28.57 50.00 66.67 16.67 20.00 60.00 60.00 100.00 40.00 60.00 20.00 50.00 0.00 0.00 100.00 0.00 66.67 66.67 66.67 50.00 50.00 100.00 50.00 50.00 50.00 100.00 100.00 50.00 100.00 0.00 50.00 50.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 100.00 0.00 100.00 100.00 100.00 0.00

3.85 42.86 26.67 7.69 66.67 16.67 25.00 60.00 22.22 0.00 14.29 0.00 66.67 50.00 0.00 0.00 20.00 60.00 60.00 60.00 60.00 40.00 25.00 25.00 25.00 100.00 33.33 66.67 66.67 33.33 100.00 50.00 50.00 100.00 0.00 100.00 50.00 100.00 50.00 50.00 0.00 0.00 50.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00

Continued on next page

45

Table 23 (continued) CWE

n

GPT

Claude

Qwen

DeepSeek

GLM

Kimi

HY4

CWE-347 CWE-364 CWE-366 CWE-369 CWE-399 CWE-407 CWE-459 CWE-460 CWE-522 CWE-538 CWE-552 CWE-613 CWE-639 CWE-668 CWE-670 CWE-680 CWE-692 CWE-789 CWE-826 CWE-843 CWE-862 CWE-918 CWE-1188 CWE-1336 Other

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 14

100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 100.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 100.00 35.71

0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 100.00 100.00 100.00 100.00 57.14

100.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 100.00 100.00 0.00 100.00 100.00 100.00 100.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 0.00 100.00 35.71

100.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 100.00 100.00 0.00 100.00 100.00 100.00 50.00

100.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 100.00 100.00 0.00 0.00 100.00 0.00 100.00 100.00 0.00 0.00 100.00 42.86

0.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 100.00 100.00 100.00 0.00 0.00 100.00 57.14

100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 100.00 0.00 0.00 100.00 100.00 100.00 50.00

Table 24 Complete CWE-specific Pass@1 (%) for

Claude Code. n counts distinct tasks, not harnesses or repeats.

CWE

n

GPT

Claude

Qwen

DeepSeek

GLM

Kimi

HY4

CWE-787 CWE-125 CWE-22 CWE-476 CWE-79 CWE-362 CWE-416 CWE-400 CWE-908 CWE-667 CWE-190 CWE-415 CWE-20 CWE-611 CWE-617 CWE-119 CWE-191 CWE-200 CWE-377 CWE-401 CWE-601 CWE-770 CWE-78 CWE-129 CWE-674 CWE-94 CWE-189 CWE-287 CWE-326 CWE-1333 CWE-23 CWE-59 CWE-74 CWE-77 CWE-284 CWE-338 CWE-352 CWE-502 CWE-532 CWE-672

26 21 15 13 12 12 12 10 9 8 7 7 6 6 6 5 5 5 5 5 5 5 4 4 4 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2

26.92 14.29 46.67 30.77 25.00 25.00 33.33 40.00 22.22 25.00 14.29 14.29 50.00 66.67 33.33 20.00 40.00 40.00 100.00 20.00 40.00 0.00 25.00 25.00 0.00 100.00 33.33 33.33 66.67 33.33 50.00 100.00 50.00 100.00 50.00 50.00 50.00 100.00 50.00 50.00

26.92 9.52 60.00 23.08 25.00 16.67 25.00 40.00 33.33 25.00 0.00 14.29 16.67 100.00 0.00 0.00 20.00 60.00 100.00 80.00 40.00 40.00 50.00 0.00 0.00 66.67 66.67 0.00 66.67 33.33 100.00 100.00 50.00 100.00 50.00 0.00 100.00 100.00 50.00 50.00

26.92 28.57 60.00 7.69 50.00 25.00 25.00 40.00 11.11 0.00 14.29 28.57 66.67 83.33 33.33 0.00 0.00 40.00 60.00 20.00 40.00 20.00 50.00 25.00 25.00 100.00 66.67 33.33 66.67 33.33 50.00 50.00 100.00 100.00 50.00 0.00 0.00 50.00 50.00 50.00

26.92 28.57 40.00 38.46 41.67 25.00 25.00 40.00 33.33 37.50 14.29 14.29 16.67 50.00 16.67 0.00 40.00 0.00 100.00 60.00 60.00 20.00 75.00 50.00 25.00 100.00 0.00 33.33 100.00 33.33 50.00 100.00 50.00 50.00 50.00 50.00 100.00 100.00 50.00 0.00

26.92 28.57 46.67 15.38 33.33 16.67 25.00 40.00 33.33 25.00 14.29 14.29 16.67 66.67 16.67 0.00 0.00 40.00 60.00 40.00 40.00 0.00 50.00 50.00 0.00 66.67 66.67 33.33 66.67 66.67 0.00 50.00 50.00 50.00 0.00 0.00 50.00 100.00 50.00 50.00

23.08 38.10 60.00 15.38 25.00 16.67 25.00 40.00 33.33 25.00 0.00 14.29 33.33 83.33 16.67 0.00 40.00 40.00 100.00 40.00 40.00 0.00 75.00 50.00 75.00 66.67 0.00 33.33 66.67 33.33 50.00 100.00 50.00 100.00 50.00 50.00 100.00 50.00 50.00 0.00

3.85 4.76 40.00 0.00 33.33 8.33 0.00 50.00 0.00 12.50 0.00 14.29 33.33 83.33 0.00 0.00 0.00 40.00 80.00 20.00 60.00 20.00 25.00 25.00 25.00 66.67 33.33 66.67 0.00 66.67 0.00 0.00 50.00 100.00 0.00 50.00 0.00 100.00 50.00 0.00

Continued on next page

46

Table 24 (continued) CWE

n

GPT

Claude

Qwen

DeepSeek

GLM

Kimi

HY4

CWE-772 CWE-835 CWE-863 CWE-29 CWE-88 CWE-91 CWE-120 CWE-122 CWE-131 CWE-193 CWE-201 CWE-203 CWE-209 CWE-250 CWE-256 CWE-264 CWE-266 CWE-295 CWE-312 CWE-327 CWE-331 CWE-347 CWE-364 CWE-366 CWE-369 CWE-399 CWE-407 CWE-459 CWE-460 CWE-522 CWE-538 CWE-552 CWE-613 CWE-639 CWE-668 CWE-670 CWE-680 CWE-692 CWE-789 CWE-826 CWE-843 CWE-862 CWE-918 CWE-1188 CWE-1336 Other

2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 14

50.00 0.00 50.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 100.00 100.00 0.00 0.00 100.00 0.00 100.00 100.00 35.71

50.00 0.00 50.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 100.00 0.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 100.00 35.71

50.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 100.00 0.00 0.00 100.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00 100.00 100.00 100.00 100.00 35.71

0.00 0.00 50.00 100.00 100.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 100.00 100.00 0.00 0.00 100.00 0.00 100.00 100.00 100.00 100.00 100.00 64.29

0.00 0.00 100.00 100.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 100.00 100.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 35.71

50.00 0.00 50.00 100.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 100.00 100.00 100.00 0.00 0.00 0.00 100.00 100.00 0.00 100.00 100.00 0.00 100.00 50.00

0.00 0.00 100.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 21.43

Table 25 Complete CWE-specific Pass@1 (%) for

Hermes Agent. n counts distinct tasks, not harnesses or repeats.

CWE

n

GPT

Claude

Qwen

DeepSeek

GLM

Kimi

HY4

CWE-787 CWE-125 CWE-22 CWE-476 CWE-79 CWE-362 CWE-416 CWE-400 CWE-908 CWE-667 CWE-190 CWE-415 CWE-20 CWE-611 CWE-617 CWE-119 CWE-191 CWE-200 CWE-377

26 21 15 13 12 12 12 10 9 8 7 7 6 6 6 5 5 5 5

19.23 19.05 33.33 15.38 33.33 8.33 16.67 40.00 0.00 0.00 14.29 14.29 16.67 50.00 16.67 0.00 20.00 20.00 80.00

19.23 9.52 40.00 30.77 33.33 8.33 33.33 30.00 33.33 25.00 0.00 28.57 33.33 83.33 16.67 0.00 60.00 60.00 60.00

19.23 14.29 40.00 0.00 75.00 16.67 0.00 30.00 0.00 0.00 0.00 0.00 16.67 33.33 0.00 0.00 0.00 60.00 100.00

26.92 14.29 46.67 7.69 33.33 8.33 25.00 20.00 22.22 12.50 14.29 28.57 16.67 66.67 0.00 0.00 40.00 60.00 80.00

7.69 4.76 40.00 7.69 33.33 16.67 25.00 30.00 11.11 0.00 0.00 0.00 33.33 50.00 0.00 0.00 0.00 20.00 80.00

11.54 9.52 33.33 30.77 25.00 0.00 16.67 30.00 11.11 0.00 0.00 14.29 16.67 50.00 16.67 0.00 20.00 20.00 100.00

0.00 14.29 26.67 7.69 58.33 0.00 0.00 30.00 11.11 0.00 14.29 0.00 33.33 50.00 16.67 0.00 0.00 20.00 80.00

Continued on next page

47

Table 25 (continued) CWE

n

GPT

Claude

Qwen

DeepSeek

GLM

Kimi

HY4

CWE-401 CWE-601 CWE-770 CWE-78 CWE-129 CWE-674 CWE-94 CWE-189 CWE-287 CWE-326 CWE-1333 CWE-23 CWE-59 CWE-74 CWE-77 CWE-284 CWE-338 CWE-352 CWE-502 CWE-532 CWE-672 CWE-772 CWE-835 CWE-863 CWE-29 CWE-88 CWE-91 CWE-120 CWE-122 CWE-131 CWE-193 CWE-201 CWE-203 CWE-209 CWE-250 CWE-256 CWE-264 CWE-266 CWE-295 CWE-312 CWE-327 CWE-331 CWE-347 CWE-364 CWE-366 CWE-369 CWE-399 CWE-407 CWE-459 CWE-460 CWE-522 CWE-538 CWE-552 CWE-613 CWE-639 CWE-668 CWE-670 CWE-680 CWE-692 CWE-789 CWE-826 CWE-843 CWE-862 CWE-918 CWE-1188 CWE-1336 Other

5 5 5 4 4 4 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 14

0.00 20.00 0.00 25.00 0.00 0.00 100.00 33.33 33.33 66.67 33.33 50.00 50.00 50.00 100.00 50.00 0.00 50.00 100.00 50.00 0.00 0.00 0.00 50.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 100.00 100.00 14.29

60.00 80.00 0.00 50.00 25.00 25.00 66.67 33.33 33.33 100.00 66.67 50.00 100.00 50.00 100.00 0.00 50.00 100.00 100.00 50.00 50.00 0.00 0.00 50.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 57.14

0.00 20.00 20.00 75.00 0.00 25.00 33.33 0.00 33.33 66.67 66.67 50.00 100.00 100.00 50.00 0.00 50.00 0.00 100.00 50.00 0.00 50.00 0.00 100.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 28.57

80.00 20.00 20.00 50.00 25.00 25.00 100.00 66.67 66.67 66.67 66.67 100.00 100.00 50.00 100.00 50.00 50.00 0.00 100.00 50.00 50.00 0.00 0.00 50.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 100.00 0.00 100.00 0.00 100.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 35.71

20.00 20.00 0.00 75.00 0.00 50.00 66.67 33.33 33.33 66.67 66.67 50.00 100.00 50.00 50.00 50.00 50.00 50.00 100.00 100.00 50.00 0.00 0.00 50.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 21.43

20.00 0.00 20.00 25.00 0.00 0.00 33.33 0.00 33.33 100.00 66.67 0.00 100.00 50.00 50.00 50.00 50.00 100.00 100.00 50.00 0.00 50.00 0.00 50.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00 100.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 100.00 100.00 14.29

20.00 40.00 0.00 75.00 0.00 25.00 66.67 33.33 66.67 33.33 33.33 0.00 50.00 100.00 100.00 0.00 50.00 0.00 50.00 100.00 0.00 0.00 0.00 50.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 0.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 100.00 100.00 0.00 100.00 0.00 0.00 0.00 100.00 0.00 0.00 100.00 14.29

48

Related documents

• CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?#1216932
• RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents#971558
• PBT-Bench: Benchmarking AI Agents on Property-Based Testing#1217464
• Archer: Towards Agentic Review for Compiler Optimizations#332603
• Growing an Agent/Prover Interface: Evolutionary Tool Design for Cost-Efficient Theorem Proving in Rocq and Lean#1144511