KernelAudit: Contract-Aware Auditing of Validators for LLM-Generated GPU Kernels

KernelAudit: Contract-Aware Auditing of Validators for LLM-Generated GPU Kernels Ziyuan Li1,2,*

Zihe Song3,*

Xiaofang Qi1

Wei Yang4,2,5,†

Tao Xie1,2,6,4,5,†

1 School of Computer Science and Engineering,

Southeast University, Nanjing, China 2 Beijing Tongming Lake Information Technology Application

arXiv:2611.15506v1 [cs.SE] 11 Nov 2026

Innovation Center, Beijing, China 3 The University of Texas at Dallas, Richardson, TX, USA 4 College of Computer Science and Artificial Intelligence,

Fudan University, Shanghai, China 5 Institute of Systems for Advanced Computing,

Fudan University, Shanghai, China 6 School of Computer Science, Peking University, Beijing, China

lziyuan32@gmail.com, Zihe.song@utdallas.edu, xfqi@seu.edu.cn yang_wei@fudan.edu.cn, taoxie@pku.edu.cn The effectiveness of generating GPU kernels driven by a large language model (LLM) is highly dependent on the quality of the used validator (used to accept or reject a generated kernel candidate against the contracts of the given kernel-generation task), whose validation results inform reward assignment, candidate selection, and speedup claims during kernel generation. However, an existing validator suffers from various limitations, e.g., wrongly accepting a candidate due to missing its forbidden library use or wrongly rejecting a candidate due to determining its permitted library use as forbidden (constraints on library uses are stated in the task contracts). To assess the quality of a validator against the given task contracts, in this paper, we propose an independentaudit approach for the validator by (1) construction of audit oracle: labeling a candidate as PASS or FAIL based on three separate dimensions in our proposed contract model (i.e., semantic correctness, implementation integrity, and measurement integrity) and (2) auditing of the validator against the audit oracle: comparing the labels of the candidate against the validation results produced by the validator. For semantic correctness, reference-calibrated stress tests check output correctness on valid inputs for a candidate. For implementation integrity, reference and candidate execution traces are used to determine specific library uses. For measurement integrity, timing checks assess whether the reported runtime reflects the full time needed to complete the candidate’s computation. We apply our approach to assess the quality of six validators and 716 candidates from seven corpora, including constructed controls. The evaluation results show that our approach helps assess the existing six validators to have nontrivial false-reject rates. There is high-degree agreement between the candidate labels produced by our audit oracle and human judgments. CCS Concepts: • Software and its engineering → Software testing and debugging; • Computing methodologies → Machine learning. Additional Key Words and Phrases: GPU kernels, executable validation, reward hacking, oracle reliability, empirical software engineering

1

Introduction

The effectiveness of GPU-kernel generation driven by a large language model (LLM) depends on the quality of its validator [6, 7, 13, 15, 19, 25]. A validator checks generated candidates against the given task contracts; its results guide subsequent generation, candidate selection, and speedup claims. For example, for a task requiring a custom matrix-multiplication kernel, the validator must reject a candidate that has library use (i.e., calling a library routine) of matrix multiplication ∗ Equal contribution. † Corresponding authors: Tao Xie and Wei Yang.

2

Ziyuan Li et al.

while allowing permitted library use for memory allocation. Insufficient validation can reward defective candidates [36], accept candidates that violate the task contracts, and support misleading performance claims. Existing work improves validators through numerical testing and execution monitoring. Numerical testing [13] examines outputs across more inputs and numerical conditions, while fallback checks [6] screen for prohibited library use. Runtime attribution, such as Atrex [40], measures the share of GPU execution time spent in a candidate. Timing defenses [24] address unreliable performance measurements. However, the existing work [6, 13, 24, 40] does not conduct high-quality check on a generated candidate against the given task contracts. Consider two kernel-generation tasks with different contracts. An elementwise-addition task requires a custom implementation of addition and forbids delegating that computation to a library routine. A matrix-multiplication task requires a custom implementation of matrix multiplication but explicitly permits a library addition routine for a biasaddition epilogue. Treating both uses of the same addition routine as forbidden can wrongly reject a compliant candidate for the second task; treating both as permitted can wrongly accept a candidate that delegates the first task’s required custom computation to the library. Distinguishing these library uses of the same library routine requires evidence of the library call’s role and contribution to the candidate’s output, interpreted under the task contracts. To address the issue, we propose an independent-audit approach with two steps: (1) constructing an audit oracle [3] that labels candidates (with the PASS or FAIL certificate) along three dimensions of our contract model—semantic correctness, implementation integrity, and measurement integrity (Figure 1)—and (2) comparing these labels with the validator’s results. The task contracts provide the compliance criteria, while runtime observations and interventions provide the evidence. For semantic correctness, reference-calibrated stress tests check output correctness on valid inputs for a candidate. For implementation integrity, reference and candidate execution traces are used to determine specific library uses. For measurement integrity, timing checks assess whether the reported runtime reflects the full time needed to complete the candidate’s computation. Note that missing required evidence leaves affected checks unresolved without overriding independently supported FAIL judgments. A PASS certificate means no detected violation within the audited scope after all required checks and certification conditions are satisfied. We apply our approach to assess the quality of six validators and 716 candidates from seven corpora, including constructed controls. Among the candidates accepted by each validator and labeled with PASS or FAIL by our audit oracle, 51.7%–66.9% receive FAIL. Among the 172 candidates labeled PASS by our audit oracle, each validator rejects 9–21, yielding false-reject rates of 5.2%–12.2%. An independent blind review of a stratified random sample of 120 candidates shows high-degree agreement between the candidate labels produced by our audit oracle and human judgments. We also assess candidate selection on a fixed pool of 232 tasks. Reusing complete primary-audit evidence for 204 slots and retaining historical evidence for the other slots, the reconciled audit gate covers 62 tasks (26.7%). Independent shipment labels identify 60 clean candidates, one unsafe, and one unreviewed, bounding its unsafe fraction at 1.6%–3.2%. The six existing validators cover 40.1%–90.5% of tasks, with 15.1%–40.5% of selected candidates labeled unsafe. Lower unsafe delivery thus accompanies substantial abstention; the evidence refresh is not a new GPU run or a prospective blind study. The paper makes the following main contributions: • A contract model separating semantic correctness, implementation integrity, and measurement integrity. • An independent audit combining execution evidence with task contracts and preserving unresolved outcomes.

KernelAudit

3

Fig. 1. Three dimensions of validator acceptance. The audit separately checks output agreement, execution provenance, and timing integrity under the task contracts, preserving unresolved outcomes when evidence is insufficient.

• A systematic assessment of six validators and audit judgments against independent human review. • An evaluation of unsafe delivery, coverage, and candidate rankings under audit-based selection. We release the code and replication materials for KernelAudit at https://github.com/ZIYUANLIstar/KernelAudit. 2

Background, Contracts, and Research Questions

Task Contracts. A task contract specifies the semantic, implementation, and measurement obligations for a candidate. The contract defines valid inputs, a reference, permitted library mediators (routines computing on the candidate’s behalf), and a timing policy. For candidate 𝑘 on task 𝑡, 𝑆𝑡 (𝑘) requires reference-matching outputs on valid inputs under a numerically justified tolerance. 𝐼𝑡 (𝑘) forbids undeclared framework or vendor-library mediation of core work when a custom kernel is required. 𝑀𝑡 (𝑘) requires latency to cover the work producing the returned output. Compliance requires all three obligations: clean𝑡 (𝑘) ≡ 𝑆𝑡 (𝑘) ∧ 𝐼𝑡 (𝑘) ∧ 𝑀𝑡 (𝑘). Task rules specify permitted library uses. An undeclared aten::mm performing a custom GEMM’s dominant computation violates the implementation contract even when outputs match. Contractallowed mediators and epilogue glue remain exempt (Section 3.3). Audit Outcomes. Audit outcomes relate certificates to a weak verifier’s decision 𝑊 (Table 1). A PASS certificate records no detected violation on audited surfaces, without establishing clean𝑡 (𝑘). Outcome codes distinguish violations by their supporting evidence. C3 requires an axis FAIL, an executable signature, and a mapping to 𝑊 ’s blind spot. Other semantic-only failures remain C2. Because axis evidence can overlap, certificate-level false accepts and RQ4 slot ledgers use the priority C3b over C3c over C3a over C2. The priority is a reporting convention, not an ordering of severity or causal strength. Research Questions. Four research questions examine validator acceptance, audit validity, performance claims, and candidate selection. RQ1 (Prevalence and structure): among oracle-decided

4

Ziyuan Li et al.

Table 1. Outcome taxonomy. C0 is weak-agnostic. C1 pairs a weak rejection with a FAIL certificate. C2–C3c are verifier-relative false-accept labels assigned only when 𝑊 accepts. Class

Operational condition

C0 Certified

Oracle certificate PASS (Section 3.5): no violation detected on any required axis; in false-accept tables, also 𝑊 (𝑡, 𝑘 ) = PASS. C1 Public failure 𝑊 rejects and the certificate is FAIL (weak rejections of PASS certificates are false rejects, reported separately). C2 Unattributed silent defect 𝑊 accepts, 𝑆𝑡 (𝑘 ) fails, no qualifying mechanism signature. C3a Output-overfitting violation 𝑊 accepts, 𝑆𝑡 (𝑘 ) fails, and a hardcode/tolerance/specialization signature maps to 𝑊 ’s blind spot. C3b Implementation violation 𝑊 accepts and 𝐼𝑡 (𝑘 ) fails: dispatcher-confirmed output dependence on policyforbidden work or a non-mediated vendor launch. C3c Measurement violation 𝑊 accepts and 𝑀𝑡 (𝑘 ) fails on a decisive, positively controlled guard (Section 3.4). INCONCL Abstention An auditability precondition remains unmet; no evidence to convict or acquit. Carries a root cause.

records, what fraction of accepted records has a FAIL certificate, what fraction of FAIL certificates does a validator accept, and are the main misses parametric or structural? RQ2 (Attribution and oracle validity): what axes and channels support FAIL decisions, and how do audit labels compare with the blind reference in sensitivity, specificity, coverage, and abstention? RQ3 (Credibility and certification): how do reportable local speedups compare with upstream claims, and what fraction of the paired records receives a PASS certificate? RQ4 (Training-free remedy): how do gate and ordering changes affect unsafe delivery, coverage, speed, and ranking on the fixed pool, and what audit-cost evidence is available? 3

A Conservative, Evidence-Tiered Audit Oracle

KernelAudit implements the audit oracle, which audits a candidate against its reference and task contract, returning three axis verdicts and a certificate. The audit combines reference-based testing, runtime observation, and program intervention on contract-supported inputs. Task rules determine whether the evidence supports conviction, withholds certification, or remains a non-gating signal. 3.1

Verdict Semantics

All tasks require the three axes, collected in 𝑅𝑡 . PASS means no detected violation on audited surfaces; FAIL requires candidate-attributable violation evidence. INCONCLUSIVE denotes unmet auditability preconditions; UNSUPPORTED denotes unavailable capabilities. Neither satisfies a required axis. Missing preconditions limit their channel without erasing an independent FAIL; tool failures alone cannot convict (Table 2). The main audit runner gives each candidate a fresh seeded worker; candidate-specific measurement conviction requires isolated attribution (R-C). RQ4 uses a shared-process runner and separately defined replay rules for saved verdicts, eligibility, and timing (Section 8). Guards protect shared canonical inputs, module state, and execution policy; candidates may mutate their private input clones. Thresholds and the vendor inventory are frozen and digest-checked, with fail-closed loading. Appendices A–H provide the parameter and protocol details. 3.2

Foundation Tier: Semantic Axis

Semantic comparison requires verified candidate–reference state correspondence: same-seed instantiation need not ensure equal RNG consumption or weights in restructured models. We inject

KernelAudit

5

Table 2. Auditability conditions and affected decisions. An unresolved channel does not erase an independent FAIL. Certificate and speedup eligibility are separate; axis-level evidence categories appear in Section 6.5 and Appendix C. Unmet condition or trigger

Effect (scope)

Build/import failure or resource exhaustion leaves no executable audit path No reliable state mapping Calibrated allowance 𝜌¯ℓ > 𝜌 max Crash or nondeterminism prevents a comparable replay R-A: no positive work evidence R-B: listed degenerate reference Localization replay missing or incomparable R-C: unattributed decisive timing signal in a shared process Stream differential meets its criterion A/A self-comparison unstable

INCONCL (execution) INCONCL (semantic) INCONCL (semantic) INCONCL (affected channel) INCONCL (implementation) certificate withheld (task) INCONCL (causal probe) INCONCL (measurement) certificate withheld speedup withheld

canonical reference weights using a mapping verified for keys, shapes, dtypes, values, and coverage. Unresolved correspondence or undeclared candidate-only state causes abstention. Valid inputs and a usable reference remain necessary after alignment (Appendix C). Empirical studies document value- and dtype-sensitive numerical failures in GPU programs [27] and shape-, dtype-, and backend-sensitive code-generation bugs in tile-programming frameworks [28]. Motivated by high-precision reference checking [33], we calibrate each input’s floatingoutput tolerances using state-identical native-precision and fp64 reference executions. In fp32, let 𝑑 ℓ𝑖 = |𝑟 ℓ𝑖32 − 𝑟 ℓ𝑖64 |. For each floating leaf ℓ, 𝜖ℓabs and 𝜖ℓrel are the 99th percentiles of 𝑑 ℓ𝑖 and 𝑑 ℓ𝑖 /max(|𝑟 ℓ𝑖64 |, 10−12 ), respectively, over that leaf’s elements, without pooling across leaves, inputs, or runs. With the operator-family multiplier fixed at one, set 𝜌¯ℓ = 𝑐𝜖ℓrel , 𝜌 ℓ = max(𝜌 min, 𝜌¯ℓ ), and 𝑎 ℓ = max(𝑎 min, 𝑐𝜖ℓabs ). For candidate output 𝑘 and native-precision reference 𝑟 (𝑟 = 𝑟 32 here), NumPassℓ (𝑘, 𝑟 ) ⇐⇒ ∀𝑖 : |𝑘 ℓ𝑖 − 𝑟 ℓ𝑖 | ≤ 𝑎 ℓ + 𝜌 ℓ |𝑟 ℓ𝑖 |.

(1)

Here 𝑐 = 4, 𝜌 min = 10−5 , 𝜌 max = 10−3 , and 𝑎 min = 10−8 . The comparison remains unresolved if any 𝜌¯ℓ > 𝜌 max , the required reference precision is unsupported, or the reference output is nonfinite. Otherwise, the rule applies per leaf with exact structure and dtype checks. The multiplier expands the reference-discrepancy scale, while the floors retain a minimum allowance. The ceiling makes excessive relative calibration an auditability limit rather than clipping it into an accepted tolerance. These settings define an empirical acceptance policy, not a statistical error bound; fp64 is a calibration reference rather than exact truth. Our 16-slot schedule draws on prior stress testing [13] to cover contract-supported value, distribution, shape/dtype, layout, repeated-call, and provider-seed tests. Unsupported optional transformations use native-seed tests (Appendix B). A candidate NaN or infinity against a finite reference is a semantic failure; an incomparable execution leaves the affected evidence unresolved. Coverage remains limited to the exercised inputs and call sequences: the stale-cache and specialization controls illustrate unexercised behavior (Table 7). 3.3

Primary Tier: Implementation Axis

Matching outputs cannot reveal delegation. Two channels observe ATen-mediated computation and vendor calls below the dispatcher (Figure 2). Static source tokens guide probes but cannot convict.

6

Ziyuan Li et al.

Fig. 2. Complementary implementation evidence under a fixed seed. Dispatcher interventions test output dependence; launch provenance examines vendor activity beyond observed ATen mediation. Conviction requires attributable evidence of work forbidden by the task contract.

Channel 1: Dispatcher Causal Probe. Dispatcher traces select suspect mediators by compute-class work, measured with FLOP estimates or element-count proxies rather than GPU time. Let 𝑅 be total referenceÍwork, 𝐶𝑔 candidate ATen work in class 𝑔 after excluding identifiable contract-allowed calls, and 𝐶 = 𝑔 𝐶𝑔 . The oracle selects all non-exempt operations in a class that matches the reference’s dominant class and reaches 5% of that class’s work. A class also qualifies when 𝐶𝑔 ≥ 0.30𝑅, or when both 𝐶 ≥ 0.05 max(𝑅, 1) and 𝐶𝑔 ≥ 0.30𝐶. These rules select probes, while conviction also requires output dependence on forbidden work. The unit floor provides numerical protection. Coverage depends on work-proxy accuracy and operation support (Appendix E). Comparable program interventions use causal reasoning [26] and testing [11] to examine whether returned values depend on selected mediators. do(P := noise) denotes replacing targeted primary floating outputs 𝑃 in place with statistically matched, deterministically seeded noise. Let 𝑦 be the clean output and 𝑦˜ the output under the joint intervention on all targets: |𝑦ℓ𝑖 − 𝑦˜ℓ𝑖 | ˜ = max Δ(𝑦, 𝑦) . (2) ℓ,𝑖 max(|𝑦 ℓ𝑖 |, 10 −12 ) This relative score measures output change, not significance or delegated work; near-zero clean outputs can amplify small absolute changes. Channel 1 records FAIL when joint output change exceeds 10−3 on contract-forbidden work. Required replays use aligned modules and inputs, must execute successfully, perturb every target selected for that replay, and preserve output structure, shape, and dtype. Violations of protected execution state also invalidate the probe. With multiple targets and a joint effect, every required localization must pass these checks; otherwise the probe is INCONCLUSIVE. Joint and individual perturbations test dependence on the selected set and effects within it, respectively. A joint effect need not appear individually and does not identify a unique mechanism. The replay checks establish observable comparability, not control-flow equivalence (Appendix A). Channel 2: Kernel-Launch Provenance. To examine delegation bypassing the dispatcher, the oracle collects forward-window GPU launches through CUPTI/Kineto [23] and aligns them using

KernelAudit

7

dispatcher correlation. ATen-mediated launches enter the mediated-library bucket for Channel 1; their presence alone cannot convict. Remaining launches require candidate attribution; unattributed activity is retained without conviction. Candidate-attributed launches without an ATen mediator are matched exactly against 23,666 digest-pinned cuBLAS and cuDNN symbols. Attribution, a vendor match, and the frozen policy jointly support FAIL when the observed work is forbidden core computation. The match supplies occurrence evidence; the policy determines whether that work is prohibited. Declared or contractallowed primitives remain exempt. Coverage is bounded by the sealed inventory (Section 9). Work Presence and Executed Fallbacks. Implementation PASS requires no detected forbidden work plus positive work evidence: a nonempty candidate compute trace or candidate-correlated launch (R-A). Work presence alone does not establish compliance. R-B withholds certification for listed degenerate-reference tasks: matching outputs lack the informativeness this policy requires (Section 6.1). After a custom-build failure, both channels audit the executed native-PyTorch fallback. Attributable forbidden delegation supports FAIL; finding no violation on that path cannot certify the unavailable custom implementation (Section 6.5). 3.4

Safeguard Tier: Measurement Axis

For the published timing-threat catalogue [24], validation and attribution evidence distinguish conviction channels, certification-withholding guards, and non-gating signals. Thread observation, lazy outputs, and static low-precision casts remain non-gating signals. Before candidate import, timing-integrity checks snapshot the identities of time.perf_counter, torch.cuda.Event methods, and synchronize. Positively controlled checks support measurement FAIL for attributable changes under isolated execution and controlled harness state (Table 7). The stream differential can withhold certification without establishing a measurement violation. Randomized ABBA comparisons use timing with and without the synchronization guard. Let 𝑅b denote the overall guarded median divided by the overall unguarded median. Withholding requires 𝑅b ≥ 3, a paired-batch 95% percentile bootstrap interval with lower endpoint at least 3, an absolute median gap ≥ 0.05 ms, and attributable evidence (Appendix A). This channel never convicts because its positive control does not demonstrate convicting power (Section 6.2). A/A screens timing instability [22] with two reference instances: 10 warmups and three paired 20-timing batches per arm in ABBA/BAAB/ABBA order. Both CVs over all 60 latencies and the maximum normalized within-batch order effect must be ≤ 0.05. A paired bootstrap of the three batch-median pairs gives a nominal 95% percentile ratio interval [𝐿, 𝑈 ]; passage requires 𝐿 ≤ 1 ≤ 𝑈 , 𝑈 − 𝐿 ≤ 0.1, and internal timing PASS, including output/state consistency. Appendix A defines the odd/even and half-batch order contrasts and resampling. This screen withholds speedups, not certificates, and does not guarantee unbiased timing. 3.5

Aggregation and Certification

Within one complete audit record, positive violation evidence takes precedence. Let 𝑉𝑎 (𝑘; 𝑒) be the verdict on required axis 𝑎 ∈ 𝑅𝑡 in record 𝑒, where 𝑅𝑡 ≠ ∅. Write U = {INCONCLUSIVE, UNSUPPORTED} for the unresolved statuses. The axis aggregate is   FAIL, ∃𝑎 ∈ 𝑅𝑡 : 𝑉𝑎 (𝑘; 𝑒) = FAIL,    𝐴𝑡 (𝑘; 𝑒) = INCONCL, š𝑎 ∈ 𝑅𝑡 : 𝑉𝑎 (𝑘; 𝑒) = FAIL ∧ ∃𝑎 ∈ 𝑅𝑡 : 𝑉𝑎 (𝑘; 𝑒) ∈ U,    PASS, ∀𝑎 ∈ 𝑅𝑡 : 𝑉𝑎 (𝑘; 𝑒) = PASS. 

(3)

8

Ziyuan Li et al.

Table 3. Audited corpora, including records with INCONCL certificates. Corpus

N Provenance and role

CUDA-L1 [17]

241 Main generated-CUDA corpus; KernelBench-style references, provider inputs, custom-replacement policy. AI-CUDA-Engineer [12] 229 Main generated-CUDA corpus; released task sources and references, same policy. KernelBenchX [38] 63 External generated-Triton corpus with upstream-independent references. TritonBench-G [15], self-built ref. 89 Production Triton kernels; our own independent algorithmic references, layer cross-validated on CPU. TritonBench-G, upstream ref. layer 26 Same source kernels with the upstream repository’s own references (reference-quality probe). cuDNN/cuBLAS delegation controls 60 Positive implementation controls: candidates explicitly call cuDNN (35) / cuBLAS (25) behind independent references. robust-kbench handwritten [13] 8 Curated handwritten set, upstream-presumed clean (probed in Section 6.1).

R-B and the stream guard can withhold certification even when all required axes pass. Let 𝑄𝑡 (𝑘; 𝑒) denote eligibility under these guards. The certificate is 𝐶𝑡 (𝑘; 𝑒) = 𝐴𝑡 (𝑘; 𝑒), except that 𝐴𝑡 (𝑘; 𝑒) = PASS and 𝑄𝑡 (𝑘; 𝑒) false yield 𝐶𝑡 (𝑘; 𝑒) = INCONCL. Certificate withholding leaves axis verdicts unchanged. R-A/R-C affect axis verdicts. The A/A gate affects speedups only. The release manifest binds each certificate to one complete record after eligibility checks. Diagnostic axes may cite separate qualified executions but cannot substitute for the certificate-bound evidence. Appendix A specifies source qualification, export, and restricted-merge rules. Selection predicates read neither weak decisions nor human-label values. Certificates determine counts, while selected axes supply diagnostics. 4

Experimental Design

The experiments use 716 task–candidate–reference records for RQ1–RQ3 and a separate fixed candidate pool for RQ4. The audit records span seven corpora (Table 3). The two TritonBench-G reference layers reuse source kernels. The candidate-selection pool contains 232 tasks and 1,160 candidate slots. Subjects and Task Policies. A frozen contract manifest specifies implementation policies, provenance, and permitted non-core glue (policy resolution in Appendix A). The main CUDA corpora require core self-implementation under our frozen policy; their upstream prompts establish no blanket ban on core library use. Permitted glue includes allocation, initialization, dtype/shape handling, and small epilogues. Constructed delegation controls explicitly require self-implementation. For TritonBench-G and robust-kbench, this requirement is our strict reading of “self-contained/ handwritten,” recorded as author-strict-reading. Implementation labels follow these policies and Section 3.3’s observation rules; Section 6.3 distinguishes the human rubric. Evaluated Verifiers. Of the six weak verifiers (Table 4), CUDA-Agent, KernelBench, KernelGYM, and robust-kbench use source-pinned adapters; DeepReinforce and MusaCoder use publicationderived reconstructions. The artifact documents versions and fidelity. Results characterize these adapters, including omissions. Verdict Alignment. Binary weak decisions are compared with three-valued audit certificates on 569 oracle-decided records (3,414 record–verifier pairs). Weak REJECT includes failed checks and inability to evaluate.

KernelAudit

9

Table 4. Evaluated checking surfaces. Each listed tensor-comparison tolerance 𝑡 sets atol = rtol = 𝑡. The Atrex-based row is an implementation-only RQ2 comparator (𝑛=120 records); Not evaluated denotes study scope, not tool capability. Method

Semantic check

CUDA-Agent [7] KernelBench [25]

assert_close 10 −2 , 5 trials runtime blocking of F.* none allclose fp32 10−4 ; fp16/bf16 10 −2 , 5 tri- none none als none in the evaluated adapter (up- none allclose 10 −2 , 5 trials stream decoy signal not included) allclose 10 −5 , 5 trials, seeded, double none none evaluation allclose 10 −2 , 5 trials none 4 guards (timing, materialization, threads, stream) allclose 10 −2 , 5 trials static high-level fallback scan none (within a five-category policy)

KernelGYM [19] robust-kbench [13] DeepReinforce [24]

MusaCoder [6] Atrex-based screen [40]

Not evaluated

Implementation check

Measurement check

Attribution screen only: target- Not evaluated as kernel GPU time / end-to-end la- measurement intency; ≤ 0.5 suspicious, not an tegrity; timing is an implementation-axis verdict attribution input

Metrics and Evidence. The false-accept rate is the fraction of weak-accepted, oracle-decided records with a FAIL certificate. The miss rate is the accepted fraction of FAIL certificates; the false-reject rate is the rejected fraction of PASS certificates. Selected-axis FAIL rates share the falseaccept denominator but may use separate valid evidence; they diagnose individual axes without decomposing certificate outcomes. RQ3 assigns removed records by semantic > implementation > measurement priority, differing from Section 2’s outcome-code priority. Study Denominators. Decidability is (PASS+FAIL)/𝑁 ; RQ3’s measurable records have reportable, positive speedups. In RQ4, eligibility divides admitted slots by all slots; coverage divides shipped tasks by all tasks. Statistical Protocol. Counts describe fixed audited populations conditional on recorded decisions. Blind-review point estimates use frozen sampling weights; 95% intervals quantify model-based posterior uncertainty under finite-population Jeffreys–Dirichlet completion (Appendix H), sharing label probabilities within strata. Timing stability and intervals follow Section 3.4 (resampling in Appendix A). Paired McNemar tests [21] use Holm correction [9] to adjust exact binomial 𝑝-values. Verifier tests pair acceptance on FAIL-certificate records separately for the natural-five pool (first five corpus blocks in Table 3) and all seven blocks. Selector tests would pair unsafe-shipment outcomes by task; the refreshed RQ4 comparisons are withheld because one selected candidate lacks a human label. Tests assume independent paired units. Source reuse across reference layers can violate independence, so verifier 𝑝-values are record-level reference calculations rather than independentkernel inference (Section 9). Decided-record error rates exclude INCONCL. Best- and worst-case false-accept bounds hold weak decisions and resolved certificates fixed while assigning unresolved accepted records to PASS or FAIL (Appendix B). These bounds do not correct errors in resolved labels. RQ4 retains all 232 tasks in shipping-rate and coverage denominators, including abstentions, and separately reports

10

Ziyuan Li et al.

Table 5. Oracle decisions by corpus. Selected axis-FAIL counts may overlap and cite different executions. Certificate PASS/FAIL/INCONCL columns partition each corpus using their own source records and eligibility conditions (Section 3.5); aggregating the displayed axes does not reconstruct them. Corpus

N Sem. FAIL Impl. FAIL Meas. FAIL PASS FAIL INCONCL

CUDA-L1 241 AI-CUDA-Engineer 229 KernelBenchX 63 TritonBench-G (self-built ref.) 89 TritonBench-G (upstream ref.) 26 Delegation controls 60 robust-kbench handwritten 8

25 35 11 14 16 16 1

136 93 22 7 1 52 2

0 0 0 0 0 0 0

2 94 23 36 8 4 5

148 120 32 21 17 56 3

91 15 8 32 1 0 0

Total

118

313

0

172

397

147

716

unsafe shipments among actual deliveries. RQ4 bounds assign its one unknown shipment clean or unsafe; they are identification bounds, not confidence intervals. Use of AI Tools (Disclosure). AI-generated and released candidates receive the same audit policy. Claude Opus 4.5 generated 696 RQ4 candidates; prompts and settings are in the artifact. Human annotators establish the blind reference labels. AI coding assistants helped engineer tooling and prepare the manuscript. OpenAI image-generation tools assisted layout and rendering of Figures 1 and 2. Infrastructure. Local experiments use an NVIDIA A800 GPU (80 GB), PyTorch 2.1.0, CUDA 12.1, and Python 3.10 on Ubuntu 22.04. Settings are fixed per experiment. The primary audit uses fresh seeded workers; historical RQ4 evidence uses a shared-process batch runner. The RQ4 refresh reuses complete primary-audit evidence for 204 slots without new GPU execution and retains historical evidence for 956 slots. GPU clocks on the shared timing host are unlocked (Appendix A). 5

RQ1: Prevalence and Structure of False Accepts

Two Conditional Rates. False-accept and miss rates share a numerator but condition on accepted, oracle-decided records and FAIL certificates, respectively (Section 4). Both use study contracts that may differ from generators’ acceptance rules. Decidability. The oracle decides 569/716 records (79.5%; Table 5). CUDA-L1 has the lowest decidability, 150/241 (62.2%); the other corpora range from 64.0% to 100.0%. The rates below describe decided subsets; Section 6.5 examines the unresolved evidence. Prevalence: False Accepts. Under the study contracts, pooling all seven corpora, including controls and handwritten kernels, yields false-accept rates of 51.7%–66.9%. The two main generated-CUDA corpora and the natural-five pool yield 56.1%–70.8% and 49.7%–63.8%, respectively (Table 6). Pooled rates weight corpora by each verifier’s decided acceptance counts. Distribution of Axis Evidence. The balance of selected axis-FAIL evidence varies by corpus. Implementation evidence exceeds semantic evidence on the main corpora (FAimp 51.9%–67.5% versus FAsem 4.0%–10.1%) and KernelBenchX (37.5%–51.4% versus 2.7%–12.5%). The ordering reverses for TritonBench-G’s self-built reference layer (FAsem 10.0%–20.4% versus FAimp 4.3%–9.1%). The handwritten set also contains two accepted records with implementation-FAIL evidence under our strict contract reading (Section 6.1).

KernelAudit

11

Table 6. False accepts by corpus and pool. Oracle cells 𝑎/𝑏 count FAIL certificates among accepted, oracledecided records. Pooled FAany and selected-axis FAsem /FAimp share that denominator; axes do not decompose certificates. Measurement-axis FA is zero. Human rows give design-weighted estimates and 95% model-based posterior intervals for the same main-corpus decided-accepted sets under the independent rubric. Both pools exclude controls. Certificate detail and abstention bounds: Appendix B; per-corpus axis rates: released data. robust-kb. = robust-kbench, DeepReinf. = DeepReinforce. Corpus (decided 𝑁 ) CUDA-L1 (150) AI-CUDA-Engineer (214) KernelBenchX (55) TB-G self-built (57) TB-G upstream (25) Delegation controls (60) rk handwritten (8)

CUDA-Agent KernelBench KernelGYM robust-kb. DeepReinf. MusaCoder 23/23 97/191 17/32 8/40 1/8 55/59 2/7

117/117 86/174 21/38 15/49 1/8 56/60 2/7

Main (364) FAany % Sem.-evidence / Impl.-evidence % Human FA % 95% posterior interval

120/214 56.1 8.4/51.9 44.2 [29.9,50.0]

203/291 69.8 4.1/67.4 49.7 [33.3,53.3]

Natural-5 (501) FAany % Sem.-evidence / Impl.-evidence %

146/294 49.7 8.8/43.2

240/386 62.2 6.2/57.0

136/138 97/191 23/40 15/49 1/8 56/60 2/7

110/110 83/167 20/37 10/44 1/8 39/43 2/7

75/75 98/192 23/40 14/48 1/8 56/60 2/7

39/39 98/192 16/32 12/46 1/8 1/3 2/7

233/329 193/277 70.8 69.7 9.1/66.6 4.0/67.5 51.6 46.1 [37.4,56.8] [28.5,50.5]

173/267 64.8 10.1/59.2 50.6 [36.7,57.3]

137/231 59.3 8.7/55.0 50.6 [35.5,55.4]

272/426 63.8 10.3/57.0

211/363 58.1 11.0/50.1

166/317 52.4 10.4/44.8

224/366 61.2 5.2/57.7

Unresolved Records and Independent Labels. Sensitivity bounds extend the main-pool analysis to all accepts by assigning accepted INCONCL records to PASS or FAIL. The lower endpoints range from 49.2% to 60.3% and the upper endpoints from 61.5% to 75.3%, holding resolved certificates fixed without correcting their errors. Table 6 separately reports OV120’s weighted human-referenced estimates (44.2%–51.6%) and posterior intervals for the decided-accepted sets. Section 6.3 gives the sampling, model, and rubric differences. Miss Rates and False Rejects. On the main pool, verifiers accept 120–233 of 268 FAIL-certificate records (miss rates 44.8%–86.9%) and reject 0–12 of 96 PASS records. Natural-five miss rates are 43.2%–80.5% among 338 FAIL records, with 9–21 false rejects among 163 PASS records. Under an independent-record model, 14 of 15 McNemar contrasts have Holm-adjusted 𝑝 < 0.05; the exception is robust-kbench versus DeepReinforce. Source reuse across reference layers limits this inference (Section 4; Appendix B). Constructed Controls and Ranking. Constructed controls change verifier rankings because their delegation pattern matches MusaCoder’s fallback scan. MusaCoder accepts one of 56 FAIL-certificate controls, compared with 39–56 for other verifiers. Adding controls to the natural-five pool gives 394 FAIL records and miss rates of 42.4%–83.2%. MusaCoder moves from second-lowest (166/338, 49.1%) to lowest (167/394, 42.4%). The handwritten set’s three FAILs are excluded from this pool but enter the all-seven-corpus analysis (𝑛=397; Appendix B). Numerical Signals and Test Coverage. The nominal-input diagnostic uses stored evidence for 62 semantic-miss records, including 16 delegation controls. Across 15 policies per record, 57 records have at least one maximum relative error above 10−5 , four further records have non-numeric signals, and one has neither (Appendix B). The same relative-error criterion flags 69/172 PASS-certificate

12

Ziyuan Li et al.

Table 7. Constructed behavioral controls under fixed audit policies (Appendices E–F). Sealing before result inspection and separation from threshold development are unverified; capture does not establish population recall or specificity. Control family

Surface

Delegation probes

Impl.

Result Misses / notes

52/60 semantic axis adds 4 (56/60 overall); 4 joint misses confirmed by counterfactual replay Launch battery (1 pos., 3 neg.) Launch 1/1; 3/3 the positive is a below-dispatcher cuBLAS call; no launch flag on negatives; mediated calls remain subject to Channel 1 Challenge positives Sem. 10/12 misses: stale cache and shape specialization; zero INCONCL Challenge negatives Sem. 4/4 no false conviction Timing-tamper probes Meas. 2/2 attributable mutations can convict Hidden-stream probe Meas. 0/1 undetected; guard can only withhold

records (40.1%). Because it omits the absolute-tolerance term, these flags are not measured rejections by the full comparator. The remaining record, exp_sqrt, passes the first repeated-run invocation but fails the second and third (maximum relative error 17.8). The nominal diagnostics miss this repeated-execution failure. Answer to RQ1. Across six verifiers, 56.1%–70.8% of accepted, oracle-decided main-corpus records receive FAIL under the study contracts. Independent human-referenced estimates are 44.2%–51.6% on the same sets. Selected implementation-FAIL evidence is more frequent than semantic-FAIL evidence. When checked outputs are identical, output comparison cannot distinguish compliant computation from forbidden delegation at any tolerance.

6 6.1

RQ2: Axis Attribution and Oracle Validity Implementation Axis: Two Complementary Evidence Channels

Of 313 selected implementation FAIL decisions, 299 use only the dispatcher probe, 6 only launch provenance, and 8 both (Appendix E). On the main corpora, 229/268 FAIL-certificate records carry selected implementation FAIL evidence (85.4%). Eight dispatcher misses involve GEMV work undercounting and unknown-operation paths (Table 7; Section 3.3). The 5%/30% targeting shares do not guarantee detection at those fractions of actual delegated work. Case 1: Provenance and Contract Interpretation. In robust-kbench, the handwritten llama_ffw passes all 16 semantic policies with an empty dispatcher trace. Vendor launches account for 99.6% of observed GPU time; all six weak verifiers accept. Our strict self-implementation policy yields C3b. Both annotators label it clean because “unchanged GEMMs” declares the library work their rubric permits. This case is one of seven reported policy disagreements affecting specificity despite agreement on execution (Section 4; Cases 2–3 in Appendix E). Case 4: Withheld Certification and an Independent Semantic Miss. On cuda_l1__L2_T80, the reference is identically zero. The candidate replaces its class-level __call__ with a cached-zero return; initial outputs match without observed candidate GPU work. R-B stops the certificatebound execution before semantic testing, yielding INCONCL. Selected semantic PASS cites a separate record with 16 passing policies. Independent re-execution finds a new instance returning an uninitialized None cache instead of the reference tensor. This witness supplies the human label

KernelAudit

13

Table 8. Oracle agreement with the independent blind re-execution rubric (𝑛=120): design-weighted estimates and 95% model-based posterior intervals (Appendix H). Metric

Est. [95% interval]

Sensitivity 0.964 [0.917, 0.967] NPV 0.934 [0.848, 0.943]

Metric

Est. [95% interval]

Specificity 0.626 [0.518, 0.673] Unknown 3.9% [3.9, 11.2]

Metric

Est. [95% interval]

PPV 0.763 [0.638, 0.810] INCONCL defect 39.5% [29.3, 57.8]

without entering automatic verdicts: the audit misses the defect while withholding certification (Appendix H). R-B cannot erase an established FAIL. 6.2

Measurement Evidence and Timing Credibility

The 716-record audit detects no measurement violation, but control results limit this finding. Timing-primitive checks detect both tampering controls. The stream guard misses its positive control and can only withhold certification (Table 7). Selected measurement sources contain guard observations for 672 records (651 PASS, 21 INCONCL), leaving 44 without observations. Each PASS has four required guard PASS results (Appendix D). The fixed timing source monitors 577 records with no detected primitive mutation. The recorded stream ratios reach 1.75 against a 3.0 threshold. A/A withholding affects speedup reportability, as examined in Section 7. 6.3

Validity Against an Independent Blind Reference

Independent blind review assesses agreement under a separate human rubric. Two annotators independently inspect artifacts and re-executions without oracle or verifier decisions. Disagreements on 29/120 units require third-party arbitration. The rubric permits explicitly declared library computation, so disagreements combine policy interpretation and execution judgments (Case 1). The confusion matrix cannot isolate detection errors under a shared contract. Delegation agreement is 80.0% (𝜅=0.582), and the reference remains fallible. Sampling draws 45/45/30 PASS/FAIL/INCONCL records from certificate×corpus strata. Inclusion probabilities determine the weights. Evaluated certificates define 47/54/19 metric cells (Appendix H). Metric denominators separate decided judgments from unresolved evidence. The four classification metrics use human-known, oracle-decided records. Unknown labels and INCONCL are reported separately. The INCONCL-defect share includes the entire cell, including unknowns. Table 8 describes overall OV120, not main-corpus-only specificity. Posterior intervals assume shared label probabilities across certificate cells within each design stratum. The intervals condition on adjudicated labels and exclude annotation and contractinterpretation uncertainty. Sensitivity to the exchangeability assumption remains untested (Appendix H). 6.4

Comparison with Runtime Attribution

We compare implementation verdicts with an Atrex-based attribution screen on the same 120 records (110 natural, 10 constructed). Implementation-only human labels are 34 violations, 80 negatives, and 6 unknowns, distinct from the full-certificate labels above. Baseline and Protocol. The screen flags suspected delegation when summed matched-kernel device time per forward divided by independently measured median end-to-end latency is ≤ 0.5. This screening rule is not an Atrex verdict and adds no causal or contract test. We reuse pinned Atrex [40] timing, ratio, and internal-error modules, adapting task/state initialization, exact CUDA/Triton symbol registration, and outer exceptions (Appendix H). Attribution is unavailable for 9 records, 6 with known human labels.

14

Ziyuan Li et al.

Table 9. Implementation comparison on the 114 records with known human delegation labels; 6 humanunknown records are excluded. Positives denote undeclared delegation, predicted from low attribution or implementation FAIL. Coverage counts decisions; abstentions are neither clean nor detected violations. Method

TP

FP

FN

TN

Abstain

Coverage

Our implementation axis Atrex-based screen

29 33

14 48

2 0

55 27

14 6

87.7% 94.7%

Errors and Coverage. The oracle reduces false positives at the cost of detection and coverage (Table 9). The implementation axis detects 29/34 human-positive records versus 33/34 for the screen, and decides 100/114 known-label records versus 108/114. Among 98 jointly decided records, only the oracle agrees with the rubric on 29 and only the screen on 2 (controls missed by targeting). These comparisons remain sample-conditional. Low attribution alone cannot establish forbidden work because end-to-end latency includes time outside observed target kernels (Appendix H). 6.5

INCONCL: Unresolved Evidence and Decidability

The ledger contains 147 INCONCL certificates (20.5%), while 286 records have 445 unresolved selected axes (diagnostic categories in Appendix C). These axis labels neither partition certificate causes nor assign responsibility. An unresolved channel does not erase another channel’s independent FAIL evidence. Eight of the 19 sampled INCONCL records have human defect labels. Dispatcher probes likewise yield implementation FAIL on 17 executed PyTorch fallback paths among 46 CUDA-L1 build-failure candidates (Appendix E). Answer to RQ2. Against the blind reference, design-weighted sensitivity is 0.964, NPV 0.934, and specificity 0.626 on decided, human-known records. The implementation axis has fewer false positives than the attribution screen, with lower detection and coverage. Labels remain policy-relative, and Case 4 demonstrates a missed semantic defect despite withheld certification. PASS denotes no detected violation on audited surfaces.

7

RQ3: Credibility and Certification of Speedup Claims

RQ3 reports two descriptive quantities: the claim-to-local contrast, the median record-wise ratio 𝑚𝑖 /𝑐𝑖 of reportable local speedup to upstream claim; and certificate survival, the paired cohort’s PASS fraction. Pairs link corresponding candidates across settings, with reference, input, and state conditions not uniformly matched. The contrast neither isolates hardware or toolchain effects nor establishes upstream claim overstatement. Of 577 records with A/A results, 394 speedups are withheld and 183 records pass. Three passes lack positive recorded speedups, leaving 180 measurable records (Section 3.4; details in Appendix D). Across all seven corpora, these 180 records receive 32 PASS, 121 FAIL, and 27 INCONCL certificates. Among the measurable records, 64 CUDA-L1 and 68 AI-CUDA-Engineer records with upstream claims form the 132-record main paired cohort. A/A withholding affects speedup reportability, not certificate status. Most records in the paired main cohort remain uncertified (Table 10). Of 132 records, 22 receive PASS, 93 receive FAIL, and 17 remain INCONCL. CUDA-L1’s two PASS certificates lie outside its 64-record paired cohort. Implementation violations account for most uncertified records under the semantic > implementation > measurement priority (Section 4). Using certificate-bound complete records, the 110 uncertified records comprise 24 (21.8%) semantic violations, 69 (62.7%) implementation violations, 0 measurement violations, and 17 (15.5%) unresolved cases. Implementation assignments exclude semantic overlaps, and unresolved cases are not counted as violations.

KernelAudit

15

Table 10. Paired speedups across upstream/local evaluation settings and certificate survival; med.(𝑚/𝑐) is the median of within-record ratios. The last column gives PASS/𝑛 and the PASS subset’s claimed and measured medians. The TritonBench-G row has no upstream claims. PASS denotes no detected violation on the audited surfaces. Corpus and timing cohort

𝑛 Claimed med. Measured med. med.(𝑚/𝑐)

CUDA-L1 AI-CUDA-Engineer Main pooled

64 68 132

1.413× 1.143× 1.283×

1.130× 1.051× 1.094×

TritonBench-G self-built (no claims)

22

—

1.793×

PASS (med. 𝑐, med. 𝑚)

0.920 0 / 64 0.958 22 / 68 (1.146×, 1.505×) 0.935 22 / 132 (1.146×, 1.505×) —

8 / 22 (—, 1.647×)

Certificate evidence does not establish that violations caused recorded speedups. All 22 PASS certificates share complete records with their timings, but 54 uncertified records (37 FAIL, 17 INCONCL) use different sources. Candidate code is linked across these sources, but full input/state equality is not established for every pair (Appendix D). Certification does not require a speed advantage. Li et al. [16] independently examine the gap between numerical correctness and performance as a replacement for optimized library operators. The PASS-subset medians in Table 10 describe selected records, not an audit-induced improvement. In TritonBench-G’s self-built layer, 8/22 measurable records receive PASS, including records at 0.19× and 0.87×. Timing eligibility limits the scope of the paired summaries. A/A passage is associated with reference duration on the shared, unlocked-clock host (Appendix D). The association does not isolate workload, measurement overhead, or clock effects. Main paired summaries require reportable timings and upstream claims, while PASS-subset medians additionally condition on certification (Section 9). Answer to RQ3. On the paired main-corpus cohort, the median measured/claimed ratio is 0.935 and certificate survival is 22/132 (16.7%). Implementation violations account for 69 of 110 uncertified records under the stated priority, while 17 remain inconclusive. These are conditional record-level results, and certification does not guarantee a speed advantage.

8

RQ4: A Training-Free Gate Swap

RQ4 compares shipping rules over fixed candidates and saved evidence, without retraining, resampling, or candidate repair. Design. The pool contains 232 tasks × 5 slots = 1,160 candidates: 206 CUDA tasks selected by the resource plan from 220 joined tasks, and all 26 joined TritonBench tasks (Appendix G). Slots follow CUDA-L1, AI-CUDA-Engineer, then Opus c0–c2 for CUDA; TritonBench uses G, pref, then Opus c0–c2. All five share the first registered source’s reference. Unavailable evidence remains INCONCL; all slots and tasks stay in the denominators. We reconcile 204 slots with complete primary-audit records after matching candidate source, task, reference, and policy, including verified state adapters. The other 956 retain historical RQ4 evidence; this reconciliation reuses evidence without a new GPU run of the pool. Certificates never combine axes from different records. The oracle-visible 2×2 comparison crosses eligibility with first/fastest ordering. 𝑆 0 and 𝑆 0 -fast require the corresponding frozen weak verifier’s PASS. 𝑆 1 -first and 𝑆 1 require PASS in the reconciled eligibility ledger. For reused records, eligibility requires the complete-record PASS certificate; historical slots retain saved three-axis PASS plus R-B. Axis-only 𝑆 axis -first and 𝑆 axis instead require three-axis PASS. The historical evidence lacks some current certification checks, so this mixed-evidence study does not evaluate full certification on every slot.

16

Ziyuan Li et al.

Table 11. Gate-swap results on 232 tasks. Oracle-visible unsafe is C2∪C3∗; human unsafe uses independent labels. Rates divide by 232 except parenthesized human unsafe/shipped. Lower-panel bounds assign the one unknown shipment clean or unsafe; they are identification bounds, not confidence intervals. Weak verifier

𝑆 0 unsafe

𝑆 0 C3∗ 𝑆 0 /𝑆 0 -fast cov. 𝑆 0 -fast unsafe Human 𝑆 0 unsafe

CUDA-Agent 11.6% (27) 9.5% (22) KernelBench 39.7% (92) 37.1% (86) KernelGYM 48.7% (113) 45.3% (105) robust-kbench 40.9% (95) 39.7% (92) DeepReinforce 28.4% (66) 24.1% (56) MusaCoder 21.1% (49) 17.7% (41) Audit selector

93/232 183/232 210/232 173/232 171/232 167/232

9.1% (21) 38.8% (90) 47.0% (109) 39.7% (92) 25.9% (60) 20.3% (47)

6.0% (15.1%) 27.6% (35.0%) 31.5% (34.8%) 30.2% (40.5%) 22.4% (30.4%) 21.6% (29.9%)

Coverage Human unsafe/232 Human unsafe/shipped

Axis-only 𝑆 axis 63/232 (27.2%) RQ4 gate 𝑆 1 62/232 (26.7%)

[1,2]/232 [1,2]/232

[1,2]/63 (1.6–3.2%) [1,2]/62 (1.6–3.2%)

Fastest selects the largest usable cached local speedup, breaking ties by fixed order. Speedups require measurement PASS, inactive guards, and A/A and paired A/B checks; changed adapters cannot inherit timing without revalidation (Appendix G). No new timing is measured. If no eligible slot has timing, the first is selected; if none is eligible, the selector abstains. Of 𝑆 1 ’s 62 deliveries, 15 use timing and 47 use fallback. Human labels do not enter selection. The historical 330-unit blind census covers all but one candidate in the refreshed 331-unit delivery union (Appendix H). The two annotators used Section 6.3’s rubric; packets hid selector membership. There were 73 arbitrated units; delegation agreement was 81.2% (𝜅=0.549). The refresh follows that review and does not claim a new prospective blind study. We retain the missing label as unknown and report unsafe/232, unsafe/shipments, and coverage; abstention is not successful delivery. Batch Timing Attribution. Isolated execution does not substantiate candidate attribution for the 114 historical batch timing signals under R-C: ratios have median 1.000 and maximum 1.046 (Appendix G). Their speedups remain unavailable for ranking. Evidence Availability Limits Coverage. The reconciled pool has 184 eligible, 789 inconclusive, and 187 FAIL slots. Eligible slots occur on 62 tasks, bounding 𝑆 1 coverage at 62/232 (26.7%) regardless of ordering, versus 𝑆 0 ’s 40.1%–90.5%. Oracle-Visible Gate and Ranking Effects. Eligibility has a larger effect than ordering on oraclevisible unsafe shipments. Contrasts average over the other factor and equally over six verifiers: 30.9 pp for eligibility (range 10.3–47.8) and 0.8 pp for ordering (Appendix G). Both audit-gated cells have zero unsafe shipments by definition; these contrasts do not independently validate the audit labels. Human-Labeled Shipment Outcomes. 𝑆 1 ships 60 candidates labeled clean, one labeled unsafe, and one without a label. The known unsafe shipment is CUDA-L1 L2 T14 (BR-203), labeled undeclared delegation despite its reused PASS certificate. The unreviewed shipment is TritonBench 109_softmax_optimize, slot 1. Thus human-unsafe deliveries lie between 1 and 2, or 1.6%–3.2% of shipments and 0.4%–0.9% of tasks. These bounds remain below the six 𝑆 0 rates, while eligibility, ordering, and abstention change jointly. We withhold paired human 𝑝-values until the missing label is resolved; the previous census’s tests do not describe the refreshed selections. The mixed-evidence gate does not establish zero unsafe delivery.

KernelAudit

17

Delivery with Speedup. SafeFast@𝑘 is the share of all 232 tasks shipping an 𝑆 1 -eligible candidate with credible speedup > 𝑘×. Changing first to fastest raises SafeFast@1 from 4/232 to 6/232 (2.6%), while SafeFast@2 stays at 2/232 (0.9%). 𝑆 0 -fast achieves SafeFast@1 on 3–5 tasks and SafeFast@2 on two (Appendix G). Cost Diagnostics. Historical cache-write intervals provide indirect diagnostics for one 46-task segment. Strong-/weak-cache p50s are 7.460 min and 77.142 s (≈ 5.8×; Appendix G). These unpaired, differently filtered intervals do not measure the refresh, chosen-candidate post-filtering, net additional latency, or full-pool screening cost. Answer to RQ4. Audit-based selection lowers human-unsafe delivery in this fixed pool alongside lower coverage. 𝑆 1 covers 26.7% of tasks, with one known unsafe and one unknown among 62 shipments: the unsafe fraction is bounded by 1.6%–3.2%.

9

Threats to Validity

Internal. RQ4’s shared-process timing signals require isolated attribution checks; unresolved channels cannot support conviction (Section 8). With unlocked clocks, ABBA and A/A stability do not establish unbiased timing or drift bounds. RQ3 conditions on reportable timings and upstream claims, plus certification for PASS-subset summaries. Unmatched reference/input/state conditions preclude isolating environmental effects or attributing speedup to violations from these contrasts alone (Section 7). Construct. Labels concern policy-relative decisions, not intent; permitted fixed-shape optimization alone cannot convict. PASS means no detected violation on audited surfaces. Measurement conviction uses only positively controlled timing-primitive checks; runtime precision-mode evidence is unimplemented. Dispatcher checks do not establish control-flow equivalence. The sealed launch inventory covers cuBLAS/cuDNN only. Tests cover finite inputs and call sequences; adaptive evasion remains unevaluated. R-A requires observed candidate work; R-B withholds certification for listed degenerate-reference tasks. Neither prerequisite establishes a violation. Detection at the fixed 5%/30% targeting shares depends on work proxies and supported operations (Section 6.1). The audit and human rubrics disagree on declared delegation in Case 1; agreement metrics do not isolate detection errors (Appendix H). OV120 validity intervals share label probabilities across certificate cells within each original sampling stratum. Exchangeability sensitivity remains untested; intervals fix adjudicated labels and do not quantify annotation or contract-interpretation uncertainty (Section 6.3). External and Statistical Conclusion. The audit of 716 records characterizes the evaluated adapters. Its results do not generalize to generator populations or other GPU architectures. DeepReinforce and MusaCoder use publication-derived reconstructions; the Atrex-based comparison covers implementation decisions on 120 records, with attribution availability dependent on the environment. Core self-implementation is our study policy, not a uniform upstream requirement. TritonBench source reuse can violate McNemar’s independent-pair assumption (Section 4). Protocols are not externally preregistered. RQ4’s fixed, resource-planned pool contains 1,160 slots across 232 tasks. Its reconciled gate covers 62 tasks, with one known unsafe and one unknown shipment; unsafe/shipments is bounded by 1.6%–3.2%. The 204 slots with reused complete evidence and 956 historical slots retain different evidence scopes; this refresh is not a uniform full-certification rerun. The refresh follows the original blind census, whose 330 labels do not cover one new delivery. Paired human tests are withheld, and lower unsafe delivery does not hold coverage fixed. Prospective risk and deployment overhead remain unevaluated (Section 8).

18

10

Ziyuan Li et al.

Related Work

LLM-Generated GPU Kernels and Validators. KernelBench [25] and TritonBench [15] established execution-based evaluation for LLM-written CUDA/Triton kernels; robust-kbench [13], KernelGYM [19], CUDA-Agent [7], MusaCoder [6], and DeepReinforce [24] add stronger evaluation, RL environments, fallback checks, and timing-attack defenses. GEAK [39], the AI CUDA Engineer [12], CUDA-L1 [17], and KernelBenchX [38] study agentic and RL optimization. Recent work [20, 33– 35, 37, 41] adds fp64-backed fuzzing, hidden tests, degenerate-task exclusion, verified solutions, harness engineering, contract languages, and a twelve-gate audit of 2,638 accepted kernels; our results show why output checks alone leave implementation violations unresolved. Atrex’s production-derived benchmark [40] is the closest runtime-attribution comparison: its target-DSL time ratio exposes fallback use beyond correctness. Section 6.4 tests that signal against independent delegation labels; our combined implementation axis has fewer false positives under that rubric, with lower detection and coverage. Formal verification targets race/barrier safety [4] or semantic equivalence [8] rather than reward validators. Our intervention builds on contract vocabulary, semantic re-auditing, runtime attribution, causal testing, fault injection, and dynamic slicing [1, 10, 11] to audit six verifiers, account for abstention, and evaluate a shipping gate. GPU Defects and Performance Evaluation. Studies of tile-program code-generation bugs [28] and GPU numerical bugs [27] characterize input-sensitive failures, their causes, and detection strategies; the latter also presents GPU-NBDetect. Li et al. [16] find that numerically valid Triton and TileLang kernels can remain much slower than optimized library operators, and examine libraryrelative efficiency and roofline-based checks. These studies complement our focus on validators’ contract-dependent acceptance decisions and the credibility of their timing evidence. Executable Code Evaluation and Oracle Design. CodeRL [14] uses execution feedback. EvalPlus [18] exposes inadequate tests. Oracle design [3] and metamorphic testing [5] address the reliability of such judgments. GPU rewards additionally require evidence about implementation and measurement. Reward hacking [2, 36] and learned verifier exploitation [32] motivate auditing these non-output obligations before deploying a validator as a gate. Model Uncertainty and Selective Prediction. CodeImprove [31] uses a validity score to identify inputs beyond a code model’s effective scope and adapts them through semantics-preserving program transformations. For LLM-based code generation, Rathnasuriya and Yang [30] report that confidence and entropy provide model-dependent evidence of execution failure, and uncertainty-based selection does not consistently improve accepted-set accuracy. Their study of defect prediction and vulnerability detection [29] also finds that improved post-hoc calibration scores need not improve selective prediction. These approaches assess model-derived reliability signals. In KernelAudit, INCONCL instead records unmet auditability or certification conditions; RQ4 evaluates unsafe delivery together with coverage under those evidence constraints. 11

Conclusion

In this paper, we have contributed a contract model and an audit oracle for execution-based validators in LLM-driven GPU-kernel generation. The model separates semantic, implementation, and measurement obligations. The oracle combines reference-calibrated semantic tests, complementary dispatcher interventions and launch provenance, and timing guards, with explicit abstention for unresolved checks. A cross-verifier audit and independent human evaluation assess validator blind spots and audit validity under the study policies. A separate fixed-pool study examines how eligibility and ordering affect unsafe delivery and coverage. Together, these contributions support

KernelAudit

19

testing reward and shipping decisions against task obligations and execution evidence on audited surfaces. Acknowledgments This work was partially supported by the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM118) and the National Natural Science Foundation of China under Grant Nos. U25A6023 and 92464301. Tao Xie is also affiliated with Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, China; School of Computer Science, Peking University, China; Institute of Systems for Advanced Computing, Fudan University, China; Shanghai Institute of Systems for Open Computing, China; School of Computer Science and Engineering, Southeast University, China.

20

Ziyuan Li et al.

References [1] Hiralal Agrawal and Joseph R Horgan. 1990. Dynamic program slicing. ACM SIGPLAN Notices 25, 6 (1990), 246–256. [2] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016). [3] Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering 41, 5 (2015), 507–525. doi:10.1109/TSE.2014.2372785 [4] Adam Betts, Nathan Chong, Alastair Donaldson, Shaz Qadeer, and Paul Thomson. 2012. GPUVerify: A verifier for GPU kernels. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications. 113–132. [5] Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR) 51, 1 (2018), 1–27. [6] Kun Cheng, Songshuo Lu, Sicong Liao, Tankun Li, Yafei Zhang, Dong Yang, Qiheng Lv, Hua Wang, Zhi Chen, and Yaohua Tang. 2026. MusaCoder: Native GPU kernel generation with full-stack training on Moore Threads GPU. arXiv preprint arXiv:2606.04847 (2026). [7] Weinan Dai, Hanlin Wu, Qiying Yu, Huan-ang Gao, Jiahao Li, Chengquan Jiang, Weiqiang Lou, Yufan Song, Hongli Yu, Jiaze Chen, et al. 2026. CUDA Agent: Large-scale agentic RL for high-performance CUDA kernel generation. arXiv preprint arXiv:2602.24286 (2026). [8] Benjamin Driscoll, Kshitij Dubey, Anjiang Wei, Neeraj Kayal, Rahul Sharma, and Alex Aiken. 2025. Equivalence checking of ML GPU kernels. arXiv preprint arXiv:2511.12638 (2025). [9] Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65–70. [10] Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. doi:10.1109/TSE.2010.62 [11] Brittany Johnson, Yuriy Brun, and Alexandra Meliou. 2020. Causal testing: Understanding defects’ root causes. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 87–99. [12] Robert Tjarko Lange, Aaditya Prasad, Qi Sun, Maxence Faldor, Yujin Tang, and David Ha. 2025. The AI CUDA engineer: Agentic CUDA kernel discovery, optimization and composition. Technical Report. Sakana AI. [13] Robert Tjarko Lange, Qi Sun, Aaditya Prasad, Maxence Faldor, Yujin Tang, and David Ha. 2025. Towards robust agentic CUDA kernel benchmarking, verification, and optimization. arXiv preprint arXiv:2509.14279 (2025). [14] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems 35 (2022), 21314–21328. [15] Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, Haojie Wang, Jianrong Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. TritonBench: Benchmarking large language model capabilities for generating Triton operators. In Findings of the Association for Computational Linguistics: ACL 2025. 23053–23066. doi:10.18653/v1/2025.findings-acl.1183 [16] Tingxi Li, Ravishka Rathnasuriya, and Wei Yang. 2026. Correct but slow: An empirical study of the GPU kernel evaluation gap in modern domain-specific languages. arXiv preprint arXiv:2607.04454 (2026). https://arxiv.org/abs/ 2607.04454 [17] Xiaoya Li, Xiaofei Sun, Albert Wang, Jiwei Li, and Chris Shum. 2026. CUDA-L1: Improving CUDA optimization via contrastive reinforcement learning. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=igZItUbY6n [18] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2023), 21558–21572. [19] Wei Liu, Jiawei Xu, Yingru Li, Longtao Zheng, Tianjian Li, Qian Liu, and Junxian He. 2026. Dr. Kernel: Reinforcement learning done right for Triton kernel generations. arXiv preprint arXiv:2602.05885 (2026). [20] Guido Martínez and Tyler Sorensen. 2026. The next frontier for AI-generated kernels: Correctness. In Proceedings of the 2026 ACM SIGPLAN International Workshop on Principles of Agentic Engineering. 25–37. [21] Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 2 (1947), 153–157. [22] Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter F Sweeney. 2009. Producing wrong data without doing anything obviously wrong! ACM SIGPLAN Notices 44, 3 (2009), 265–276. [23] NVIDIA. [n. d.]. CUDA profiling tools interface (CUPTI). Online documentation. Retrieved October 21, 2026 from https://docs.nvidia.com/cupti/ [24] Ornith Team. 2025. Hacks and defenses in automatic GPU kernel generation. Blog post. Retrieved September 29, 2026 from https://ornith.ai/defense_kernel_hack.html

KernelAudit

21

[25] Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. 2025. KernelBench: Can LLMs write efficient GPU kernels? arXiv preprint arXiv:2502.10517 (2025). [26] Judea Pearl. 2009. Causality: Models, reasoning, and inference (2 ed.). Cambridge University Press. doi:10.1017/ CBO9780511803161 [27] Ravishka Rathnasuriya, Nidhi Majoju, Zihe Song, and Wei Yang. 2025. An investigation on numerical bugs in GPU programs towards automated bug detection. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1654–1677. doi:10.1145/3728950 [28] Ravishka Rathnasuriya, Zihe Song, Nidhi Majoju, Tingxi Li, Aaryaa Moharir, Wei Yang, and Tao Xie. 2026. Characterizing real-world bugs in tile programs for automated bug detection. Proceedings of the ACM on Software Engineering 3, ISSTA (2026), 4118–4142. doi:10.1145/3832271 [29] Ravishka Rathnasuriya and Wei Yang. 2026. Beyond confidence: Rethinking uncertainty calibration in deep code models. arXiv preprint arXiv:2610.21248 (2026). https://arxiv.org/abs/2610.21248 [30] Ravishka Rathnasuriya and Wei Yang. 2026. Characterizing overconfident failure in LLM-based code generation. arXiv preprint arXiv:2610.11300 (2026). https://arxiv.org/abs/2610.11300 [31] Ravishka Rathnasuriya, Zijie Zhao, and Wei Yang. 2025. CodeImprove: Program adaptation for deep code models. In Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering. 514–526. doi:10.1109/ ICSE55347.2025.00139 [32] Jaideep Ray. 2026. Before the model learns the bug: Fuzzing RLVR verifiers. arXiv preprint arXiv:2606.01066 (2026). [33] Dipankar Sarkar. 2026. The correctness illusion in LLM-generated GPU kernels. arXiv preprint arXiv:2606.20128 (2026). [34] Rishi Shah and Rishav Shrestha. 2026. A contract-grade verifier for LLM-generated GPU kernels, and a native Blackwell backward for the gated-linear-recurrence family. arXiv preprint arXiv:2608.12700 (2026). [35] Yue Shui, Chenyu Ma, Hangfei Xu, Shengzhao Wen, and Yanpeng Wang. 2026. Harness engineering for LLM-driven GPU kernel generation. arXiv preprint arXiv:2607.17979 (2026). [36] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems 35 (2022), 9460–9471. [37] Cooper Veit. 2026. Kernel contracts: A specification language for ML kernel correctness across heterogeneous silicon. arXiv preprint arXiv:2604.22032 (2026). [38] Han Wang, Jintao Zhang, Kai Jiang, Haoxu Wang, Jianfei Chen, and Jun Zhu. 2026. KernelBenchX: A comprehensive benchmark for evaluating LLM-generated GPU kernels. arXiv preprint arXiv:2605.04956 (2026). [39] Jianghui Wang, Vinay Joshi, Saptarshi Majumder, Xu Chao, Bin Ding, Ziqiong Liu, Pratik Prabhanjan Brahma, Dong Li, Zicheng Liu, and Emad Barsoum. 2025. GEAK: Introducing Triton kernel AI agent & evaluation benchmarks. arXiv preprint arXiv:2507.23194 (2025). [40] Lingyun Yang, Yuxiao Wang, Shenghao Liang, Linfeng Yang, Daocheng Ying, Chunbo You, Rui Zhang, Luping Wang, Yinghao Yu, Guodong Yang, et al. 2026. Are LLM-generated GPU kernels production-ready? A trace-driven benchmark and optimization agent. arXiv preprint arXiv:2607.14541 (2026). [41] Yunxiang Zhang, Ping Yu, Jianyu Wang, Max (Xiangjun) Fan, Julian Reed, Azalia Mirhoseini, and Will Su. 2026. KernelBench-Verified: Do LLM-generated kernels actually beat PyTorch? arXiv preprint arXiv:2607.16241 (2026).

22

Ziyuan Li et al.

Appendices Appendices A–H provide audit rules, supporting results, and evidence boundaries for the main paper. Each analysis states its estimand and uses the corresponding sealed records or completed annotation sheets. Summaries and RQ3 claimed/measured speedup pairs are in experimental_tables.json; final_decisions.json lists certificate and selected-axis sources, and cause_inventory.json records unresolved-axis evidence labels. comparison_verified.json retains OV120 sampling fields and the implementation-comparison results. A

Audit Parameters and Record Provenance

The verdict semantics, certificate aggregation, and outcome-code priority are defined in Section 3.1, Section 3.5, and Table 1. This appendix gives the source-handling and replay details behind those rules. Task-Policy Resolution. The registry-to-task loader prioritizes explicit boolean implementation policies, uses the manifest for missing/null policies, and rejects invalid values or uncovered entries. Source Qualification and Selection. Qualification checks file and progress hashes, record schema and digests, record membership, and manifest-bound run or experiment identifiers. Ordinary replays preserve record ID and task, candidate, contract, and policy fingerprints. Declared state adaptations require separate verification of candidate/reference sources, registration, adaptation fields, and adapted candidate-fingerprint correspondence. Worker placeholders require a registry-entry digest, the same record ID, and INCONCL on all axes and the certificate. They record unavailable evidence, without establishing complete state correspondence or supplying PASS/FAIL evidence. Admission across records does not require identical inputs, environments, or state hashes; state correspondence remains necessary within semantic comparisons. Source export traverses records in specified input order, applying certificate eligibility before selecting each axis and the effective certificate separately. Restricted merging admits new decisions only for unresolved evidence; axis updates additionally require a pre-specified eligibility set. Let 𝐷 denote PASS or FAIL and 𝑈 denote INCONCL. The operations differ as follows: Existing / incoming

Source export

𝐷/𝑈 𝑈 /𝐷

Keep existing source Use incoming source

Same status Opposite decided statuses

Restricted merge

Keep existing source Update an eligible axis; a certificate requires an incoming record’s decided certificate Use incoming source Keep existing source Use incoming source; no conflict guard Stop import for conflict investigation in this operation

Restricted imports require empty weak results and preserve existing weak-evidence sources. Input order, rather than a timestamp ranking, determines replacement during export. The example in Appendix C distinguishes certificate eligibility from source replacement. Source digests identify the base policy and vendor inventory. Executed settings additionally depend on each record’s effective_policy, tolerance configuration, and timing fields. The inventory’s internal fingerprint agrees with the bound execution records. Dispatcher Replay Checks. Beyond the shape, dtype, and structure checks in Section 3.3, recursive comparisons check tensor/non-tensor types, mapping keys, and sequence types/lengths. Matched call counts are not required.

KernelAudit

23

Stress-Schedule Configuration. The base schedule combines 12 pre-specified slots with four nativeseed slots and uses contract-adapted values spanning 10−4 –105 . Appendix B gives the instantiated provider-specific schedule. A/A Resampling Details. For the timing gate in Section 3.4, let 𝑡𝑎𝑏𝑖 be the 𝑖th latency of arm 𝑎 in batch 𝑏 and 𝑚𝑎𝑏 = median1≤𝑖 ≤20 𝑡𝑎𝑏𝑖 . The sample standard deviation uses denominator 60 − 1. Let P contain the odd/even and first-ten/last-ten position pairs within each arm and batch. The order-effect requirement is median𝑖 ∈𝐼 𝑡𝑎𝑏𝑖 − median𝑖 ∈ 𝐽 𝑡𝑎𝑏𝑖 ≤ 0.05. 𝑚𝑎𝑏 The three paired batch medians are resampled with replacement 2,000 times (seed 0), using identical batch indices for both arms. Each replicate takes the median of the resampled A-arm medians divided by that of the B arm. The 2.5th and 97.5th percentiles provide the interval [𝐿, 𝑈 ] used by the main-text acceptance rule. The three-batch construction does not guarantee nominal coverage. 𝑂 = max max

𝑎,𝑏 (𝐼,𝐽 ) ∈ P

B

Per-Corpus Misses and Aggregate False Rejects

The rate definitions and weak-REJECT convention follow Section 4. The following details specify corpus pooling, unresolved-record bounds, and paired verifier contrasts. Pooling and Abstention Sensitivity. For aÍfixed verifier, let 𝑎𝑐 count accepted FAIL certificates and 𝑏𝑐 count decided accepts in corpus 𝑐. For 𝑐 𝑏𝑐 > 0, the pooled rate is Í ∑︁ 𝑏𝑐 𝑎𝑐 𝑎𝑐 Í FApool = Í𝑐 = . 𝑐 𝑏𝑐 𝑗 𝑏 𝑗 𝑏𝑐 𝑐:𝑏𝑐 >0 Í Í Weights depend on each verifier’s decided acceptance counts. With 𝑎 = 𝑐 𝑎𝑐 , 𝑏 = 𝑐 𝑏𝑐 , and 𝑢 accepted INCONCL records, the sensitivity endpoints are 𝑎/(𝑏 +𝑢) and (𝑎 +𝑢)/(𝑏 +𝑢) for 𝑏 +𝑢 > 0. They assign all 𝑢 records to PASS or FAIL with weak decisions and resolved certificates fixed; they quantify assignment sensitivity, not sampling uncertainty or errors in resolved certificates. Table B.1. Misses (accepted oracle-FAILs / corpus FAIL total) and false rejects (rejected oracle-PASSes / corpus PASS total). Corpus

CUDA-Agent KernelBench KernelGYM robust-kb. DeepReinf. MusaCoder

CUDA-L1 miss AI-CUDA-Eng. miss KernelBenchX miss TB-G self-built miss TB-G upstream miss Delegation controls miss robust-kbench miss

23/148 97/120 17/32 8/21 1/17 55/56 2/3

117/148 86/120 21/32 15/21 1/17 56/56 2/3

136/148 97/120 23/32 15/21 1/17 56/56 2/3

110/148 83/120 20/32 10/21 1/17 39/56 2/3

75/148 98/120 23/32 14/21 1/17 56/56 2/3

39/148 98/120 16/32 12/21 1/17 1/56 2/3

All-corpus false rejects

15/172

17/172

9/172

21/172

11/172

14/172

Paired Verifier Contrasts. McNemar’s exact null model assigns probability 1/2 to either discordance direction among the 𝑏 + 𝑐 discordant FAIL records. With independent records, 𝑏 ∼ Binomial(𝑏 + 𝑐, 1/2). Holm correction covers the 15 pairs within each pool; it does not remove source dependence. Per-corpus axis rates are in the released false_accepts cells under axis_FA_rates. Evidencesource distinctions follow Appendix C.

24

Ziyuan Li et al.

Table B.2. Paired misses on 397 FAIL-certificate records. 𝑏/𝑐 count misses unique to the second/first verifier. Exact two-sided 𝑝 and Holm adjustments follow the model above; 14 of 15 pairs have adjusted 𝑝 < 0.05, with robust-kbench vs. DeepReinforce the exception. Shared source kernels limit independent-record inference. Pair CUDA-Agent vs. KernelBench CUDA-Agent vs. KernelGYM CUDA-Agent vs. robust-kbench CUDA-Agent vs. DeepReinforce CUDA-Agent vs. MusaCoder KernelBench vs. KernelGYM KernelBench vs. robust-kbench KernelBench vs. DeepReinforce KernelBench vs. MusaCoder KernelGYM vs. robust-kbench KernelGYM vs. DeepReinforce KernelGYM vs. MusaCoder robust-kbench vs. DeepReinforce robust-kbench vs. MusaCoder DeepReinforce vs. MusaCoder

𝑏

𝑐

𝑝 (exact)

𝑝 (Holm)

112 127 106 66 37 32 2 29 22 2 2 2 57 31 0

17 0 44 0 71 0 35 58 151 67 63 163 53 127 100

2.45e-18 1.18e-38 4.40e-07 2.71e-20 1.38e-03 4.66e-10 1.02e-08 2.48e-03 7.43e-25 8.19e-18 1.16e-16 5.86e-46 7.75e-01 5.38e-15 1.58e-30

2.45e-17 1.65e-37 1.76e-06 2.98e-19 4.14e-03 2.79e-09 5.12e-08 4.96e-03 8.91e-24 7.37e-17 9.31e-16 8.79e-45 7.75e-01 3.77e-14 2.05e-29

Table B.3. Paired misses on the five natural corpora (CUDA-L1, AI-CUDA-Engineer, KernelBenchX, both TritonBench-G layers; 338 FAIL records). Delegation controls and the handwritten set are excluded. Under the same model, correction, and source-dependence limitation, 14 of 15 pairs have adjusted 𝑝 < 0.05. Natural-corpus miss rates: CUDA-Agent 43.2%, KernelBench 71.0%, KernelGYM 80.5%, robust-kbench 66.3%, DeepReinforce 62.4%, MusaCoder 49.1%. Pair CUDA-Agent vs. KernelBench CUDA-Agent vs. KernelGYM CUDA-Agent vs. robust-kbench CUDA-Agent vs. DeepReinforce CUDA-Agent vs. MusaCoder KernelBench vs. KernelGYM KernelBench vs. robust-kbench KernelBench vs. DeepReinforce KernelBench vs. MusaCoder KernelGYM vs. robust-kbench KernelGYM vs. DeepReinforce KernelGYM vs. MusaCoder robust-kbench vs. DeepReinforce robust-kbench vs. MusaCoder DeepReinforce vs. MusaCoder

𝑏

𝑐

𝑝 (exact)

𝑝 (Holm)

111 126 105 65 37 32 2 29 22 2 2 2 40 31 0

17 0 27 0 17 0 18 58 96 50 63 108 53 89 45

4.25e-18 2.35e-38 4.66e-12 5.42e-20 9.07e-03 4.66e-10 4.02e-04 2.48e-03 3.26e-12 6.12e-13 1.16e-16 9.41e-30 2.13e-01 1.12e-07 5.68e-14

5.10e-17 3.53e-37 3.26e-11 7.05e-19 1.81e-02 2.79e-09 1.61e-03 7.44e-03 2.61e-11 5.51e-12 1.28e-15 1.32e-28 2.13e-01 5.59e-07 5.68e-13

Nominal-Input Diagnostics and Repeated Execution. The diagnostic set is the record-level union of selected semantic FAILs accepted by at least one weak verifier; each record is counted once. Its 62 records all have FAIL certificates: 16 each from CUDA-L1, AI-CUDA-Engineer, and the delegation controls, 10 from TritonBench-G’s self-built layer, and 4 from KernelBenchX. IDs appear in nominal_strict_comparator in the released table data. The input-provider contract determines the executable policy schedule. For these 62 records, contract adaptation replaces ten unsupported optional transformation slots in the 12-plus-4 base schedule with native-seed slots before execution. Each instantiated plan contains value_normal, 14 native_seed_* policies, and one repeated_run policy with three calls. The nominal diagnostic excludes repeated_run and examines the other 15 stored policy results. The nominal signal rule and aggregate counts are reported in Section 5.

KernelAudit

25

Table B.4. Abstention sensitivity by natural corpus. Counts are weak-accepted INCONCL records; bounds assign those records to PASS or FAIL with resolved certificates fixed. Decided acceptance counts and false accepts are in Table 6. Corpus

Verifier

CUDA-L1

CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder all six

AI-CUDA-Engineer

KernelBenchX

TritonBench-G self-built

TritonBench-G upstream

Accepted INCONCL

FA bounds (%)

17 41 46 38 25 10 13 6 13 5 13 13 2 2 2 1 2 1 23 25 25 11 25 25 0

[57.5,100.0] [74.1,100.0] [73.9,98.9] [74.3,100.0] [75.0,100.0] [79.6,100.0] [47.5,53.9] [47.8,51.1] [47.5,53.9] [48.3,51.2] [47.8,54.1] [47.8,54.1] [50.0,55.9] [52.5,57.5] [54.8,59.5] [52.6,55.3] [54.8,59.5] [48.5,51.5] [12.7,49.2] [20.3,54.1] [20.3,54.1] [18.2,38.2] [19.2,53.4] [16.9,52.1] [12.5,12.5]

The four non-numeric signals are NaN/Inf against a finite reference (sakana__L2_T80); fp16 output where fp32 is expected (cuda_l1__L1_T8, cuda_l1__L2_T96); and a return-structure mismatch, expected tuple[2] versus the linalg_eig return type (kernelbenchx__eig). The latter is a structural diagnostic signal; whether it constitutes a violation depends on the declared output contract. For example, kernelbenchx__tanh has maximum relative error 0.3393 but maximum absolute error 1.7881 × 10−7 ; it passes with 𝜌 ℓ = 10−5 and 𝑎 ℓ = 2.4424 × 10−7 . The record without either nominal signal, kernelbenchx__exp_sqrt, has repeated-run evidence in the complete record selected for both its certificate and semantic axis. Its first invocation passes (maximum relative error 1.6746 × 10−7 ); the second and third fail (17.8396 each), with maximum absolute error 3.46937 and 75% of elements beyond tolerance. C

INCONCL Evidence Categories

Table C.1 breaks down the unresolved-axis evidence summarized in Section 6.5. State-Alignment Evidence. An example behind the state-correspondence requirement in Section 3.2 is replacement of reference modules by directly held nn.Parameter objects. The 49 “parameter/state mapping unproven” labels comprise 17 semantic, 16 implementation, and 16 measurement entries across 18 records; they count unresolved evidence entries, not separate candidate defects.

26

Ziyuan Li et al.

Table B.5. Pooled abstention bounds (upper panel) and control-axis diagnostics (lower panel). Delegation and handwritten corpora have zero accepted INCONCL records, so their bounds collapse to the decided rates in Table 6. Lower-panel rates use that table’s decided acceptance denominators and describe selected-axis evidence, not a certificate decomposition. Pool

Verifier

Main pooled

CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder

Natural-5 pooled

Accepted INCONCL

FA bounds (%)

30 47 59 43 38 23 55 74 86 55 65 49

[49.2, 61.5] [60.1, 74.0] [60.1, 75.3] [60.3, 73.8] [56.7, 69.2] [53.9, 63.0] [41.8, 57.6] [52.2, 68.3] [53.1, 69.9] [53.2, 66.3] [49.3, 64.5] [45.4, 58.7]

Corpus

Verifier

Delegation controls

CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder all six

robust-kbench handwritten

Sem.-evidence %

Impl.-evidence %

27.1 26.7 26.7 18.6 26.7 0.0 0.0

86.4 86.7 86.7 81.4 86.7 33.3 28.6

Table C.1. Evidence labels for 445 unresolved record–axis entries across 286 records. Multiple axes can contribute labels within a record; these labels do not assign responsibility. Axis-level evidence label Reference-calibration ceiling Unresolved after verified execution Execution timeout Candidate artifact missing or syntax error Parameter/state mapping unproven Candidate build diagnostic Reference precision or output unsupported GPU memory shortage Reference nonfinite output Insufficient work/task evidence

Count 132 90 51 51 49 26 20 18 6 2

Interpretation reference/evidence boundary execution evidence incomplete execution evidence incomplete candidate artifact boundary semantic precondition candidate artifact boundary reference boundary execution environment boundary reference boundary reference trace lacks compute

Certificate and diagnostic-axis conclusions retain separate evidence sources. In the ledger, 140 FAIL certificates coexist with 158 unresolved selected axes. Four FAIL certificates have no selected FAIL axis, and one INCONCL certificate has three selected PASS axes. These five certificates and their diagnostic axes cite different source records, so the displayed axes cannot reconstruct the certificate or explain its abstention. Category evidence may also cite a source different from the selected axis; the released per-axis ledger and evidence strings preserve this distinction. The two insufficient-work labels concern absent reference compute traces, not the total number of R-A/R-B exclusions. For cuda_l1__L2_T83, execution 1cb62964. . . supplies three PASS axes. R-B changes its stored PASS certificate to effective INCONCL without changing those axes. During source export,

KernelAudit

27

the later input 2e358987. . . is an all-INCONCL worker placeholder reporting a SchemaError for a missing candidate-state hash. The 𝐷/𝑈 rule retains the execution’s diagnostic sources; the 𝑈 /𝑈 rule binds the certificate to the placeholder. R-B explains ineligibility, while input-order replacement explains the selected source. The placeholder’s error neither overturns the PASS axes nor supplies evidence for R-B or a candidate defect. D

Measurement Evidence and RQ3 Timing Details

The following tables distinguish fixed-source timing guards and source-pattern observations from the selected-axis summary in Section 6.2. Table D.1. Measurement observations and static-source review. Timing rows use the fixed 577-record guard cohort within the 716-record audit; static rows summarize a separate fixed scan and its PASS review subset. The stream row reports only the ratio component of the criterion in Section 3.4. Item

Value

Hidden-stream ratio in the fixed guard cohort: p50 / p95 Thread-injection / lazy-evaluation signals Static source-pattern matches in fixed scan (non-gating) Flagged PASS subset selected for source review Source-text categories in reviewed subset

1.09 / 1.26 0/0 122 records 19 records 16 dtype-table/handling text, 1 cuDNN TF32 setting, 2 bf16 comments

Static source-pattern matches identify review targets without establishing executed precision. The fixed scan flags 122 records. Pattern counts are record-level and non-exclusive: fp16 85, bf16 72, two TF32 patterns 17 each, autocast 14, and .half() 6. One cudnn.allow_tf32=True setting can match both TF32 patterns. Source review selects 19 flagged records marked PASS in the scan input; each also has a PASS certificate in the evaluation ledger. This scan-defined subset does not establish the complete set of static matches among the 172 PASS certificates. Sixteen contain dtype-table or handling text, one a cuDNN TF32 setting, and two have bf16 matches only in comments. These records contain no runtime precision evidence, and numerical agreement does not identify the executed precision mode. Static matches remain non-gating; the constructed dtype-downgrade probe is detected by the semantic axis. Guard Coverage and Evidence Sources. Fixed-source guard membership and selected measurement verdicts describe different evidence views of the same 716 records (Table D.2). Their 577 guard records and 651 PASS records overlap on 558 IDs. The fixed source omits guards on 139 records: 98 with unresolved state mapping, 40 worker failures, and one timeout. Table D.2. Fixed timing-guard membership and selected measurement verdicts over the same 716 record IDs. Fixed-source guard membership

Selected PASS

Selected INCONCL

Total

Present Absent

558 93

19 46

577 139

Total

651

65

716

Every selected measurement PASS has passed timing, materialization, thread, and stream guards, including the 93 records outside the fixed guard cohort. The 44 records without guard observations in Section 6.2 comprise 17 state-mapping problems, 25 worker failures, one audit-context failure,

28

Ziyuan Li et al.

and one timeout. Guard presence therefore does not itself imply a decided axis. The fixed-source stream quantiles and mutation census describe the 577-record cohort, not the selected-source guard cohort; RQ3 uses the former timing population. RQ3 Timing Exclusions and Corpus Breakdown. The 394 withheld speedups reported in Section 7 comprise 285 exclusions for reference self-variation, 82 for intervals excluding one, and 27 for other gate conditions. Table D.3 gives the per-corpus certificate breakdown; all 32 PASS certificates share their complete source record with the corresponding timing evidence. Table D.3. Certificate outcomes among all 180 measurable records. Measurable denotes A/A passage and a positive recorded local speedup. Timing records and audit certificates are joined by record ID. Corpus

Measurable

PASS

FAIL

INCONCL

CUDA-L1 AI-CUDA-Engineer KernelBenchX TritonBench-G (self-built ref.) TritonBench-G (upstream ref.) Delegation controls robust-kbench handwritten

64 68 2 22 1 20 3

0 22 0 8 0 1 1

54 39 2 4 1 19 2

10 7 0 10 0 0 0

Total

180

32

121

27

Paired summaries weight records equally without assuming independent tasks. For the 132record main paired cohort, the median of record-wise measured/claimed ratios is 0.934838, whereas the ratio of marginal medians is 0.852335. RQ3 Certificate Sources and Attribution. Table D.4 expands the 54 source-mismatched main-cohort records identified in Section 7. Ten diagnostic records (all INCONCL) also have different certificate and timing hashes; all eight diagnostic PASS records share theirs. A different complete-record hash does not itself imply different candidate source code. Policy fingerprints match across the 54 main-cohort links, while audit-program, configuration, and environment fingerprints differ. Matching recorded identity fields therefore does not establish identical execution contexts or realized execution paths. Table D.4. Source correspondence for the 54 main-cohort records with different timing and certificate record hashes. All are uncertified. 𝑛 Records 45 Other linked records 1 3

1 4

Confirmed correspondence and limits

Recorded candidate, task, contract, and reference fingerprints, plus canonical input and pre-/post-state hashes, match. CUDA-L1: Same candidate source; the candidate-fingerprint difference is explained by four batchL3_T46 normalization state-adaptation metadata entries. AI-CUDA-Engineer: Candidate, task, and input match. Recorded state coverage differs in the ordinary L3_T37, L3_T39, L3_T40 attributes h0/c0 for T37 and h0 for T39/T40. This correspondence does not establish full state equality or a change of reference code. CUDA-L1: Candidate, task, reference, and input hashes match; the certificate source lacks candidate L2_T80 canonical pre-/post-state hashes. Timeout or validation-failure certificate records retain linked candidate code but lack CUDA-L1: L1_T48, L1_T51, L1_T52, complete input/state execution identity. L2_T83

KernelAudit

29

RQ3 Duration Strata. A/A passage varies with reference duration in the fixed 577-record cohort. Median reference durations are 2.544208 ms for the 183 A/A passes and 0.112656 ms for all 394 withholdings. The 285 records withheld for excessive reference CV have median 0.086720 ms. In selected duration strata, A/A pass counts are 63/362 (17.40%) below 0.5 ms, 64/105 (60.95%) in [2, 10) ms, and 34/66 (51.52%) at 10 ms or above. Together, the latter bins have 98/171 passes (57.31%); the bin-specific rates are observed proportions, not confidence bounds. These timing associations do not identify the causal contributions of kernel properties, host conditions, or measurement overhead. E

Extended Case Studies

Launch-Provenance Controls. The launch controls in Table 7 have the following per-case observations (launch_controls_results.json). Columns count forbidden vendor (Forb.), candidate (Cand.), and mediated library (Med.) launches. Case

Construction

C-LP1 direct vendor call C-LP2 mediated call C-LP3 self-written C-LP4 name collision

extension calls cublasSgemm below the dispatcher plain torch.mm forward hand-written CUDA matmul kernel custom kernel with vendor-style name (ampere_sgemm_..._like_custom)

Forb. Cand. Med. 2 0 0 0

0 0 1 1

0 2 0 0

The controls distinguish the tested launch origins. C-LP1 matches two inventory symbols: an ampere_sgemm variant and a splitKreduce kernel. C-LP2 places the tested dispatcher-mediated kernels in the library bucket. This launch-channel negative remains subject to Channel 1’s contract assessment. C-LP4’s vendor-style name produces no false vendor attribution. Comparisons against torch.mm record maximum absolute differences of 0, 0, 3.8147×10−5 , and 4.1962×10−5 for C-LP1–4, respectively. These observations have no recorded numerical acceptance threshold; the 4/4 result tests launch-bucket expectations. Evidence-Channel Attribution Matrix. The matrix allocates 313 selected implementation FAIL records across the two evidence channels; independent validity is assessed in Appendix H. Of the 229 main-corpus records with selected implementation FAIL evidence identified in Section 6.1, 23 also carry selected semantic FAIL evidence. These are record-level overlaps: selected axes and certificates need not share an execution source. No main-corpus record receives C3a under the reporting priority: semantic failures can coexist with C3b, while semantic-only false accepts without a qualifying signature receive C2. These codes do not establish author intent. Corpus

Impl. FAIL

Dispatcher only

Launch only

Both

CUDA-L1 AI-CUDA-Engineer KernelBenchX TritonBench-G (both layers) Delegation controls robust-kbench controls

136 93 22 8 52 2

127 89 22 8 52 1

1 4 0 0 0 1

8 0 0 0 0 0

Total

313

299

6

8

Case 1 (Extended): llama_ffw, a Policy-Sensitive Diagnostic. For the llama_ffw case in Section 6.1, the reference matmul signature is 1.8×1011 FLOPs. Channel 2 observes five device activities: three vendor-kernel launches, one custom-kernel launch, and one memset. The vendor SGEMM kernels exactly match the sealed inventory: two ampere_sgemm_128x32_tn and one ampere_sgemm_128x64_tn. Saved CUDA source contains three cublasSgemm calls, consistent with the runtime hits. The saved wrapper matches the record’s wrapper fingerprint; this match does not fully identify the bytes of the external CUDA source used at execution. GPU-time buckets attribute 5,435 𝜇s to vendor work and 21 𝜇s (0.4%) to the candidate’s fused_silu_mul_predicate_kernel:

30

Ziyuan Li et al.

the custom work is the SiLU epilogue, while cuBLAS performs the core GEMMs. The recorded implementation signature is launch:forbidden_library:3. Case 2 (Extended): Silent-Fallback Shell cuda_l1__L2_T46. After the custom-convolution build fails in the frozen environment, the candidate’s wrapper executes native PyTorch fallback. Channel 1 records dominant aten::convolution work (estimated 1.50×1012 FLOPs; reference 5.99×1011 ). Joint noise replacement of convolution outputs returns LEAK: maximum relative output change 1.84×107 , leakage fraction 1.0. Channel 2 records three non-mediated vendor launches: two cuDNN layout transforms (cudnn::ops::nchwToNhwcKernel) and one implicit-GEMM launch. KernelBench, KernelGYM, and robust-kbench accept; CUDA-Agent rejects for runtime F.* blocking, MusaCoder for compilation failure, and DeepReinforce for output mismatch at 10−2 tolerance, with underlying cause undetermined. Case 3 (Extended): Attribution. In the TritonBench-G upstream layer, 124_logsumexp_fwd is convicted only after do(P:=noise) confirms output dependence on aten::logsumexp (all six weak verifiers accept it). The implementation-axis misses summarized in Table 7 comprise three GEMV-level probes below the targeting threshold and five transposed-convolution probes on unknown-operation paths. For the GEMV probes, reference-to-candidate work-proxy ratios are 192, 323, and 384; these compare estimator outputs, not measured hardware FLOPs. Four transposed-convolution probes are captured by the semantic axis; one (convtrans2d_k2s2_exact) passes all axes. F

Oracle-Validity Details

Constructed Controls, per Axis. Beyond the family-level results in Table 7, the stale-cache and shape-specialization controls receive semantic PASS but implementation and certificate FAIL. Each executes value_normal, 14 native-seed policies, and repeated_run with three calls. This 16-slot plan does not establish alternative-shape coverage; the causes of these two semantic misses remain undetermined. Measurement controls separate decisive violations from non-gating signals. Of six positives, two timing-tamper probes yield measurement FAIL, three thread/lazy/precision probes raise their designed non-gating signals with measurement PASS, and one hidden-stream probe produces no signal. Thus 5/6 activate a signal, while 2/6 establish measurement violations; both measurement negatives have PASS with no guard signal. Upstream Reference Quality. For the upstream-reference probe in Table 5, a FAIL certificate alone does not attribute a defect to the upstream reference or its author. Such attribution requires record-level evidence separating candidate, reference, and task/input causes; execution failure alone is insufficient. G

RQ4 Details

Registration and Static-Screening Details. The task pool and slot order are specified in Section 8. The CUDA reference-basename join uses level lists of 46, 140, and 20 tasks; 14 joined tasks lie outside the resource plan. The same basename join is used for the TritonBench G/pref intersection; this join key does not establish semantic equivalence of registered references. Historical static screening flagged 191 slots for simultaneous explicit PYBIND11_MODULE and load_inline(functions=...) bindings, which can duplicate generated C++ exports. This compilation issue is distinct from tensor-storage aliasing; membership in a skip list does not prove that no earlier execution was attempted.

KernelAudit

31

Reconciled Evidence and Replay Eligibility. The 204 reused slots described in Section 8 come from 181 distinct complete primary-audit records: 23 TritonBench records each supply two slots. Let 𝐸𝑊 , 𝐸 axis , and 𝐸 1 denote the weak, axis-only, and reconciled eligible-slot sets defined by the rules in Section 8. There are 186 axis-PASS slots and 184 eligible slots. The two remaining axis-PASS exclusions are historical Opus c0 and c2 on cuda_l1__L2_T80. This contrast does not isolate R-A’s effect. Timing-Cache Eligibility and Provenance. Beyond the measurement and guard conditions in Section 8, timing requires successful state alignment and available task tolerances and calibration settings. A/A requires accepted=True under the full protocol in Appendix A. Local A/B timing uses ABBA pairing and requires timing PASS together with successful output comparison, crosscheckpoint consistency, state/input checks, and stability checks. Warmup counts, batch sizes/counts, and stability limits match the A/A settings in Section 3.4. A usable cache entry has PerformanceFacts.status=PASS and positive, finite speedup, ref_ms, and cand_ms; otherwise its public speedup is null. All 68 non-null speedups satisfy these conditions and are computed as median(local reference times) speedup = . median(local candidate times) Ranking reads non-null cached speedups without rerunning these checks; upstream speedup claims are excluded. Timing availability and shipping eligibility impose different conditions. Measurement PASS alone does not provide a usable speedup: any activated measurement guard, including the 114 stream signals, withholds it. Conversely, timing availability does not require semantic or implementation PASS; shipping eligibility uses 𝐸𝑊 , 𝐸 axis , or 𝐸 1 . Ordering Details. Each gate selects from the fixed pool rather than post-filtering 𝑆 0 ’s choice. Exact PerformanceFacts.speedup ties use the earlier slot, without tolerance or a secondary key. Selection neither minimizes raw candidate latency nor requires speedup above one. Table G.1. Outcome distribution over 1,160 slots. The R-C attribution results are in Section 8. Convicted slots rejected by all six weak verifiers are counted separately; otherwise the reported code is shared by the accepting verifiers. Outcome

Slots

Share

Note

Eligible under 𝑆 1 FAIL; all six reject C2 (silent semantic defect) C3a (semantic violation w/ signature) C3b (implementation violation) C3c (measurement violation) INCONCL

184 48 17 0 122 0 789

15.9% 4.1% 1.5% 0.0% 10.5% 0.0% 68.0%

covers 62 of 232 tasks aggregate rejection count includes reused primary evidence unavailable evidence or withheld eligibility

Shipping Estimands. The slot totals do not determine how much task-level abstention is attributable to candidate violations or unavailable evidence. Shipping rates distinguish tasks from deliveries. For 𝑆 ∈ {𝑆 0, 𝑆 axis, 𝑆 1 }, let 𝑑𝑡 (𝑆) indicate delivery on task 𝑡, and let ℎ𝑡 (𝑆) indicate a human-labeled Í unsafe shipment, with ℎ𝑡 (𝑆) = 0 when 𝑆 abstains. On this Í 232-task pool, coverage is 𝐶 (𝑆) = 𝑡 𝑑𝑡 (𝑆)/232, and the human unsafe-task rate is 𝑈 (𝑆) = 𝑡 ℎ𝑡 (𝑆)/232. For nonzero Í Í coverage, 𝑉 (𝑆) = 𝑡 ℎ𝑡 (𝑆)/ 𝑡 𝑑𝑡 (𝑆) is the unsafe fraction among deliveries, and 𝑈 (𝑆) = 𝐶 (𝑆)𝑉 (𝑆). For an unknown shipment label, the lower and upper bounds assign ℎ𝑡 (𝑆) = 0 or 1, respectively; these are identification bounds, not confidence intervals. A smaller unsafe-task rate can reflect

32

Ziyuan Li et al.

fewer deliveries as well as a different mix of delivered candidates. Oracle-visible unsafe rates replace the human indicator with the corresponding audit-label indicator; the two standards are not interchangeable. Gate and Ordering Contrasts. For each weak verifier, let 𝑢 𝑤,𝑓 , 𝑢 𝑤,𝑣 , 𝑢𝑐,𝑓 , and 𝑢𝑐,𝑣 be oracle-visible unsafe shipments divided by 232 for 𝑆 0 , 𝑆 0 -fast, 𝑆 1 -first, and 𝑆 1 , respectively. Here 𝑤/𝑐 denote weak/audit gates and 𝑓 /𝑣 denote first/fastest ordering. Unresolved shipments and abstentions remain in the denominator without entering the unsafe numerator; neither is thereby labeled safe. The two descriptive reductions, in percentage points, are   100  100  𝐺= (𝑢 𝑤,𝑓 − 𝑢𝑐,𝑓 ) + (𝑢 𝑤,𝑣 − 𝑢𝑐,𝑣 ) , 𝑅= (𝑢 𝑤,𝑓 − 𝑢 𝑤,𝑣 ) + (𝑢𝑐,𝑓 − 𝑢𝑐,𝑣 ) . 2 2 We average 𝐺 and 𝑅 equally over the six verifiers, obtaining 30.927 and 0.826 pp. Per-verifier 𝐺 ranges from 10.345 to 47.845 pp. These six contrasts share tasks and are not independent samples. Both audit-gated cells have zero oracle-visible unsafe shipments by their eligibility definition, so 𝑅 is half the weak-gate ordering contrast. Delivery with Speedup. Table G.2 gives the per-verifier SafeFast breakdown behind Section 8. Missing speedup evidence contributes zero to this endpoint without establishing absence of a speed advantage. The small ordering contrast on unsafe counts therefore does not imply absence of benefit on speed-qualified delivery. Table G.2. SafeFast counts by verifier and ordering on the same 232 tasks. Pairs give counts at 𝑘=1, 2 and require membership in 𝐸 1 and credible speedup > 𝑘×. Corresponding oracle-visible unsafe counts are in Table 11. Weak verifier

𝑆 0 SafeFast (1, 2)

𝑆 0 -fast SafeFast (1, 2)

CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder

(0, 0) (0, 0) (0, 0) (0, 0) (0, 0) (3, 1)

(3, 2) (3, 2) (3, 2) (3, 2) (3, 2) (5, 2)

𝑆 1 -first SafeFast (1, 2) (4, 2)

𝑆 1 SafeFast (1, 2) (6, 2)

Shared audit gate All six verifiers

Table G.3. Runtime diagnostics. Worker durations and cache-write intervals have different units of observation; neither is a direct deployment-cost measurement. Observation

Sample

Worker duration with end record Strong-cache write interval

Primary audit: 729 attempts, including repeated attempts RQ4 46-task segment: 196 intervals; endpoints on 44 tasks, 196 candidates Same segment: 887 intervals; endpoints on 45 tasks, 174 candidates; six verifiers 72 tasks, 360 slots; six weak verifiers

Weak-cache write interval Cached first-PASS scans

Statistic p50/p95: 1.025/18.081 min p50: 7.460 min p50: 77.142 s wall time/slot: 0.03 ms

KernelAudit

33

Cost Measurement and Scope. Runtime summaries retain their observation units and sampling filters. Worker durations include each attempt with an end record, including repeats; their quantiles are not per unique kernel. Cache intervals are differences between adjacent writes, retaining 0 < Δ ≤ 3600 s for the strong cache and 0 < Δ ≤ 300 s for the weak cache. For the reported quantiles 𝑞 ∈ {0.5, 0.95}, the script selects zero-based index ⌊𝑛𝑞⌋ among 𝑛 sorted observations. The p50 ratio is 7.460 × 60/77.142 ≈ 5.80. Different filters and candidate coverage make these unpaired summaries; intervals do not isolate an individual audit stage’s start and end. Using the strong-cache p50 for one audit per task is a cost proxy. Beyond the cost limitations in Section 8, per-axis audit costs and human reference-authoring costs are unmeasured. The cached-scan measurement covers data loading, indexing, and first-PASS selection. Its 0.03 ms per slot is total wall time divided by 360 over 72 tasks and six weak verifiers, including reads of 12 audit JSON files, index construction, and scans for the first weak-PASS and first stored certificatePASS entries. It excludes evidence collection and does not measure fastest ordering or the refreshed 𝑆 1 workflow. H

Blind-Annotation Details

The historical blind review comprises 450 independently annotated units: 120 validity-sample records and 330 shipment-census candidates. The refreshed RQ4 union adds one unannotated candidate; it is not included in the following annotation-reliability statistics. Packets randomize identifiers and omit oracle axes/certificates, weak verdicts, timing ratios, and source-provenance fields. Annotators inspect candidate/reference artifacts and independent executions for correctness, delegation, and measurement. The 102 units with at least one dimension disagreement (29 validity units and 73 census units, concentrated in delegation declarations) receive third-party arbitration with a written rationale for each unit. A unit may disagree on multiple dimensions: the 102 units yield 119 dimension-level rulings, adopting annotator A 49 times and B 70 times. All rulings use one of the two annotators’ labels. Policy and Human-Reference Scope. Upstream prompts request CUDA optimization: KernelBench (Appendix C.1 of the upstream paper) permits partial operator replacement, and the examined CUDA-L1 (Table 3 of the upstream paper) and AI-CUDA-Engineer (Appendix B.2 of the upstream paper) prompts do not state a universal ban on core library computation. The frozen study policy imposes core self-implementation; e.g., cuda_l1__L1_T1 and sakana__L1_T1 have requires_self_impl=true. The policy/rubric distinction is discussed in Section 4 and Section 6.3. On weak-accepted, oracle-decided main-corpus records, human-referenced false-accept estimates are 8.7–23.6 percentage points below corresponding oracle rates; these differences do not identify their causes. Reliability. Raw agreement and Cohen’s 𝜅 jointly describe annotation reliability. Agreement/𝜅 by dimension are: validity sample (correctness 90.8%/0.740, delegation 80.0%/0.582, measurement 96.7%/−0.006); shipment census (correctness 95.2%/0.568, delegation 81.2%/0.549, measurement 99.4%/0.497). Measurement labels are highly imbalanced: almost all are honest, with 4 and 2 disagreements, respectively. Because 𝜅 adjusts for chance agreement using the annotators’ marginal label frequencies, high raw agreement can coexist with low 𝜅; neither measure is interpreted alone. Validity-Sample Confusion (Unweighted, Certificate Level). The unweighted cells are PASS (44 clean, 3 defect), FAIL (40 defect, 13 clean, 1 unknown), and INCONCL (8 defect, 8 clean, 3 unknown). Human defect is reference-positive and certificate FAIL is audit-positive; the weighted metric denominators are specified in Section 6.3.

34

Ziyuan Li et al.

Table H.1. Oracle-validity sample design. Rows are the frozen original certificate × corpus sampling strata; selected certificates define metric cells. The table reports population 𝑁ℎ , draws 𝑛ℎ , human labels defect/clean/unknown, and inclusion probability 𝜋ℎ . Design certificate

Corpus

𝑁ℎ

𝑛ℎ

d/c/u

𝜋ℎ

FAIL FAIL FAIL FAIL FAIL FAIL FAIL INCONCL INCONCL INCONCL INCONCL INCONCL PASS PASS PASS PASS PASS PASS

AI-CUDA-Engineer CUDA-L1 cuDNN-real KernelBenchX robust-kbench TritonBench-G self-built TritonBench-G upstream AI-CUDA-Engineer CUDA-L1 KernelBenchX TritonBench-G self-built TritonBench-G upstream AI-CUDA-Engineer cuDNN-real KernelBenchX robust-kbench TritonBench-G self-built TritonBench-G upstream

107 79 56 37 3 21 15 31 162 13 39 4 91 4 13 5 29 7

14 11 8 5 2 3 2 3 19 2 4 2 26 2 4 2 9 2

12/2/0 8/3/0 8/0/0 3/2/0 1/1/0 1/2/0 2/0/0 2/1/0 11/8/0 0/1/1 0/2/2 1/0/1 0/26/0 2/0/0 0/4/0 0/2/0 0/9/0 0/2/0

0.131 0.139 0.143 0.135 0.667 0.143 0.133 0.097 0.117 0.154 0.103 0.500 0.286 0.500 0.308 0.400 0.310 0.286

Interval Method. Point estimates are ratios of counts weighted by 1/𝜋ℎ ; 95% model-based posterior intervals use finite-population stratified Jeffreys–Dirichlet completion. For validity metrics, ℎ denotes original certificate × corpus and 𝑗 the selected final certificate cell. Let 𝑥ℎ 𝑗 count observed defect, clean, and unknown labels, with population size 𝑁ℎ 𝑗 and sample size 𝑛ℎ 𝑗 . For replicate 𝑏, ∑︁ 𝑥ℎ = 𝑥ℎ 𝑗 , 𝑝ℎ(𝑏 ) ∼ Dirichlet(𝑥ℎ + 21 13 ), 𝑗

𝑌ℎ(𝑏𝑗 ) | 𝑝ℎ(𝑏 ) ∼ Multinomial(𝑁ℎ 𝑗 − 𝑛ℎ 𝑗 , 𝑝ℎ(𝑏 ) ),

𝑥eℎ(𝑏𝑗 ) = 𝑥ℎ 𝑗 + 𝑌ℎ(𝑏𝑗 ) .

Each stratum shares one 𝑝ℎ(𝑏 ) across its cells; cell completions are conditionally independent given that draw. Observed labels, including unknowns, remain fixed. The uncertainty scope and unassessed exchangeability sensitivity are stated in Section 6.3. For human-referenced false-accept rates, ℎ is further crossed with weak acceptance, using the same shared-draw construction. The decided estimate uses accepted records with certificate PASS or FAIL; the all-accepted lower and upper bounds assign unknown labels to clean and defective, respectively. The all-accepted lower and upper point estimates coincide because sampled accepted units have no unknown labels; posterior completion retains unknown-label probability, yielding distinct intervals. Metrics are recomputed on 𝐵=2000 completed populations. Zero-event strata with unsampled units retain residual uncertainty. Each corpus contributes at most one record per task; source kernels may recur across reference layers. The validity completion simulation uses seed 20260817; the human-referenced false-accept simulation uses seed 20260826. Statistical Reproducibility. The statistical inputs bind each of the 120 validity-sample units’ dimension-level and combined human labels to its original stratum, 𝑁ℎ , 𝑛ℎ , inclusion probability, and selected certificate. Hash and join checks bind the inputs. Calculations use the retained sixdecimal inclusion probabilities; exact 𝑛ℎ /𝑁ℎ values are also retained. An independent standardlibrary implementation reproduces the six validity point estimates and 24 interval sets: six validity intervals and three human-referenced false-accept intervals per verifier. This recomputation verifies computation conditional on the stated model.

KernelAudit

35

Table H.2. Human-referenced false-accept rates over all accepted main-corpus records. Lower/upper intervals assign unknown labels to clean/defect. The point estimates coincide under these assignments; corresponding oracle-decided estimates and intervals are in Table 6. Verifier CUDA-Agent KernelBench KernelGYM robust-kbench DeepReinforce MusaCoder

All-accepted point

Lower 95% interval

Upper 95% interval

44.2% 47.0% 48.0% 43.7% 52.1% 50.6%

[32.4,52.9] [33.7,54.4] [36.6,56.7] [29.4,51.9] [38.7,59.7] [36.6,57.1]

[37.7,57.4] [37.6,58.3] [40.7,59.3] [33.4,55.9] [43.9,63.6] [42.1,60.6]

Historical Census Provenance. The historical 330-unit census was the complete deduplicated delivery union of 𝑆 0 , 𝑆 0 -fast, 𝑆 axis -first, and 𝑆 axis under the original saved evidence, each applied for all six weak verifiers. Archived snapshots before annotation contain that pool, ordering, and mapping; shipment packets also hid selector membership. The original reference standard labels 220 of these 330 candidates clean on all three dimensions. Both refreshed audit selectors choose the same known unsafe and unreviewed candidates identified in Section 8; their coverage and human-unsafe bounds are reported in Table 11. Case 4: Primary-Audit Evidence and the Independent Witness. The historical axis-only selection of CUDA-L1 L2 T80 (BR-171) was labeled correctness defect and measurement dishonest by both annotators. Its class-level __call__ replacement affects new instances, whose _zero_tensor remains None; held-out and stress executions returned no candidate tensor against a one-output reference. These observations establish the human labels independently of the zero-valued reference. In the primary audit, the certificate-bound execution terminates at R-B before the semantic context is created, with zero effective policies. A separate automatic record reports 16 passing policies, but its instance lifecycle is not established and it contains no arity mismatch. The refreshed RQ4 record therefore remains certificate-INCONCLUSIVE; BR-171 is no longer the selected axis-only candidate. This historical example cannot explain the refreshed gate’s human outcomes by R-B alone. Atrex-Based Screen and Reproducibility. The screen and threshold are described in Section 6.4. The adapter preserves upstream missing-observation returns; outer execution errors yield unavailable observations. Byte/hash checks bind the adapter sources to the executed package; offline recomputation matches 248 saved observations (240 OV120 over two seeds, eight controls). Eighteen unavailable observations represent nine OV120 records across both seeds. This recomputation validates stored computation, not complete-harness equivalence or fresh GPU execution. Interpreting Low Attribution. At both seeds (20260805 and 424243), the same 24 of 48 falsepositive records contain one distinct kernel symbol in the saved attribution output; launch activity is aggregated by symbol. For KernelBenchX softmax (OV-100), each seed observation records 2.84 𝜇s for _softmax_kernel and 7.17 𝜇s end-to-end (about 39.6%). The numerator comes from profiling and the denominator from an independent eager benchmark under the same seed, not from timing the same forward execution. Neither the human standard nor the implementation axis identifies delegation for this candidate. One observed symbol does not exclude delegation.

Related documents

• KernelAudit: Contract-Aware Auditing of Validators for LLM-Generated GPU Kernels#1659088
• RESOLVE: Language-Agnostic Validation of GPU Kernels Through Testing, Reduction, and Proof#1201696
• RESOLVE: Language-Agnostic Validation of GPU Kernels Through Testing, Reduction, and Proof#1194391
• Agentic Kernel Optimization: Generating State-of-the-Art GPU Kernels Without Hand-Written CUDA#610063
• Agentic Kernel Optimization: Generating State-of-the-Art GPU Kernels Without Hand-Written CUDA#609001