KernelMutate: Mutation-Guided Validation of Generated GPU Kernels

KernelMutate: Mutation-Guided Validation of Generated GPU Kernels Ziyuan Li1,2,† Zihe Song3,† Shuxiao Xie4,2 Mingfang Ji4,2 Xiaofang Qi1 Wei Yang4,2,5,* Tao Xie1,2,6,4,5,* 1 School of Computer Science and Engineering,

arXiv:2611.15441v1 [cs.SE] 11 Nov 2026

Southeast University, Nanjing, China 2 Beijing Tongming Lake Information Technology Application

Innovation Center, Beijing, China 3 The University of Texas at Dallas, Richardson, TX, USA 4 College of Computer Science and Artificial Intelligence,

Fudan University, Shanghai, China 5 Institute of Systems for Advanced Computing,

Fudan University, Shanghai, China 6 School of Computer Science, Peking University, Beijing, China

lziyuan32@gmail.com, Zihe.song@utdallas.edu 26113050150@m.fudan.edu.cn, 26113050055@m.fudan.edu.cn xfqi@seu.edu.cn, yang_wei@fudan.edu.cn, taoxie@pku.edu.cn

Generated GPU implementations are commonly accepted after a few differential tests, although replacement obligations may remain unchecked. We present KernelMutate, an empirical study of validation reliability using controlled faults, released implementations, and replacement tests. A five-draw anchor accepts 548 of 838 CUDA candidates. The full suite finds native FP32 witnesses for 136 accepted candidates and dtype-only findings for another 362, mostly low-precision execution limitations without established publisher guarantees. Source-guided testing identifies the full suite’s 692 candidates at 26.1% lower charged cost, but yields only dtype evidence for 16 accepted candidates with native witnesses in the full suite. Independently observed context and boundary findings cover the same candidate set. Task-held-out template matching reveals differences in ranking and observation coverage: global top-eight recommendations match more probes than classconditioned lists (51 versus 48 of 75), while conditional analyses restrict the apparent advantage at smaller limits. These template-matching results are offline measurements, not deployed-suite gains. Independent controls and replacement cases expose numerical-policy, binding, and custom-execution boundaries. Validation therefore needs to report witness domains and observation coverage alongside candidate yield and cost. CCS Concepts: • Software and its engineering → Software testing and debugging; Empirical software validation. Additional Key Words and Phrases: GPU kernels, differential testing, mutation testing, validation reliability, generated code

1

INTRODUCTION

Kernel validators are essential for assessing whether generated GPU implementations preserve the behavior of reference computations. Kernel-generation systems compare candidate implementations with references on sampled inputs before measuring performance [8, 11, 15]. In AI-Driven Research for Systems (ADRS) workflows, accepted candidates receive performance feedback that guides further generation (Figure 1) [2]. When validation misses a discrepancy, the resulting performance feedback can favor an implementation that changes the intended computation. Conversely, passing numerical checks does not establish replacement efficiency: DSL kernels can remain substantially slower than optimized library operators [10]. ∗ Corresponding authors: Tao Xie and Wei Yang. † Equal contribution.

2

Ziyuan Li et al.

Improvements to kernel validators address both the range of test conditions and the reliability of correctness checks. Tests vary initialization, configurations, and input distributions to compare candidate and reference behavior across different conditions [9, 32]. Studies of GPU numerical bugs and tile-program code generation further identify input values, shapes, dtypes, and backend behavior as important failure-triggering conditions [20, 21]. Interpreting the observed discrepancies requires criteria that distinguish implementation errors from acceptable numerical variation. Tolerance calibration sets comparison thresholds according to the operator and dtype [26]. Beyond numerical tolerances, contract-based checks examine properties such as preservation of non-finite values. Known-correct implementations help assess false alarms [29], while mutation analysis uses controlled faults to assess the ability of test cases to detect errors [4]. The remaining difficulty is choosing informative test cases within a limited budget and interpreting the resulting evidence for replacement. Repeated samples from one distribution can leave relevant behavior untested. For example, tests with only nonnegative inputs cannot expose an in-place ReLU that overwrites negative input values while the reference preserves its inputs. Covering more conditions consumes additional testing resources, so validation needs a way to prioritize test cases. Test selection alone does not determine what a failure or a pass establishes. A numerical failure under the native FP32 contract differs from a candidate’s rejection of an FP16 input. The FP16 rejection concerns support for that dtype and does not by itself establish incorrect FP32 behavior. A passing comparison shows that the executed code meets the comparison criteria on the tested inputs. We present KernelMutate, a study of validation reliability that combines fault-informed test selection with checks on the scope of validation evidence. Controlled mutations identify conditions that expose implementation differences. We organize the observed associations between fault classes and test conditions into a fault-to-test map. M-dir uses the map and candidate source sites to prioritize test templates, then fills the remaining budget with general tests. M-full provides a fixed suite of value and execution-context tests for comparison. To interpret the outcomes, we record each failure under the study’s validation rules as a witness, retaining the failure stage and tested domain. The records distinguish failures under the native FP32 contract from failures involving additional FP16 or BF16 inputs. A separate replacement study examines reference bindings and actual custom execution in the tested contexts. Evaluator LLM / ADRS Generator

Validator

proposal

pass only

small‐sample correctness gate

score / feedback

Profiler performance

fail, reject

(a) Conventional ADRS evaluation

Evaluator Offline fault study → fault‐to‐test map

LLM / ADRS Generator

Enhanced Validator proposal

score / feedback

Fault‐site analysis

Test‐plan construction

pass only Profiler Execution & verdicts performance fail, reject

(b) KernelMutate‐strengthened evaluation

Fig. 1. Conceptual placement of KernelMutate in a kernel-generation workflow, using an offline fault-to-test map. Only accepted candidates reach performance measurement. Inconclusive outcomes (not shown) remain distinct from failures.

KernelMutate: GPU Kernel Validation

3

Our experiments show that broader testing finds witnesses missed by repeated random sampling, while M-dir incurs less charged testing cost than M-full. Across 838 released CUDA candidates, M-dir and M-full find witnesses for the same 692 candidates, compared with 179 under up to 32 IID draws. Combining the recorded findings from independent boundary and dtype tests also covers the same candidate set. Under the study’s cost-accounting rules, M-dir incurs 26.1% lower testing cost than M-full. The comparison between M-dir and M-full evaluates complete strategies and does not isolate the effect of source guidance. Interpreting the candidate counts requires distinguishing the tested domains. Of 548 candidates accepted by five random draws, M-full finds native FP32 witnesses for 136 and only additionaldtype witnesses for another 362. Most additional-dtype witnesses concern low-precision rejections without established publisher guarantees of support. For 16 of these accepted candidates, Mfull finds native FP32 witnesses while M-dir finds only additional-dtype witnesses. A separate replacement check finds a shape error in a custom implementation that the passing wrapper tests did not execute. This work contributes empirical evidence, a fault-informed testing method, and an analysis of what validation outcomes establish: • Evidence of validation gaps. We study 838 released CUDA candidates and 1,646 controlled probes to characterize native FP32 failures, additional-dtype limitations, and conditions that expose surviving probes. • Fault-informed test construction. We develop an empirical fault-to-test map and budgeted source-guided plans, evaluated through task-held-out recommendations and comparisons of witness yield, tested domain, and cost. • Analysis of validation outcomes. We use independent controls and replacement tests to examine how numerical policies, reference bindings, and actual custom execution affect validation outcomes. We release the code and replication materials at https://github.com/ZIYUANLI-star/KernelMutate. Appendix A provides the extended evidence tables, comparison protocols, and witness analyses. 2

BACKGROUND AND MOTIVATION

Our validation contract records the candidate entry point, paired reference, input domain, corresponding state, execution mode, and output policy. A generated implementation may replace a PyTorch [17] operation, fused block, or model; KernelBench supplies executable references and input generators [15]. The output policy permits specified differences, including floating-point variation from precision and evaluation order [5, Chs. 2 and 4]. Output comparisons require a valid reference and corresponding state; sampled agreement supports only the tested context. A native FP32 witness records a scored failure under the study’s FP32 contract; an additional-dtype witness concerns requested FP16 or BF16. These labels identify the requested dtype, not the failure cause (Section 4.3). Additional-dtype findings need not violate original publisher specifications. C3-23530’s LeNet-5 agrees on tested FP32 inputs but rejects Half tensors before custom CUDA launch (Table 13). Further FP32 draws cannot reveal that rejection. Execution also limits the evidence: C5-01485 defines custom CUDA, yet its published forward() only calls PyTorch layers (Section 5.7). Wrapper agreement cannot validate the dormant function. Even when the intended code executes under a valid contract, particular values or precision may be needed to expose discrepancies, motivating targeted tests.

4

3

Ziyuan Li et al.

CONTROLLED FAULT PROBES

We use controlled mutations to identify conditions that expose implementation differences and to construct an empirical prior for test selection [3, 6]. Figure 2 outlines probe construction, validity checks, and evidence classification. Controlled fault probes 90 accepted CUDA hosts → 1,646 first‐order probes

Static equivalence evidence 9 cases; anchor skipped Fixed‐harness scope

Dynamic controls + anchor

571 excluded

Passing H‐R control R‐M: ≤ 5 draws; atol = rtol = 10⁻² 539 killed / 527 survived 527 survivors

Unusable comparison or failed control

Survivor searches: H‐M on reference‐generated inputs Screening: up to 32 calls 20 random + 12 directed

no witness

Enhanced search: up to 103 calls 40 random + 21 value policies × 3 seeds no witness / incomplete

Observed H‐M difference

Unresolved

Does not alone establish R‐M failure

Survival does not establish equivalence

Searches use atol = rtol = 10⁻²; budgets are upper plans. FP16 follow‐up extends the domain. H: unmodified host M: mutant R: task reference

Fig. 2. Controlled-probe analysis. Nine cases with harness-scoped equivalence evidence bypass the anchor; dynamic records distinguish exclusions, anchor kills, and survivors eligible for further search. Budgets denote planned calls.

Probe construction. C1 contains 1,646 first-order probes from 90 previously accepted CUDA hosts, selecting the highest recorded speedup among available turns for each selected KernelBench L1/L2 task. Each probe applies one source rewrite at a Python AST or CUDA-pattern site, retaining at most three per host–operator pair. The 16 operators span arithmetic, GPU parallelism, ML numerical behavior, and broadcast/layout assumptions. Appendix A.5 lists operators, transformations, and observed counts. Counts reflect this construction, not natural-fault frequencies. Comparison and evidence. Dynamic evidence requires a usable comparison and a passing control for the unmodified host against the paired reference. The anchor compares mutant with reference on up to five FP32 draws at atol = rtol = 10−2 . Host controls establish agreement on their recorded inputs; they do not validate every input used by the survivor searches. Equivalence is scoped to the fixed harness [12]. Each of the nine screened cases changes an integer used only by candidate get_inputs, leaving CUDA, initialization, forward computation, and state unchanged. The harness calls reference generators instead, so compared behavior is unaffected (Appendix A.5). These cases bypass dynamic mutant testing; finite survival alone does not establish equivalence. Two searches seek inputs that distinguish dynamic anchor survivors from their unmodified hosts. Screening plans up to 32 calls (20 random + 12 directed). Probes without a witness are eligible for enhanced search with up to 103 calls (40 random + 21 value policies × three seeds). Both searches use task-reference inputs and 10−2 absolute/relative tolerances. Approximate agreement

KernelMutate: GPU Kernel Validation

5

is not transitive: a host-relative witness need not be a reference-relative failure. Two additional FP16 witnesses extend the observed domain. Search probes without witnesses remain unresolved; timeouts and incomplete plans are recorded separately. Section 5.5 gives the complete accounting. Fault-to-test prior. Candidate testing uses one fixed screening-derived map with 607 templatesupporting probes. Its singleton F-IDX association with the last-element boundary template lacks admissible support: the recorded rejection reason is nondeterminism. The template also has support from F-BOUND (8/9) and F-CONST (3/5), so it cannot be attributed solely to that association. The artifact binds the map and candidate plans; Section 5.5 measures the related witness dependence. Task-held-out recommendation analysis. We separately measure transfer of confirmed fault conditions by five-fold cross-fitting over whole tasks, whose probes share a host and reference. The training roster contains 539 anchor kills and 68 confirmed screening witnesses; one lacks a template identity, leaving 606 probes with usable template evidence. Evaluation includes all 75 primary witnessed survivors, with seven enhanced-search cases used only for evaluation. The two additional FP16 cases remain outside this population. Training-fold records rank templates by fault class or globally after pooling classes, using Equation (1); ties use training mean time, then template identity. We compare caps from one to eight without filling exhausted lists. Coverage requires an identified recorded witness; unexecuted or inconclusive templates remain unknown. Probe classes are known from construction, unlike the multiple source-derived hypotheses for natural candidates. This measures template matching across seeds, without candidate replay. 4

FAULT-GUIDED CANDIDATE VALIDATION

Given a candidate, paired reference, validation contract, and invocation budget 𝐵, KernelMutate constructs directed (M-dir) or full (M-full) plans (Figure 3) and records their verdicts. 2) Test Execution

1) Test Construction Candidate + reference contract + budget

value_stress

dtype_stress

21‐policy library Values + boundaries

FP16 / BF16 Portability domain

Fingerprint + fault map Directed allocation then general fill

training_stress

repeated_run

Training‐mode tests Diagnostic here

Repeated execution Diagnostic here

M‐full

config_stress

M‐dir

General pressure suite Fixed template order Alternative plans for the same candidate

Layout + batch variation Batch‐mode tests are diagnostic Reference, state and input checks Contract‐specific interface + numerical oracle

3) Verdicts Per attempt

PASS Agreement within the tested contract

FAIL Valid mismatch or candidate failure INCONCLUSIVE

No usable verdict was established Eligible FAIL → witness

Planned budget: 32 calls, including 24 score‐eligible calls; other modes are diagnostic.

Fig. 3. Evaluated plans across five testing dimensions. M-dir uses source sites and the fault-to-test map; M-full follows a fixed sequence. Validation stages determine verdicts; only score-eligible failures contribute discoveries. The plans also include unscored diagnostics.

6

4.1

Ziyuan Li et al.

Source Fingerprints and an Empirical Prior

Source fingerprints identify eligible fault classes, excluding sites matching prior-equivalence patterns. A class requires both a remaining site and an effective map template. Site counts determine inclusion, not weights or budget shares; matches establish neither a defect nor execution of matched code. The map ranks templates by conditional exposure rates. A template comprises policy, mode, and parameters; a case adds a seed. Let 𝑊 𝑓 contain class-𝑓 probes with identified witness templates in the map’s input records (Section 3). For template 𝑡, 𝐸 𝑓 ,𝑡 ⊆ 𝑊 𝑓 contains probes with a conclusive execution of 𝑡, excluding inconclusive and invalid-input outcomes. 𝐾 𝑓 ,𝑡 ⊆ 𝐸 𝑓 ,𝑡 contains those with a witness under the stage’s comparison: mutant–reference for anchor kills, mutant–host for screening differences. Each probe counts once per set across seeds. For nonempty 𝐸 𝑓 ,𝑡 , 𝑞(𝑓 , 𝑡) = |𝐾 𝑓 ,𝑡 |/|𝐸 𝑓 ,𝑡 |.

(1)

Class 𝑓 retains templates with |𝐾 𝑓 ,𝑡 | > 0, storing four-decimal scores and both counts. Counts reveal sparse support without correcting selection effects; the ranking is not a natural-candidate defect probability. 4.2

Constructing and Executing a Plan

M-dir combines directed selection and general tests within 𝐵 planned calls. Represented classes are prioritized by their best map score; their descending-score template lists are traversed round-robin, deduplicating templates. Ordinary cases cost one call; repeated cases cost their specified repetitions. A heuristic caps directed selection at ⌈0.7𝐵⌉ calls. At this cap or list exhaustion, unused templates are appended from a fixed general sequence covering IID/structured values, boundaries, dtype, repetition, and batch configurations. Cases exceeding a stage’s remaining budget are skipped while selection continues. Residual capacity cycles through the general sequence. The experimental plans specify every case’s order and seed, including repeated selections of a template. General tests also permit construction without source matches or effective templates; neither equal class allocation nor optimal selection is guaranteed. M-full follows a fixed sequence: 21 value policies covering magnitudes, sign patterns, sparsity, boundaries, and reduction-sensitive inputs, followed by dtype and execution-context cases. Plans and score eligibility appear in Table 3; execution and completion rules appear in Section 5.1, and cost accounting in Section 5.3. The comparison evaluates complete constructions and execution coverage, not simply reordered tests. 4.3

Validity and Verdicts

The main-study oracle uses differential testing to compare observable behavior [13]. With matched inputs and state, it compares returned values, input side effects, and post-call module state. Nested structure and all corresponding leaves must agree, including tensor shape, dtype, device, and layout type; stride and aliasing equality are not required. NaN positions must match, with matching NaNs accepted; infinity positions and signs must match. Integer and Boolean elements require exact equality. Finite floating-point reference and candidate elements 𝑟𝑖 , 𝑐𝑖 must satisfy |𝑟𝑖 − 𝑐𝑖 | ≤ atol + rtol|𝑐𝑖 |, using torch.isclose(reference, candidate) [19]. Absolute tolerance handles near-zero values; relative tolerance scales with candidate magnitude. We preset atol = rtol = 10−4 for FP32 and 10−2 for FP16/BF16 as acceptance policies, not universal numerical error bounds. Verdicts follow the stage reached and the evidence available at that stage. Reference import/setup failures yield Inconclusive. Candidate import failures (after successful reference import) and candidate model setup failures yield Fail. Model setup includes class lookup, construction, device/dtype

KernelMutate: GPU Kernel Validation

7

conversion, and mode setting. These Fail verdicts require neither reference forward execution nor completed state binding; the invocation-stage resource exclusion does not apply. Failures in validator state binding, snapshots, or input preparation yield Inconclusive, as do parent timeouts. During invocation, reference failures or recognized memory resource errors yield Inconclusive; other candidate exceptions after reference success, including compilation within forward, yield Fail. Valid comparisons yield Fail for numerical/interface mismatches and Pass for agreement. A strategy counts each candidate once upon any selected, score-eligible Fail. Records retain failure stage, case/source identities, policy, seed, mode, parameters, and available timing/error evidence. Within a strategy, native FP32 evidence takes precedence; dtype-only means additionaldtype failures without an observed native witness (Table 6). Passing observations concern only exercised inputs and callable; replacement tests check custom execution separately (Section 5.7). 5 EVALUATION We study the reliability of small-sample acceptance, the benefit and cost of stronger tests, and the interpretation of their outcomes through five research questions: RQ1: What native and additional-dtype failures remain among naturally generated candidates accepted by a small suite? RQ2: How do input strategies compare in witness yield and cost, and how much does numerical tolerance explain? RQ3: How do validators behave on independently labeled bugs and controls? RQ4: Which controlled faults escape the anchor, and how well do their test conditions transfer across tasks? RQ5: Which suite components retain observed discoveries, and why does dtype require explicit testing? A complementary replacement study examines network integration, reference binding, and custom execution. 5.1

Study Design

Subjects and selection. We evaluate 838 collected CUDA implementations from C2, C3, and C5 across 238 task identifiers [8, 11, 28]. Eligibility requires nonempty source, a resolvable KernelBench task, and static evidence of custom CUDA. Whitespace-normalized source hashes remove withincorpus duplicates. C2 includes all 85 eligible accepted records. C3 contributes 227 candidates, selecting the lexicographically smallest full stable source ID among each task’s eligible accepted records. C5 supplies 526 eligible candidates accepted in its published baseline-evaluation pool. Collection acceptance labels are distinct from B1u acceptance. C1, C4, and C6 have distinct roles and are excluded from natural-candidate denominators (Table 1). C1 informs test construction; C4 is inventory only. C6 retains third-party bug/control labels [25] and compares outputs to its supplied FP64 reference via run(dict). Candidates are identified by corpus, task, and stable source ID; different implementations of a task remain separate. We report comparisons and intervals for the 753-candidate C3+C5 panel alongside the full 838-candidate population. C2, C3, and C5 overlap with C1 on 38, 90, and 85 tasks, covering 38, 90, and 316 candidates, respectively. Task-held-out cross-fitting evaluates recommendations; natural-candidate experiments evaluate complete strategies on the collected population. The sample is nonrandom; corpus membership does not establish source disjointness. Comparison domains and validity. Main observations concern the specified callable and its validation stages under the study contract. Contracts fix recorded input shapes and adopt unrestricted

8

Ziyuan Li et al.

Table 1. Corpus inventory and roles. Counts denote implementations or probes, not necessarily distinct tasks. ID

Collection

Backend

Count

Experimental role

C1

KernelBench L1/L2 mutations CUDA-L1 AI-CUDAEngineer TritonBench-G KernelBenchsamples

CUDA

1,646

CUDA CUDA

85 227

Controlled fault conditions and evidence for test construction Natural candidates and replacement study Natural candidates and replacement study

Triton CUDA

184 526

Inventory only: paired references not established Natural candidates and replacement study

Triton/CUDA, CPU

26

Third-party labels: ten seeded bugs and 16 controls

C2 C3 C4 C5 C6

gpuemu

finite values and determinism as study defaults, not independently audited publisher specifications. Native FP32 witnesses concern this contract; additional FP16/BF16 findings concern replacement support. Low-precision obligations remain unestablished for C2/C3; C5’s recovered public evaluation setting is FP32. B1u, B3–B6, M-full, and M-dir share the numerical/interface oracle in Section 4.3. Output comparisons require usable reference and state bindings; candidate import/setup failures follow the separate stage rules there. Replacement tests separately check custom execution. Strategies and budgets. The strategy panel separates additional sampling, value and context variation, and complete fault-informed constructions (Table 2). B7–B9 retain the external policies’ inputs and decision rules [9, 31]; B10 checks dynamic CUDA properties [14]. B12 applies fixed tolerance to the full-suite schedule on the selected cohort in Section 5.3. Table 2. Validator strategies and roles. IDs remain fixed across studies; interfaces depend on the corpus. ID

Strategy

Input construction and experimental role

B1 B1u B3 B4 B5

Default-style anchor Dtype-aware anchor IID-32 Value-diversified Boundary values

B6

Composite context

B7 B8 B9

robust-kbench-style KernelBenchX-style Differential fuzzing

B10 B11

Compute Sanitizer M-dir

B12 B13

M-fixed M-full

Five IID inputs; fixed atol = rtol = 10−2 Five IID inputs; FP32 tolerance 10−4 , FP16/BF16 10−2 Unchanged input distribution and execution context; 32 seeds Eight value distributions × four seeds Eight boundary policies × four seeds: zeros, extreme magnitudes, and boundary elements Eight dtype cases (FP16/BF16 × four seeds), eight training cases (IID/near-zero × four), and 16 batch cases ({1, 4, 16, 64}× four); no fault prior External strategy port; its supported inputs and oracle External strategy port; its supported inputs and oracle Seeded FP64 CPU-reference port; operator- and dtype-calibrated tolerances Dynamic CUDA error/hazard checking Code fingerprints and fault-informed construction; nominal ≈ 70% directed allocation with general budget fill Full-suite input schedule with fixed 10−2 absolute/relative tolerance 21 value policies plus dtype, training, repeated-execution, and configuration tests; 32-call limit

KernelMutate: GPU Kernel Validation

9

Plans distinguish logical cases, nominal invocations, and score-eligible invocations (Table 3). Invocations count candidate calls, not CUDA launches; repeated cases consume multiple calls. Nonanchor controlled plans allow at most 32 calls, with at most 23 directed calls for M-dir. Training, batch-mode, and repeated-mode diagnostics remain unscored. Main runs stop at the first Fail per candidate–strategy pair or after five consecutive timeouts; each test-case worker has a 180-second parent-process timeout. Completion runs attempt eligible cases omitted by these guards within the same plan, with failure stopping disabled. Reference/validator preparation failures and timeouts remain inconclusive. Each candidate–strategy–case contributes at most one selected observation; confirmations do not enlarge denominators. Table 3. Per-candidate plans for 838 natural candidates. Logical cases, nominal invocations, and score-eligible invocations are counted separately.

Strategy

Logical cases

Nominal calls

Eligible calls

B1u B3 B4 B5 B6 M-full

5 32 32 32 32 31

5 32 32 32 32 32

M-dir

30

32

Eligible composition

Other planned calls

5 32 32 32 8 24

Five IID 32 IID 32 value 32 boundary Eight dtype 21 value; two dtype; one layout

24

20 value; four dtype

None None None None Eight training; 16 batch Two training; four batch; two repeat Four batch; four repeat

Measures and inference. For natural candidates, witness yield is |𝐷𝑠 |/𝑁 : candidates with a scoreeligible failure divided by the full population, including inconclusive candidates. It is not defect recall. First-witness time accumulates recorded calls through the first scored failure, including preceding unscored calls; reported medians condition on 𝐷𝑠 and use the upper middle observation for even counts. Cost separately aggregates charged testing wall time per 1,000 targets under the scopes in Table 7. C3+C5 yield-difference intervals use paired task-cluster bootstrap resampling: tasks are drawn with replacement, retaining their implementations and paired strategy outcomes (10,000 resamples, seed 20260819). The intervals measure resampling uncertainty, not execution variability, and do not correct nonrandom selection. The main and replacement studies use A800 (PyTorch 2.1.2, CUDA 12.1) and RTX 4090 (PyTorch 2.5.1, CUDA 12.4), respectively; B9 uses CPU. Cross-platform timings are not pooled. The main-study records do not establish global TF32, autocast, or reduced-precision-reduction settings; replacement settings are stated separately. OpenAI Codex assisted implementation of CPU scripts for witness aggregation, cost accounting, cohort selection, and offline rankings. Outputs were checked against recorded identities, hashes, and separate aggregation implementations. 5.2

RQ1: Evidence After Small-Sample Acceptance

All 548 B1u-accepted candidates have exactly five valid Pass observations and no inconclusive outcome. Of 838 candidates, B1u rejects 177 and leaves 113 undecidable. Acceptance requires at least one valid Pass and no Fail; all-inconclusive candidates are not accepted. We classify accepted candidates using shared-oracle evidence pooled from B3–B6, M-full, and Mdir (Table 4). Native FP32 evidence takes precedence; dtype-only means additional-dtype evidence

10

Ziyuan Li et al.

without a pooled native witness. Remaining candidates with B7–B10 findings are external-only; the rest have no qualifying finding. Precedence yields disjoint reporting categories despite overlapping witness sets. Table 4. Findings pooled across strategies for 548 B1u-accepted candidates. Percentages use 548; dtype-only findings are additional-domain evidence. Evidence category

C2

C3

C5

Total (%)

Native FP32/value: B3–B5, M-full, M-dir Dtype only: B6, M-full, M-dir External validators only: B7–B10 No qualifying finding

9 4 8 33

24 47 0 0

103 311 4 5

136 (24.8) 362 (66.1) 12 (2.2) 38 (6.9)

B1u-accepted candidates

54

71

423

548 (100.0)

Native symptoms include finite-value and NaN/Inf-position differences in outputs or side effects, rather than source-level defect categories (Appendix A.1). Additional-dtype findings mostly concern execution support. Among the 362 dtype-only candidates, 359 fail to execute on low-precision inputs while the reference succeeds; three C2 candidates return output discrepancies. The disjoint native and dtype-only categories give 498/548 (90.9%) with shared-oracle witnesses. Ten B9 and two sanitizer-only findings raise this count to 510/548 (93.1%) across mixed rules. Neither fraction estimates native-defect prevalence; the 38 residual candidates are not certified correct. Interpreting native witnesses. Native witnesses expose different contract divergences. In C2 L1P19, relu_ mutates negative inputs while the reference’s relu preserves them, a side effect invisible on nonnegative draws. In C2 L1-P37, denormals expose differing infinity positions: the candidate adds 10−12 before reciprocal square root, whereas the reference divides by an unregularized norm. The recorded size selects PyTorch fallback. These cases distinguish input-side-effect semantics from numerical conventions; neither establishes an executed custom-CUDA defect. Appendix A.1 gives identities and error evidence. Answer. Small-sample acceptance leaves both native discrepancies and additional-dtype limitations unchecked. The latter concern replacement support; a violation of the original specification additionally requires an established obligation. 5.3

RQ2: Effectiveness, Cost, and Numerical Strictness

Discovery under shared validation rules. On all 838 candidates, IID-32 yields 179 detections versus 177 for the five-draw anchor: three new detections and one not reproduced, a net gain of two (Table 5). The runs use different seed sets. Value-diversified and boundary strategies yield 289 and 309. All 682 B6 detections arise in its eight planned eligible dtype cases; training/batch diagnostics remain unscored. M-full and M-dir detect the same 692 candidates: all 179 IID-32 detections plus 513 others (3.87 times its yield). On the collected 753-candidate C3+C5 panel, their yield gain over IID-32 is 66.4 percentage points, with a paired task-cluster 95% interval of [60.8,71.5] over 238 task clusters. Gains over B4 and B5 are 52.7 [46.6,58.7] and 50.3 [44.1,56.4] points. These are panel-specific yield differences, not defect recall. The 692 discoveries have disjoint stage categories: both strategies flag 58 candidates during preparation (three import, 55 model setup); M-full additionally records 409 candidate-invocation

KernelMutate: GPU Kernel Validation

11

Table 5. Discovery by corpus and comparison population. Panel (a) shares an oracle; panel (b) retains each external strategy’s own decision rule. (a) Shared-protocol findings: count (percentage) Strategy C2 (85) C3 (227)

C5 (526)

C3+C5 (753)

All (838)

B1u B3 B4 B5 B6 M-full M-dir

65 (12.4) 66 (12.5) 150 (28.5) 164 (31.2) 492 (93.5) 492 (93.5) 492 (93.5)

160 (21.2) 162 (21.5) 265 (35.2) 283 (37.6) 662 (87.9) 662 (87.9) 662 (87.9)

177 (21.1) 179 (21.4) 289 (34.5) 309 (36.9) 682 (81.4) 692 (82.6) 692 (82.6)

17 (20.0) 17 (20.0) 24 (28.2) 26 (30.6) 20 (23.5) 30 (35.3) 30 (35.3)

95 (41.9) 96 (42.3) 115 (50.7) 119 (52.4) 170 (74.9) 170 (74.9) 170 (74.9)

(b) External findings: counts under the respective oracles Strategy Decidable /838 C2 C3

C5

C3+C5

All

B7 B8 B9 B10

115 144 308 41

171 201 449 50

189 222 474 50

628 643 496 778

18 21 25 0

56 57 141 9

failures and 225 observable-behavior mismatches, versus 424 and 210 for M-dir. The remaining 96 candidates are all-inconclusive because state binding is unusable; 50 have passing observations without a failure. Observed domains and overlapping discoveries. The strategies cover the same 498 accepted candidates but provide different evidence (Table 6): 16 have native witnesses in M-full and only dtype witnesses in M-dir. Native evidence takes precedence within each strategy. Construction and stopping change observation opportunities, so these partitions do not measure latent domain recall. Table 6. Observed witness domains by strategy. Native and dtype-only are disjoint within each strategy; early stopping limits which domains are observed. All candidates (838)

B1u-accepted (548)

Strategy

Native

Dtype-only

Native

Dtype-only

B1u B3 B4 B5 B6 M-full M-dir

177 179 289 309 0 313 297

0 0 0 0 682 379 395

0 3 113 132 0 136 120

0 0 0 0 491 362 378

Independently executed B6 and B5 satisfy 𝐷 𝐵6 ∪ 𝐷 𝐵5 = 𝐷 M-dir = 𝐷 M-full over all 838 candidates. All ten additions beyond B6 are native C2 discrepancies; B5 finds all ten and B4 nine. This observed union supplies no measured budget or cost for a combined strategy and does not isolate a unique discovery benefit of source fingerprints or fault information. External strategies. Table 5(b) reports findings and decidable coverage under each external strategy’s inputs, oracle, and validity conditions. Nominal 32-call charges do not equalize decidable

12

Ziyuan Li et al.

comparisons. Sanitizer findings require positive dynamic error/hazard counts. All 838 targets remain in the population, but these results cannot form a common semantic-recall ranking. Testing cost. Charged cost aggregates retained in-scope time rather than total physical execution. It includes original-phase selected observations and each completion stage’s in-scope prefix through its first Fail in plan order. Across 838 candidates, M-full and M-dir charge 65.14 and 48.14 GPU-node hours (26.1% lower; Table 7). Their first-witness medians over the shared 692 detected candidates are 108.67 and 91.45 seconds; at 80 seconds, M-full finds 257 versus 234 for M-dir. Different plans and stopping preclude a controlled reordering interpretation. Counting all selected in-scope time or all selected time, including diagnostics, gives differences of 25.36% and 25.41%, respectively. These scope checks exclude unretained completion attempts and do not recover total physical execution; Appendix A.2 documents record selection and accounting. Table 7. Recorded timing and charged cost. Medians condition on detection; Total is node wall-hours per 1,000 targets, except B10.

Strategy

Node

Median first (s)

Detected by 80 s

Total

B1u B3 B4 B5 B6 B7 B8 B9 B10 M-full M-dir

GPU GPU GPU GPU GPU GPU GPU CPU GPU GPU GPU

52.1 52.7 54.3 55.7 52.9 — — — — 108.7 91.4

154 150 218 199 638 — — — — 257 234

24.7 87.0 112.0 114.4 23.7 37.6 ≥ 121.9 185.4 89.1† 77.7 57.5

Dashes denote unavailable comparable timings. B8 has timing for 834/838 targets, hence a lower bound. †B10 uses 69.32 recorded hours over 778 executed candidates (89.1 h/1k), or 82.7 h/1k over 838 targets without imputing the other 60. Other rows use 838. Appendix A.2 gives first-call and remaining costs; these do not isolate initialization and execution.

B6 finds 682 candidates at 23.7 h/1k, including 638 by 80 seconds. Its recorded dtype prefix retains all 682 at 17.9 h/1k over 838 targets; standalone prefix execution was not measured. B6’s total includes diagnostics excluded by M-full/M-dir accounting. B7/B9 timings cover all 838 attempts, including failures; whole-batch durations cannot recover their first-failure timestamps. Input schedule versus oracle. The oracle comparison uses 668 candidates selected by a prior raw kill flag in the anchor or M-full, regardless of final score eligibility (170 C3, 498 C5; Table 8). The other 85 candidates in the 753-candidate panel did not meet this condition and were not selected for fixed-tolerance reruns. Within this selected cohort, the dtype-aware anchor detects all 109 fixed-oracle discoveries plus 51 others, and both full-suite executions detect the same 661 candidates. Higher full-suite yield thus persists with fixed tolerance in this cohort; it does not estimate the oracle’s contribution to gains over the full population. Case-level verdicts differ despite equal full-suite discovery sets (Appendix A.3). These are matched cases from independent executions, not rescored saved outputs; stopping also changes which cases are observed.

KernelMutate: GPU Kernel Validation

13

Table 8. Oracle sensitivity on 668 candidates selected by prior raw kill flags, using aligned plans from independent executions. Dtype-aware tolerances tighten FP32 to 10−4 ; FP16/BF16 remain at 10−2 . Input schedule Five-draw IID Full suite, up to 32 calls

Fixed 10−2

Dtype-aware

B1: 109/668 M-fixed: 661/668

B1u: 160/668 M-full: 661/668

Answer. Directed construction retains the full suite’s discovery set at lower charged in-scope cost, with different witness domains. The baseline overlap precludes claiming a unique discovery benefit of fault information. Broader testing retains its yield gain under both tolerances on the selected oracle cohort. 5.4

RQ3: Independent Labels and Control Alarms

The C6 campaign adapts eleven strategies to gpuemu’s ten seeded bugs and 16 controls. FP64 reference outputs are rounded to the input dtype. Ordinary adaptations use the operator/dtype metadata as both absolute and relative tolerances, with the relative term scaled by the reference; B1 fixes both at 10−2 , whereas B9 uses the metadata as an absolute bound only. Thus the adaptations do not share one numerical gate. B6 uses a dtype/shape grid. Any failure flags a subject; every subject–strategy pair has a conclusive observation. All eleven strategies detect all ten bugs (Table 9), so these saturated results cannot establish superior recall. M-dir flags two of 16 controls versus seven for M-full; several baselines raise none. Both plan 32 calls per subject. M-full records 806 decidable observations; M-dir records 728, with 52 additional BF16 cases inconclusive because input conversion fails before kernel invocation.

Table 9. Seeded-bug detection and control alarms in the adapted C6 campaign. Each row tests ten seeded bugs and 16 controls; control labels remain unchanged. Strategy B1, B1u, B3, B4, B6, B7, B9 (each) B5 Boundary values B8 KernelBenchX-style M-full M-dir

Bugs detected /10

Controls flagged /16

10 10 10 10 10

0 7 1 7 2

Anchors use their gpuemu adaptations. Compute Sanitizer and M-fixed are not additional C6 rows.

Control alarms have unresolved numerical significance. M-full’s seven flagged controls yield 14 FP32 near-overflow failures: six NaN-pattern mismatches and eight tolerance violations. Mdir’s three FP32 alarms involve all-negative/reduction-adversarial attention_triton and reductionadversarial NumPy matmul. Absent outputs and elementwise errors prevent further adjudication; independent labels remain unchanged. Appendix A.4 gives policies, tolerances, and reconstructed shapes. Answer. M-dir flags fewer controls than M-full with the same saturated bug detection. Unequal test exposure prevents isolating a guidance effect; the results establish neither zero false positives nor superiority to zero-alarm baselines.

14

5.5

Ziyuan Li et al.

RQ4: Controlled Faults and Test-Condition Transfer

Table 10 separates excluded probes, harness-scoped equivalents, anchor kills, witnessed survivors, and unresolved cases. Appendix Sections A.5–A.7 give operator, control, and witness-condition details. Table 10. Complete C1 dispositions. The 77 witnessed survivors comprise 68 screening, seven enhanced-search, and two additional FP16 witnesses. Disposition

Probes

Evidence and scope

Excluded

571

Anchor-killed Harness-equivalent Witnessed survivors Unresolved

539 9 77 450

105 compile/stillborn; 441 host-control failures; 25 other unusable comparisons Mutant–reference failures under the anchor Static dependency evidence; anchor not executed Host-relative differences in the stated search domains No established equivalence or distinguishing witness

Total

1,646

All generated probes

Of 459 probes without a screening witness, 453 have enhanced-search records; six lack an executable plan. After the two additional FP16 witnesses, 450 remain unresolved. Searches compare mutant with host (Section 3), so a witness need not be a same-reference anchor failure. Timeouts and incomplete plans do not establish equivalence. There are 616 probes with established non-equivalence: 539 anchor kills and 77 witnessed survivors. The conditional fraction not exposed by the anchor is 77/616 = 12.5% over the expanded witness domain, including two FP16 cases beyond the FP32 anchor. It is not a same-domain FP32 false-negative rate. Excluding 450 unresolved probes restricts that estimate rather than resolving them. Saved witness and trial fields partition the 77 probes into 69 structured-value, five randominput, two additional FP16 cases, and one confirmed crash without a template identity. These observation categories do not establish mutually exclusive causes: random-input witnesses do not prove tolerance absorption, and comparison-pair changes limit input or precision attribution. Dependence on prior support. The last-element template occurs twice in all 838 natural-candidate M-dir plans, with 716 selected scored observations across 565 candidates: 503 Pass, 211 Inconclusive, and two Fail. Masking its observations retains 690/692 discoveries and 295/297 native discoveries. The two affected candidates each have a one-element FP32 tolerance violation (Appendix A.6). This masking analysis measures fixed-witness dependence on the template, not deletion of one association: other classes and general fill can select it, and association-level construction traces are unavailable. Task-held-out recommendations. Under Section 3’s confirmed-evidence protocol, class-conditioned recommendations cover 48/75 = 64.0% of the primary witnesses. Seven probes have insufficient coverage evidence; 20 have conclusive records for every recommended template without a witness among them. The missing-template probe remains in the denominator. Global top-eight covers 51, with insufficient evidence for the remaining 24. Observation availability differs across the rankings. At cap three, class lists have conclusive records for 139/192 recommendations versus 104/225 globally. To assess ordering on shared evidence, we intersect each probe’s two top-eight lists with its conclusive templates, retain both rankings, and take the first 𝑘. Among 73 probes with nonempty pools, class/global coverage is 28/8 at 𝑘 = 1,

KernelMutate: GPU Kernel Validation

15

Table 11. Offline recommendation evidence on 75 held-out probes. “No witness” means all recommended templates have conclusive records without a witness; unknown means insufficient evidence. Recommendation totals are not executed calls. Per-probe cap

1

2

3

4

5

6

7

8

Class covered /75 Class no witness Class unknown Global covered /75 Global no witness Global unknown Class recommendations Global recommendations

11 20 44 0 6 69 75 75

29 20 26 3 5 67 146 150

46 20 9 22 2 51 192 225

46 20 9 35 2 38 216 300

48 20 7 35 2 38 228 375

48 20 7 41 1 33 228 450

48 20 7 48 0 27 228 525

48 20 7 51 0 24 228 600

45/47 at 𝑘 = 2, and 47/47 at 𝑘 = 3. Pools contain at most four templates, making equality at larger 𝑘 inevitable. This conditioning excludes unique and unobserved templates and does not correct informative missingness or early stopping. Only six probes have fully conclusive records for both unchanged cap-one lists, and none at caps seven or eight (Appendix A.6). Answer. Witness conditions and recorded coverage are measurable, but the ranking comparison depends on observation support. Task-held-out matching and its conditional sensitivity do not establish deployed detection or runtime gains. 5.6

RQ5: Component Contribution and Dtype

The removal study conditions on 609 previously detected C3+C5 candidates: 165 C3 and 444 C5, comprising 401 level-1 and 208 level-2 implementations. Each variant removes one dimension and applies its remaining-suite and budget-fill construction. Table 12 measures retention on this selected cohort, not full-population discovery. Table 12. Component analysis on 609 detected candidates. The first three numeric columns describe executed removal-and-refill constructions. The last masks observed M-full witnesses, without constructing or executing new tests.

Construction

Detections

M-full Without value

609 609

Without dtype

269

Without training Without repeat Without configuration

607 606 602

Retention (%)

Inconclusive

Candidates after masking

100.0 100.0 44.2 (at most 44.8) 99.7 99.5 98.9

0 0

609 339

4

270

0 0 0

609 609 609

Witness masking asks which observed failures depend on a dimension. The fixed M-full evidence contains 270 value-domain and 339 dtype-domain candidates; masking either retains the other. No other category contributes scored witnesses. Early stopping leaves tests unobserved, so masking cannot predict a reconstructed plan. Executed removal-and-refill changes testing opportunities. Removing dtype loses 340 detections; even if all four inconclusive cases yielded witnesses, at least 336/609 = 55.2% remain lost. Losses

16

Ziyuan Li et al.

after removing training, repeat, or configuration do not isolate those dimensions: their diagnostics are unscored, while reconstruction changes eligible tests. Table 13. Dtype cases with native controls and execution evidence. Additional-dtype support is distinguished from the publisher’s original guarantee. Candidate / task

FP32 control

Additional-dtype observation

C3-23530 / LeNet-5

48 applicable inputs agree; four custom ReLU launches 32 original-distribution inputs agree

FP16/BF16 reject the first input at a Float pointer check, before custom launch FP16 CUDA executes; 723,115 boundary elements exceed tolerance (max absolute difference 1.0), masked by readout; BF16 rejected

C3-00995 / Matmul

C3-23530’s dispatch rejection in Table 13 also occurs in the main campaign after 21 passing FP32 policies, with a valid reference. For C3-00995, scalar-type accumulation is consistent with precision loss, but a unique repair-based cause is unestablished; its FP32 large-value witness precludes a dtype-exclusive attribution. Appendix A.8 gives additional case evidence. Answer. Dtype is necessary to retain this suite’s observed discoveries on the selected population. Passing FP32 controls neither exercise low-precision dispatch nor establish accurate computation after that dispatch succeeds. 5.7

Replacement Study: Binding and Execution Boundaries

Design and controls. We attempt 838 candidates in three requested dtypes, totaling 2,514 configurations. Network candidates retain their model; applicable operators use a deterministic shallow readout, such as ReLU and adaptive average pooling to eight features. Unsuitable cases retain explicitly identified direct comparisons. Inputs are synthetic tensors, not labeled images; outcomes concern tensors, not classification accuracy. Each configuration keeps its shape and plans at most 32 original-distribution plus 16 applicable structured inputs, with early stopping and a 300-second timeout. Candidate and reference receive synchronized corresponding state and identical inputs. C3 uses the paired functional model’s default Python path as reference and compiled entry as candidate; C2/C5 use a reference copy and synchronized published implementation. Execution uses evaluation mode, no gradients/autocast, and disabled TF32. Absolute/relative tolerances are (10−2, 10−2 ) for FP32/FP16 and (10−2, 5 × 10−2 ) for BF16, separate from the main oracle. Native-branch guards, explicit attachment of an existing custom entry, or direct execution evidence address fallback; published algorithms are not repaired. Requested and actual dtype, custom execution, and comparison validity are separate observations. Reference/state controls support the specified bindings without proving every adapter correct; attribution also uses source, execution traces, and reproducible witnesses. Complete outcomes. The 18 final-output discrepancies and two runtime failures involve 17 candidates. Fourteen local/direct differences comprise nine masked downstream and five without a readout. Appendix Table S9 enumerates the 19 subcategories of the 523 other outcomes. Agreement remains finite evidence. Evidence beyond main-study non-detection. Among 146 candidates without a scored M-full failure, 96 are all-inconclusive and 50 have only passing observations of the tested callable. Joining stable identities yields the four FP32 final-output/runtime witnesses in Table 15. The first three

KernelMutate: GPU Kernel Validation

17

Table 14. All 2,514 replacement outcomes by corpus and requested dtype. Agreement requires evidence of custom execution. Every terminal category remains in the denominator.

Corpus

Dtype

Configs

Agree

Final diff.

Runtime

Local/ direct

Dtype reject

Other

C2 C2 C2 C3 C3 C3 C5 C5 C5

FP32 FP16 BF16 FP32 FP16 BF16 FP32 FP16 BF16

85 85 85 227 227 227 526 526 526

4 0 0 135 7 0 496 0 0

3 2 1 9 0 0 3 0 0

0 0 0 1 0 0 1 0 0

1 0 0 5 1 0 7 0 0

0 6 7 0 138 148 0 508 508

77 77 77 77 81 79 19 18 18

Total

—

2,514

642

18

2

14

1,315

523

require binding changes; C3-17320 also changes reference. C5-00631’s failure at input 37 therefore does not establish budget as its main-study obstacle. Four further candidates have only downstreammasked differences and remain outside this category. Table 15. All four FP32 final-output/runtime witnesses among candidates without a scored M-full failure. Cross-study changes restrict false-negative attribution. Candidate

Replacement witness

Comparison scope

C3-17320

Linear–Sigmoid–Sum–LogSumExp returns [1] instead of scalar []

C3-20564

GEMM–GroupNorm–Min–Bias readout returns [128, 8] instead of [1, 8] LayerNorm on constant-0.1 input: final max absolute difference ≈ 0.128; first failure at test 37 MobileNetV1 custom Conv+BN+ReLU returns three dimensions; next layer raises dimension error

Main binding inconclusive; recovered reference uses a different operation sequence Main state binding inconclusive; replacement binding permits comparison Main parameter binding inconclusive

C5-00631

C5-01485

Main Pass runs PyTorch-only forward; replacement executes the dormant custom entry

A dormant custom implementation. C5-01485’s 24 main Pass observations exercise PyTorch-only forward. Its custom function returns [32, 224, 224] instead of [10, 32, 112, 112]: the wrapper allocates {out_channels, height, width}, omitting batch and stride reduction. One custom launch and passing reference/state controls support this attribution (Appendix A.9). Numerical strictness cannot repair bindings or invoke an unattached function. Replacement witnesses remain separate from main counts: a replacement claim needs the intended custom code executing with usable bindings and explicit input obligations. 6

THREATS TO VALIDITY

Construct validity. Study contracts preset input domains, determinism, and tolerances without establishing publisher obligations. We distinguish native FP32, additional-dtype, and external-tool

18

Ziyuan Li et al.

findings. Yield includes preparation and invocation failures alongside comparison mismatches, so it is not defect recall. C6 control alarms are unadjudicated and do not establish precision on natural candidates (Section 5.4). Internal validity. Protocol choices affect attribution even with reference/state controls. Replacement witnesses have distinct bindings and execution paths (Section 5.7). The unsupported prior association is localized, but template masking cannot identify the effect of removing that association on plan construction (Section 5.5). No executed source-agnostic strategy holds eligible budget, dtype allocation, and stopping constant; guidance and ordering effects remain unisolated. C6 exposure differs, and removal-and-refill conditions on 609 detected candidates. External validity. The natural collections are nonrandom KernelBench-derived samples with incomplete generation provenance and task overlap with C1. Task-held-out cross-fitting measures confirmed-probe template coverage, not task-disjoint natural-candidate performance. Host selection, exclusions, and unresolved probes restrict witness-condition frequencies; C6 is a small adapted seeded corpus. Fixed shapes, synthetic inputs, and selected replacement networks leave broader workloads and GPU architectures untested. Replacement results establish neither online optimization convergence nor end-to-end search-cost benefits. Conclusion validity. Task-cluster intervals address dependence, not execution variability or nonrandom selection. Prior raw kill selection and independent executions limit oracle-panel inferences. Offline rankings aggregate seeds over partial observations; conditioning on shared observable templates does not remove selection from stopping or timeouts. Probe classes are known, whereas natural fingerprints supply hypotheses. Source-ledger checks bound original-record selection, but unretained completion attempts and global main-study precision settings remain unverified. Charged costs describe retained observations, and first-witness medians condition on detection. Confirmations add no independent units. 7

RELATED WORK

Kernel generation and validation. KernelBench [15] compares sampled outputs with PyTorch and measures runtime; released CUDA collections [8, 11, 28] supply our candidates. robust-kbench [9] varies initialization and configuration and uses LLM screening. KernelBenchX [31] separates call validity, output agreement, and efficiency; KernelBench-Verified [32] adds hidden distributions and a TF32-enabled baseline. Our strategy ports retain their respective inputs and oracles. Li et al. [10] study Triton and TileLang kernels that pass numerical checks yet run substantially slower than library baselines, separating replacement efficiency from numerical agreement. Our evaluation concerns validation outcomes and testing cost, not the execution efficiency of replacement kernels. GPU and tile-program bug characterization. Rathnasuriya et al. [20] characterize 397 GPU numerical bugs and develop GPU-NBDetect using bug-triggering inputs and numerical oracles. Their tile-program study [21] analyzes 301 code-generation bugs, including dependencies on tensor shapes, types, and backend transformations. These studies characterize observed defects; our controlled probes measure which faults survive a specified validator and which test conditions expose them. Oracle measurement and candidate audit. Du et al. [4] combine mutation analysis, validity checks, per-problem suite selection, and taxonomy-informed generation, holding out mutants within each problem. Our analysis holds out whole tasks and compares class-conditioned with global rankings under partial observation. Shah and Shrestha [29] audit accepted kernels using applicable contract gates and positive controls. Our findings connect equal candidate yield with different witness

KernelMutate: GPU Kernel Validation

19

domains, recommendation limits with observation coverage, and replacement verdicts with binding and custom execution. Numerical testing and verification. The gpuemu studies examine high-precision references [25], tolerance calibration [26], and input generation [27]; we retain their labels when assessing alarms. ProofWright [1] proves memory safety and race freedom under inferred preconditions, with semantic equivalence for restricted element-wise kernels. Compute Sanitizer [14] detects dynamic safety hazards. These guarantees differ from the numerical witnesses measured here. Mutation analysis and fault-informed testing. MUTGPU [33] targets CUDA C; CLTestCheck [18] assesses OpenCL tests through coverage, fault seeding, and schedule amplification. Mu2 [30] uses online mutation feedback during Java greybox fuzzing. Our source-guided selection uses an offline prior without mutating new candidates. Equivalent-mutant techniques include compiler checks [16] and targeted suppression [7]; we retain the distinction between harness-scoped equivalence and unresolved survival [12]. Uncertainty in code models. CodeImprove [24] combines a model-derived input-validity score with semantics-preserving program transformations to improve code-model predictions. A study of code generation finds that confidence and entropy provide incomplete, model-dependent evidence of execution failure [23]. For defect prediction and vulnerability detection, improved calibration scores do not consistently improve selective prediction [22]. These approaches assess or improve model predictions; KernelMutate records execution-based witnesses under explicit input and oracle contracts. Our Inconclusive verdict denotes missing valid execution evidence, rather than a threshold on model confidence. 8

CONCLUSION

KernelMutate shows that sampled acceptance leaves native discrepancies and additional-dtype limitations unchecked, while wrapper agreement can leave custom code untested. Recommendation comparisons depend on both ranking and observable evidence; shared-template analyses limit generalization from recorded coverage. Replacement decisions therefore need witness domains, comparison contracts, and actual execution paths alongside candidate yield and charged cost. ACKNOWLEDGMENTS This work was partially supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM118) and National Natural Science Foundation of China under Grant Nos. U25A6023 and 92464301. Tao Xie is also affiliated with Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, China; Shanghai Institute of Systems for Open Computing, China. REFERENCES [1] Bodhisatwa Chatterjee, Drew Zagieboylo, Sana Damani, Siva Hari, and Christos Kozyrakis. 2025. ProofWright: Towards Agentic Formal Verification of CUDA. https://doi.org/10.48550/arXiv.2511.12294 arXiv:2511.12294v2 [2] Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Shubham Agarwal, Mert Cemri, Bowen Wang, Alex Krentsel, Tian Xia, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya Agrawal, Ashwin Naren, Shulu Li, Ruiying Ma, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica. 2025. Let the barbarians in: How AI can accelerate systems performance research. arXiv:2512.14806 [3] Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34–41. https://doi.org/10.1109/C-M.1978.218136 [4] Mingzhe Du, Anh Tuan Luu, Dong Huang, and See-Kiong Ng. 2026. Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles. arXiv:2609.22220v1 [cs.LG] https://arxiv.org/abs/2609.22220v1

20

Ziyuan Li et al.

[5] Nicholas J. Higham. 2002. Accuracy and Stability of Numerical Algorithms (2 ed.). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA. https://doi.org/10.1137/1.9780898718027 [6] Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. https://doi.org/10.1109/TSE.2010.62 [7] Benjamin Kushigian, Samuel J. Kaufman, Ryan Featherman, Hannah Potter, Ardi Madadi, and René Just. 2024. Equivalent Mutants in the Wild: Identifying and Efficiently Suppressing Equivalent Mutants for Java Programs. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 654–665. https: //doi.org/10.1145/3650212.3680310 [8] Robert Tjarko Lange, Aaditya Prasad, Qi Sun, Maxence Faldor, Yujin Tang, and David Ha. 2025. The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition. Sakana AI technical report and accompanying kernel archive. https://huggingface.co/datasets/SakanaAI/AI-CUDA-Engineer-Archive [9] Robert Tjarko Lange, Qi Sun, Aaditya Prasad, Maxence Faldor, Yujin Tang, and David Ha. 2025. Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization. arXiv:2509.14279v1 https://arxiv.org/abs/2509.14279v1 [10] Tingxi Li, Ravishka Rathnasuriya, and Wei Yang. 2026. Correct but Slow: An Empirical Study of the GPU Kernel Evaluation Gap in Modern Domain-Specific Languages. https://doi.org/10.48550/arXiv.2607.04454 arXiv:2607.04454v3 [cs.SE] [11] Xiaoya Li, Xiaofei Sun, Albert Wang, Jiwei Li, and Chris Shum. 2026. CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning. In The Fourteenth International Conference on Learning Representations. https: //proceedings.iclr.cc/paper_files/paper/2026/hash/c94bbbef466ab1b2cfa100e41413b3a8-Abstract-Conference.html [12] Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Józala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering 40, 1 (2014), 23–42. https://doi.org/10.1109/TSE.2013.44 [13] William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal 10, 1 (1998), 100–107. https://www.cs.tufts.edu/comp/150FP/archive/bill-mckeeman/DifferentailTesting.pdf [14] NVIDIA. 2026. Compute Sanitizer. NVIDIA Corporation. https://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/ index.html Accessed 2026-09-29. [15] Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. 2025. KernelBench: Can LLMs Write Efficient GPU Kernels?. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 47356–47415. https://proceedings.mlr.press/ v267/ouyang25a.html [16] Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. 936–946. https://doi.org/10.1109/ICSE.2015.103 [17] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, Vol. 32. 8026–8037. https://papers.nips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740Abstract.html [18] Chao Peng and Ajitha Rajan. 2019. CLTestCheck: Measuring Test Effectiveness for GPU Kernels. In Fundamental Approaches to Software Engineering (Lecture Notes in Computer Science, Vol. 11424). Springer, 315–331. https://doi.org/ 10.1007/978-3-030-16722-6_19 [19] PyTorch Contributors. 2023. PyTorch 2.1.2 Documentation: torch.isclose. PyTorch. https://github.com/pytorch/pytorch/ blob/v2.1.2/torch/_torch_docs.py#L5418-L5439 Documentation source at tag v2.1.2. Accessed 2026-10-02. [20] Ravishka Rathnasuriya, Nidhi Majoju, Zihe Song, and Wei Yang. 2025. An Investigation on Numerical Bugs in GPU Programs Towards Automated Bug Detection. Proceedings of the ACM on Software Engineering 2, ISSTA (July 2025), 1654–1677. https://doi.org/10.1145/3728950 [21] Ravishka Rathnasuriya, Zihe Song, Nidhi Majoju, Tingxi Li, Aaryaa Moharir, Wei Yang, and Tao Xie. 2026. Characterizing Real-World Bugs in Tile Programs for Automated Bug Detection. Proceedings of the ACM on Software Engineering 3, ISSTA (Oct. 2026), 4118–4142. https://doi.org/10.1145/3832271 [22] Ravishka Rathnasuriya and Wei Yang. 2026. Beyond Confidence: Rethinking Uncertainty Calibration in Deep Code Models. https://doi.org/10.48550/arXiv.2610.21248 arXiv:2610.21248v1 [cs.SE] [23] Ravishka Rathnasuriya and Wei Yang. 2026. Characterizing Overconfident Failure in LLM-Based Code Generation. https://doi.org/10.48550/arXiv.2610.11300 arXiv:2610.11300v1 [cs.SE] [24] Ravishka Rathnasuriya, Zijie Zhao, and Wei Yang. 2025. CodeImprove: Program Adaptation for Deep Code Models. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 514–526. https://doi.org/10.1109/ ICSE55347.2025.00139

KernelMutate: GPU Kernel Validation

21

[25] Dipankar Sarkar. 2026. The Correctness Illusion in LLM-Generated GPU Kernels. arXiv:2606.20128v1 https: //arxiv.org/abs/2606.20128v1 [26] Dipankar Sarkar. 2026. Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels. https://doi.org/10. 48550/arXiv.2607.16228 arXiv:2607.16228v1 [27] Dipankar Sarkar. 2026. Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs. https: //doi.org/10.48550/arXiv.2606.27396 arXiv:2606.27396v1 [28] Scaling Intelligence. 2025. KernelBench Samples. Dataset. https://huggingface.co/datasets/ScalingIntelligence/ kernelbench-samples Accessed 2026-09-29. [29] Rishi Shah and Rishav Shrestha. 2026. A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family. https://doi.org/10.48550/arXiv.2608.12700 arXiv:2608.12700v2 [cs.LG] [30] Vasudev Vikram, Isabella Laybourn, Ao Li, Nicole Nair, Kelton OBrien, Rafaello Sanna, and Rohan Padhye. 2023. Guiding Greybox Fuzzing with Mutation Testing. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 929–941. https://doi.org/10.1145/3597926.3598107 [31] Han Wang, Jintao Zhang, Kai Jiang, Haoxu Wang, Jianfei Chen, and Jun Zhu. 2026. KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels. https://doi.org/10.48550/arXiv.2605.04956 arXiv:2605.04956v2 [32] Yunxiang Zhang, Ping Yu, Jianyu Wang, Max (Xiangjun) Fan, Julian Reed, Azalia Mirhoseini, and Will Su. 2026. KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch? https://doi.org/10.48550/arXiv.2607.16241 arXiv:2607.16241v1 [33] Qianqian Zhu and Andy Zaidman. 2020. Massively Parallel, Highly Efficient, but What about the Test Suite Quality? Applying Mutation Testing to GPU Programs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification. 209–219. https://doi.org/10.1109/ICST46399.2020.00030

22

A

Ziyuan Li et al.

SUPPLEMENTARY EVIDENCE

This appendix provides witness configurations, record-selection details, and additional breakdowns supporting the main results. The study populations, validation contracts, and aggregate comparisons are specified in Sections 2 to 5. A.1

Native-Witness Details

The following configurations and recorded symptoms supplement the native-witness analysis in Section 5.2. The accepted candidates with native FP32 witnesses comprise 100 level-1, 28 level-2, and eight level-3 implementations. Across the controlled strategies’ observed native tests, 95 candidates have only finite-value tolerance violations, 30 only NaN/Inf-position mismatches, and 11 both. These are symptom categories, not source-level causes. The three cases below use FP32, evaluation mode, and seed zero; their original observations are bound to candidate/reference source hashes. For C2__KB_L1_P19__d0640b9cce2e0f4a, the large-magnitude input has shape [4096, 393216], with gradients disabled. The post-call input comparison reports 805,319,693 of 1,610,612,736 elements above the threshold and maximum absolute difference 5767.48. For C2__KB_L1_P37__f84d95d3af89255b, denormal inputs of shape [112, 64, 512, 512] produce Inf-position differences at all 1,879,048,192 output elements. This size exceeds the candidate’s 1,048,576-element custom-CUDA guard. The saved message neither identifies which side contains Inf nor establishes the reference’s numerical validity in this domain. For C2__KB_L2_P93__683a5034e986671e, a large-magnitude input yields 403 threshold violations among 276,889,600 output elements and a reported maximum absolute difference of 2.28085 × 10−4 . Its cause remains unadjudicated. The finite comparisons use atol = rtol = 10−4 . Counts and rounded error magnitudes come from saved oracle messages; no saved-output rescoring or numerical replay underlies these descriptions. The three C2 additional-dtype cases that return discrepant outputs consist of one output-dtype mismatch, one finite-value tolerance violation, and one NaN-position mismatch. Failure-stage classification uses recorded stages and oracle verdicts; comparison failures are retained when an ancillary state-restore error also occurs. A.2

Cost Components and Record Reconstruction

Table S1 gives the recorded cost components underlying the totals in Table 7. First-call cost sums the full wall time of each candidate’s first invocation in the original ledger; the remaining cost is charged Total minus First call. These components do not isolate initialization and execution. Only strategies with available component measurements are listed.

KernelMutate: GPU Kernel Validation

23

Table S1. First-call and remaining charged costs, in GPU-node wall-hours per 1,000 targets. First-call Remaining cost charged cost

Configuration B1u anchor

13.4

11.3

B3 IID-32

13.4

73.7

B4 Value-diversified

14.8

97.2

B5 Boundary values

14.9

99.5

B6 Composite execution-context arm

13.6

10.0

M-full

14.8

62.9

M-dir

17.3

40.2

Selection and ledger fields. Original records are deduplicated by candidate and test identity, preferring Fail. Within each completion stage, selected in-scope observations follow original plan order through its first Fail. Out-of-scope calls, superseded duplicate attempts, and completion calls beyond that prefix are excluded; charged failures, timeouts, and inconclusive observations remain. The ledger binds each selected observation to candidate, strategy, case identity, phase, source order, wall time, charge flag, and exclusion reason. Six provenance-matched source ledgers contain 88,547 records with 88,547 distinct candidate– strategy–case keys. First-recorded and Fail-preferring selection therefore choose identical original records; this agreement is checked on source ledgers, not inferred from the deduplicated export. Candidate and contract hashes match registered subjects, and directed records match the saved plans. File order is available, but global physical timestamps are not. Skipped-slot completion. Completion supplies 214 previously unobserved cases and fills 37 timeoutstop slots. Each replaced slot has zero wall time, zero observed candidate invocations, and the reason “skipped after 5 consecutive parent-timeout cases in this subject/strategy pair.” These are unexecuted placeholders, not competing earlier executions. The 18 M-full and 19 M-dir slots belong to one candidate; paired case and contract identities agree. Retained completion gives 16 inconclusive and two Fail outcomes for M-full, and 15 inconclusive and four Fail outcomes for M-dir. Unselected completion ledgers are outside this audit, so it does not establish the absence of all unrecorded repeated attempts. Table S2. Original-record and completion-slot accounting. Strategy M-full M-dir

Original Skipped slots records 13,179 9,294

18 19

New cases

Filled slots

Selected records

26 188

18 19

13,205 9,482

Alternative time scopes. Table S3 uses the same selected observation export throughout. The second row includes all in-scope completion records without the charged-prefix cutoff; the third also includes diagnostics. These are alternative time scopes, not selections among unretained

24

Ziyuan Li et al.

attempts or measurements of full physical execution. The CPU reconstruction does not replay GPU candidates. Table S3. Recorded hours under the alternative accounting scopes. Selected-record scope Charged prefixes All in-scope records All records, including diagnostics

A.3

M-full hours

M-dir hours

Relative difference

65.14337 65.20060 66.75913

48.14322 48.66410 49.79841

26.10% 25.36% 25.41%

Oracle-Control Selection and Matched Cases

The selection rule and candidate-level results are reported in Section 5.3 and Table 8. The unselected records comprise 57 C3 and 28 C5 candidates. Their preliminary anchor dispositions are 82 state-binding failures (57 C3, 25 C5), one C5 parent-process timeout, and two C5 passes with no preliminary M-full kill. These are selection-time dispositions, not outcomes of unexecuted fixed-tolerance controls. Matching cases by candidate identity, policy, seed, mode, parameters, and replicate yields 2,706 anchor pairs and 8,802 suite pairs across independent executions. Anchor matches contain 51 Passto-Fail transitions and none in reverse. Suite matches contain 58 Pass-to-Fail, one Fail-to-Pass, and seven Inconclusive-to-Pass transitions. Saved outputs were not rescored, and the different stopping points affect which cases can be matched. A.4

Control-Alarm Configurations

For the C6 comparison in Section 5.4, the ordinary adaptation calls np.allclose(candidate, ideal), scaling the relative term by the reference result. The 286 subject–strategy outcomes aggregate 7,670 observations, with at least one Pass or Fail observation for every pair. The three M-dir alarm records use FP32 and shape variant zero. For attention_triton, atol = rtol = 10−3 ; for the NumPy matmul case, atol = rtol = 5 × 10−2 . All three are tolerance violations against the corpus reference, not NaN-pattern mismatches. Table S4 gives their seeds and shapes, reconstructed deterministically from the saved runner and metadata rather than recovered from logged tensors. Table S4. Seeds and reconstructed shapes for the three M-dir control alarms. Control

Input policy

attention_triton attention_triton matmul

All negative Reduction adversarial Reduction adversarial

Seed Reconstructed shapes 2093511732 𝑄 : [1, 8]; 𝐾, 𝑉 : [256, 8] 505391955 𝑄 : [8, 8]; 𝐾, 𝑉 : [256, 8] 1239010171 𝐴 : [2, 256]; 𝐵 : [256, 64]

The original rows retain categorical failures but no output tensors, maximum errors, or affectedelement counts. These quantities remain unknown. Available corpus-source hashes identify recovered files, not the source bytes at historical execution time. The configurations identify reported alarms without adjudicating their numerical significance.

KernelMutate: GPU Kernel Validation

A.5

25

Controlled-Mutation Details

Table S5 expands the aggregate dispositions in Table 10 by fault family. Table S5. C1 probe dispositions by fault family; aggregate totals are in Table 10. Disposition

A: arithmetic

B: GPU parallelism

C: ML numerical behavior

D: Broadcast/ layout

Generated probes

757

702

178

9

Compile/stillborn

56

3

45

1

Host-control failure

196

172

70

3

Other excluded

20

5

0

0

Anchor-killed

186

337

13

3

Harness-scoped equivalence

9

0

0

0

Survived, later falsified

25

42

10

0

Unresolved

265

143

40

2

Host controls. The 441 host-control exclusions arise from 22 distinct hosts: eight with statebinding failures, eight with output mismatches, three with candidate execution failures, one with an instantiation failure, and two with CUDA configuration or argument errors. The other 68 hosts pass control. Dynamic evidence uses these passing controls; one independently discharged static-equivalence probe belongs to a host without a valid dynamic control, so the retained probe population spans 69 host identities. Operator catalog. Host identities agree with the source inventory. The operator families contain three arithmetic, four GPU-parallelism, seven ML-numerical, and two broadcast/layout operators. Probe records identify 15 operators with instances; reduction_reorder has none. Table S6 pairs recorded counts with target constructs and rewrites described by the available implementation. Its exact source bytes are not bound to the C1 generation-time implementation, so the descriptions neither establish the complete matching constraints used for this batch nor guarantee its regeneration. Table S6. Mutation-operator counts and descriptive rewrites. Family Operator A A

arith_replace relop_replace

A

const_perturb

B B B

index_replace sync_remove mask_boundary

B

launch_config_mutate

Probes Descriptive target and rewrite 268 Binary arithmetic: + ↔ − and × ↔ / 267 Relational operators: strict/non-strict and equality/inequality swaps 222 Numeric literals: integer ±1; floating point ×1.01 or ×0.99 267 Program/thread/block indices: change the selected axis 104 Matched GPU barriers: remove the synchronization call 219 Boundary guards: change comparison strictness or offset the bound 112 Matched grid/block sizing expressions: add or subtract one Continued on next page

26

Ziyuan Li et al.

Table S6. Mutation-operator counts and descriptive rewrites. (continued) Family Operator C

stab_remove

C

acc_downgrade

C C

epsilon_modify scale_modify

C C C D

cast_remove reduction_reorder init_modify broadcast_unsafe

D

layout_assume

Total

Probes Descriptive target and rewrite 7 Matched max-subtraction stabilization: remove the subtraction 46 Matched accumulator or conversion expressions: lower precision 32 Matched small epsilon literals: replace by zero or 10−2 33 Matched inverse-square-root scaling: remove or alter the scaling 38 Explicit dtype conversions: remove the matched cast 0 Matched sums: reverse the reduction input order 22 Min/max initializers: replace ±∞ with finite values 4 Explicit expand/broadcast/unsqueeze: remove the matched operation 5 Matched contiguity conversions or checks: remove the call 1,646

Static-screen checks. The dead_host_constant screen examines assignments outside classes and functions in parseable Python source. For const_perturb changes, it checks that the assigned name does not occur directly in the Model/ModelNew class AST. This syntactic check alone does not establish dependency isolation: the full source difference, other uses, CUDA strings, initialization, forward computation, and model state must also be considered. For the nine scoped cases described in Section 3, the unchanged CUDA strings do not interpolate the altered constant. Non-transitivity example. Using the task reference 𝑅, host 𝐻 , and mutant 𝑀 from Section 3, define agreement by |𝑎 − 𝑏 | ≤ 0.01 + 0.01|𝑏 |. For 𝑅 = 1, 𝐻 = 1.019, and 𝑀 = 0.981, both 𝐻 and 𝑀 agree with 𝑅, but |𝑀 − 𝐻 | = 0.038 > 0.02019. This example illustrates why a host-relative witness does not imply a reference-relative failure. A.6

Recommendation Reproducibility and Observation Coverage

Fold and ranking details. The task-held-out construction in Section 3 assigns folds by hashing task identity with the fixed e1-crossfit-v1 prefix. Missing template identities remain missing; only explicitly recorded random trials map to IID. For tie-breaking, mean training time is rounded to 0.1 ms before canonical template identity is compared. The artifact’s complete 75-probe manifest identifies templates, folds, and recommendation outcomes. Conclusive observations. The coverage partitions and recommendation counts are in Table 11. Class-conditioned lists contain one to five recommendations (mean 3.04 per probe). Table S7 supplies the additional probe–template observation counts. Its denominators count recommended pairs; the numerators count pairs with conclusive records. Joint-complete probes have two nonempty, fully conclusive recommendation lists.

KernelMutate: GPU Kernel Validation

27

Table S7. Observation coverage of recommended probe–template pairs. Cap

Class templates conclusive/total

Global templates conclusive/total

Joint-complete probes

1 2 3 4 5 6 7 8

31/75 100/146 139/192 155/216 160/228 160/228 160/228 160/228

6/75 58/150 104/225 127/300 139/375 152/450 164/525 178/600

6 5 2 2 2 1 0 0

Conclusive-template proportions are 41.3% versus 8.0% at cap one and 70.2% versus 29.7% at cap eight. Neither list witnesses a failure among the joint-complete probes at any cap; these cohorts shrink from six to zero and do not support an equivalence conclusion. For the common observablepool analysis in Section 5.5, the total recommendations per ranking are 73, 130, and 136 at 𝑘 = 1, 2, 3. At 𝑘 ≥ 4, both exhaust the pools and select the same 138 probe–template pairs. Deployed-prior traceability. The recorded unsupported association is L1_P37__index_replace__11, whose stored F-IDX support for boundary_last_element is 1/1; its admissibility record rejects the last-element witness for kernel nondeterminism at that position. This explanation reports the recorded reason, not a new nondeterminism experiment. Confirmed F-IDX evidence contains 156 probes, with head-heavy support 6/6 and IID support 150/156, but no admissible last-element witness. These support statistics do not replace the deployed prior. All 9,482 selected directed observations match the saved plans by original test identity, policy, mode, parameters, seed, and candidate/task. The two last-element cases per directed plan give 1,676 planned cases. In addition to the full-population masking results in Section 5.5, the B1u-accepted cohort retains 496/498 discoveries and 118/120 native discoveries. The affected C5 L1-P37 source IDs are 14e554b1d46601b9 and 67d428bcecfba425. Each has one FP32 output element violating tolerance at shape [112, 64, 512, 512], with reported maximum absolute differences 1.35496 × 10−4 and 1.35243 × 10−4 , respectively. These remain unadjudicated numerical witnesses. The schedules do not identify each case’s generating association; order, seeds, stopping, discovery, and cost under a regenerated plan remain unmeasured. A.7

Witness Classification and Traceability

The named-stress witnesses counted in Section 5.5 comprise 62 screening and seven enhancedsearch records. A screening witness requires the label WITNESSED_NON_EQUIVALENT. Enhancedsearch witnesses require FALSIFIED in the probe’s terminal record, selected by its latest recorded completion time. The two FP16 observations have matched passing host controls and join through the recorded subject-to-probe map. The case classifier reads divergence.round_type and divergence.policy: an explicit stress round with a named policy is a stress witness, and an explicit random round is a random witness. FP16 cases use the directly recorded dtype. A reproduced-crash record without a template retains that unknown identity. The artifact binds all 77 probes to source-file and original-line hashes, anchor records, and available matching trials.

28

Ziyuan Li et al.

For example, L2_P81__index_replace__3 has round_type=random, a null policy, and seed 10000. L1_P95__relop_replace__5 records a reproduced crash without a divergence template, so no IID or stress policy is imputed. A.8

Additional Dtype Evidence

Table S8 records the input and output shapes for the two cases summarized in Table 13. Table S8. Operation and shape details for the dtype cases in Table 13. Candidate

Operation/network and input-to-output shapes

C3-23530

LeNet-5 with four ReLU replacements; [1,1,32,32] → [1,10] logits

C3-00995

torch.matmul(A,B), A:[16,1024,2048], B:[2048,768]; boundary [16,1024,768], readout [16,8]

For C3-00995, the FP16 custom path returns FP16 outputs. An additional case, C5-01707, uses LeNet with a custom ReLU and input shape [1, 1, 32, 32], producing [1, 10] logits. All 48 applicable FP32 inputs agree with custom execution established. FP16 and BF16 fail at the custom Float pointer requirement. This case indicates an additional-dtype portability limitation, separate from the recovered FP32 evaluation setting. A.9

Replacement Outcome Details

Table S9 expands the Other category in Table 14. Codes preserve the recorded configuration classification, with each configuration contributing to exactly one row. Adapter/reference limitations and lack of evidence for custom execution remain separate from candidate computation failures.

KernelMutate: GPU Kernel Validation

29

Table S9. The 19 recorded terminal categories within the 523 Other outcomes. Code

Terminal outcome

BUILD CUPY SO SYNTAX UNDEFINED CONSTRUCTOR FORWARD CSTATE RSTATE NO_ENTRY PLACEHOLDER DISABLED ATEN_ONLY CUBLAS REF_DTYPE REF_NONFINITE LAUNCH PROCESS TIMEOUT

Extension build failure CuPy dependency or execution path unavailable Shared-library loading failure Candidate source syntax failure Undefined name during candidate initialization Candidate constructor argument mismatch Adapter-generated call argument mismatch Candidate state mapping unavailable Reference state mapping unavailable No callable binding to the custom entry Placeholder implementation Custom computation disabled by the original configuration ATen-only execution Library delegation without handwritten-kernel execution evidence Reference dtype incompatibility Nonfinite reference output Custom-path launch failure Process termination without a valid result Timeout before planned completion

Configurations

Total

174 12 3 3 9 36 87 81 81 4 3 9 1 1 14 1 1 1 2 523

The five direct-comparison cases without an applicable downstream readout are C2-00096, C310177, C3-10342, C3-10422, and C3-11178. These cases do not establish downstream agreement. Table S10 supplies the input and reference-output shapes for the four witnesses in Table 15. Table S10. Input and expected-output shapes for the witnesses in Table 15. Candidate

Input → expected output

C3-17320

[128,10] → scalar []

C3-20564

[128,512] → [1,8]

C5-00631

[16,64,256,256] → [16,8]

C5-01485

[10,3,224,224] → [10,1000]

For C3-20564, the boundary comparison has shapes [1, 256, 128, 1] versus [128, 1]. For C5-00631, the failure at input 37 follows 36 passing inputs. For C5-01485, the published forward() invokes self.model followed by the classifier. The native reference block is PyTorch Conv2d + BatchNorm2d + ReLU. The replacement invokes the published conv_bn_relu_cuda and then the remaining MobileNetV1 layers. The subsequent layer reports expected 4D input (got 3D input). These entry bindings complement the custom-execution evidence in Section 5.7.

Related documents

• KernelMutate: Mutation-Guided Validation of Generated GPU Kernels#1659023
• WarpGuard: Towards Control-Flow Attestation for Heterogeneous CPU-GPU Execution#370276
• RESOLVE: Language-Agnostic Validation of GPU Kernels Through Testing, Reduction, and Proof#1201696
• RESOLVE: Language-Agnostic Validation of GPU Kernels Through Testing, Reduction, and Proof#1194391
• Equivalence Checking of ML GPU Kernels#612878