VEHBench: A Stage-Local Diagnostic Benchmark for LLM-Assisted Vibration Energy Harvester Design
Depeng Su1 1
Yuyu Luo2,*
Guobiao Hu1,*
Internet of Things Thrust, Information Hub
2
Data Science and Analytics Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) * Corresponding authors: {yuyuluo,guobiaohu}@hkust-gz.edu.cn
arXiv:2607.18181v1 [cs.CL] 20 Jul 2026
Abstract Battery-free Internet of Things (IoT) requires iterative design of vibration energy harvesters (VEHs) under coupled physical constraints, while LLMs are emerging as interface layers for engineering workflows. However, existing engineering benchmarks primarily assess final artifact validity, offering limited insights into how LLMs behave across different stages of coupled physical design. We introduce VEHBench, an engineering-native diagnostic benchmark for LLM-assisted VEH design, featuring 763 literature-grounded tasks scored by an analytical physical oracle. VEHBench evaluates four design roles: specification triage, verifierguided search, corrupted-state recovery, and policy-conditioned selection. Experimental results reveal that LLM capability is strongly stage-dependent: no single model consistently dominates the entire workflow, and response-control profiles expose distinct behavioral patterns across design roles. VEHBench thus provides a stage-aware foundation for evaluating, selecting, routing, and improving verifier-grounded engineering LLMs. The benchmark artifact is available at https://huggingface.co/datasets/AnonymousVehbench/vehbench.
1
Introduction
Battery-free IoT reduces maintenance for dense, long-lived sensing, making vibration energy harvester (VEH) design a recurring need for low-power sensors (Citroni et al., 2024, Zeadally et al., 2020, Naifar et al., 2024). VEHs convert ambient vibration into power, but each deployment can impose different vibration spectra, power targets, size limits, materials, packaging conditions, and safety margins (Safaei et al., 2019, Sadaf et al., 2024, 2025). LLMs are increasingly used as engineering workflow interfaces to translate requirements, review feedback, revise candidates, and compare trade-offs; this paper explores how assistance should be evaluated in physics-grounded VEH design. VEH provides a compelling testbed for evaluating LLM-assisted engineering design since it involves tightly coupled mechanical–electrical interactions and demands expertise across structural dynamics, smart materials, circuit design, and embedded systems. For example, a cantilever-based piezoelectric energy harvester is a representative VEH due to its canonical role in the literature. Its performance is jointly governed by structural parameters (e.g., beam geometry, proof mass, and material properties) and electrical parameters (e.g., load resistance, capacitance, and electromechanical coupling), leading to strong trade-offs between objectives such as power output and bandwidth. As a result, VEH differs fundamentally from retrieval-based tasks or single-objective optimization. Figure 1 illustrates how an LLM can assist this coupled workflow by interpreting requirements, proposing design modifications, analyzing verifier feedback, and comparing feasible alternatives while the physical oracle remains outside the model. In this work, we propose a VEHBench to evaluate workflow assistance under physical constraints, not certified autonomous hardware design. Preprint.
Step 1: Requirement Brief + P1 Triage Design a battery-free VEH for structural health monitoring.
Vibration: 20-200 Hz
Target: > 50 uW at 120 Hz Size:< 250 x 10 mm
Size: < 250x10mm
mp
Power: > 50 uW
power
Material / safety constraints
Strain (ε)
VEH
candidate.json
Piezoelectric layer
Charge (Q)
Polarization
i = dQ/dt
electromech. coupling
VEH-powered node design plan Battery-free SHM node
c LLM
power IoT chip
Step 2-5: VEHBench Verifier-mediated co-design loop Step 2: Step 3: Structured VEH Candidate LLM Structured Proposal L mp b Parse requirements piezoelectric element t Propose actions { "type": "cantilever", "L":2.2e-1, "b": 8e-3, "t": 0.5e-3, "m_p": 5.8e-3, "R_L": 1e6 "C":2.2e-9
RL
Start Design brief is actionable
Need Info missing specs
P1 Triage Gate
Infeasible reject/ stop
Step 4&5: External Verification and Verifier-to-LLM Metric
RL
C
Resonance Stress
load match electrical
Limit / Target
Result
±5% of 120 Hz ≤ 40.0 MPa
Power output >50 uW @120 Hz
policycard candidate ID: D-18 score: 0.72/1.00 trade-off: power , stress edit targets: reduce σmax increase Pout keep size, f n strategy: fine-tune / refine or replan (new idea)
σ max : 52.3 > 40.0 MPa P out : 18.7 < 50.0 uW fn (Hz) σmax (MPa) Pout (uW) η (%) P3 Recovery Structured Proposal 62.1 52.3 18.7 121.3 failure analysis Fine-tune / Refine Replan /New Idea P3 Recovery (escape local failure) (keep structure,adjust parameters) OR LLM (change structure or working principle) LLM P2 Search (iterative optimization) k em ∝ d bt3 31 L
1 R L ≈ ωnCp
{
Step 6: P4 Selection and Final Recommendation How to Select the Best Fit Candidate LLM P4 Selection evaluate & rank feasible candidates
Candidate Trade-off Overview σmax fn Trade-off Characteristic
ID
pout
(uW)
(MPa)
(Hz)
D-27
71.5
40.0
120.3
2) compare trade-offs
D-31
22.7
32.1
119.6
3) Choose candidate that best satisfies req.
D-18
70.8
36.5
120.1 Balanced (overall Rbest)
1) Map priority: power /stress /size /freq. match
P P
High P, High σ (best for power-critical)
R
Low P, Low σ (best for reliability)
Final VEH Design ready for validation piezoelectric m p element RL
R
P
Figure 1: Coupled VEH co-design workflow and LLM intervention points. VEHBench evaluates how an LLM assists the verifier-grounded design loop through stage-local roles rather than treating the model as a standalone hardware designer.
Current engineering evaluations have progressed by introducing external verification, including simulation-backed and physics-grounded checks (Guo et al., 2025b, Xia et al., 2025, Jadhav and Farimani, 2024, Doris et al., 2025, Jain et al., 2025). These benchmarks move beyond textual judgment by asking whether generated artifacts satisfy engineering constraints, but endpoint validity alone does not reveal whether a model checked brief sufficiency before search, made bounded edits after physical feedback, recovered from a misleading trajectory, or followed an explicit policy when several candidates were already feasible. These are workflow failures, not just solver failures. Recent AI evaluation has shown the value of decomposing aggregate ability into task-local roles and error types (Srivastava et al., 2023, Liang et al., 2023, Kiela et al., 2021); engineering design needs this idea in an engineering-native diagnostic layer, where stages are workflow boundaries with different trusted states, admissible actions, verifier signals, and failure consequences. For LLM-assisted engineering, a benchmark should therefore evaluate whether the model behaves appropriately at the design stage where it is used, rather than only whether the final artifact passes. We study three research questions. • Q1: How can we evaluate LLMs in specialized engineering design tasks? Because standardized datasets for VEH co-design remain scarce, meaningful evaluation requires the construction of an engineering-native benchmark incorporating real constraints, external verification, and stage-specific workflow assessment. • Q2: How capable are current LLMs in VEH co-design, and what insights can be gained from analyzing their failure modes? This requires separating design responsibilities and diagnosing behavior beyond aggregate scores. • Q3: How can VEHBench support engineering applications, identify targets for improvement, and enable broader evaluation? This requires evidence that stage-local results can inform model selection, routing, adaptation, and future verifier-grounded evaluations. The first VEH-focused diagnostic benchmark. To answer these questions, we introduce VEHBench, an engineering-native diagnostic benchmark that evaluates LLM-assisted vibration energy harvester co-design through stage-local, verifier-grounded design tasks. Its 763 cases are not meant to compete with web-scale language benchmarks in volume; they are literature-grounded, oraclecheckable, manifest-traced engineering probes. VEHBench is built from literature-derived VEH 2
design anchors and scored by an analytical oracle that checks physical feasibility and objective quality. Rather than treating VEH design as one endpoint task, the benchmark decomposes the workflow into four design roles: specification triage, verifier-guided search, corrupted-state recovery, and policyconditioned selection. The detailed probe definitions are introduced in the benchmark construction section; the introduction uses them only to establish the evaluation problem. Our contributions include: • Benchmark framework and VEHBench construction. We propose an engineering-native diagnostic benchmark for LLM-assisted design and instantiate it as VEHBench for coupled mechanical–electrical VEH co-design. The benchmark combines literature-grounded task construction, external analytical verification, and stage-specific evaluation of design workflow behavior. • Empirical findings and interpretation framework. We systematically evaluate current LLMs across the VEH design workflow and analyze their behavior through response-control profiles. This framework links empirical results to interpretable model behavioral characteristics, including action discipline, bounded editing, feedback conditioning, state recovery, and policy execution. • Stage-aware guidance for engineering application and model improvement targets. We demonstrate how the benchmark can guide practical LLM-assisted design: engineers can select, prompt, route, or adapt models according to different stages of the design workflow, while AI researchers can identify critical capability gaps for engineering agents, such as specification triage, verifier-guided search, corrupted-state recovery, or policy-conditioned ranking.
2
Related Work
LLM-assisted engineering design. Recent engineering benchmarks evaluate LLMs on tasks closer to design practice. EngDesign studies whether models satisfy engineering design requirements; BuildArena emphasizes artifact construction and interaction with executable feedback; mechanical-design agent studies test iterative CAD/CAE-style reasoning; DesignQA targets engineering-document understanding; and MSEval evaluates material-selection behavior (Guo et al., 2025b, Xia et al., 2025, Jadhav and Farimani, 2024, Doris et al., 2025, Jain et al., 2025). Together, these works move LLM evaluation beyond linguistic plausibility toward engineering constraints, external evidence, and physically meaningful artifacts. Their main readout, however, usually remains final satisfaction, artifact validity, or task completion, making it hard to identify which design-stage behavior failed. Vibration energy harvesting and VEHBench positioning. VEH is a meaningful first substrate for LLM-assisted engineering evaluation because it is practically motivated, constraint-coupled, and analytically auditable. Modern reviews emphasize vibration and cantilever piezoelectric harvesters as compact power sources for wireless, embedded, MEMS, and IoT systems (Safaei et al., 2019, Maamer et al., 2019, Sadaf et al., 2024, 2025). VEHBench follows the engineering-benchmark direction but shifts the readout from final artifact validity to stage-local design behavior. It therefore complements existing benchmarks by using an analytically checkable domain to expose where workflow behavior fails rather than treating the LLM as a single end-to-end designer; Appendix L expands the comparison.
3
VEHBench: Design and Construction
3.1
Design Goals, Domain Scope, and Anchors
Goals. VEHBench evaluates LLM assistance in verifier-grounded engineering design, not autonomous hardware certification. We set four goals. (G1) Engineering-grounded controllability: tasks should embed realistic coupled design constraints while remaining compact enough for reproducible, deterministic evaluation. (G2) Stage-local diagnosticity: the benchmark should isolate failures at distinct workflow boundaries—specification, search, recovery, and selection—rather than collapse behavior into a single final-artifact score. (G3) External and reproducible scoring: all metrics are computed by an analytical physical oracle from structured outputs and persisted logs, without human labels or LLM-as-a-judge. (G4) Auditable extensibility: construction preserves source provenance, manifests, splits, oracle traces, prompts, and evaluator metadata, so the same scaffold can extend to richer simulators or other coupled domains. 3
Table 1: VEHBench probe inventory. Probe Design stage
Input state
Model action
What it measures
Practical use
P1
Specification triage
Propose, abstain, or request information
Action discipline
Gate search before design begins
P2
Search
VEH brief with complete, missing, or infeasible fields Seed design plus oracle feedback
Make bounded edits
Improve candidate designs
P3
Recovery
P4
Selection
Corrupted or misleading design trajectory Feasible candidate pool plus explicit policy
Reset, re-anchor, or stabilize Rank or select candidates
Feedback conditioning and feasible improvement State recovery Policy execution
Prevent failed-trajectory cascade Choose a final design under deployment priorities
Domain scope and anchors. We instantiate this framework on early-stage cantilever VEH codesign, a compact but coupled battery-free IoT setting with deployment-specific vibration spectra, power targets, size limits, materials, packaging, and safety margins (Citroni et al., 2024, Zeadally et al., 2020, Sadaf et al., 2025). VEHBench does not replace FEM, manufacturing review, or hardware certification; it evaluates four stage-local roles under analytical verification—specification triage, verifier-guided search, corrupted-state recovery, and policy-conditioned selection—summarized in Table 1. A design anchor is a literature-derived VEH physical state that has been cleaned into normalized variables, units, bounds, assumptions, objectives, and constraints so that it can be scored by the external oracle. These anchors are the source of the stage-local probes rather than free-form prompt templates. 3.2
Probe Design: Matching Design Stages to Measurements
The four roles span distinct information boundaries within a design workflow: a design brief, verifier feedback during search, a corrupted trajectory, and a pool of feasible alternatives. Each boundary imposes a different trusted state, admissible action, and failure consequence. VEHBench turns these distinctions into probes by changing the trusted state, allowed action, and metric while keeping the physical design substrate fixed. The probes are therefore not four difficulty levels of the same task; they are four different design roles over the same coupled physical object. This design makes the benchmark useful not only for comparing models, but also for deciding where and how an LLM should be inserted into a verifier-grounded engineering workflow. VEHBench defines four task families, summarized in Table 1. P1 occurs before search: the trusted state is only the design brief, the model must decide whether to propose, abstain, or request missing information, and the metric measures action discipline. P2 occurs during search: the trusted state is a seed design plus oracle feedback, the model may make bounded candidate edits, and the metric measures feedback-conditioned feasible improvement. P3 occurs after the design history has become unreliable: the model must reset, re-anchor, or stabilize rather than blindly continue, and the metric measures recovery from contaminated state. P4 occurs after feasibility has already been established: the trusted state is an oracle-feasible candidate pool, the model ranks or selects under an explicit policy, and the metric measures policy execution. The same VEH object is shared; the model role and failure consequence change. 3.3
Anchor-Based Construction Pipeline with Built-in Quality Control
VEHBench uses anchor-based probe generation. Instead of freely writing engineering prompts, we start from literature-derived VEH design anchors, clean them into oracle-checkable physical states, and then generate stage-specific probes around those states. This keeps the benchmark grounded in engineering literature while allowing controlled variation over workflow stage. Figure 2 summarizes the full construction path from literature audit to manifest-backed release. The construction pipeline has six quality-controlled steps. Step 1: Literature audit and provenance control. We audit VEH papers and extract fields relevant to cantilever energy harvesting, including geometry, material, layer structure, excitation, target frequency, load resistance, power, stress, displacement, and feasibility constraints. Each extracted record is tracked by source metadata so that later tasks can be traced back to their construction path. 4
Literature Sources 209 papers
Unit / Assumption Checks
Stage-Local Probe Packaging P1 Specification Triage P2 Verifier-Guided Search
Completeness Checks
P3 Corrupted-State Recovery
Similarity Deduplication
P4 Policy-Conditioned Selection
Schema Normalization
Quality Assurance Field Extraction & Schema Standardization
Split / Source Separation E=mc�
� x dx
Oracle Review & Anchor Filtering
QC feedback
Literature Audit & Provenance Control
Automatic Screening
Manifest & Traceability Schema & Score-Trace Inspection Data Contamination Check
Standardized Anchor Pool 2090 extracted field rows
Cleaned Anchor Set 71 oracle-reviewed 52 retained anchors
Final VEHBench Task Bank 763 Tasks
Figure 2: Benchmark construction pipeline from literature audit to manifest-backed release, with quality-control gates throughout.
Step 2: Issue screening and unit control. Extracted records are screened for missing units, inconsistent assumptions, incomplete variables, ambiguous operating conditions, and unsupported formulations. Records that cannot be made physically checkable are excluded before task generation. Step 3: Oracle review and physics control. Candidate anchors are reviewed against the analytical VEH oracle. Only anchors whose assumptions and variables can support feasibility and objective scoring are retained. This prevents task labels depending on unverifiable or internally inconsistent physics. Step 4: Anchor cleaning and schema control. Accepted anchors are converted into canonical design contexts with normalized variables, bounds, assumptions, objectives, and constraint definitions. This step ensures that tasks differ because of design role, not accidental formatting or schema artifacts. Step 5: Probe packaging and split control. Cleaned anchors are packaged into P1–P4 task families with stage-specific prompts, admissible actions, metrics, and oracle-facing metadata. P2– P4 use source-anchor separation for optimizer-facing splits, while P1 is constructed as a matched certification-style triage stage. Step 6: Release packaging and reproducibility control. The final tasks are released with manifests, evaluator-facing metadata, oracle traces, prompt templates, generation scripts, split reports, and model-log formats. This makes each task auditable at the source, schema, split, and evaluation levels. 3.4 Oracle Scoring, Data Quality, and Release VEHBench keeps physical evaluation outside the language model. Each task is generated from a cleaned literature anchor and specifies design variables, bounds, objectives, budgets, and any exposed design history. Model outputs are parsed into structured actions and passed to the analytical oracle, which returns feasibility, active violations, objective values, and task-specific feedback. The 763 released tasks are derived from a 209-paper audit and 52 cleaned anchors, screened for unit and assumption consistency and packaged into stage-local probes. The resulting benchmark is therefore high-density and high-quality: every task traces to a real VEH parameter set, and every score is computed by a deterministic physical verifier rather than by human judgment. This pipeline protects two forms of validity: physical validity, via deterministic oracle scoring; and diagnostic validity, via stage-specific probe design. The oracle supports P1 triage, P2 feasible search and objective quality, P3 corrupted-state recovery, and P4 policy-conditioned ranking over an oracle-feasible pool. The release mirrors this quality chain with task banks, manifests, evaluator code, prompts, splits, and per-model logs, making every reported score traceable to a source anchor and deterministic physical verifier.
4
Empirical Findings: Stage Boundaries Create Distinct Failure Regimes
4.1
Experimental Setup
We evaluate 12 complete model runs on the 763-task VEHBench benchmark. Each run covers P1–P4. Full prompts, splits, run modes, and metric definitions are reported in the appendix. 5
Table 2: Headline VEHBench results over 12 complete model runs. Columns report P1-Composite, P2 final feasible power ratio, P3-Success, and P4 Kendall τb ; higher is better. Model qwen3-max gemini-3.1-pro-preview o4-mini deepseek-r1 gpt-5.4 hunyuan-hy3-preview deepseek-v3 llama-3.3-70b mimo-v2.5-pro qwen3.6-plus deepseek-v4-pro claude-4.6-sonnet
P1 Comp. ↑
P2 ratio ↑
P3 Succ. ↑
P4 Tau ↑
0.574 0.549 0.518 0.504 0.428 0.425 0.369 0.361 0.317 0.306 0.306 0.207
0.1955 0.3904 0.1551 0.2413 0.1329 0.1609 0.1565 0.1197 0.1073 0.2063 0.1294 0.2394
30.1% 37.2% 26.3% 42.9% 42.3% 47.4% 35.3% 3.2% 45.5% 27.6% 28.8% 16.0%
0.835 0.824 0.780 0.833 0.887 0.839 0.860 0.714 0.843 0.877 0.794 0.840
Gemini-3.1-Pro
P1 P2
over-refusal missing-info miss
policy mismatch
infeasible closure
post-escape failure
utility loss
Qwen3-Max over-action
policy mismatch
policy mismatch
0.0
0.2
0.4
0.6
0.8
1.0
cascade
GPT-5.4 over-action over-refusal 1.0 0.5
missing-info miss infeasible closure utility loss
dead budget cascade
escape failure
Hunyuan-Hy3 over-action over-refusal
top-choice miss
missing-info miss
policy mismatch
infeasible closure
post-escape failure
utility loss
dead budget
Mean error rate across 12 models
post-escape failure
over-refusal 1.0
0.5
post-escape failure
top-choice miss
infeasible closure
escape failure
top-choice miss
post-escape failure
missing-info miss
policy mismatch
utility loss cascade
cascade
P3
1.0
dead budget
dead budget
top-choice miss
over-refusal 0.5
escape failure
P4
over-action
top-choice miss
over-action
missing-info miss infeasible closure utility loss
dead budget
escape failure
1.0 0.5
cascade
escape failure
Figure 3: Stage-level error families and representative profiles. Bars show 12-model mean error rates; radar panels illustrate model-specific signatures on the same axes.
Scoring and profiles. All scores are deterministic functions of persisted JSONL logs; no score uses human annotation or LLM-as-a-judge. Headline metrics are P1-Composite, P2 final feasible power ratio, P3-Success, and P4 Kendall τb . P2 assigns zero to infeasible finals; P3 counts final feasible recovery after the full budget; P4 ranks a fixed feasible pool against oracle policy order. Higher is better for headline scores; Figure 3 reports lower-is-better, non-exclusive error rates. Statistical intervals use nonparametric bootstrap, Wilson intervals, and paired bootstrap where appropriate. To diagnose stage-specific behavior and failure modes, we extract response-control profiles from the same logs: action discipline, edit style, feedback conditioning, state-reset effort, and policy execution. Profiles are used only for diagnosis, not ranking; Appendix A.1 indexes all metrics and Appendix F gives full definitions. 4.2
Finding 1: Stage Boundaries Change Rankings and Failure Modes
Table 2 shows that no model leads the full VEHBench workflow. The four stages have four different winners: qwen3-max for P1 triage, gemini-3.1-pro-preview for P2 search, hunyuan-hy3-preview for P3 recovery, and gpt-5.4 for P4 selection. Stage leaderboards are weak proxies for one another: pairwise Spearman correlations range from −0.26 to 0.38 (Appendix Table 30). The immediate question is therefore not only which model wins, but how each stage fails. Figure 3 answers this by grouping stage-specific, non-exclusive diagnostic error rates across the 12-model roster. The radar panels are illustrative rather than additional headline metrics: they show that the same stage-level error axes produce different model-specific failure signatures. P1 is an entry-control problem. We diagnose triage failures with over-action, over-refusal, and missinginformation miss, rather than raw accuracy, because proposing can look superficially acceptable while failing safety-critical action cases (Appendix H.1; Table 16). Across the 12-model roster, mean over-action is 0.683, mean missing-information miss is 0.635, and mean over-refusal is only 0.085 6
Table 3: Profile–stage Spearman correlations, computed from Appendix Tables 29 and 18–21. Profile axis Action discipline Edit style Feedback conditioning State-reset effort Policy execution
P1 Comp. P2 ratio P3 Succ. P4 Tau Diagnostic tension 0.972 0.158 0.543 -0.074 -0.133
0.273 0.839 0.741 -0.392 0.552
0.396 -0.189 0.587 0.217 0.217
-0.119 -0.077 0.217 0.119 0.762
Safe gating can suppress exploratory design updates Local edits can follow corrupted trajectory history Feedback guides search; stale context can mislead Reset helps recovery but disrupts search continuity Policy discipline can constrain design exploration
(Appendix Table 32). The dominant P1 error is therefore not excessive caution; it is entering design when the brief is infeasible or underspecified. qwen3-max has the best P1 composite because it suppresses unsafe entry and missing-information misses relative to most models (Appendix Table 18). P2 is a physical search-closure problem. We separate infeasible closure, destructive or invalid updates, and utility loss so that well-formed edits alone are not rewarded. Mean infeasible closure is 0.478 and mean utility loss is 0.644, while destructive and protocol-invalid proxies are near zero (Appendix Table 32). Most models can follow the output protocol but still fail to close the physical search loop; gemini-3.1-pro-preview is distinctive because its closure failure is only 0.038, even though useful-power loss remains substantial (Appendix Table 19). P3 deserves a separate readout because its failures occur after exposure to a corrupted trajectory. Its diagnostics separate escape, cascade, dead budget, and post-escape recovery. Mean escape failure is only 0.191, but mean post-escape failure is 0.609 (Appendix Table 32). The bottleneck is therefore not mainly noticing the trap; it is stabilizing after escape. hunyuan-hy3-preview wins P3 because its post-escape failure is much lower than the roster average (Appendix Table 20). This post-escape gap bridges to the profile and state-interface analyses below. P4 has a different surface again. Mean policy mismatch is 0.709, while dominance error is 0.008 and parse failure is 0.001 (Appendix Table 32). P4 failures are therefore not mainly formatting, infeasiblecandidate, or Pareto-dominance errors; models usually parse and compare feasible candidates, but fail to execute the stated engineering policy (Appendix Table 21). These four surface failures motivate the response-control diagnosis in Finding 2. 4.3
Finding 2: Profiles Diagnose Stage-Specific Failure Modes
Response-control profiles add a diagnostic layer to the surface errors in Finding 1: they map failures to behavioral signals in the logs. They are not independent causal variables. Several profile indicators are drawn from the same metric families as the stages they diagnose, so strong intended diagonal correlations should be read as instrumentation checks rather than causal discovery. The useful signal is the full sign pattern, the off-diagonal transfer or conflict, and cases where profile effort decouples from task outcome. The complete scaling rule, indicator sets, and 12-model profile table are in Appendix F, Appendix Table 15, and Appendix Table 29. Table 3 separates three patterns. First, the intended diagonals check that the profiles summarize the stages they diagnose: action discipline tracks P1, edit style tracks P2, and policy execution tracks P4. Second, some priors transfer across boundaries. Feedback conditioning is positive for P1, P2, and P3, making it the closest profile dimension to a general engineering-assistance signal; policy execution also transfers to P2, suggesting that fixed-candidate evaluation and verifier-guided search both reward disciplined use of external constraints. Third, some priors conflict across boundaries. State-reset effort is negatively associated with P2, and edit style is weakly negative with P3: behaviors that help clean search are not automatically useful when the trajectory state is corrupted. The P1 profile explains why accuracy is insufficient for entry control. qwen3-max has the best P1 composite (0.574) despite lower raw accuracy than o4-mini (65.4% vs. 70.0%) because its action discipline is more balanced across propose, missing-information, and infeasibility cases (Appendix Tables 18 and 29). claude-4.6-sonnet has 57.5% raw accuracy but a much weaker P1 composite (0.207), with ACS 0.057 and MDS 0.000 (Appendix Table 18). Action discipline diagnoses the difference between being often correct and being safe at the gate. The P2 profile diagnoses closure as directed search, not formatting. gemini-3.1-pro-preview combines top edit style (0.753) with the highest feedback conditioning (0.508), and its closure failure is only 0.038 (Appendix Tables 29 and 32). llama-3.3-70b has moderate edit style (0.586) but the lowest 7
Table 4: P3 state-reset effort versus recovery outcome.
P3-Success rate (%)
(a)
80
State reset
P3 Succ.
Escape
Replan
First feas.
RecQ
deepseek-v4-pro qwen3-max deepseek-v3 hunyuan-hy3-preview mimo-v2.5-pro
0.737 0.646 0.629 0.599 0.627
28.8% 30.1% 35.3% 47.4% 45.5%
90.4% 85.9% 82.7% 77.6% 63.5%
99.4% 49.4% 57.7% 7.7% 17.9%
27.6% 22.4% 23.1% 40.4% 41.0%
0.199 0.261 0.287 0.488 0.626
Model mean
0.540
31.9%
80.9%
23.7%
21.5%
0.275
Raw history
+16.7
60
(b)
State summary
+2.7
+0.0
+33.3
40 20 0
Gemini-3.1pro-preview
DeepSeek-R1
GPT-5.4
40 30
-21.9
20
-21.6
10 0
Claude-4.6sonnet
Raw history State summary
50
Cascade rate (%)
Model
-29.4 Gemini-3.1pro-preview
DeepSeek-R1
GPT-5.4
-1.4 Claude-4.6sonnet
Figure 4: P3 state-summary intervention. It improves recovery for most models and reduces cascade at the corrupted-state boundary; full numeric results are in Appendix Table 14.
feedback conditioning (0.326), and its closure failure rises to 0.630 (Appendix Tables 29 and 32). Both models edit; the difference is whether edits follow verifier feedback. The P4 profile diagnoses policy mismatch rather than parser weakness. gpt-5.4 has the best P4 Tau (0.887) and high policy execution (0.743), while qwen3.6-plus has the strongest policy execution profile score (0.761; Appendix Table 29) and best exact-match rate (Appendix Table 21). In contrast, llama-3.3-70b has the lowest policy execution score (0.622) and weakest P4 Tau (0.714) (Appendix Tables 29 and 21). P4 measures evaluator-role discipline after search has been removed. The clearest decoupling case is P3. Table 4 shows why the state-reset profile should be read as recovery effort, not recovery success; its source columns are expanded in Appendix Tables 29 and 20. deepseek-v4-pro has the highest state-reset score and replans in 99.4% of tasks, but its P3 success is only 28.8% and its recovery quality is 0.199. In contrast, hunyuan-hy3-preview and mimo-v2.5-pro replan much less aggressively but reach feasible states and preserve objective quality more often. P3 therefore requires both recognizing contaminated state and stabilizing after reset. Model-level profiles then identify usable failure modes rather than a global best model. gemini3.1-pro-preview is interaction-oriented, matching its P2 lead. gpt-5.4 is evaluator-oriented, with the highest policy execution (0.743) and P4 Tau (0.887), but weaker edit style (0.482) and P2 ratio (0.133) (Appendix Tables 29, 19, and 21). These patterns expose two actionable failure classes: representation/interface failures, where recovery effort does not stabilize state, and boundary-fit failures, where a useful prior at one stage does not transfer to another. 4.4
Finding 3: Diagnostics Translate into Engineering Controls
The two failure classes above point to two controls. For representation failure, we change the P3 state interface. For boundary-fit tension, we analyze stage-aware model selection. Control 1: State interface design. The P3 post-escape gap suggests a representation failure: the model may try to recover, but raw corrupted history remains in context and prevents stabilization. We therefore replace raw trajectory history with a verifier-authored state summary that preserves the current trusted proposal, best feasible candidate, and active violations while removing the contaminated conversational trace. On a stratified 36-task subset, this state-interface change improves model-mean recovery from 50.0% to 63.2% and lowers cascade from 28.1% to 9.5% (Figure 4). The response is profile-dependent: claude-4.6-sonnet gains 33.3 pts in success, suggesting that raw-history format was suppressing usable design behavior; gemini-3.1-pro-preview gains 16.7 pts, consistent with its strong edit-style profile benefiting from a cleaner state; gpt-5.4 holds success fixed but reduces cascade to 0.0%; deepseek-r1 gains only 2.8 pts. The mechanism is therefore not “longer prompting.” It is stage-local state control. 8
Table 5: Stage-aware selection on full 12-model held-out splits. Stage cells report raw held-out scores; Mean norm. averages scores after dividing each stage by its held-out stage-best score. Strategy
P1
P2
P3
P4
Mean norm.
P1 specialist: qwen3-max P2 specialist: gemini-3.1-pro-preview P3 specialist: hunyuan-hy3-preview P4 specialist: gpt-5.4 Validation-selected router
0.594 0.185 23.9% 0.552 0.385 32.6% 0.413 0.157 45.7% 0.454 0.148 38.0% 0.552 (gemini-3.1- 0.385 (gemini-3.1- 42.4% pro-preview) pro-preview) (mimo-v2.5-pro)
0.839 0.821 0.815 0.888 0.821 (deepseek-v3)
0.738 0.892 0.755 0.745 0.945
Ex-post stage-best envelope
0.594 (qwen3-max)
0.385 (gemini-3.1- 45.7% (hunyuanpro-preview) hy3-preview)
0.888 (gpt-5.4)
1.000
Control 2: Stage-aware selection simulation. Finding 2’s profile-stage matrix shows why routing is needed: state-reset effort is negatively associated with P2, and edit style is weakly negative with P3. Table 5 turns this into a deployment view using full 12-model held-out splits. Stage specialists win locally but pay a cross-stage tax: qwen3-max, hunyuan-hy3-preview, and gpt-5.4 drop to mean normalized scores of 0.738, 0.755, and 0.745 when forced across the workflow. gemini-3.1-propreview is the best single-model fallback not because it dominates, but because its weakness is less severe, giving a mean of 0.892. The insight is that aggregate deployment quality depends less on peak strength than on avoiding boundary-specific collapse. The split-based routing simulation in Appendix K.2 separates realized routing gain from remaining headroom. The validation-selected router improves mean normalized held-out score from 0.892 to 0.945, almost entirely by switching P3 from gemini-3.1-pro-preview to mimo-v2.5-pro and raising normalized recovery from 0.714 to 0.929. The ex-post envelope remains 1.000, leaving 0.055 headroom from validation misses of held-out stage leaders. Thus VEHBench supports two concrete controls: state-interface design for representation failures and stage-aware routing for profile-boundary mismatch.
4.5
Robustness and Scope Checks
We run five controls. Three address internal threats: a selection–generation audit confirms ranking is not reducible to generation (26.1–60.9 pt A–B gaps); a query-matched CMA-ES baseline rules out P2 as pure numerical search (ratio 0.099 vs. the strongest LLM); and a prompt-control audit finds no consistent correction of known deficits. Two support construct validity: thinking-mode coverage helps recovery but does not predict the four stage leaders; the circuit pilot reproduces stage spreads under a second engineering domain. Full results are reported in Appendix P–R.
5
Discussion
The empirical findings suggest that engineering LLMs should be evaluated and deployed through stage-specific use of external verification, not as monolithic designers. VEHBench is complementary to simulation-backed evaluation: verifiers determine physical validity, while diagnostic probes show whether the model used that validity signal appropriately at each design stage. History is a control variable, not just context. In clean search, raw trajectory history can carry useful constraint information; in corrupted recovery, the same history can become contamination. The statesummary intervention supports stage-specific state representation: preserve trajectory during clean search, sanitize state during recovery, and expose feasible candidates plus policy during selection. The broader contribution is the diagnostic chain, not any single model ranking. Surface error decomposition identifies what fails, response-control profiles diagnose the behavior behind failure, and stage-local controls test whether interface or routing changes can improve outcomes. These profiles are operational diagnostics, not causal claims about cognition. VEHBench remains scoped to analytically verifiable cantilever VEH co-design and does not replace FEM or hardware certification; the next step is to instantiate the same scaffold with richer simulators and other coupled domains. 9
6
Conclusion
We introduced VEHBench, an engineering-native diagnostic benchmark for LLM-assisted VEH co-design. It combines literature-grounded task construction, analytical verification, and stage-local evaluation of design behavior. The central conclusion is that LLM capability in engineering design is role-dependent: aggregate rank is less informative than stage compatibility. VEHBench provides both an evaluation tool and a design guide for verifier-grounded engineering agents: select, route, adapt, and identify improvement targets according to the stage where they are used.
10
References Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Pieter Gijsbers, Joan Giner-Miguelez, Nitisha Jain, Michael Kuchnik, et al. Croissant: A metadata format for ML-ready datasets. In Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning, pages 1–6, 2024. doi: 10.1145/3650203.3663326. Alibaba Cloud Model Studio. Using deep thinking models. https://www.alibabacloud.com/ help/en/model-studio/deep-thinking, 2026. Accessed April 29, 2026. Rocco Citroni, Fabio Mangini, and Fabrizio Frezza. Efficient integration of ultra-low power techniques and energy harvesting in self-sufficient devices: A comprehensive overview of current progress and future directions. Sensors, 24(14):4471, 2024. doi: 10.3390/s24144471. DeepSeek. DeepSeek-V4 preview release. news260424, 2026. Accessed April 29, 2026.
https://api-docs.deepseek.com/news/
Anna C. Doris, Daniele Grandi, Ryan Tomich, Md Ferdous Alam, Hyunmin Cheong, and Faez Ahmed. DesignQA: A multimodal benchmark for evaluating large language models’ understanding of engineering documentation. Journal of Computing and Information Science in Engineering, 2025. arXiv:2404.07917. Florian E. Dorner, Vivian Y. Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data. arXiv preprint arXiv:2410.13341, 2025. Accepted at ICLR 2025. Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall, New York, 1993. Alper Erturk and Daniel J. Inman. A distributed parameter electromechanical model for cantilevered piezoelectric energy harvesters. Journal of Vibration and Acoustics, 130(4):041002, 2008. doi: 10.1115/1.2890402. Alper Erturk and Daniel J. Inman. An experimentally validated bimorph cantilever model for piezoelectric energy harvesting from base excitations. Smart Materials and Structures, 18(2): 025009, 2009. doi: 10.1088/0964-1726/18/2/025009. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12): 86–92, 2021. doi: 10.1145/3458723. Google AI for Developers. Gemini thinking. https://ai.google.dev/gemini-api/docs/ thinking, 2026. Accessed April 18, 2026. Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633–638, 2025a. doi: 10.1038/s41586-025-09422-z. Xingang Guo et al. Toward engineering AGI: Benchmarking the engineering design capabilities of LLMs. arXiv preprint arXiv:2509.16204, 2025b. Accepted at NeurIPS 2025 Datasets & Benchmarks Track. Nikolaus Hansen. The CMA evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772, 2016. Updated version, 2023. Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software, 36(1):114–144, 2021. doi: 10.1080/10556788.2020.1808977. Yayati Jadhav and Amir Barati Farimani. Large language model agent as a mechanical designer. arXiv preprint arXiv:2404.17525, 2024. Yash Patawari Jain et al. MSEval: A dataset for material selection in conceptual design to evaluate algorithmic models. Journal of Mechanical Design, 2025. arXiv:2407.09719. 11
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110–4124, 2021. doi: 10.18653/v1/2021.naacl-main.324. Percy Liang, Rishi Bommasani, Tony Lee, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2023. Bilel Maamer, Ayda Boughamoura, Ahmed M. F. El-Bab, Loïc A. Francis, and Fathi Tounsi. A review on design improvements and techniques for mechanical energy harvesting using piezoelectric and electromagnetic schemes. Energy Conversion and Management, 199:111973, 2019. doi: 10.1016/j.enconman.2019.111973. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 220–229, 2019. doi: 10.1145/3287560.3287596. Slim Naifar, Olfa Kanoun, and Carlo Trigona. Energy harvesting technologies and applications for the Internet of Things and wireless sensor networks. Sensors, 24(14):4688, 2024. doi: 10.3390/s24144688. OpenAI. o4-mini model. https://developers.openai.com/api/docs/models/o4-mini, 2026. Accessed April 18, 2026. Sophia Xiao Pu, Sitao Cheng, Xin Eric Wang, and William Yang Wang. Dynamic evaluation for oversensitivity in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2337–2344, 2025. doi: 10.18653/v1/2025.findings-emnlp.126. Qwen Team. Qwen3.6-Plus: Towards real world agents. https://qwen.ai/blog?id=qwen3.6, 2026. Accessed April 29, 2026. Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. Asef Ishraq Sadaf, Riaz Ahmed, and Hossain Ahmed. Cantilever configurations in vibrationbased piezoelectric energy harvesting: A comprehensive review on beam shapes and multi-beam formations. Smart Materials and Structures, 33(12):123001, 2024. doi: 10.1088/1361-665X/ ad8b87. Asef Ishraq Sadaf, Mohammad Mohiuddin, Riaz Ahmed, Daniel Meade, Khaleda Akter, and Hossain Ahmed. Harnessing vibrations: A review on structural architecture and design ideology of the cantilever beam based piezoelectric energy harvesters. Applied Energy, 396:126286, 2025. doi: 10.1016/j.apenergy.2025.126286. Mohsen Safaei, Henry A. Sodano, and Steven R. Anton. A review of energy harvesting using piezoelectric materials: State-of-the-art a decade later (2008–2018). Smart Materials and Structures, 28 (11):113001, 2019. doi: 10.1088/1361-665X/ab36e4. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research (TMLR), 2023. arXiv:2206.04615. Tencent Hy Team. Hy3-preview model card. https://huggingface.co/tencent/Hy3-preview, 2026. Accessed April 29, 2026. Zhong Lin Wang and Jinhui Song. Piezoelectric nanogenerators based on zinc oxide nanowire arrays. Science, 312(5771):242–246, 2006. doi: 10.1126/science.1124005. C. B. Williams and R. B. Yates. Analysis of a micro-electric generator for microsystems. Sensors and Actuators A: Physical, 52(1–3):8–11, 1996. doi: 10.1016/0924-4247(96)80118-X. 12
Tian Xia, Tianrun Gao, Wenhao Deng, Long Wei, Xiaowei Qian, Yixian Jiang, Chenglei Yu, and Tailin Wu. BuildArena: A physics-aligned interactive benchmark of LLMs for engineering construction. arXiv preprint arXiv:2510.16559, 2025. Lang Xiong, Nishant Bhargava, Jianhang Hong, Jeremy Chang, Haihao Liu, Vasu Sharma, and Kevin Zhu. Probe-rewrite-evaluate: A workflow for reliable benchmarks and quantifying evaluation awareness. arXiv preprint arXiv:2509.00591, 2025. doi: 10.48550/arXiv.2509.00591. Sheng Xu, Yong Qin, Chen Xu, Yaguang Wei, Rusen Yang, and Zhong Lin Wang. Self-powered nanowire devices. Nature Nanotechnology, 5:366–373, 2010. doi: 10.1038/nnano.2010.46. Sherali Zeadally, Faisal Karim Shaikh, Anum Talpur, and Quan Z. Sheng. Design architectures for energy harvesting in the Internet of Things. Renewable and Sustainable Energy Reviews, 128: 109901, 2020. doi: 10.1016/j.rser.2020.109901.
13
Appendix A
Supplementary Roadmap
The appendix supports the benchmark claims in the main paper rather than serving as a storage room for extra leaderboards. It is organized into four support blocks: • Dataset and artifact audit: construction provenance, artifact boundary, contamination scope, P3 intervention sampling, and release contents. • Physical oracle, metric, and statistical definitions: closed-form VEH equations, uncertainty intervals, calibration references, and formal definitions for every headline and diagnostic metric. • Full results and split checks: complete P1–P4 full-bank tables, split-resolved tables, reasoning/thinking coverage, response-control profiles, and stage-aware selection/error decomposition. • Mechanism and cross-domain audits: extended related work, prompts, failure cases, selection– generation isomorphism, Tier 3 controlled-prompt results, CMA-ES, and the circuit audit. A.1
Master Metric Index
Tables 6–7 provide a compact index of the metrics used in Section 4 and the main appendix result tables. The index is a lookup layer: the Definition column points to the appendix section or table that gives the formula, scaling rule, or source calculation. Table 6: P1–P4 metrics used in main results. Metric
Type
Used in
P1: specification triage P1-Composite P1
Probe
Headline
Table 2; Finding 1
Accuracy
P1
Diagnostic
Macro-F1
P1
Diagnostic
Finding 2; App. bles App. tables
ACS
P1
Diagnostic
App. tables
MDS
P1
Diagnostic
IDS
P1
Diagnostic
Finding 2; App. bles App. tables
Subtype F1
P1
Diagnostic
App. tables
Over-action
P1
Error family Fig. 3; Finding 1
Over-refusal
P1
Error family Fig. 3; Finding 1
Missing-info miss
P1
Error family Fig. 3; Finding 1
P2: verifier-guided design search P2 ratio / P2b P2 Headline Final feasible rate
P2
CondRatio
P2
Utility loss
P2
Infeasible closure
P2
Destructive edit P2 Invalid/no-op proxy P2 P2a first feasible P2 Improvement rate / P2 P2c
Formula sketch / meaning
Definition
Weighted score combining Macro-F1, ACS, MDS, App. F.1 IDS, and subtype F1; higher means safer entry control. ta- Fraction of P1 tasks whose predicted action equals App. F.1 the gold triage action. Unweighted mean of F1 over propose, infeasible, App. F.1 and request actions. Propose recall multiplied by one minus spurious- App. F.1 propose rate. ta- Missing-info request recall multiplied by one mi- App. F.1 nus spurious-request rate. Infeasible recall multiplied by one minus spurious- App. F.1 infeasible rate. Macro-F1 over six subtype families, used as a App. F.1 small component of P1-Composite. Rate of proposing design when the gold action is App. F.1; App. H.1 not propose. Spurious request plus spurious infeasible rates on App. H.1 tasks whose gold action is different. 1− missing-info recall; denominator is gold App. H.1 missing-info cases.
Mean Fi Pi /PiBKF ; infeasible final designs con- App. F.2; App. H.1 tribute zero. Diagnostic Finding 2; App. ta- Fraction of P2 tasks whose final candidate satisfies App. F.2 bles all oracle constraints. Diagnostic App. tables Mean objective ratio only among tasks that close App. H.1 feasibly. Error family Fig. 3; Finding 1 1− CondRatio, clipped to [0, 1]; objective short- App. H.1 fall after feasible closure. Error family Fig. 3; Finding 1 1− final feasible rate; failure to close the physical App. H.1 search loop. Error family Fig. 3 Feasible-to-infeasible transition rate after an edit. App. H.1 Error family Fig. 3 Protocol-invalid rate used as the stable released App. K.3 proxy for invalid or no-op behavior. Diagnostic App. tables Fraction feasible after the first update step. App. F.2 Diagnostic App. tables Fraction of consecutive proposals with higher nor- App. H.1 malized utility. Table 2; Finding 1
Continued on next page
14
Metric
Probe
Type
Used in
Formula sketch / meaning
AUC
P2
Diagnostic
App. tables
Queries
P2
Diagnostic
App. tables
Mean best-so-far normalized objective over the App. H.1 fixed query budget. Mean number of oracle calls used by the model App. H.1 trajectory.
Definition
P3: corrupted-state recovery P3-Success P3 Headline
Table 2; Finding 1
Recovery quality / P3 RecQ First feasible P3
Diagnostic
Final feasibility after the corrupted-state recovery App. F.3 budget. Table 4; App. tables Objective ratio among successful final recoveries. App. F.3; App. H.1
Diagnostic
App. tables
Escape rate
P3
Cascade rate
P3
Dead-budget rate
P3
Replan rate
P3
Escape failure P3 Post-escape failure P3
Whether the trajectory reaches feasibility before App. H.1 the final step. Diagnostic Finding 1; App. ta- Fraction of tasks where the model moves away App. F.3; App. H.1 bles from the known corrupted trap. Diagnostic Fig. 4; App. tables Fraction of escaped tasks that introduce a new cou- App. F.3; App. H.1 pled violation. Diagnostic Fig. 3; App. tables Fraction of tasks with no meaningful recovery ac- App. F.3; App. H.1 tion. Diagnostic Table 4; App. tables Fraction of tasks with explicit reset or replanning App. H.1 behavior. Error family Fig. 3; Finding 1 1− escape rate. App. H.1 Error family Fig. 3; Findings 1–3 1 − Success/ max(Escape, ϵ), clipped to App. H.1 [0, 1].
P4: policy-conditioned selection Full Tau P4 Headline
Table 2; Finding 1
Exact match Top-1 accuracy Top-2 set accuracy
P4 P4 P4
Diagnostic Diagnostic Diagnostic
App. tables App. tables App. tables
Pareto Tau
P4
Diagnostic
App. tables
BARS
P4
Diagnostic
App. tables
Policy mismatch Dominance error Parse failure Top-choice miss
P4 P4 P4 P4
Error family Error family Error family Error family
Fig. 3; Finding 1 Fig. 3 Fig. 3 App. error tables
Kendall τb between model ranking and oracle pol- App. F.4; App. H.1 icy ranking over the full pool. Full-order exact match to the oracle policy order. App. F.4; App. H.1 Whether the oracle-best candidate is ranked first. App. F.4; App. H.1 Whether the unordered top-two candidate set is App. F.4; App. H.1 preserved. Ranking consistency on Pareto-relevant compar- App. H.1 isons. Balanced-active ranking score combining tau, App. F.4; App. H.1 policy-sensitive pairs, and exact match. 1− all-policy-sensitive pair accuracy. App. H.1 Pareto-violation rate. App. H.1 Parse-error rate in the ranking output. App. H.1 1− top-1 accuracy. App. H.1
Table 7: Profile, intervention, routing, and robustness metrics. Metric
Analysis
Used in
Formula sketch / meaning
Action discipline
Profile
Table 3; Finding 2
Edit style
Profile
Table 3; Finding 2
Feedback conditioning
Profile
Table 3; Finding 2
State-reset effort
Profile
Tables 3, 4
Policy execution
Profile
Table 3; Finding 2
Direction-normalized average of P1 action alignment, App. F.5; Table 15 macro-F1, recalls, propose precision, and non-invalid rate. Direction-normalized average of bounded local edits, App. F.5; Table 15 feasibility preservation, directed updates, final feasibility, non-destructive edits, and protocol validity. Direction-normalized average of violation reduc- App. F.5; Table 15 tion, utility improvement, best-so-far AUC, and postfeedback feasibility signals. Direction-normalized average of escape, replan, es- App. F.5; Table 15 cape quality, non-cascade, non-dead-budget, and statesummary gains. Direction-normalized average of P4 tau/BARS, policy- App. F.5; Table 15 sensitive pair accuracy, exact/top-1 accuracy, and nonerror terms.
State-summary ∆ suc- Intervention cess State-summary ∆ cas- Intervention cade Intervention subset Intervention
Fig. 4; Finding 3
Mean normalized score Routing
Table 5; Finding 3
Validation-selected Routing router Ex-post stage-best enve- Routing lope Router gain Routing
Table 5; Finding 3
Definition
Paired task-level P3-Success difference between state- Table 14 summary and raw-history prompts. Fig. 4; Finding 3 Paired task-level cascade-rate difference between Table 14 state-summary and raw-history prompts. Finding 3; App. Stratified 36-task P3 subset balanced across splits and Table 11 sampling subtype families.
Table 5; Finding 3 Finding 3
Average over stages after dividing each held-out stage App. K.2 score by its held-out stage-best score. Per-stage model chosen on validation split and evalu- App. K.2 ated only on held-out splits. Ex-post held-out best model per stage; upper bound, App. K.2 not deployable selection. Mean normalized score of validation router minus App. K.2 best single-model fallback.
15
Metric
Analysis
Used in
Formula sketch / meaning
Definition
Remaining headroom
Routing
Finding 3
1− validation-router mean normalized score.
App. K.2
Selection-generation gap Robustness CMA-ES P2 ratio
Robustness
Controlled-prompt delta Robustness Thinking-mode cover- Robustness age Circuit audit spread Robustness
Sec. 4.5
A-selection success minus B-generation success on App. P isomorphic probe groups. Sec. 4.5 Query-matched optimizer final feasible objective ratio App. M under the same oracle. Sec. 4.5 Targeted prompt condition minus neutral/default con- App. Q dition on pre-registered weak cells. Sec. 4.5; App. tables Grouped stage means by whether thinking mode was Tables 27–28 used in the reported run. Sec. 4.5 Stage-leader spread reproduced under a second closed- App. R form engineering domain.
Reporting conventions. All headline metrics are higher-is-better. Error-family rates are lower-isbetter and non-exclusive: they can use different denominators and are not intended to sum to one. Unless stated otherwise, roster means average per-model rates over the 12 complete P1–P4 runs. Held-out routing combines test_id and test_ood by task-count weights; validation splits are used only for model selection in the router simulation.
B
VEH Analytical Oracle Formulas
This section gives the closed-form equations used by the analytical VEH oracle. The oracle is a fast single-mode Euler–Bernoulli model for a unimorph piezoelectric cantilever with one substrate layer, one piezoelectric layer, a possible tip mass, harmonic base excitation, and a resistive electrical load. It is the external verifier used to compute physical feasibility and objective values; it is not a FEM model or a hardware-certification model. The electromechanical structure follows cantilevered piezoelectric energy-harvester models (Erturk and Inman, 2008, 2009); broader self-powered nanogenerator context is cited separately in the appendix discussion (Wang and Song, 2006, Xu et al., 2010). The implementation follows the same order as diagbench.physics.oracle.PiezoelectricOracle. Table 8: Symbols used by the VEH analytical oracle. All internal computations use SI units; task files expose lengths in mm, thicknesses in µm, tip mass in g, excitation in Hz and g, and output power in µW. Symbol
Meaning
L, b hs , hp
beam length and width m substrate and piezo layer m thickness tip mass and load resis- kg, Ω tance excitation frequency and Hz, rad/s angular frequency base acceleration ampli- g, m/s2 tude substrate and piezo Pa Young’s moduli substrate and piezo densi- kg/m3 ties piezo strain and stress m/V, C/m2 constants piezo permittivity at con- F/m stant stress mechanical damping ratio – transformed neutral-axis m location from substrate bottom composite bending stiff- N m2 ness first-mode effective mass kg resonant angular fre- rad/s, Hz quency and frequency electromechanical cou- N/V = C/m pling coefficient internal capacitance and F, Ω electrical load impedance tip displacement and load m, V voltage amplitudes average load power, root W, Pa, % stress, frequency error
mt , RL fe , ω ag , a Es , E p ρs , ρp d31 , e31 ϵT 33 ζ ȳ
EI meff ωr , fr θ Cp , Ze W, V P, σ, ηf
Unit
Notes from beam_length_mm, beam_width_mm from substrate_thickness_um, piezo_thickness_um from tip_mass_g, load_resistance_ohm ω = 2πfe a = 9.80665 ag material-table values material-table values e31 = d31 Ep −12 ϵT F/m 33 = ϵ0 ϵ33,r , ϵ0 = 8.854187817×10
default 0.01 unless specified in the task environment modulus-weighted centroid
parallel-axis theorem over substrate and piezo layers meff = 0.2357mb + mt first-mode clamped-free approximation single-mode unimorph coupling Ze = RL /(1 + jωRL Cp ) harmonic steady-state amplitudes task outputs are µW, MPa, and %
16
Input conversion and material parameters.
The task design vector is
x = (Lmm , bmm , hs,µm , hp,µm , mt,g , RL ),
(A-1)
and the excitation context is (fe , ag ). The oracle converts to SI units by L = 10−3 Lmm ,
b = 10−3 bmm ,
hs = 10−6 hs,µm ,
hp = 10−6 hp,µm ,
mt = 10−3 mt,g ,
a = 9.80665 ag .
(A-2)
The piezo material supplies (d31 , ϵ33,r , Ep , ρp ) and the substrate material supplies (Es , ρs ). Unless specified, the default pair is PZT-5A on stainless steel. The oracle supports PZT-5A, PZT-5H, MFCM8528, and PVDF for the piezo layer, and stainless steel, aluminum, brass, and titanium for the substrate. Composite section and bending stiffness. Let As = bhs and Ap = bhp be the layer areas. The layer centroids measured upward from the substrate bottom are ys =
hs , 2
yp = hs +
hp . 2
(A-3)
The transformed neutral axis is the modulus-weighted centroid ȳ =
Es As ys + Ep Ap yp . Es As + Ep Ap
(A-4)
The second moments of area about the neutral axis are Is =
bh3s + As (ys − ȳ)2 , 12
Ip =
bh3p + Ap (yp − ȳ)2 , 12
(A-5)
and the composite bending stiffness is EI = Es Is + Ep Ip . Equivalent mass and resonance.
(A-6)
The distributed beam mass is mb = (ρs hs + ρp hp )bL.
(A-7)
The first-mode effective mass uses the standard clamped-free cantilever factor meff = αϕ mb + mt ,
αϕ = 0.2357.
The analytical first-mode resonance is approximated by r 3EI ωr ωr = , fr = . 3 meff L 2π
(A-8)
(A-9)
The frequency-matching diagnostic used by the constraints is ηf = 100 Electromechanical coupling and capacitance.
|fr − fe | . fe
(A-10)
The piezoelectric stress constant is
e31 = d31 Ep .
(A-11)
For full piezo coverage and a tip-normalized first bending mode, the root-slope factor is γ1 = 1.3765. The single-mode electromechanical coupling coefficient is γ1 (A-12) θ = −e31 b (yp − ȳ) . L The internal capacitance of the piezo layer is Cp = ϵT33
bL , hp
ϵT33 = ϵ0 ϵ33,r . 17
(A-13)
Electrical load and harmonic response. For excitation angular frequency ω = 2πfe , the resistive load and piezo capacitance form the complex electrical impedance Ze (ω) =
RL . 1 + jωRL Cp
(A-14)
The coupled single-mode denominator is D(ω) = meff (ωr2 − ω 2 ) + j 2ζmeff ωr ω + θ2 jωZe (ω). The steady-state tip displacement amplitude under base acceleration is meff a . W = |D(ω)|
(A-15)
(A-16)
This is the dynamic frequency-response displacement used by the implementation, not the static cantilever deflection F L3 /(3EI). The voltage amplitude across the load is V = |θ| ω |Ze (ω)| W,
(A-17)
V2 , 2RL
(A-18)
and the average load power is P =
PµW = 106 P.
√ 2 /RL with Vrms = V / 2. The Here V is a peak harmonic amplitude. Equivalently, P = Vrms reported tip displacement is Wmm = 103 W. (A-19) Root stress and feasibility constraints. The oracle uses a root-bending approximation for substrate tensile-face stress. With cs = ȳ denoting the distance from the neutral axis to the bottom substrate face, 3W σ = Es cs 2 , σMPa = 10−6 σ. (A-20) L Given task limits σmax , Wmax , ηmax , and Pmin , the oracle reports signed slacks sσ = σmax − σMPa , sW = Wmax − Wmm , sf = ηmax − ηf , sP = PµW − Pmin .
(A-21)
Positive slack means the corresponding constraint is satisfied. A candidate is feasible iff sσ ≥ 0,
sW ≥ 0,
sf ≥ 0,
sP ≥ 0.
(A-22)
If a task omits a limit, the implementation uses the default values σmax = 50 MPa, Wmax = 5 mm, ηmax = 5%, and Pmin = 1 µW. The oracle returns (fr , PµW , σMPa , Wmm , ηf ), the four signed slacks above, and the intermediate values meff , EI, θ, and Cp for audit.
C
Construction Audit and Artifact Availability
The current VEH suite is not a loose aggregation of prompts. Its optimizer-facing stages are traceable to a concrete literature-extraction and admission pipeline. The upstream extraction audit covers 209 papers and 2090 field rows, with 151 papers carrying at least one issue and 77 carrying a high-risk issue. From this audit, 71 cantilever candidates are forwarded to oracle review, and the current P2 backbone admits 52 cleaned structure-review anchors. The current benchmark then packages these anchors into split-specific task banks whose manifests store source tables, input manifest hashes, seeds, and artifact SHA256 values.
18
Table 9: Construction audit for the current benchmark release. The table separates source scale, split policy, and release evidence so that the benchmark can be audited as a dataset rather than only as a set of scores. Stage
Upstream source
P1
vetted VEH anchor con- three matched partitions un- 240 = 80/80/80 texts rendered through der three seeds; subtypeeight certification tem- balanced and not DOIplates disjoint 52 cleaned structure- optimizer-facing splits 208 = 64/40/104 review anchors from the are DOI-disjoint: 26 extraction audit oracle-domain anchors feed dev/test_id, 26 projected out-of-domain-kept anchors feed test_ood dual-constraint trap pro- inherits P2 optimizer-facing 156 = 64/40/52 jection from P2 split boundary policy-conditioned rank- inherits P2 optimizer-facing 159 = 93/36/30 ing pools derived from split boundary P2 BKF records
P2
P3 P4
Split policy
Final inventory
Audit detail legacy dev/test_id/test_ood labels are bookkeeping only; partitions use 20/19/19 anchor contexts with pairwise overlap 19/19/18 8 unit-suspect papers retained with explicit flags; admission report records source-anchor counts and no dropped task ids
manifests record source P2 task bank, BKF table, split manifest, and trap policy 53 realized base pools across the release; manifests record profile counts, top-1 distinctness, and source-task lineage for each pool
Table 10: Task-statistics overview for the released VEH benchmark snapshot. Counts are computed from the released JSONL task banks and their split manifests. Dimension
Released snapshot
Probe inventory Split sizes Subtype coverage
P1/P2/P3/P4 contain 240/208/156/159 tasks. P1: 80/80/80; P2: 64/40/104; P3: 64/40/52; P4: 93/36/30 over dev/test_id/test_ood. P1 has 8 triage subtypes; P2 has 4 search subtypes; P3 uses the dual-constraint trap family; P4 uses policyconditioned full-ranking pools. P1: 60 base, 30 moderate, 150 hard; P2: 52 easy, 52 medium, 104 hard; P3: 156 hard; P4: 12 easy, 51 medium, 96 hard. P1 uses 20 matched anchor contexts; P2 uses 52 cleaned anchors; P3 traces to 50 source anchors; P4 realizes 29 source-anchor ranking pools. P2 has 104 oracle-domain and 104 out-of-domain-kept tasks; P3 has 104/52; P4 has 129/30; P1 is a matched triage stress stage rather than a DOI-disjoint source split.
Difficulty coverage Source anchors Source-group coverage
The release package is designed around these manifests and is available through the artifact page at huggingface.co/datasets/AnonymousVehbench/vehbench. VEHBench is the benchmark name used in the paper; diagbench is the implementation package and code namespace for the same released artifact, so mixed names in code paths do not indicate a separate benchmark or incompatible evaluator. diagbench-734D is the historical repository slug used during peer review, not the benchmark title. The artifact contains the exact task JSONL files and manifests for P1–P4, the oracle and evaluator code, the task-generation scripts, the admission and split reports, and the per-model JSONL run logs used for the main and appendix tables. The intended artifact boundary is therefore the same as the paper boundary: each reported benchmark table should be traceable to a specific manifest-backed snapshot rather than to an undocumented local run. The release preserves one auxiliary model-output directory under anonymized naming for audit completeness, so repository directory counts may exceed the 12-model roster used in the paper tables. All reported tables use the frozen 12 complete P1–P4 model runs; incomplete, deprecated, or auxiliary snapshots are retained only as release evidence and are not imputed into scores. Contamination resistance in the current release is deliberately scoped. It means DOI/title-hash deduplication in the upstream paper registry, DOI-disjoint source-anchor separation for P2–P4, manifest-level lineage tracking, and evaluation on post-fix noleak P1 runs only. It does not mean that we claim web-scale near-duplicate search over model pretraining corpora or model-specific pretraining decontamination. P1 is intentionally outside the source-disjoint claim and should be read as a matched robustness stress stage rather than as a held-out generalization benchmark.
D
P3 Intervention Provenance
The P3 intervention in the main paper is intentionally not a rerun over the entire 156-task bank. It is a frozen protocol audit on a 36-task subset selected by stratified sampling over split × source_p2_subtype. The selection rule is: draw three tasks per observed stratum, then top up to 12 tasks per split from the remaining pool while preserving split balance. This keeps every observed family visible while preventing the intervention table from collapsing to a single split.
19
Table 11: Full-bank versus intervention-subset composition for P3. The 36-task intervention subset is stratified rather than ad hoc: every observed split/subtype family is retained, and the final subset is exactly balanced across splits. Split
Full 156-task bank
36-task intervention subset
dev
64 tasks: boundary_binding 20, paper_like 17, power_tight 12, resonance_tuned 15 40 tasks: boundary_binding 10, paper_like 4, power_tight 15, resonance_tuned 11 52 tasks: projected_boundary 18, projected_resonance 34
12 tasks: 3 / 3 / 3 / 3 across the same four strata
test_id
test_ood
E
12 tasks: 3 / 3 / 3 / 3 across the same four strata 12 tasks: projected_boundary 3, projected_resonance 9 after split-balanced top-up
Statistical Validity
Headline gaps should be paired with uncertainty rather than interpreted as isolated point estimates. For continuous aggregates such as P1-Composite, P2 final feasible power ratio, and P4 Kendall-style scores, we use nonparametric bootstrap intervals over task rows (Efron and Tibshirani, 1993). For binary proportions such as P3-Success, we report Wilson intervals. Paired bootstrap deltas are used for the main near ties and for the P3 intervention audit so that uncertainty reflects within-task covariance rather than independent resampling. Table 12: Headline uncertainty table for the four main probes. Probe
Metric
Best
95% CI
Key delta
Note
P1
P1-Composite
0.574
[0.500, 0.641]
not a trivial propose-bias win
P2
feasible power ratio
0.3904
[0.3598, 0.4235]
P3
P3-Success
47.4%
[39.8, 55.2]%
P4
Kendall Tau
0.887
[0.862, 0.911]
qwen3-max − gemini3.1-pro-preview: +0.025 [-0.078, 0.121] gemini-3.1-pro-preview − claude-4.6-sonnet: +0.1510 [0.1117, 0.1891] hunyuan-hy3-preview − mimo-v2.5-pro: +1.9 pts [-3.2, 7.1] gpt-5.4 − qwen3.6-plus: +0.013 [-0.024, 0.050]
separates closure from anchoring
near tie at the recovery frontier
calibrates ranking claims
Table 13: Near-tie paired tests that matter most to the main paper. Pair
Test object
Why this pair matters
hunyuan-hy3-preview vs mimo- P3-Success / RecoveryQuality v2.5-pro gpt-5.4 vs qwen3.6-plus P4 Full Tau / Exact qwen3-max vs gemini-3.1-pro- P1-Composite / P2 headline preview
current recovery frontier; similar success, different recovery quality structural ranking versus endpoint preference matching certification-versus-closure split
Table 14: P3 intervention numeric results. Delta columns report paired task-bootstrap 95% intervals for state_summary versus raw history. Succ. raw/sum.
∆ success
Casc. raw/sum.
∆ cascade
deepseek-r1 gpt-5.4 gemini-3.1-pro-preview claude-4.6-sonnet
55.6 / 58.3 63.9 / 63.9 50.0 / 66.7 30.6 / 63.9
+2.8 pts [-11.1, +16.7] +0.0 pts [-13.9, +13.9] +16.7 pts [-2.8, +36.1] +33.3 pts [+16.7, +50.0]
30.3 / 8.7 29.4 / 0.0 46.9 / 25.0 5.7 / 4.3
-21.6 pts [-37.5, -7.2] -29.4 pts [-45.5, -14.7] -21.9 pts [-43.6, +0.5] -1.4 pts [-8.6, +4.4]
Model mean
50.0 / 63.2
+13.2 pts [+4.9, +21.5]
28.1 / 9.5
-18.6 pts [-26.6, -10.6]
Model
20
F
Metric Definitions
This section defines the metrics used in the main paper and appendix. All scores are computed from persisted JSONL logs by deterministic evaluator code; no metric uses human annotation or LLM-as-a-judge. F.1
P1: Credible Triage
P1 is a one-shot three-action decision problem with {propose_design, declare_infeasible, request_missing_info}.
gold
labels
in
Accuracy and Macro-F1. Accuracy is micro-averaged over all P1 tasks. Macro-F1 is the unweighted mean of the three action-level F1 scores: 1X 2 Precc Recc Macro-F1 = F1,c , F1,c = . (A-23) 3 Precc + Recc c∈C
If Precc + Recc = 0, the corresponding F1,c is defined as zero. Discipline scores. Let Ar , Mr , and Ir denote recall for propose_design, request_missing_info, and declare_infeasible. Let As , Ms , and Is denote the corresponding spurious-action rates on tasks whose gold label is not that action. Then: ACS = Ar (1 − As ), (A-24) MDS = Mr (1 − Ms ), (A-25) IDS = Ir (1 − Is ). (A-26) ACS penalizes habitual proposing, MDS penalizes over-requesting information, and IDS penalizes over-refusal. P1-Composite. The P1 headline metric is the weighted certification score used by the evaluator: P1Comp = 0.40 F 13class + 0.20 ACS + 0.15 MDS (A-27) + 0.15 IDS + 0.10 F 1subtype . F.2
P2: Verifier-Guided Design Search
For a P2 trajectory, let xt be the design at step t, V (xt ) ≥ 0 the total normalized oracle violation, and T the final evaluated step. First and final feasibility. P2a is the fraction of tasks where V (x1 ) = 0. Final feasible rate is the fraction where V (xT ) = 0. P2b final feasible power ratio. For VEH tasks, the headline metric is: NP2 h i 1 X (i) P2b = 1 V (xT ) = 0 NP2 i=1 (i)
·
(A-28)
Pout (xT )
. (i) Pout (xBKF ) The circuit audit uses the analogous final feasible objective score returned by the circuit oracle. Conditional objective ratio. (i) obj(xT ) (i) i: V (xT )=0 . (i) |{i : V (xT ) = 0}|
P CondObj =
(A-29)
Trajectory diagnostics. Violation reduction consistency is the fraction of transitions with V (xt+1 ) ≤ V (xt ) − 10−6 . Directed update rate is the fraction of violated-constraint updates that move the relevant oracle metric in the feedback-implied direction. Feasibility preservation is the fraction of feasible-to-feasible transitions. Mean log edit delta is: 1 X |log(xt+1,d /xt,d )| δedit = . (A-30) |D| log(xmax /xmin d d ) d∈D Over-edit and no-op rates threshold this quantity in the corresponding evaluator. 21
F.3
P3: Post-Trap Recovery
P3 starts from a corrupted trajectory state. Escape rate is the fraction of tasks where at least one step reduces violation relative to the trapped state. Cascade rate is the fraction of escaped tasks where a new structurally coupled violation appears after the escape move. Dead-budget rate is the fraction of tasks with no meaningful post-trap design change. P3-Success and recovery quality. P3Success =
i 1 X h (i) 1 V (xT ) = 0 . NP3 i (i) obj(xT ) (i) i: V (xT )=0 . (i) |{i : V (xT ) = 0}|
(A-31)
P RecoveryQuality =
(A-32)
The raw-history versus state-summary delta is the paired task-level difference in P3-Success between the two prompt representations, reported with paired bootstrap intervals. F.4
P4: Policy-Conditioned Ranking
Each P4 task contains five oracle-feasible candidates. Full Kendall τ is Kendall τb between the model ranking and oracle ranking. Exact match, top-1 accuracy, top-2 set accuracy, pairwise accuracy, and policy-flip accuracy are computed directly from the predicted order. BARS. For the VEH P4-full bank, the headline BARS score is computed only on balanced-active rows: BARS = 0.55 τBA + 0.25 PairAccBA (A-33) + 0.20 ExactBA , where BA denotes the balanced-active subset. For the circuit audit, the analogous ranking score uses scaled Kendall (τ + 1)/2, policy-flip accuracy, and exact match with the same weights. F.5
Response-Control Profile Scores
Profile scores are log-derived diagnostics, not extra model prompts and not human annotations. They summarize how a model behaves in the released logs: whether it acts too readily or too cautiously before search, whether edits are local or destructive, whether verifier feedback changes the next action, whether corrupted history triggers reset-like behavior, and whether fixed candidates are ranked according to the stated policy. For each dimension D, the evaluator collects a fixed indicator set MD , orients every indicator so higher is better, min-max scales it across the compared models, and averages: X 1 ProfileD (m) = zq (m). (A-34) |MD | q∈MD
For an indicator q, the normalized value is: zq (m) =
sq (m) − minm′ sq (m′ ) , maxm′ sq (m′ ) − minm′ sq (m′ ) + ϵ
(A-35)
after applying an orientation transform when lower raw values are better (for example parse-error, cascade, over-edit, or spurious-action rates). We use ϵ = 10−12 only to avoid division by zero; constant indicators therefore contribute no cross-model separation. The compared set for the main paper is the 12 complete P1–P4 model runs. The profiles are used only for diagnosis and correlation analysis; they do not change the task-level scores in Tables 2 or 18–21.
22
Table 15: Indicators used for response-control profile extraction. All indicators are direction-normalized before averaging. Dimension
Indicators
Action discipline Edit style
P1 action-distribution alignment, macro-F1, missing recall, infeasible recall, propose precision, non-invalid rate P2 bounded-local-edit rate, feasibility preservation, directed update rate, final feasible rate, non-destructive edit rate, protocol-valid rate P2 violation reduction, P2 utility improvement, P2 best-so-far AUC, P3 violation reduction, P3 post-feedback feasible rate P3 trap escape, explicit replan, escape quality, non-cascade rate, non-dead-budget rate, state-summary success gain, state-summary cascade reduction P4 scaled full Tau, balanced-active BARS, balanced-active policy-sensitive pair accuracy, exact match, top-1 accuracy, non-Pareto-violation rate, non-parse-error rate
Feedback conditioning State-reset effort Policy execution
Interpretation. The profile dimensions are intentionally behavioral and stage-local. Action discipline is expected to align with P1 because P1 is an entry-control problem. Edit style and feedback conditioning are expected to align with P2 because P2 rewards bounded verifier-guided design search. State-reset effort is expected to matter most in P3 because P3 contaminates the history itself, but it measures recovery-oriented effort rather than final recovery success. Policy execution is expected to align with P4 because search has been removed and only policy-conditioned ordering remains. In the main results, the strongest positive alignments follow the intended diagnostic pattern: action discipline tracks P1-Composite, edit style tracks P2 final feasible power ratio, and policy execution tracks P4 Tau/BARS. Weak or negative off-diagonal relationships are part of the finding: a response-control prior that is helpful at one trusted-state boundary can be neutral or harmful at another.
G
Calibration References
Weak baselines are useful here because the benchmark is deliberately structured. For P1, the main concern is degenerate label-prior matching. For P2, the main concern is whether headline gains reflect true improvement beyond the literature-derived seed or only feasibility preservation. For P3, the relevant weak behavior is continuity with the corrupted state rather than explicit reset. For P4, the relevant floor is chance agreement with a five-item ordering. Table 16: Calibration references corresponding to the main-text weak baselines. These are not competitors to the frontier roster; they anchor how much of each stage can be explained by trivial behavior. Probe
Baseline reference
Score
Interpretation
P1
alwayspropose_design on the dev split
P2
anchor-fixed heuristic on the 208-task main bank
56.3% accuracy, 0.240 macro-F1, 0.0 worst-action recall 100% feasible coverage, 0.3065 mean power ratio
P3
last-state continuation / no-reset heuristic
a degenerate action prior can look superficially acceptable on raw accuracy, which is why P1Composite and subtype-balanced reporting are necessary many frontier models fail to preserve even the trivial feasible anchor, while gemini-3.1-propreview is the only one to beat this heuristic on the headline utility ratio the useful floor is not a single scalar success rate but the failure mode produced by trusting corrupted state; P3 therefore reports escape, cascade, dead-budget, and recovery jointly
P4
uniform random permutation over five feasible candidates
0% explicit reset by construction; weak reference for escape, cascade, and recovery columns E[τ ] = 0, exact 0.83%, top-1 20%
23
observed ranking scores are far above chance, so P4 is not a random-agreement artifact
H
Full Leaderboard Tables
The appendix carries the full all-model tables for each main probe so that the main-paper mechanism claims remain auditable. All four tables use the frozen 12-model complete roster; runs without complete P1–P4 coverage are excluded rather than imputed. This block should be read as a diagnostic audit rather than four extra leaderboards. Read each table column-wise: the headline metric gives the rank, while the diagnostic columns identify the failure mode behind that rank. The full evaluator formulas are collected in Appendix F; the dictionary below states how to compute the columns used in Tables 18–21 and the diagnostic error rates used in Figure 3. H.1
Metric and Diagnostic-Rate Dictionary
All quantities are computed from released JSONL task logs and oracle traces. Unless otherwise stated, rates are first computed per model and then averaged across the 12 complete models when the main text reports a roster mean. Diagnostic error families are non-exclusive: they use different denominators and are not intended to sum to one. For example, if a missing-information P1 task is incorrectly answered with propose_design, it is both a missing-information miss and an over-action event. P1 column definitions. Let gi be the gold P1 action and âi the model action for task i, with actions P in {propose_design, request_missing_info, declare_infeasible}. Accuracy is N −1 i 1[âi = gi ]. Macro-F1 is the unweighted mean of the three action-level F1 scores. For action c, recall and spurious-action rate are: |{i : gi = c and âi = c}| Recc = , (A-36) |{i : gi = c}| |{i : gi ̸= c and âi = c}| Spurc = . (A-37) |{i : gi ̸= c}| The table columns ACS, MDS, and IDS are: ACS = Recpropose (1 − Spurpropose ),
(A-38)
MDS = Recrequest (1 − Spurrequest ),
(A-39)
IDS = Recinfeasible (1 − Spurinfeasible ).
(A-40)
Subtype F1 averages F1 over the released P1 subtypes. P1-Composite is 0.40 macro-F1 +0.20 ACS +0.15 MDS +0.15 IDS +0.10 subtype F1. The P1 error-family rates in Figure 3 are: over-action = Spurpropose , over-refusal = clip[0,1] (Spurrequest + Spurinfeasible ), and missing-information miss = 1 − Recrequest . P2 column definitions. For P2 task i, let Fi ∈ {0, 1} indicate final oracle feasibility, Pi be the final output power, and PiBKF > 0 be the P benchmark-known feasible reference power. P2a is first-step feasibility. Final feasible rate is N −1 i Fi . The headline P2b final feasible power ratio is: Pi 1 X P2b = Fi BKF , (A-41) N i Pi so infeasible final designs contribute zero. Conditional ratio is: X 1 Pi CondRatio = P , BKF i Fi i:F =1 Pi
(A-42)
i
which measures objective quality only among closed feasible searches. Impr. is the trajectory improvement rate: the fraction of consecutive proposal steps whose utility increases, where utility is normalized power minus penalties for active frequency, stress, and displacement violations. AUC is the mean best-so-far normalized objective over the fixed query budget; Queries is the mean number of oracle calls used. The P2 error-family rates are infeasible closure = 1− final feasible rate, destructive edit = feasible-to-infeasible transition rate, invalid/no-op proxy = protocol-invalid rate, and utility loss = 1 − CondRatio. Utility loss therefore asks how much objective quality remains missing after successful feasible closure; it is not mutually exclusive with closure failure. 24
P3 column definitions. For P3, Success is final feasibility after the corrupted-state recovery budget. RecQ is the objective ratio among successful recoveries. First feas. is whether the trajectory reaches feasibility before the final step. Escape is the fraction of tasks where the model moves away from the known corrupted trap. Cascade is the fraction of escaped tasks that introduce a new coupled violation. Dead marks trajectories with no meaningful recovery action, and Replan marks explicit reset or replanning behavior. The P3 error-family rates are escape failure = 1− escape rate, cascade = cascade rate, dead budget = dead-budget rate, and post-escape failure = 1 − Success/ max(Escape, ϵ), clipped to [0, 1]. Post-escape failure isolates models that notice the trap but still fail to stabilize a feasible design. P4 column definitions. For P4, all candidates are oracle-feasible before ranking. Full Tau is Kendall τb between the model order and the oracle policy order over the full five-item pool. Exact is full-order exact match. Top1 and Top2 measure whether the best candidate and the top-two set are preserved. Pareto Tau measures ordering consistency on Pareto-relevant comparisons. BARS is the balanced-active ranking score defined in Appendix F. Parse / viol. reports parse failure and attempted candidate-modification or dominance-violation behavior. The P4 error-family rates are policy mismatch = 1− all-policy-sensitive pair accuracy, dominance error = Pareto-violation rate, parse failure = parse-error rate, and top-choice miss = 1− top-1 accuracy. Table 17: Interpretive map for the appendix full-table block. Each table is tied to a benchmark claim rather than presented as metric inventory alone. Probe
Core columns
Diagnostic 1
Diagnostic 2
Diagnostic 3
Diagnostic 4
Primary mechanism claim served
P1
Acc / Macro-F1 / ACS / MDS / IDS / subtype F1 / composite P2a / P2b / final feasible rate / conditional ratio / Impr. (P2c) / queries success / recovery quality / trap escape / cascade / dead budget / explicit replan Full Tau / Exact / Top-1 / Top2 / Pareto Tau / BARS
over-act prior
hard infeasibility misses
missing-info discipline
balanced selectivity
raw accuracy does not equal credible triage
anchoring
closure
best-so-far improvement
query efficiency
start is not finish
no-escape failure
escaped-butunrecovered
false recovery
post-escape quality
escape is not recovery
structural order
strict endpoint ranking
policyactive ranking
parse stability
structural ranking is not policy sensitivity
P2
P3
P4
25
P1 metrics and readout. P1 is a one-shot triage task. Accuracy counts exact agreement with the oracle label; macro-F1 averages the three action labels so that a model cannot score well by favoring the majority action. ACS, MDS, and IDS are discipline scores for propose_design, request_missing_info, and declare_infeasible; each multiplies recall by one minus the corresponding spurious-action rate. Subtype F1 checks whether performance holds across nearfeasible, missing-info, and infeasible subtypes. The headline P1-Composite combines these terms, so it rewards balanced entry discipline rather than raw willingness to produce a design. The table shows why this is necessary: several models have similar raw accuracy, but qwen3-max and gemini-3.1-propreview separate because their discipline scores are more balanced, while claude-4.6-sonnet exposes over-refusal or action-mismatch behavior that accuracy alone would under-explain. Table 18: P1 full table. Compact headers keep certification metrics readable. Model
Acc
qwen3-max 65.4% gemini-3.1-pro-preview 69.2% o4-mini 70.0% deepseek-r1 66.2% gpt-5.4 43.3% hunyuan-hy3-preview 66.2% deepseek-v3 64.6% llama-3.3-70b 62.5% mimo-v2.5-pro 61.7% qwen3.6-plus 61.3% deepseek-v4-pro 61.3% claude-4.6-sonnet 57.5%
F1
ACS
MDS
IDS
Subtype F1
Comp.
0.647 0.636 0.623 0.594 0.486 0.539 0.483 0.467 0.433 0.422 0.422 0.305
0.429 0.430 0.388 0.360 0.224 0.295 0.206 0.242 0.152 0.143 0.143 0.057
0.533 0.667 0.396 0.510 0.657 0.333 0.352 0.200 0.244 0.200 0.200 0.000
0.486 0.218 0.361 0.298 0.223 0.200 0.100 0.192 0.083 0.100 0.100 0.100
0.768 0.762 0.776 0.730 0.569 0.700 0.668 0.669 0.644 0.641 0.641 0.583
0.574 0.549 0.518 0.504 0.428 0.425 0.369 0.361 0.317 0.306 0.306 0.207
Audit note. The exact tie between qwen3.6-plus and deepseek-v4-pro is preserved from the frozen scoring snapshot rather than broken by post-hoc rounding or reranking. Exact vector ties are therefore treated as tied diagnostic rows, not as evidence for an ordering between the two models. Takeaway. P1 separates credible entry control from raw action frequency: the strongest rows combine reasonable proposing with missing-information and infeasibility discipline, while weak rows expose over-refusal or over-action even when accuracy is not catastrophic.
P2 metrics and readout. P2 is a verifier-guided design-search task under oracle feedback. P2a is first-step feasibility, which measures whether the first edit lands directly in a feasible region. P2b is the headline final feasible power ratio: the final design contributes only if it is feasible, and its output power is normalized by the benchmark-known feasible reference. Final feasibility separates closure from objective quality, conditional ratio reports quality only among successful searches, improvement measures best-so-far utility gain, AUC summarizes trajectory quality across the budget, and queries records how many oracle calls were needed. The main conclusion is that search closure and utility are not the same: gemini-3.1-pro-preview dominates P2b because it combines high final feasibility with high feasible utility, while models with respectable first-step feasibility can still finish with low final ratio if their subsequent edits overfit one constraint or lose power. Table 19: P2 full table. P2a is first-step feasibility; P2b is the headline final feasible power ratio. Model
P2a
gemini-3.1-pro-preview 26.0% deepseek-r1 29.8% claude-4.6-sonnet 13.0% qwen3.6-plus 30.8% qwen3-max 23.6% hunyuan-hy3-preview 2.4% deepseek-v3 7.2% o4-mini 22.6% gpt-5.4 7.2% deepseek-v4-pro 30.3% llama-3.3-70b 23.6% mimo-v2.5-pro 26.4%
P2b
Final feas.
Cond. ratio
Impr.
AUC
Queries
0.3904 0.2413 0.2394 0.2063 0.1955 0.1609 0.1565 0.1551 0.1329 0.1294 0.1197 0.1073
96.2% 67.8% 62.5% 61.1% 55.8% 48.6% 44.2% 42.8% 36.1% 41.3% 37.0% 33.2%
0.4101 0.3585 0.3891 0.3432 0.3537 0.3380 0.3576 0.3665 0.3841 0.3205 0.3277 0.3282
0.7902 0.7901 0.7037 0.6466 0.6649 0.8033 0.5907 0.7571 0.6828 0.6167 0.6673 0.6714
0.3388 0.2999 0.2797 0.2880 0.2894 0.2480 0.2766 0.3119 0.2708 0.2935 0.2895 0.3075
2.63 3.47 4.41 3.87 4.21 4.79 4.68 3.59 4.75 4.26 4.38 4.50
Takeaway. P2 is won by feasible closure plus useful final power, not by first-step feasibility alone; gemini-3.1-pro-preview’s lead is therefore a design-search trajectory result rather than an anchoring artifact.
26
P3 metrics and readout. P3 starts from a corrupted trajectory. Success is final feasibility at the end of the recovery budget; recovery quality reports the objective quality among successful recoveries; first feasible identifies whether a model ever reaches a feasible state early in the recovery trace. Escape measures whether the model moves away from the known trap, cascade records whether the escape introduces a new coupled violation, dead-budget marks trajectories with no meaningful recovery action, and replan counts explicit reset or replanning behavior. These columns show that P3 is not ordinary clean design search: claude-4.6-sonnet is the clearest diagnostic row, with 96.8% escape but only 0.6% first-feasible and 16.0% final success. It often notices that the raw history is wrong, but it rarely stabilizes back into a feasible design. deepseek-r1 and gpt-5.4 combine high escape with better final recovery; hunyuan-hy3-preview and mimo-v2.5-pro lead success through different trade-offs in recovery quality and cascade control. Table 20: P3 full table. Columns separate recovery success, escape, cascade, dead-budget, and replanning behavior. Model
Succ.
RecQ
hunyuan-hy3-preview 47.4% 0.4876 mimo-v2.5-pro 45.5% 0.6263 deepseek-r1 42.9% 0.2708 gpt-5.4 42.3% 0.3723 gemini-3.1-pro-preview 37.2% 0.3194 deepseek-v3 35.3% 0.2868 qwen3-max 30.1% 0.2612 deepseek-v4-pro 28.8% 0.1986 qwen3.6-plus 27.6% 0.1453 o4-mini 26.3% 0.1389 claude-4.6-sonnet 16.0% 0.0960 llama-3.3-70b 3.2% 0.0926
First feas.
Escape
Cascade
Dead
Replan
40.4% 41.0% 22.4% 26.3% 26.9% 23.1% 22.4% 27.6% 12.8% 12.2% 0.6% 2.6%
77.6% 63.5% 92.3% 90.4% 92.3% 82.7% 85.9% 90.4% 94.9% 69.2% 96.8% 34.6%
9.1% 4.0% 26.4% 21.3% 50.7% 27.1% 12.7% 17.0% 23.0% 17.6% 5.3% 3.7%
25.3% 26.7% 25.1% 25.2% 25.8% 27.6% 25.7% 24.3% 25.6% 26.6% 25.7% 14.3%
7.7% 17.9% 26.3% 6.4% 10.9% 57.7% 49.4% 99.4% 0.6% 8.3% 0.0% 0.0%
Takeaway. P3 shows that escaping a corrupted trajectory is not the same as recovering: high escape with low success marks continuity or cascade failure, while successful recovery requires both reset and stabilization.
P4 metrics and readout. P4 removes search from the problem: all candidates are oracle-feasible before the model ranks them. Full Kendall Tau measures global order agreement with the oracle policy ranking; exact match requires the entire five-item order to be correct; top-1 and top-2 set accuracy measure whether the highest-priority choices are preserved; Pareto Tau isolates ranking consistency on Pareto-relevant comparisons; BARS emphasizes balanced-active policy-sensitive rows; parse and violation rates record formatting failures and attempts to alter fixed candidates. The table shows that P4 is not reducible to one ranking statistic. gpt-5.4 leads Full Tau, qwen3.6-plus is strongest on exact match, and gemini-3.1-pro-preview has a high BARS despite lower Tau, indicating that policy execution has multiple sub-behaviors rather than a single scalar notion of “ranking skill.” Table 21: P4 full table. Compact notation keeps ranking, Pareto, and formatting diagnostics in one table. Model
Exact
Top1
Top2
Pareto
BARS
Parse / viol.
gpt-5.4 0.887 57.2% qwen3.6-plus 0.877 66.3% deepseek-v3 0.860 54.7% mimo-v2.5-pro 0.843 52.8% claude-4.6-sonnet 0.840 56.6% hunyuan-hy3-preview 0.839 50.9% qwen3-max 0.835 49.7% deepseek-r1 0.833 56.0% gemini-3.1-pro-preview 0.824 54.1% deepseek-v4-pro 0.794 43.4% o4-mini 0.780 50.0% llama-3.3-70b 0.714 34.0%
Tau
83.0% 79.1% 77.4% 72.3% 72.3% 77.4% 78.0% 71.7% 71.7% 60.4% 60.0% 54.1%
81.1% 83.4% 79.2% 79.2% 79.2% 79.9% 76.7% 76.7% 78.6% 74.2% 72.0% 65.4%
1.000 1.000 0.984 0.993 1.000 0.943 0.983 1.000 0.975 0.977 1.000 0.959
0.6258 0.6912 0.5751 0.6012 0.6781 0.5610 0.5367 0.6835 0.6988 0.5832 0.5988 0.5386
0.0% / 0.0% 0.0% / 0.0% 0.0% / 0.8% 0.0% / 0.3% 0.0% / 0.0% 0.0% / 2.8% 0.0% / 0.8% 0.0% / 0.0% 1.3% / 1.3% 0.0% / 1.2% 0.0% / 0.0% 0.0% / 2.0%
Takeaway. P4 ranking has multiple sub-behaviors: global order quality, exact endpoint order, topchoice preservation, and policy-sensitive BARS can diverge, which is why P4 is not reducible to one scalar preference score. 27
I
Split-Resolved Tables
This block checks that the main-paper claims are not driven by a single split. The P2 closure pattern and P4 non-monotonic ranking pattern remain visible across splits, while P3 contains genuine near ties that should be read with the confidence intervals in Appendix 12. P1 uses matched partitions rather than DOI-disjoint source splits, so its split labels audit robustness rather than held-out generalization. The split-resolved audit tables below use the same 12 complete P1–P4 model runs as the main leaderboard and stage-aware selection simulation. Exact split-invariant rows, such as qwen3.6-plus and claude-4.6-sonnet in P1, are retained as audit signals from the released snapshot. Each table reports the main stage headline plus one supporting diagnostic in a split-resolved view. Table 22: Split and subtype reporting plan. The appendix presents split-resolved views as mechanism audits rather than as extra leaderboards. Probe
Split-resolved view
P1
matched-partition infeasible-margin, composite and infeasible-structural, diagnostics (legacy missing-info families dev/test_id/test_ood labels) dev / test_id / test_ood paper-like, resonanceheadline and feasible tuned, power-tight, rate boundary-binding dev / test_id / test_ood trap families and cascade success and recovery diagnostics quality dev / test_id / test_ood balanced, performanceFull Tau and Exact first, reliability-first, BARS
P2 P3 P4
Subtype families
Main claim audited certification robustness is not driven by one matched partition or blocker family closure dominates anchoring as the main search distinction corrupted-state recovery is not reducible to nominal design-search quality structural ranking and policysensitive ranking remain partially separable
Table 23: P1 matched-partition table. Each cell reports P1-Composite and raw accuracy. Model qwen3-max gemini-3.1-pro-preview o4-mini deepseek-r1 gpt-5.4 hunyuan-hy3-preview deepseek-v3 llama-3.3-70b mimo-v2.5-pro qwen3.6-plus deepseek-v4-pro claude-4.6-sonnet
dev
test_id
test_ood
0.532 / 61.3% 0.545 / 70.0% 0.542 / 71.2% 0.517 / 68.8% 0.374 / 37.5% 0.448 / 67.5% 0.357 / 63.7% 0.362 / 61.3% 0.336 / 62.5% 0.306 / 61.3% 0.275 / 60.0% 0.207 / 57.5%
0.622 / 70.0% 0.525 / 66.2% 0.509 / 70.0% 0.512 / 66.2% 0.469 / 47.5% 0.422 / 66.2% 0.385 / 65.0% 0.393 / 65.0% 0.308 / 61.3% 0.306 / 61.3% 0.336 / 62.5% 0.207 / 57.5%
0.567 / 65.0% 0.578 / 71.2% 0.502 / 68.8% 0.482 / 63.7% 0.440 / 45.0% 0.404 / 65.0% 0.364 / 65.0% 0.328 / 61.3% 0.306 / 61.3% 0.306 / 61.3% 0.306 / 61.3% 0.207 / 57.5%
Table 24: P2 split-resolved table. Each cell reports the final feasible power ratio and final feasible coverage. Model gemini-3.1-pro-preview deepseek-r1 claude-4.6-sonnet qwen3.6-plus qwen3-max hunyuan-hy3-preview deepseek-v3 o4-mini gpt-5.4 deepseek-v4-pro llama-3.3-70b mimo-v2.5-pro
dev
test_id
test_ood
0.4017 / 98.4% 0.2560 / 62.5% 0.2199 / 54.7% 0.2355 / 57.8% 0.2182 / 50.0% 0.1689 / 43.8% 0.2052 / 48.4% 0.1587 / 42.2% 0.0991 / 26.6% 0.1891 / 51.6% 0.1781 / 43.8% 0.1682 / 40.6%
0.4324 / 90.0% 0.3049 / 82.5% 0.3095 / 75.0% 0.2316 / 65.0% 0.2486 / 60.0% 0.1703 / 50.0% 0.1761 / 47.5% 0.1220 / 35.0% 0.0541 / 12.5% 0.1165 / 40.0% 0.1046 / 32.5% 0.0819 / 30.0%
0.3672 / 97.1% 0.2078 / 65.4% 0.2245 / 62.5% 0.1786 / 61.5% 0.1612 / 57.7% 0.1523 / 51.0% 0.1189 / 40.4% 0.1655 / 46.2% 0.1841 / 51.0% 0.0977 / 35.6% 0.0896 / 34.6% 0.0796 / 29.8%
28
Table 25: P3 split-resolved table. Each cell reports final recovered feasible rate and post-escape recovery quality. Model hunyuan-hy3-preview mimo-v2.5-pro deepseek-r1 gpt-5.4 gemini-3.1-pro-preview deepseek-v3 qwen3-max deepseek-v4-pro qwen3.6-plus o4-mini claude-4.6-sonnet llama-3.3-70b
dev
test_id
test_ood
50.0% / 0.4811 50.0% / 0.6591 45.3% / 0.3103 48.4% / 0.4322 43.8% / 0.3770 42.2% / 0.2368 39.1% / 0.3182 37.5% / 0.2456 28.1% / 0.1774 28.1% / 0.1556 20.3% / 0.1270 4.7% / 0.1429
42.5% / 0.5333 40.0% / 0.5435 37.5% / 0.2105 40.0% / 0.3816 32.5% / 0.2917 27.5% / 0.3571 22.5% / 0.1912 25.0% / 0.1714 27.5% / 0.1389 30.0% / 0.1667 10.0% / 0.0513 0.0% / 0.0000
48.1% / 0.4605 44.2% / 0.6406 44.2% / 0.2708 36.5% / 0.2841 32.7% / 0.2660 32.7% / 0.3068 25.0% / 0.2444 21.2% / 0.1633 26.9% / 0.1100 21.2% / 0.0972 15.4% / 0.0918 3.8% / 0.1429
Table 26: P4 split-resolved table. Each cell reports Full Tau and Exact. Model gpt-5.4 qwen3.6-plus deepseek-v3 mimo-v2.5-pro claude-4.6-sonnet hunyuan-hy3-preview qwen3-max deepseek-r1 gemini-3.1-pro-preview deepseek-v4-pro o4-mini llama-3.3-70b
J
dev
test_id
test_ood
0.886 / 53.8% 0.867 / 65.6% 0.888 / 57.0% 0.852 / 51.6% 0.832 / 59.1% 0.856 / 51.6% 0.832 / 49.5% 0.834 / 57.0% 0.826 / 55.9% 0.804 / 44.1% 0.778 / 47.3% 0.720 / 35.5%
0.900 / 58.3% 0.856 / 52.8% 0.783 / 47.2% 0.794 / 44.4% 0.833 / 41.7% 0.767 / 41.7% 0.850 / 47.2% 0.800 / 44.4% 0.778 / 38.9% 0.772 / 38.9% 0.733 / 44.4% 0.667 / 22.2%
0.873 / 66.7% 0.920 / 80.0% 0.867 / 56.7% 0.873 / 66.7% 0.873 / 66.7% 0.873 / 60.0% 0.827 / 53.3% 0.867 / 66.7% 0.873 / 66.7% 0.787 / 46.7% 0.800 / 56.7% 0.753 / 43.3%
Reasoning Coverage and Response-Control Profiles
This block remains secondary evidence rather than a substitute for the model-by-model tables. It is a negative control for the response-control profile account: if the paper were only measuring whether thinking mode was on, the stage patterns should collapse under this grouping. Instead, thinking mode helps some workflow regimes more than others and still does not erase stage dissociation.
29
Table 27: Reasoning/thinking run-mode audit for the 12 complete P1–P4 model runs. The column records whether thinking mode was used in the reported run. Model
Public / run evidence
o4-mini
Yes
reasoning-labeled row
Yes
thinking support active in the reported run
Yes
key within-family contrast is R1 versus V3
Yes
run manifests are retained as the source of truth for provider settings
No
manifest records thinking=disabled for JSON/action-schema stability
No
manifest records thinking=disabled for JSON/action-schema stability DashScope call used provider default; no explicit enable_thinking=True override
deepseek-v3 claude-4.6-sonnet
OpenAI documentation identifies it as a reasoning model (OpenAI, 2026). Google documentation describes internal thinking / thought-summary support (Google AI for Developers, 2026). DeepSeek documentation positions R1 as a reasoning-oriented model (Guo et al., 2025a). Public provider pages describe Hy3 as a fast/slow or multi-mode reasoning model (Tencent Hy Team, 2026). DeepSeek documentation describes V4 as supporting Thinking and Non-Thinking modes (DeepSeek, 2026). Xiaomi Mimo API row is treated as a thinking-capable provider model in the run plan. Alibaba documentation describes Qwen3-Max as hybrid-thinking with thinking disabled by default (Alibaba Cloud Model Studio, 2026). Qwen documentation describes explicit enable_thinking=True usage (Qwen Team, 2026). Frontier GPT row with provider-default reasoning effort in our logs. No reasoning-first grouping used here. No reasoning-first grouping used here.
llama-3.3-70b
No reasoning-first grouping used here.
No
gemini-3.1-pro-preview
deepseek-r1 hunyuan-hy3-preview
deepseek-v4-pro
mimo-v2.5-pro qwen3-max
qwen3.6-plus
gpt-5.4
Think used
30
No
Caveat
No
DashScope call used provider default without an explicit enable_thinking flag
No
reported as no explicit thinking-mode run
No No
used as the within-family non-R1 contrast grouped by observed profile, not by hidden implementation details open model baseline row in the complete P1–P4 roster
Table 28: Stage-wise group means on the 12 complete model runs, grouped only by whether thinking mode was used in the reported run. Stage
Think
No-think
P1 Composite P2 headline P3 Success (%) P4 Full Tau
0.499 0.237 38.5 0.819
0.359 0.161 28.6 0.831
31
Table 29: Stable profile scores on the 12 complete P1–P4 runs. Scores are diagnostic boundary-fit indicators, not a monolithic leaderboard. Column aliases: Action=action discipline, Edit=edit style, Feedback=feedback conditioning, State=state-reset effort, Policy=policy execution. Model
Action
Edit
Feedback
State
Policy
qwen3-max gemini-3.1-pro-preview gpt-5.4 deepseek-r1 o4-mini hunyuan-hy3-preview deepseek-v3 llama-3.3-70b mimo-v2.5-pro deepseek-v4-pro qwen3.6-plus claude-4.6-sonnet
0.737 0.668 0.657 0.652 0.635 0.564 0.536 0.528 0.499 0.491 0.491 0.428
0.571 0.753 0.482 0.667 0.470 0.485 0.493 0.586 0.385 0.404 0.613 0.622
0.407 0.508 0.406 0.455 0.391 0.429 0.389 0.326 0.391 0.332 0.404 0.392
0.646 0.425 0.453 0.455 0.495 0.599 0.629 0.452 0.627 0.737 0.523 0.443
0.679 0.729 0.743 0.728 0.666 0.688 0.708 0.622 0.707 0.666 0.761 0.730
32
K
Stage-Aware Selection and Error Decomposition
This appendix reports the two analyses used in Section 4 to connect stage-local scores to actionable model selection and visible failure modes. Both analyses reuse existing split scores and oracle-log profile metrics; no new model calls are introduced. K.1
Stage-Rank Correlation Matrix
Table 30 reports the full Spearman rank-correlation matrix among the four stage leaderboards. The off-diagonal correlations range from −0.26 to 0.38, showing that no single stage ordering is a reliable proxy for the others. Table 30: Spearman rank correlations between stage leaderboards.
P1 P2 P3 P4
K.2
P1
P2
P3
P4
1.00 0.29 0.31 -0.26
0.29 1.00 0.06 0.07
0.31 0.06 1.00 0.38
-0.26 0.07 0.38 1.00
Stage-Aware Selection Simulation
The router simulation uses the 12 complete P1–P4 model runs. Model selection is performed on the validation split. Held-out evaluation combines test_id and test_ood with task-count weights: P1 80/80, P2 40/104, P3 40/52, and P4 36/30. Stage scores are normalized by the held-out best score for that stage before averaging. Validation ties are broken by the corresponding validation auxiliary metric: P1 accuracy, P2 final feasible rate, P3 recovery quality, and P4 exact match. Table 31: Full stage-aware selection simulation. The selected model column records the model used for each stage. Strategy Validation aggregate leader
P1
0.552 (gemini-3.1pro-preview) Best held-out single model 0.552 (gemini-3.1pro-preview) Validation-selected stage router 0.552 (gemini-3.1pro-preview) Ex-post stage-best envelope 0.594 (qwen3-max)
P2
P3
0.385 (gemini-3.1pro-preview) 0.385 (gemini-3.1pro-preview) 0.385 (gemini-3.1pro-preview) 0.385 (gemini-3.1pro-preview)
32.6% (gemini-3.10.821 (gemini-3.1pro-preview) pro-preview) 32.6% (gemini-3.10.821 (gemini-3.1pro-preview) pro-preview) 42.4% 0.821 (deepseek-v3) (mimo-v2.5-pro) 45.7% (hunyuan-hy3- 0.888 (gpt-5.4) preview)
P4
Mean norm. 0.892 0.892 0.945 1.000
The validation-selected stage router improves over the validation aggregate leader because it assigns different models to different design roles instead of forcing one model to cover the full workflow. In the full 12-model pool, the aggregate validation leader and the best held-out single model are both gemini-3.1-pro-preview, with a mean normalized held-out score of 0.892. This equality is not a duplicate-row error: gemini-3.1-pro-preview is both the validation aggregate choice and the ex-post best single model under the held-out pool. The validation-selected router keeps gemini-3.1-propreview for P1 and P2, switches P3 to mimo-v2.5-pro, and selects deepseek-v3 for P4; this raises mean normalized held-out score to 0.945. The realized gain comes almost entirely from P3, where held-out recovery improves from 32.6% under the aggregate leader to 42.4% under the routed choice. The router does not reach the ex-post stage-best envelope because validation selection misses the held-out P1 leader (qwen3-max), the held-out P3 leader (hunyuan-hy3-preview), and the held-out P4 leader (gpt-5.4). This is the desired diagnostic behavior: the router is deployable because it uses only validation information, while the envelope reports remaining headroom. Worked normalization example. Table 5 reports raw held-out stage scores and a normalized mean. For qwen3-max, the raw held-out scores are 0.594 on P1, 0.185 on P2, 23.9% on P3, and 0.839 on P4. The held-out stage-best denominators are 0.594, 0.385, 45.7%, and 0.888, respectively. Its mean normalized score is therefore 1 0.594 0.185 0.239 0.839 + + + = 0.738. 4 0.594 0.385 0.457 0.888 33
The ex-post stage-best envelope has mean normalized score 1.000 by construction because each stage uses its own held-out best model as the denominator. K.3
Stage-Wise Error Decomposition
Table 32 expands the compact main-text decomposition to all 12 full-coverage models. The error families are non-exclusive diagnostic rates; they are not a partition of all failures and should not be summed within a row. The exact column formulas are given in Appendix H.1. P2 invalid/no-op uses protocol-invalid rate as the stable released proxy; raw identical-edit no-op extraction is not required for the reported scores. Table 32: Stage-wise error decomposition by model. All values are rates in [0,1]; lower is better. Model qwen3-max gemini-3.1-pro-preview o4-mini deepseek-r1 gpt-5.4 hunyuan-hy3-preview deepseek-v3 llama-3.3-70b mimo-v2.5-pro qwen3.6-plus deepseek-v4-pro claude-4.6-sonnet
P1 over-act P1 over-ref. P1 miss P2 no close P2 destr. P2 inv. P2 util. loss P3 no escape P3 casc. P3 dead P3 post-esc. fail P4 policy P4 dom. P4 parse 0.390 0.524 0.600 0.581 0.343 0.705 0.790 0.752 0.848 0.857 0.857 0.943
0.233 0.067 0.027 0.088 0.552 0.000 0.010 0.039 0.000 0.000 0.000 0.000
0.467 0.333 0.600 0.444 0.311 0.667 0.644 0.800 0.756 0.800 0.800 1.000
0.442 0.038 0.572 0.322 0.639 0.514 0.558 0.630 0.668 0.389 0.587 0.375
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
0.000 0.000 0.005 0.000 0.000 0.005 0.000 0.000 0.000 0.000 0.000 0.005
0.646 0.590 0.634 0.642 0.616 0.662 0.642 0.672 0.672 0.657 0.680 0.611
0.141 0.077 0.308 0.077 0.096 0.224 0.173 0.654 0.365 0.051 0.096 0.032
0.127 0.507 0.176 0.264 0.213 0.091 0.271 0.037 0.040 0.230 0.170 0.053
0.257 0.258 0.266 0.251 0.252 0.253 0.276 0.143 0.267 0.256 0.243 0.257
0.649 0.597 0.620 0.535 0.532 0.388 0.574 0.907 0.283 0.709 0.681 0.834
0.701 0.711 0.879 0.719 0.566 0.657 0.617 0.851 0.657 0.661 0.783 0.710
0.008 0.013 0.000 0.000 0.000 0.028 0.008 0.020 0.003 0.000 0.012 0.000
0.000 0.013 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
These error families explain what the response-control profiles summarize: action discipline appears as over-action or over-refusal; bounded editing and feedback conditioning appear as infeasible closure, destructive edits, invalid updates, and utility loss; state-reset effort appears as escape, cascade, dead-budget, and post-escape failures; and policy execution appears as policy mismatch, dominance error, parse failure, and top-choice failure. Readout. The main text reports the cross-model means and interpretation. This table provides the per-model values used to identify which failure family drives each stage score and to support the profile readout.
34
L
Extended Related Work Table 33: Positioning VEHBench among LLM-assisted engineering design benchmarks.
Benchmark / line of What it evaluates work
How validity is checked
Broad engineering design benchmarks Artifact / simulation benchmarks
Rubric / task-specific checks Simulator / physical feedback
General engineering design capability Whether generated artifacts work
Iterative mechanical design or CAD/CAE reasoning Engineering QA / docu- Understanding mentation benchmarks engineering documents Materials / selection Choosing suitable benchmarks materials or options Mechanical-design agents
VEHBench
LLM-assisted VEH co-design
What remains hard to see
VEHBench distinction
Which design-stage behavior failed Whether failure came from triage, search, recovery, or selection Simulation or Cross-stage role expert/problem checks dissociation
Stage-local workflow diagnosis Verifier used as workflow boundary
QA labels / expert answers
Moves from comprehension to design assistance Separates generation, recovery, and selection
Design-loop behavior
Labels / property rules Search/recovery behavior before selection Analytical physical N/A oracle
Separate assistant roles
Engineering-native diagnostic benchmark
VEHBench sits between engineering-design benchmarks and benchmark-validity work. General agent and software benchmarks are useful context for endpoint task completion, but they are not the primary comparators for this paper because VEHBench targets coupled physical design rather than coding, web navigation, or generic tool use. The relevant comparison is therefore not whether an agent completes a broad interactive task, but whether an engineering assistant behaves reliably at a specific verifier-grounded design boundary: pre-search triage, verifier-guided design search, corrupted-state recovery, or policy-conditioned selection. Appendix Table 33 gives the benchmark-positioning summary; this appendix expands the surrounding context. A growing line of work evaluates LLMs on engineering design tasks that require external physical or simulation-based verification. EngDesign, BuildArena, mechanical-design agents, DesignQA, and MSEval evaluate design satisfaction, simulation loops, document reasoning, or material-selection behavior (Guo et al., 2025b, Xia et al., 2025, Jadhav and Farimani, 2024, Doris et al., 2025, Jain et al., 2025). VEHBench differs by treating design as a sequence of decision regimes rather than one endpoint objective. Its VEH domain is not arbitrary: vibration and piezoelectric energy harvesting is a canonical small-scale power setting for wireless and embedded systems, from early micro-generator analysis to modern cantilever-based, mechanical, and IoT-oriented harvesting surveys (Williams and Yates, 1996, Safaei et al., 2019, Maamer et al., 2019, Sadaf et al., 2024, 2025, Citroni et al., 2024, Zeadally et al., 2020, Naifar et al., 2024). Cantilevered piezoelectric models provide the closed-form electromechanical structure used by the oracle (Erturk and Inman, 2008, 2009), and highimpact nanogenerator work further established piezoelectric transduction as a route to self-powered microsystems (Wang and Song, 2006, Xu et al., 2010). This makes VEH a useful closed-form engineering domain: it is physically meaningful, constraint-coupled, and analytically auditable. VEHBench also builds on benchmark design and evaluation-validity work. BIG-bench, HELM, Dynabench, datasheets, model cards, Croissant, and benchmark-accountability work motivate transparent task construction, metadata, contamination boundaries, and reproducibility claims (Srivastava et al., 2023, Liang et al., 2023, Kiela et al., 2021, Gebru et al., 2021, Mitchell et al., 2019, Akhtar et al., 2024, Raji et al., 2021). LLM-as-judge and prompt-sensitivity studies caution that evaluation can fail when the judge is comparable to the tested model or when models recognize evaluation contexts (Dorner et al., 2025, Xiong et al., 2025, Pu et al., 2025). VEHBench responds by using an external physical oracle and by reporting a controlled-prompt audit rather than relying on conversational closure or LLM judgment. Finally, VEHBench is adjacent to black-box optimization, but not reducible to it. COCO/BBOB and CMA-ES provide optimization context and baselines (Hansen et al., 2021, Hansen, 2016). VEHBench’s target is broader: numerical search matters in P2, but P1 tests actionability, P3 tests recovery after state contamination, and P4 tests policy-conditioned selection.
35
M
CMA-ES Classical Optimization Baseline
CMA-ES (Hansen, 2016) is run under the same analytical oracle as the LLM agents, with population size 8 and initial sigma 0.15. The 6-query run covers the released split-resolved P2 subset used for this optimizer audit (156 tasks), while the relaxed 40-query run is a dev-subset ceiling check (64 tasks). These runs are therefore calibration references rather than full-bank competitors to the model results in Table 2. Table 34: CMA-ES P2 baseline results. The 6-query budget matches the LLM agents on the 156-task optimizeraudit subset; 40 queries tests a 64-task dev-subset optimizer ceiling. The anchor-fixed heuristic (0 oracle queries) and gemini-3.1-pro-preview (2.63 avg. queries) are shown for context. Method CMA-ES CMA-ES Anchor-fixed heuristic gemini-3.1-pro-preview (LLM)
Budget (queries)
Final feasible rate
Mean power ratio
Mean queries
6 40 0 6
0.269 0.578 1.000 0.962
0.099 0.305 0.306 0.390
6.0 40.0 0.0 2.63
CMA-ES under 6 queries underperforms most frontier LLM agents on P2. With 40 queries on the dev subset it matches the anchor-fixed heuristic (0.305 vs. 0.306) but remains below gemini-3.1-propreview (0.390). This supports two complementary interpretations: (1) LLM agents carry physical priors from pretraining that accelerate feasibility-preserving search beyond what a pure numerical optimizer achieves under the same oracle budget; and (2) P2 is not explained away by a generic optimizer within the benchmark’s query constraints, so the observed inter-model differences reflect genuine behavioral variation rather than a ceiling effect.
36
N
Prompt Templates, Oracle Protocol, and Task Examples
N.1
Evaluation Harness Protocol
All model runs use a shared evaluation harness. Each task instance specifies the design variables, variable bounds, task objective, and query budget. The harness parses the first valid JSON object in each model response, normalizes field aliases (e.g., R_ohm vs. R_value), and applies a fixed retry policy: up to two retries for transient malformed or length-truncated outputs before counting the row as a parse failure. Iterative probes (P2, P3) run under bounded query budgets. P2 can stop on feasible closure; P3 preserves the full recovery trace. All runs use temperature 0.0, and provider-level default max output tokens (typically 4096–8192 depending on the provider). Run dates, provider endpoints, and model version strings are recorded in per-model run manifests released with the benchmark artifact. N.2
P1 Prompt Template (Triage)
The P1 prompt presents the task specification and asks the model to output exactly one action. P1 Prompt: Specification Triage System: You are an engineering triage agent. Given a design specification, choose whether the task is feasible, infeasible, or underspecified. Do not output numeric design variables; output one action label as a single JSON object. User: { "task": "Evaluate whether the following VEH design specification is feasible, infeasible, or missing critical information.", "spec": { "beam_length_mm": [20, 80], "tip_mass_g": [0.5, 5.0], "substrate_thickness_um": [100, 500], "piezo_thickness_um": [50, 300], "target_frequency_hz": 50, "target_power_uw": 15, "excitation_acceleration_g": 0.5, "material": "PZT-5A", "substrate": "stainless_steel" }, "allowed_actions": ["propose_design", "declare_infeasible", "request_missing_info"] } Expected output format: { "action": "propose_design | declare_infeasible | request_missing_info", "reason": "<brief justification>", "missing_fields": ["spec.xxx", ...] }
The missing_fields array is only required for request_missing_info. P1 evaluator scoring is described in Appendix F.
37
N.3
P2 Prompt Template (Verifier-Guided Design Search)
P2 runs as a multi-turn loop. Each turn presents the current design state and the oracle’s constraint feedback. P2 Prompt: Verifier-Guided Design Search System: You are an engineering design agent. You are given a design task with coupled physical constraints. At each step, propose an improved design. The verifier will return feasibility and violation information. Your goal is to reach a feasible design within the query budget while maximizing the objective. User (turn 1): { "task": "Design a piezoelectric cantilever VEH meeting the following constraints.", "design_variables": ["beam_length_mm", "tip_mass_g", "substrate_thickness_um", "piezo_thickness_um", "beam_width_mm", "load_resistance_ohm"], "variable_bounds": { "beam_length_mm": [20, 80], "tip_mass_g": [0.5, 5.0], "substrate_thickness_um": [100, 500], "piezo_thickness_um": [50, 300], "beam_width_mm": [5, 30], "load_resistance_ohm": [1000, 100000] }, "initial_design": { "beam_length_mm": 50, "tip_mass_g": 2.0, "substrate_thickness_um": 300, "piezo_thickness_um": 150, "beam_width_mm": 15, "load_resistance_ohm": 10000 }, "objective": "maximize output power (uw)", "query_budget": 6 } Oracle response (example, after turn 1): { "feasible": false, "metrics": { "resonant_freq_hz": 72.3, "load_power_uw": 3.2, "tip_stress_mpa": 12.4, "tip_disp_mm": 0.8 }, "violations": [ {"constraint": "frequency_target", "value": 72.3, "target": 50.0, "tolerance": 0.05, "message": "frequency too high; increase mass or length"} ], "total_violation": 8.47, "oracle_feedback": "frequency=72.3 Hz exceeds [47.5, 52.5] Hz target; tip stress=12.4 MPa within limit." } Model output format (each turn): { "design": { "beam_length_mm": 55.0, "tip_mass_g": 2.5, "substrate_thickness_um": 280, "piezo_thickness_um": 150, "beam_width_mm": 15, "load_resistance_ohm": 10000 }, "action": "continue", "reason": "Increased beam length and tip mass to reduce resonant frequency toward 50 Hz target." }
38
N.4
P3 Prompt Template (Post-Trap Recovery)
P3 is structurally identical to P2 but pre-loads a corrupted trajectory into the conversation context. The corruption varies by trap type: unit-flip, wrong-formula-direction, false-feasibility, topology-trap, verifier-ignored, or progressive-contamination. The model receives the full pseudo-history as if it were prior conversation turns. P3 Prompt: Corrupted-State Recovery System: You are an engineering recovery agent. You are given a design task, a prior trajectory, and verifier feedback. Some earlier trajectory steps may be corrupted or physically misleading. Your goal is to recover a feasible design within the remaining query budget. You may reuse a trusted prior design, reset to an earlier state, replan, or propose a new bounded edit, but the final response must be a single valid JSON object. User: { "task": "Recover a feasible VEH design from the following trajectory.", "design_variables": ["beam_length_mm", "tip_mass_g", "substrate_thickness_um", "piezo_thickness_um", "beam_width_mm", "load_resistance_ohm"], "variable_bounds": {...}, "objective": "maximize output power (uw) subject to verifier constraints", "query_budget_remaining": 4, "corrupted_history": [ { "step": 0, "design": {...}, "verifier_response": {...}, "note": "prior assistant claimed frequency was too low" }, { "step": 1, "design": {...}, "verifier_response": { "feasible": false, "oracle_feedback": "frequency is too high; the prior direction increased violation" } } ], "allowed_actions": ["propose_design", "replan", "reset_to_prior", "declare_infeasible"] } Model output format: { "action": "propose_design | replan | reset_to_prior | declare_infeasible", "design": { "beam_length_mm": 60.0, "tip_mass_g": 2.8, "substrate_thickness_um": 260, "piezo_thickness_um": 150, "beam_width_mm": 15, "load_resistance_ohm": 10000 }, "trusted_state_decision": "discard step 1 because verifier feedback contradicts the claimed update direction", "reason": "Reset from the corrupted direction and reduce frequency without introducing stress or displacement violations." }
For the state-summary intervention, the raw trajectory is replaced with a deterministic verifierauthored summary containing only already-observed trusted fields: current step, latest proposal, latest verifier state, best-so-far feasible proposal (if any), short-horizon objective and violation trends, and whether new violations were introduced. The oracle evaluator is unchanged.
39
N.5
P4 Prompt Template (Policy-Conditioned Ranking) P4 Prompt: Policy-Conditioned Selection
System: You are an engineering evaluator. Given a set of oracle-feasible design candidates and a stated ranking policy, rank the candidates according to the policy. Do not modify the designs or propose new ones. User: { "task": "Rank the following feasible VEH designs according to the stated policy.", "policy": "Prioritize output power, then tip-stress margin, then component cost. All candidates are feasible.", "candidates": [ {"id": "A", "design": {...}}, {"id": "B", "design": {...}}, {"id": "C", "design": {...}}, {"id": "D", "design": {...}}, {"id": "E", "design": {...}} ] } Expected output: { "ranking": ["B", "A", "D", "E", "C"], "reason": "B has highest power; A second on power with better stress margin than D; E and C follow by power." }
N.6
Oracle Specification (VEH Domain)
The VEH oracle follows the single-mode piezoelectric cantilever model defined in Appendix B. That section gives the complete symbol table, composite-section calculation, electromechanical coupling, dynamic frequency-response displacement, average load-power formula, root-stress approximation, and feasibility slacks. We do not repeat a shortened formula list here because the prompt templates use oracle feedback fields rather than asking the model to execute the equations manually. The release includes the oracle code, task manifests, and validation artifacts needed to audit these computations. N.7
Example Task Walkthrough (VEH P2)
We illustrate a complete P2 trajectory for a single task to make the evaluation protocol concrete. Task. Design a PZT-5A cantilever VEH with target frequency 50 Hz, target power ≥ 15 µW, excitation 0.5 g, stress limit 50 MPa, displacement limit 2 mm. Budget: 6 queries. Initial design: L = 50 mm, mt = 2.0 g, hs = 300 µm, hp = 150 µm, w = 15 mm, RL = 10 kΩ. Step 1. Model proposes {L : 55, mt : 2.5, hs : 280, hp : 150, w : 15, RL : 10000}. Oracle returns: fr = 63.8 Hz (violation: too high), Pout = 4.1 µW, stress 11.2 MPa, displacement 0.7 mm. Feasible: false. Total violation: 5.24. Step 2. Model proposes {L : 65, mt : 3.0, hs : 250, hp : 120, w : 15, RL : 10000}. Oracle returns: fr = 48.7 Hz (within [47.5, 52.5]), Pout = 9.8 µW, stress 9.1 MPa, displacement 0.8 mm. Feasible: true. Violation: 0. Objective: 9.8/15 = 0.653 (relative to BKF). End of trajectory (feasible closure). This task is considered a P2 success. The model used 2 queries, achieved feasible closure, and the final feasible power ratio is 0.653. N.8
Example Task Walkthrough (Circuit P3 Dual-Trap)
Task. RC low-pass filter, target fc = 1 kHz ±2%, source current limit 0.5 mA, R ∈ [1kΩ, 100kΩ], C ∈ [1nF, 1µF], Vin = 5 V. Corrupted history (phase-1 trap): prior steps repeatedly increased C under the false assumption that fc was too low; actual fc = 3.18 kHz (too high). Current bad design: R = 5kΩ, C = 100 nF. Budget: 5 queries. 40
Step 1. Model recognizes the contradiction between oracle feedback (“fc too high”) and corrupted history (“fc too low”). Proposes {R : 20000, C : 7.96nF}. Oracle: fc = 1000 Hz (feasible), Isource = 0.25 mA (feasible). Phase 1 escaped. Step 2 (phase-2 trap). With R = 20kΩ, the model’s edit has triggered the near-boundary margin: a second constraint on component tolerance (previously dormant) now requires C ≥ 1.5 nF. Oracle returns a new violation. Model must adjust R downward slightly and C upward. Step 3. Model proposes {R : 15000, C : 10.6nF}. Oracle: fc = 1001 Hz (feasible), Isource = 0.33 mA (feasible). All constraints satisfied. Recovery success: true. Cascade: false. This task demonstrates the progressive-dual-trap mechanism: the model must escape the primary corrupted state (phase 1) and then handle a secondary coupled constraint triggered by the escape move itself (phase 2). N.9
Run Configuration
All model runs share the following configuration unless noted otherwise in per-model manifests: • Temperature: 0.0 (deterministic decoding). • Max output tokens: provider default (4096–8192). • Retry policy: up to 2 retries on parse failure; row counted as parse error after 3 consecutive failures. • P2 stop rule: trajectory terminates on feasible closure or budget exhaustion. • P3 stop rule: trajectory runs full budget; no early stop on feasibility. • Provider endpoints and model version strings are recorded in per-model run manifests. • Core roster run dates: April 2026. Thinking-model extension run dates: April 25–27, 2026. Circuit audit run dates: April 28–30, 2026.
41
O
Representative Failure Cases
The following notes give one auditable failure case per probe. They are not additional metrics; they make the response-control interpretation concrete and show what the probe-specific scores count as failure. We use prose rather than a table because the cases contain long task identifiers and multi-sentence readouts. P1: optimistic over-proposal. In audit case p1::dev::infeasible_hard_conflict::0000:: s42, the gold action is declare_infeasible, but the model chooses propose_design. The verifier-side label makes the failure observable: continuation is counted as a spurious propose and unsafe entry into search. This is an action-prior failure. P1 is therefore not rewarding raw willingness to generate a design; it rewards disciplined entry control. P2: over-edit / wandering. In the macro-unimorph tip-mass design-search case derived from DOI 10.3390/mi14020421 (case 0016), the trajectory makes large moves across violation families instead of preserving the feasible neighborhood. The oracle trace shows oscillation and failure to re-enter feasibility within budget; over-edit and low directed-update scores capture this behavior. This is an edit-style failure: the model can react to feedback, but the reaction is too global and can chase one constraint while breaking another. P3: continuity trap. In audit case p3::dev::0093::0000, the exposed history has already moved substrate thickness in a harmful direction. Continuity-biased runs keep treating that corrupted history as trusted state. P3 separates this into escape, cascade, dead-budget, and final-success fields: merely noticing the trap is not enough if the model then cascades or fails to return to a feasible design. This is why the state-summary intervention is diagnostic; it tests whether verifier-authored state isolation reduces raw-history continuity bias. P4: policy mismatch. In audit case vehbench::test_id::p4::0032, all candidates are physically feasible, but the model ranks a local trade-off neighborhood as B > C > E > A > D while the oracle policy ranks C > E > D > A > B. The resulting full Kendall tau is −1.0 and the dominance-violation rate is 1.0. The error is therefore not search failure. It is preference-execution failure: the model does not follow the policy-conditioned ordering over feasible designs. Together, these cases clarify why VEHBench reports stage-specific metrics instead of a single endpoint score. A P1 over-proposal can look productive but is unsafe triage; a P2 over-edit can contain valid engineering arithmetic but still destroy design-search feasibility; a P3 continuity trap can escape one violation and cascade into another; and a P4 policy mismatch can occur after the physical verifier has already accepted every candidate.
42
P
Selection–Generation Isomorphism Audit
The following tables document the frozen cleanup protocol behind the SG-gap analysis in the main text. Residual failures after one cleanup pass and at most one targeted rerun are reported as final outcomes rather than recursively cleaned away. P.1
Selection–Generation Isomorphism Audit
This audit bank tests whether the same underlying feasible design remains accessible when the response regime changes. We construct 46 probe groups from the hard/medium P2–P4 overlap and instantiate each group in three matched forms: A-selection (pick the one feasible candidate from a five-way pool), B-generation (synthesize a feasible candidate from a near-feasible seed), and C-completion (fill in two masked variables of a gold feasible design). We freeze the protocol at one cleanup pass plus at most one targeted rerun for malformed outputs; residual failures after that pass are reported as final failures rather than recursively cleaned away. Table 35: Isomorphic probe results after one cleanup pass and at most one targeted rerun. Success rates are group-level percentages over 46 probe groups. SG-gap is A-selection minus B-generation. gemini-3.1-propreview remains partially parse-limited only on B-generation; all other model/form cells are parse-clean after the frozen cleanup protocol. Model
A-sel. B-gen. C-comp. SG-gap Parse-B Lat. (s) Bad rows
claude-4.6-sonnet gpt-5.4 gemini-3.1-pro-preview qwen3.6-plus
41.3% 32.6% 65.2% 52.2%
4.3% 6.5% 4.3% 8.7%
17.4% 21.7% 30.4% 15.2%
37.0 pts 100.0% 26.1 pts 100.0% 60.9 pts 41.3% 43.5 pts 100.0%
18.1 2.4 29.8 70.0
0 0 27 0
For gemini-3.1-pro-preview, B-generation is deliberately reported on the full 46-group denominator because parse robustness is part of the generation regime. Among the 19 parse-clean B-generation rows, its feasible-generation success is 2/19 = 10.5%; the full-denominator value remains 2/46 = 4.3%. This distinction prevents the SG-gap from being misread as only a semantic generation failure or only a formatting failure. Table 36: Protocol log for the isomorphic probe reruns. Residual bad rows after the first targeted rerun are treated as part of the measured behavior, not recursively retried until clean. Model
First-pass issue
Final residual after one rerun Readout
gpt-5.4
no malformed-output cleanup needed
0 bad rows
claude-4.6-sonnet
transient transport failures in the first pass, cleaned once
0 bad rows
gemini-3.1-pro-preview
first pass mixed timeouts and malformed JSON
27 bad rows, generation
qwen3.6-plus
four timeout rows in the first pass
0 bad rows
43
all
stable baseline; all A/B/C cells remain parse-clean clean selector after one cleanup pass; low B-generation remains a capability result, not a format artifact B- residual failures are no longer timeout-dominated; they remain half-written JSON under the hardest six-variable generation form fully clean after one cleanup pass, but still by far the slowest model in the audit
Q
Tier 3 Controlled-Prompt Audit — Full Results
The main text (Section 4) reports the four primary confirmatory cells with targeted-minus-neutral paired bootstrap deltas. This appendix provides the complete per-condition results, the C4 per-model split, and per-task pattern decomposition that support the interpretation advanced in the main text.
Q.1
Design and Execution
The experiment was fully pre-registered. Four confirmatory cells pair known deficit models with their weak stages: C1 (gpt-5.4, P2), C2 (claude-4.6-sonnet, P3), C3 (gemini-3.1-pro-preview, P4), and C4 (gpt-5.4 and claude-4.6-sonnet, P1). Each cell samples 36–60 tasks under frozen seed 42 with subtypelevel stratification. Five prompt conditions are tested per cell: default (the standard benchmark prompt), neutral (generic structured execution without boundary-specific diagnosis), targeted (boundary-specific diagnostic prompt designed to address the known deficit), wrong-boundary (a mismatched diagnostic prompt from a different stage), and for C2, a state-summary-plus variant stacking targeted with default. All prompts were frozen before execution; no iterative refinement occurred. The analysis uses 10,000 paired task-level bootstrap resamples with 95% confidence intervals. Table 37: Tier 3 prompt-condition logic. The audit asks whether boundary-specific prompt controls can erase the profile deficits identified in the main benchmark. Condition
Purpose
Used where
Prediction if profiles are prompt artifacts
default neutral
standard benchmark prompt controls for extra structure and length boundary-specific control instruction mismatched control block targeted P3 plus verifierauthored summary
all cells all cells
baseline behavior small generic improvement at most
all cells
large gain over neutral
C1, C3, C4 C2
no gain or degradation extra gain if reset wording adds value beyond state representation
targeted wrong-boundary state-summary-plus
Q.2
Prompt Conditions
The default condition uses the standard P1–P4 benchmark prompt templates in Appendix N. The other conditions prepend one frozen control block before the standard task prompt. The neutral block controls for extra structure and length without naming a boundary-specific failure mode: Neutral Control Prompt SYSTEM CONTROL: Structured task execution. Break down the task into steps. For each step: - State what you are doing. - Execute the step. - Verify the result before proceeding. Complete all steps before outputting the final answer.
The targeted prompts are boundary-specific. P1 targets entry discipline: Targeted P1 Control Prompt SYSTEM CONTROL: Triage protocol. 1. List every missing parameter needed for a valid design decision. 2. Check whether any hard constraints conflict. 3. Determine whether a feasible design exists within stated bounds. Only output ‘propose_design‘ if all three conditions pass. Output ‘request_missing_info‘ if condition 1 fails. Output ‘declare_infeasible‘ if condition 2 or 3 fails.
P2 targets bounded verifier-guided design search rather than broad regeneration: 44
Targeted P2 Control Prompt SYSTEM CONTROL: Verifier-guided design search. - Edit exactly ONE design variable per iteration. - Preserve constraints that are currently satisfied unless the verifier feedback implies a necessary trade-off. - Target the constraint with the largest violation first. - Report which constraint you targeted and whether it improved.
P3 targets skeptical reset under corrupted history: Targeted P3 Control Prompt SYSTEM CONTROL: Corrupted trajectory recovery. The history below may contain steps that moved the design in a harmful direction. Before acting: 1. Identify the direction the design moved over the last 3 steps. 2. Locate the best feasible design in the trajectory (if any). 3. Explicitly state which portion of the history you are discarding. Proceed from the best feasible state, not from the last step.
P4 targets evaluator-role discipline: Targeted P4 Control Prompt SYSTEM CONTROL: Policy-conditioned evaluation. You are an evaluator. Do NOT propose or modify designs. All candidates are fixed and must be treated as given. 1. Extract from the policy the primary objective and any secondary objectives, with their priority ordering. 2. Compare candidates only on policy-relevant quantities. 3. Rank candidates according to the policy’s priority order. Ignore personal design preferences not stated in the policy.
The wrong-boundary condition intentionally applies an incompatible control block: P1 and P2 receive the P4 evaluator prompt, while P4 receives the P2 bounded-edit prompt. This tests whether improvement comes from generic instruction length or from matching the control block to the trusted-state boundary. The C2 state-summary-plus condition stacks the P3 targeted prompt with a verifier-authored state summary, allowing us to check whether explicit reset instructions add value beyond the state representation itself. Q.3
Full Per-Condition Results
45
Table 38: Tier 3 Phase A complete per-condition results. All metrics are primary confirmatory endpoints per cell. The C1 targeted secondary objective is retained with an audit flag because it is a scale outlier; the confirmatory endpoint for C1 is feasible rate. Cell
Model
Condition
n
Primary metric
Secondary
Parse
C1
gpt-5.4
default neutral targeted wrong
40 40 40 40
0.150 feasible 0.175 feasible 0.150 feasible 0.075 feasible
61.0 mean obj 4.3 mean obj 461.8† mean obj 2.1 mean obj
0.000 0.000 0.000 0.000
C2
claude-4.6-sonnet
default neutral targeted wrong s+
36 36 36 36 36
0.167 success 0.389 success 0.194 success 0.222 success 0.222 success
0.694 cascade 0.528 cascade 0.778 cascade 0.694 cascade 0.750 cascade
0.000 0.000 0.000 0.000 0.000
C3
gemini-3.1-pro-preview
default neutral targeted wrong
50 50 50 50
0.692 τ 0.692 τ 0.708 τ 0.700 τ
0.320 exact 0.320 exact 0.320 exact 0.300 exact
0.000 0.000 0.000 0.000
C4
claude-4.6-sonnet
default neutral targeted wrong
60 60 60 60
0.650 acc 0.617 acc 0.567 acc 0.633 acc
— — — —
0.050 0.050 0.183 0.050
gpt-5.4
default neutral targeted wrong
60 60 60 60
0.433 acc 0.400 acc 0.467 acc 0.400 acc
— — — —
0.000 0.000 0.000 0.000
Table 38 reports the complete per-condition primary metric. The C1 targeted mean-objective value marked with † is a secondary-display scale outlier relative to the rest of the C1 rows; we leave it visible for auditability but do not use it for the Tier 3 conclusion. Four patterns merit attention beyond the main-text targeted-minus-neutral deltas: 1. C2 neutral improvement. The generic structured neutral prompt lifts claude-4.6-sonnet P3 success from 0.167 to 0.389 (95% CI for neutral−default: [0.083, 0.361]), while reducing cascade from 0.694 to 0.528. This is the largest single-condition effect in the experiment and suggests that claude-4.6-sonnet benefits from structured execution guidance but not from skeptical-reset framing. 2. C2 cascade–success inversion. Cascade rate under targeted (0.778) is the highest across all C2 conditions, and targeted is the only condition where cascade exceeds the default rate. Pertask analysis identifies four tasks with cascade exclusively under targeted, consistent with the interpretation that the skeptical-reset prompt induces overcorrection. 3. C2 state-summary-plus does not rescue targeted. The s+ condition raises success only from 0.194 to 0.222 relative to targeted and leaves cascade high at 0.750. This is informative because it shows that simply stacking state-summary representation onto the same targeted wording does not recover the neutral-prompt gain. 4. C4 parse-error interaction. claude-4.6-sonnet’s parse-error rate under targeted P1 (0.183) is more than triple the rate under other conditions (0.050). Parse errors concentrate in infeasible_margin tasks (4/7) and declare_infeasible gold labels (5/14). Excluding parse errors, claude-4.6-sonnet’s classification accuracy under targeted is 0.694, exceeding neutral (0.617), indicating that the targeted prompt content aids reasoning while its length impairs output-format compliance. Q.4
Per-Task Decomposition
C2 per-task transitions. Of 36 tasks, 20 fail under all three main conditions; 7 succeed under neutral but fail under default and targeted; 4 succeed under all conditions. The neutral→targeted transition loses 9 successes and gains 2. Cascade under targeted increases in 10 tasks relative to neutral and decreases in only 1. C3 per-task decomposition. The +0.016 mean τ gain is driven entirely by one task whose τ moves from −0.4 (negatively correlated ranking) under neutral to +0.4 under targeted; the remaining 49 tasks 46
show identical τ values across both conditions. Top-1 accuracy improves on 2 tasks (94%→100%). The targeted prompt thus eliminates a single ranking failure rather than systematically improving order quality. C1 subtype pattern. gpt-5.4 never solves boundary_binding tasks under any condition (0/40). The paper_like subtype accounts for most feasible cases across all conditions (19/22 total feasible designs). This subtype specificity, together with the uniformly low feasible rates (0.075–0.175), indicates a capability floor that prompt variation does not shift. Q.5
Interpretive Summary
The four-cell audit supports the following interpretation, which we advance in the main text: behavioral fingerprints identified by Tier 1 profiles are not shallow prompt artifacts. If they were, targeted boundary-specific prompts should erase them. Instead, C1 shows no effect, C2 shows significant reversal, C3 shows a one-task marginal effect, and C4 shows model-dependent interaction. The resilience of stage dissociation to prompt-level control reinforces the benchmark’s construct validity: what VEHBench measures is durable prior-boundary compatibility, not prompt sensitivity.
47
R
Circuit Audit: Cross-Domain Construct Validity
This appendix reports the final circuit audit described in Section 4. The audit is a construct-validity check, not an independent circuit benchmark: it asks whether the same P1–P4 diagnostic regimes remain discriminative under a second closed-form engineering domain. The released task bank hardens P1 with near-boundary infeasible and missing-information cases, hardens P2 with dualconstraint design-search tasks where fixing one violation can break another, and uses matched P3/P4 recovery and ranking banks to test corrupted-state recovery and policy-conditioned selection. P1/P2 use the hardened final audit bank, while P3/P4 use the retained original audit bank to preserve the paper-facing snapshot. Table 39: Circuit audit task inventory. The task bank changes the physics while preserving the P1–P4 decision boundaries. Probe
n
Source
P1
32 final circuit audit
P2
32 final circuit audit
P3
18 final circuit audit
P4
24 final circuit audit
Hardening
Validation target
Near-boundary infeasible cases; lower raw propose_design prior; subtype-balanced entry decisions Dual constraints; update directions can conflict; objective-preserving feasible design search is rewarded Progressive dual traps; escape can trigger a second violation Policy-flip ranking tasks under feasible candidate pools
Action-discipline transfer
Edit style and feedback conditioning
State-reset behavior and trap sensitivity Policy execution
Why P1-Composite is the circuit P1 headline. The circuit P1 bank follows the VEH P1 hardening principle: raw accuracy is too easy to inflate when the action distribution contains many feasible proposals. The bank therefore lowers the trivial propose_design prior and adds near-boundary infeasible/request cases. P1-Composite is used as the headline because it jointly rewards correct entry actions and penalizes spurious proposals, unsafe proposals, missed missing-information cases, and missed infeasibility cases. This makes P1 an entry-discipline metric rather than a willingness-togenerate metric. Table 40: Circuit P1 action triage. P1-Comp. is the headline; abbreviations: Spur.=spurious propose, Unsafe=unsafe propose, Req.=request recall, Inf.=infeasible recall. Model
n
P1-Comp.
Acc.
F1
Spur.
Unsafe
Req.
Inf.
Parse
qwen3-max hunyuan-hy3-preview deepseek-r1 gemini-3.1-pro-preview gpt-5.4 claude-4.6-sonnet
32 32 32 32 32 32
0.971 0.944 0.974 0.974 0.923 0.932
0.969 0.938 0.969 0.969 0.906 0.938
0.965 0.933 0.971 0.971 0.917 0.940
0.000 0.000 0.000 0.000 0.000 0.031
0.000 0.000 0.000 0.000 0.000 0.031
1.000 1.000 1.000 1.000 1.000 1.000
1.000 1.000 1.000 1.000 1.000 0.875
0.000 0.000 0.000 0.000 0.000 0.000
Table 41: Circuit P2 verifier-guided design search. P2b is the final feasible objective score; abbreviations: Feas.=final feasible rate, Dir.=directed update, O-edit=over-edit. Model
n
Feas.
P2b
Dir.
O-edit
Parse
qwen3-max hunyuan-hy3-preview deepseek-r1 gemini-3.1-pro-preview gpt-5.4 claude-4.6-sonnet
32 32 32 32 32 32
0.906 0.938 0.969 1.000 0.875 0.906
0.494 0.512 0.594 0.662 0.457 0.487
0.938 0.914 1.000 0.969 0.883 0.914
0.400 0.771 0.605 0.938 0.489 0.646
0.000 0.000 0.000 0.000 0.000 0.000
48
Table 42: Circuit P3 corrupted-state recovery retained from the original bank. Abbreviations: Esc.=escape, Replan=explicit replan, Reset=history reset, Casc.=cascade, Dead=dead budget, Succ.=final success. Model
n
Esc.
Replan
Reset
Casc.
Dead
Succ.
Rec. q.
Parse
qwen3-max gemini-3.1-pro-preview gpt-5.4 deepseek-r1 hunyuan-hy3-preview claude-4.6-sonnet
18 18 18 18 18 18
0.944 1.000 0.944 0.778 0.889 1.000
0.000 0.000 0.111 0.000 0.000 0.000
0.000 0.000 0.111 0.000 0.000 0.000
0.471 0.000 0.176 0.000 0.062 0.000
0.000 0.000 0.056 0.000 0.000 0.000
0.778 1.000 0.778 0.722 0.778 1.000
0.366 0.611 0.401 0.314 0.364 0.568
0.000 0.000 0.000 0.000 0.000 0.000
Table 43: Circuit P4 policy-conditioned ranking. Abbreviations: Top2=set agreement for the top two candidates; Flip=policy-flip accuracy. Model
n
Tau
Exact
Top1
Top2
Pair
Flip
BARS
Parse
qwen3-max gemini-3.1-pro-preview gpt-5.4 deepseek-r1 hunyuan-hy3-preview claude-4.6-sonnet
24 24 24 24 24 24
0.608 0.475 0.625 0.542 0.492 0.608
0.375 0.250 0.375 0.292 0.208 0.375
0.583 0.500 0.583 0.583 0.458 0.542
0.583 0.500 0.625 0.500 0.417 0.500
0.804 0.738 0.812 0.771 0.746 0.804
0.769 0.679 0.801 0.716 0.647 0.759
0.710 0.625 0.722 0.661 0.614 0.707
0.000 0.000 0.000 0.000 0.000 0.000
Table 44: Circuit profile scores. Action/edit scores use the updated P1/P2 audit; state/policy scores are retained from the original P3/P4 banks. Model
Action
Edit
State
Policy
qwen3-max hunyuan-hy3-preview deepseek-r1 gemini-3.1-pro-preview gpt-5.4 claude-4.6-sonnet
0.983 0.968 0.985 0.985 0.956 0.962
0.877 0.792 0.579 0.766 0.830 0.815
0.650 0.721 0.700 0.800 0.720 0.800
0.706 0.612 0.672 0.633 0.714 0.696
Table 45 summarizes the signature patterns that reproduce across VEH and circuit domains. Table 45: Cross-domain pattern reproduction. The circuit audit checks whether the same diagnostic reversals appear under different physics. Pattern
VEH observation
Circuit observation
gpt-5.4 P4 lead / weaker search P4 Tau 0.887 (1st); P2 ratio 0.133 gemini-3.1-pro-preview P2 lead P2 ratio 0.390 (1st); P4 Tau 0.824 (7/12) / weak P4 No single model dominates qwen3-max P1; gemini-3.1-pro-preview P2; hunyuan-hy3-preview P3; gpt-5.4 P4
P3 trap-geometry sensitivity
Multi-step contamination penalizes continuity-biased models (gemini-3.1-pro-preview cascade 0.507 in VEH)
Profile diagonal direction
Action discipline tracks P1; edit style tracks P2; policy execution tracks P4
retained P4 Tau 0.625 (1st); circuit P2b 0.457 (6/6) circuit P2b 0.662 (1st); retained P4 Tau 0.475 (6/6) gemini-3.1-pro-preview/deepseek-r1 P1; gemini-3.1-pro-preview P2; gemini-3.1-pro-preview/claude-4.6-sonnet retained P3; gpt-5.4 retained P4 Formula-level circuit traps give gemini-3.1-pro-preview zero cascade, while qwen3-max cascades at 0.471; this is a trap-geometry contrast, not a claim that the same model-level cascade mechanism transfers unchanged P1-Composite, P2b, and P3 success remain separable under the circuit oracle
What the circuit audit proves and does not prove. The audit supports a narrow construct-validity claim: when the physics changes from VEH to closed-form circuit design, the P1–P4 scaffold still produces separable action, design-search, recovery, and ranking behaviors. It also shows that responsecontrol reversals such as gemini-3.1-pro-preview’s search/ranking split and gpt-5.4’s ranking/search split are not artifacts of VEH-specific formulas. It does not establish stable circuit-domain model rankings, cover modern circuit design broadly, or replace a full circuit benchmark; the task count and family coverage are intentionally pilot-scale.
49
NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction (§1) state that VEHBench evaluates LLMassisted vibration energy harvester co-design through four verifier-grounded design roles (P1–P4), that frontier models dissociate across these roles, that response-control profiles provide a diagnostic account of the dissociation, and that stage-local results support stageaware model selection and control. Each claim is supported by the experimental evidence in §4, with robustness and scope checks reported in the appendix and limitations discussed in §5. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Section 5 states the main scope boundary: VEHBench is scoped to analytically verifiable cantilever VEH co-design and does not replace FEM or hardware certification. The appendix details the construct-validity circuit pilot, P1 matched certification split, P3 intervention subset, SG-gap audit, prompt-control audit, profile definitions, uncertainty, split coverage, and closed-API snapshot drift (Appendix R, Appendix P, Appendix Q, Appendix F). 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [N/A] Justification: This paper does not include theoretical results. It is a benchmark and empirical evaluation paper. The VEH oracle uses closed-form engineering equations documented in Appendix B; the circuit construct-validity audit is documented in Appendix R. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper? Answer: [Yes] Justification: Section 3 describes the benchmark construction pipeline (209-paper extraction audit, 52 cleaned anchors, source-anchor separated optimizer-facing splits, P1 matched certification-style triage, manifest-backed provenance). Section 4.1 describes the experimental setup (12 complete model runs, headline metrics, shared oracle). The Appendix provides full metric definitions, P1–P4 prompt templates, oracle specifications, run configuration, and the CMA-ES baseline configuration. All task banks, evaluator code, and per-model JSONL logs are released with the artifact. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The release artifact is available at huggingface.co/datasets/AnonymousVehbench/vehbench. The artifact includes P1–P4 JSONL task banks, manifests, oracle and evaluator code, generation scripts, split reports, prompt templates, per-model JSONL logs, licenses, and Croissant RAI metadata. The dataset is released under CC-BY-4.0; code under MIT license. Build and evaluation commands are provided in the Appendix and artifact README. 50
6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: This paper evaluates pre-trained frontier LLMs as engineering agents; no training is performed. Evaluation details are specified in §4.1 and the Appendix: temperature 0.0, max output tokens provider-default (4096–8192), up to 2 retries on parse failure, P2 stops on feasible closure, P3 uses full budget. Task splits (dev/test_id/test_ood) are described in §3. Model provider endpoints and run dates are recorded in per-model manifests. All task generation uses fixed random seeds. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Table 2 reports the main headline scores, while Appendix Table 12 reports 95% confidence intervals and the key paired deltas for the best point estimates. Appendix Table 14 reports paired task-bootstrap 95% confidence intervals for the P3 state-summary intervention. Spearman profile-stage and stage-rank correlations are reported as descriptive diagnostics. The CI methodology (non-parametric bootstrap, Wilson intervals for binary proportions, and paired resampling for deltas) is described in the Appendix. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: All experiments use API-based inference with no local GPU compute. Model runs use provider APIs (temperature 0.0, max output tokens 4096–8192). The analytical oracle (VEH) runs at 0.5 ms per evaluation on CPU; the circuit oracle is faster. CMA-ES baseline on the full P2 bank (208 tasks × 40 oracle calls) completes in under 1 second on a standard laptop. Total experimental API cost is approximately USD 300–500 across all models and probes. No training runs were performed. 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics? Answer: [Yes] Justification: The research conforms to the NeurIPS Code of Ethics. No human subjects were involved. The benchmark uses publicly available engineering design principles and published literature data; no private or sensitive data was collected. The LLMs evaluated are publicly accessible models. The artifact is released under open licenses (CC-BY-4.0 for data, MIT for code) with documented intended use as an engineering-agent diagnostic benchmark, explicitly noting that it is not intended for production safety certification. 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Section 3 describes the release boundary and states that VEHBench is intended as a diagnostic tool for engineering-agent evaluation, not for production safety certification or as a standalone engineering solver. The Discussion (§5) provides deployment guidance for verifier-gated and stage-routed systems. Potential positive impact includes more auditable engineering-agent evaluation; the main negative risk is over-trusting benchmark scores as evidence of safe real-world hardware design. This is mitigated by intended-use documentation, analytical-oracle scope limits, and release metadata. 51
11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse? Answer: [N/A] Justification: This paper releases a benchmark dataset and evaluation code, not a deployable model. The dataset consists of closed-form engineering design tasks (VEH and circuit domains) with analytical oracles. It poses no risk of misuse for disinformation, surveillance, or generation of harmful content. The artifact documentation includes an intended use statement explicitly scoping the benchmark to research on engineering-agent evaluation. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Prior engineering benchmarks and task families (EngDesign, BuildArena, mechanical-design agents, DesignQA, and MSEval) are cited in §2, with expanded related work in the Appendix. The VEH domain is supported by modern VEH/IoT harvesting reviews, Erturk–Inman analytical VEH references, and self-powered piezoelectric nanogenerator work. Evaluated LLM models are cited with provider documentation references where available. The CMA-ES baseline is cited through the CMA-ES reference. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: VEHBench is a new benchmark. Documentation includes task schemas (Appendix N), metric definitions with formulas (Appendix F), construction pipeline and split policy (§3), P1–P4 prompt templates (Appendix N), oracle specifications with equations (Appendix B), Croissant RAI metadata, release manifests, JSONL task banks, evaluator/oracle code, model logs, and manifest-backed provenance. The artifact is released under CC-BY-4.0 (data) and MIT (code). 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: This paper does not involve crowdsourcing or human subjects research. All evaluations are automated via LLM API calls to pre-trained models. The benchmark uses an analytical physics oracle rather than human annotation for ground truth. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: This paper does not involve human subjects research. All data is either derived from published engineering literature or synthetically generated via deterministic oracles. No human participants were recruited, surveyed, or studied. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? 52
Answer: [Yes] Justification: LLMs are the primary subjects of evaluation in this benchmark paper. All 12 evaluated frontier models are LLMs used as engineering design agents. Their usage is fully described in §4.1 (experimental setup) and Appendix N (prompt templates, run configuration). LLM assistants were also used during manuscript preparation for code/debugging assistance, figure integration, formatting migration, and language editing; all scientific claims, reported numbers, and final text were reviewed by the authors.
53