B UILD - AND -F IND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
arXiv:2605.06136v1 [cs.SE] 7 May 2026
Jhen-Ke Lin National Yang Ming Chiao Tung University [email protected]
Abstract Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agentmanaged: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce B UILD - AND -F IND, a protocol for evaluating whether downstream agents can recover those intended choices from generated repositories, and how much inspection that recovery requires. For each task, a builder sees a hidden repository specification and creates a codebase; a finder sees only the codebase and a specification-traced multiple-choice question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Accuracy and stability act as gates: effort is interpreted only when recovery succeeds reliably. Among artifacts from which the same intent can be recovered, lower effort by the same finder suggests that the artifact makes that intent easier to locate. Questiononly and spec-only controls quantify generic priors and specification access, while audits separate omitted claims from finder failures and check whether correct answers cite artifact evidence. In the released high-prior task pack, recovery accuracy is near saturation, so inspection effort and finder-specific effects provide the main panel-local comparison. We release the harness, tasks, generated artifacts, records, canonical tables, reports, scripts, metadata, licenses, and evidence audit to support auditable extensions and private task packs.
1
Introduction
Passing tests establishes behavioral evidence without guaranteeing that code is easy to understand. Empirical work on ChatGPT-generated programs already treats correctness and code quality as separable concerns: beyond checking behavioral outcomes, Liu et al. analyze style and maintainability issues in generated Java and Python programs [Liu et al., 2024b]. As coding agents move from function synthesis to repository-level work, generated codebases increasingly become intermediate artifacts: one agent writes them, another inspects them, and later agents audit, modify, or extend them. A repository that hides design intent can impose extra inspection burden even when visible tests pass; one that exposes intent can make the same design easier to recover. This mirrors a familiar software-engineering principle: maintainable code combines correctness with structure that helps future readers understand why it is written that way. Readability and structure are established software-quality concerns [Buse and Weimer, 2010, Börstler et al., 2023], and Parnas’s modularization criteria frame decomposition as a tool for system comprehensibility [Parnas, 1972]. In agent-managed repositories, the future reader may be another agent. The corresponding question is Preprint.
whether the repository exposes intended behavior and design choices clearly enough for later agents to inspect, audit, and use it as context. We separate three layers of agent-written software evaluation: behavioral correctness, artifact-side intent recoverability, and downstream modification success. Existing coding benchmarks primarily target behavioral correctness, including unit-test passing in HumanEval and MBPP [Chen et al., 2021, Austin et al., 2021], harder programming tasks in APPS and DS-1000 [Hendrycks et al., 2021, Lai et al., 2023], and issue-resolution patches in SWE-bench [Jimenez et al., 2024]. B UILD - AND -F IND targets the middle layer: before asking whether a later agent can safely modify a repository, we ask whether it can recover the intended behaviors and design choices from the repository itself. The target construct is artifact-evidenced recovery. A correct answer should be distinguishable from alternatives using evidence in generated source files, tests, configuration, or documentation. This gives artifact-side evaluation a sharp boundary. Functional correctness asks whether the repository behaves correctly; downstream modification success asks whether a later agent can safely change it; artifact-evidenced recovery asks whether intended decisions are visible in the artifact before any modification is attempted. In high-recovery regimes, accuracy alone is a coarse measure of artifact legibility. If two generated repositories both allow downstream agents to recover the same intended decisions, the amount of inspection required for that recovery becomes informative. An artifact that exposes intent through clear structure, tests, configuration, or documentation should require less finder inspection than one that buries the same intent. B UILD - AND -F IND therefore treats inspection effort as an agent-facing proxy for artifact legibility, interpreted only after recovery and stability gates. The central idea is to treat generated code as external memory and as a communication medium. A hidden specification defines intended behaviors and design choices; a builder encodes them in a codebase; and a finder, seeing only the codebase and a question bank, tries to recover them. Accurate recovery with low inspection effort, after stability checks, indicates that the repository made the intended choices easier for downstream agents to find. The builder-finder split separates implementation ability from artifact recovery while exposing interactions between artifacts and inspection policies. This paper makes four contributions. 1. Evaluation target. We define artifact-evidenced recovery of intended repository behavior and design choices as a distinct layer between behavioral correctness and downstream modification success. 2. Builder–finder protocol. Builders see hidden specifications and produce repositories; finders see only those repositories plus specification-traced question banks. 3. Measurement contract. We report recovery accuracy, stability, implementation coverage, and inspection effort; effort is interpreted as a proxy for artifact legibility only after recovery and stability gates. 4. Calibration and release. We include question-only, spec-only, compile-failed, low-prior, evidence-audit, and builder–finder affinity diagnostics, and release the harness, artifacts, records, canonical tables, reports, scripts, metadata, licenses, and evidence audits needed for audit. Empirical scope. B UILD - AND -F IND treats controls as part of the evaluation contract: they separate ordinary engineering priors, specification access, and artifact-conditioned recovery. Section 4.2 reports the calibration numbers; the empirical claims in this release are panel-local and priorconditioned.
2
Related work
Functional code generation. HumanEval [Chen et al., 2021] and MBPP [Austin et al., 2021] established compact unit-test-based evaluations for code models. APPS [Hendrycks et al., 2021], DS-1000 [Lai et al., 2023], and LiveCodeBench [Jain et al., 2025] broaden the task distribution and contamination controls. These benchmarks answer whether generated code satisfies behavioral tests. Our protocol asks a different question: after a repository is generated, does it expose the intended decisions clearly enough for a downstream agent to recover them? 2
Code understanding and code QA. Code-understanding and code-QA benchmarks evaluate adjacent capabilities. CodeXGLUE [Lu et al., 2021] combines program-understanding and generation tasks, InfiBench [Li et al., 2024] evaluates code-related QA, and repository-level QA benchmarks such as SpyderCodeQA, CodeRepoQA, and SWE-QA test semantic, dependency, intention, and multi-hop repository understanding [Strich et al., 2024, Hu et al., 2025, Peng et al., 2025]. These tasks usually start from an existing repository. B UILD - AND -F IND first asks a builder to create a repository from a hidden specification, then asks what information that generated artifact carries for a finder that never sees the specification. Repository-level software engineering. SWE-bench [Jimenez et al., 2024] and SWE-bench Verified [OpenAI, 2024] evaluate model patches on real GitHub issues. RepoBench [Liu et al., 2024a] targets repository-level retrieval and completion. These settings capture realistic software context through patch correctness or completion endpoints. Our endpoint is different: the produced repository itself becomes the object that future agents must inspect. Program comprehension and maintainability. Program-comprehension and maintainability work studies how readers ask questions about code and reason about behavior [Sillito et al., 2006, Ko and Myers, 2008, 2010, Buse and Weimer, 2010, Oliveira et al., 2020]. B UILD - AND -F IND uses this perspective to define an intermediate evaluation target. Before measuring whether a later agent can safely modify a generated repository, we measure whether it can recover the decisions that the repository was meant to express. Interactive and long-horizon agent benchmarks. Interactive and long-horizon benchmarks such as τ -bench, MLE-bench, PaperBench, and Terminal-Bench evaluate agents that act through tools, APIs, terminals, research workflows, and sandboxed environments [Yao et al., 2025, Chan et al., 2025, Starace et al., 2025, Merrill et al., 2026]. These benchmarks share our focus on agentic, tool-mediated work. Their primary outcome is task completion in an environment; our benchmark asks whether the artifact produced by one agent makes intended decisions recoverable for another agent, and how much inspection that recovery requires.
3
Protocol design
3.1
Builder-finder protocol
Each task t contains a hidden specification with intended behaviors and design choices. It also contains a question bank whose gold answers are traced to specific items in that specification. These traces make the recovery target auditable, following the broader role of traceability in softwareintensive systems [Cleland-Huang et al., 2014]. The question-bank form is also motivated by program-comprehension work showing that developers ask concrete questions during evolution tasks [Sillito et al., 2006], including why and why-not questions about program behavior [Ko and Myers, 2008, 2010]. A builder b receives the task prompt and constructs a self-contained codebase Ab,t,r on trial r. Build validation records compile checks and structural metadata. A finder f then receives only the artifact and the question bank; the hidden specification remains unavailable. Functional tests ask whether the code behaves as intended. B UILD - AND -F IND asks whether the intended behavior and design choices are recoverable from the artifact. Appendix A states the build, find, and orchestration steps as pseudocode. The finder inspects files, makes tool calls, and writes a structured answer file. Each question is a four-option multiple-choice item with deterministic option shuffling per run. Exact-match grading avoids free-form answer parsing and keeps partial runs auditable. The intended construct is artifact-evidenced recovery: a correct answer should be distinguishable from plausible alternatives by evidence in source files, tests, configuration, or documentation. Exact-match grading keeps the formal score deterministic. Specification traces, prior controls, compile-failed stress diagnostics, and the post-hoc artifact-evidence audit in Appendix I test whether that deterministic score matches the intended construct. The audit counts a correct answer as evidence-supported only when the cited file, symbol, or snippet-level rationale distinguishes the selected option from plausible alternatives. 3
We refer to these artifact-conditioned runs as the formal condition; some tables abbreviate them as formal runs or rows. 3.2
Finders as measurement instruments
Finders play two roles. In the benchmark, they are calibrated measurement instruments. In the target deployment setting, they approximate downstream agents that must use generated repositories as external memory. The measured result therefore depends on both the artifact and the finder: a weak finder can make good artifacts look opaque, while a very strong finder can mask artifact differences. This mirrors a broader evaluation problem: assessing LLM-generated code and text is difficult, and LLM-as-a-judge methods require calibration for reliable use [Wang et al., 2025]. We therefore analyze finder validity before comparing artifacts. A useful finder panel has high artifactconditioned recovery, passes control checks, retains performance on low-prior or implementationsignal questions, and spans more than one model family. We also report builder–finder affinity: pair-specific recovery or effort effects after accounting for the builder’s average artifact legibility and the finder’s average inspection ability. Affinity is panel-local and diagnostic. Appendix E reports the per-finder calibration. 3.3
Prior and specification controls
Software tasks contain legitimate engineering priors. A finder may infer that a database should use write-ahead logging or that a web server should separate routing from middleware without reading a particular artifact. The benchmark therefore quantifies prior answerability with controls while preserving natural requirements. In the question-only control, a finder receives only the question bank. In the spec-only control, it receives the same specification given to builders. Thus the specification is hidden from artifactconditioned finders and deliberately exposed in this control. These controls give artifact-conditioned accuracy its prior and specification context. In this task pack, question-only exact-match accuracy is high, so accuracy serves primarily as a recovery gate and calibration signal; conditional inspection effort carries the main artifact-legibility comparison. The effort analysis then asks a narrower question: among successful recoveries on compile-passing artifacts, how much inspection does the same finder spend to recover the same intended decisions? Compile-failed artifacts are reported as stress diagnostics. 3.4
Recovery and inspection-effort metrics
We interpret inspection effort as a proxy for agent-facing artifact legibility. The proxy is meaningful only when recovery succeeds reliably: among artifacts encoding the same intended design, lower effort suggests that the design is easier for the finder to locate. Consequently, inspection effort is never interpreted without recovery accuracy, repeatability, and coverage. The scoring set is restricted to claims that the artifact actually implements as the gold specification behavior. Let Qt be the question set for task t. For an artifact Ab,t,r , let Q+ b,t,r ⊆ Qt be the subset of questions whose artifact-question audit label is implemented-gold. Finder recovery metrics use this audited scoring set. Non-gold and ambiguous pairs enter builder-side implementation coverage reports and are excluded from finder recovery scoring. Per-run answer accuracy is X 1 1{ŷb,f,t,r,q = yt,q }. (1) Acc(b, f, t, r) = + |Qb,t,r | + q∈Qb,t,r
Here ŷ is the finder’s answer and y is the gold option. A run is all-correct when all questions in Q+ b,t,r are answered correctly. The observed inspection effort for a run is kept unchanged: the finder spent the full recorded inspection bytes, while accuracy and all-correct status are computed over the audited scoring set. Accuracy is a validity check and context signal: it says whether recovery happened, while artifact legibility is evaluated through conditional inspection effort. The reported effort unit is novel agent inspection bytes: new artifact bytes retrieved or requested by the finder through retrieval and tool calls, excluding system prompts, tool schemas, transport overhead, and replayed context. Bytes provide a tokenizer-independent unit; tokenization is model-specific, 4
so the same retrieved source span can yield different token counts for different finders. Because raw bytes are finder-specific, builder artifacts are compared only within the same finder-task context. Conditional Rb is the geometric mean, over contributing all-correct finder-task cells, of the builder’s mean all-correct effort divided by the minimum contributing builder effort in that same finder-task cell. Missing all-correct cells are excluded from Rb and reported as coverage failures. We also report a task-level recovery gate. For each builder–finder–task cell, the gate uses simple mean audited answer recovery over repeated find runs. The number of contributing cells is reported as a separate coverage quantity. This keeps the recovery gate separate from inspection effort; it asks whether recovered answers are reliable where the artifact exposes implemented-gold claims. Appendix F gives the full byte accounting and metric equations. The three reported views answer different methodological questions. Exact-match accuracy asks whether recovery occurred at all. Conditional Rb asks, among successful recoveries, how much inspection effort the same finder needed for each artifact. The task-level recovery gate asks whether the cell is recoverable enough for effort to be interpreted. Reporting all three avoids treating a low-effort but unreliable artifact as equivalent to a consistently recoverable one.
4
Empirical validation
We report controls before effort so artifact legibility is interpreted only after prior, coverage, and recovery gates. The empirical section therefore starts with prior and stress calibration, then reports repeatability, implementation-aware recovery, conditional effort, rank stability, affinity, and low-prior evidence audits. 4.1
Study setup
The reported study uses two executable repository task families, scratch_minidb and scratch_nanoweb, with 15 questions per task. We evaluate Claude Opus 4.7 and Sonnet 4.6, GPT-5.5 and GPT-5.4-mini, MiMo-v2.5-pro, and MiMo-v2.5, each at high and low reasoning effort. Each builder-task cell has two build trials; each status-ok artifact whose release path exists is inspected by each finder for three trials. The artifact-present release contains 48 builds, 1728 artifact-conditioned find records, and 25920 raw question rows. The compile-pass primary panel contains 41 builds, 1476 find records, and 22140 raw question rows. Seven artifact-present builds failed the compile probe and are retained only as stress-test diagnostics. After the compile and artifact-presence checks, we audited each artifact-question pair to verify whether the generated artifact implements the gold specification claim being scored. The audit uses a twostage process. First, an automated conformance-triage script compares stable answers from selected base finders, codex_gpt5_5 and claude_code_opus_4_7, over repeated artifact-conditioned runs; both selected base finders are also members of the reported finder panel. Artifact-question pairs that meet the stable selected-finder agreement policy receive a consensus prelabel. Second, rows in the manual-review queue are inspected and merged as manual-review labels. The final release records this provenance in label_source: 696 of 720 artifact-question rows are consensus_prelabel, and 24 are manual_review. The audit labels pairs as implemented-gold, implemented-non-gold, or absent/ambiguous. Finder recovery metrics are computed on implemented-gold pairs; non-gold and ambiguous pairs are reported separately as builder-side implementation coverage. The audit constructs the scoring set to avoid penalizing finders for gold claims that a builder artifact did not implement. In the compile-pass primary panel, this gives 592 audited scoring pairs and 21312 scored finder-answer rows. This audit changes the recovery question but does not hide builder failures. Across artifact-present builds, builder-side implementation coverage ranges from 85.0% to 100.0%, with four builders at 100% coverage. Artifact quality is therefore interpreted through implementation coverage, recovery, stability, and effort jointly. Task specifications and question banks were schema-validated and manually reviewed, and the 48 generated artifacts were audited for direct answer leakage; we found no such patterns. Appendix G 5
Table 1: Prior, artifact, and stress calibration. Artifact-conditioned rows use the audited scoring set; ∆ is lift over question-only. Slice
Task
Build
Find
Accuracy
Perfect
∆ vs. question-only
Question-only Spec-only Artifact-present, all Compile-pass primary Compile-failed excluded
all all all all all
– – 48 41 7
– – 1728 1476 252
94.5% 99.9% 98.9% 98.9% 98.8%
– – 86.4% 86.4% 86.1%
– +5.4 pp +4.4 pp +4.4 pp +4.2 pp
Table 2: Repeatability controls. Formal rows use the audited scoring set; agreement requires all three attempts in a matched cell to be correct. Slice Formal, all tasks Spec-only control Question-only control
Build
Reliable 3-trial Find Answer rows agreement questions
41 1476 – 72 – 72
21312 1080 1080
97.5% 99.7% 86.7%
96.7% 100.0% 66.7%
decomposes compile-probe exclusions, and Appendix H reports the audit scope and qualitative artifact patterns. 4.2
Prior and stress calibration
Table 1 calibrates priors, specification access, artifact-conditioned recovery, and compile-probe exclusions. The high question-only baseline is a central limitation of this task pack: accuracy still verifies recovery, but it is not the main discriminative signal. Effort and affinity are interpreted under this calibration and restricted to compile-passing artifacts. 4.3
Stability and repeatability
Repeated trials estimate reliability across attempts. Table 2 reports repeatability controls for the formal artifact-conditioned condition and the question-only and spec-only controls. Low-effort but unstable artifacts therefore receive weaker evidence than artifacts that are both recoverable and repeatable. 4.4
Implementation-aware recovery gates
Before interpreting effort, we combine builder-side implementation coverage with downstream recovery on implemented-gold claims. This prevents an artifact from looking strong merely because unimplemented claims were excluded from finder scoring. Implementation-aware downstream recovery ranges from 84.0% to 99.4% across builders. Because audited-row downstream recovery is uniformly high (97.3–99.5%), most variation comes from builder-side implementation coverage (85.0–100.0%), not from finder failure on implemented claims. Appendix J gives the full builder-level view. Table 3 reports the complementary task-level recovery gate. Entries are computed before inspectioneffort normalization or task-length compounding, reserving effort analysis for Figure 1. The table separates the two task families because aggregate recovery can hide task-specific gaps. Among reported builder–task entries with implemented-gold evidence, audited answer recovery is already high: every reported entry exceeds 96%, and most are near or above 98%. This near-saturation motivates the conditional inspection-effort analysis in Section 4.5. The parenthesized cell count identifies whether the value rests on the full finder panel for that task. 4.5
Conditional inspection effort
Figure 1 reports the builder effort score, Rb , on cells where a finder answered all questions correctly at least once. High- and low-effort panels are computed separately so that the denominator for each finder-task cell is defined inside a single effort regime. 6
Table 3: Task-level recovery gate. Entries report mean audited answer recovery; parentheses give contributing finder cells. Low effort
High effort Builder
minidb
GPT-5.5 GPT-5.4-mini Opus 4.7 Sonnet 4.6 MiMo 2.5 Pro MiMo 2.5
Builder
nanoweb
GPT-5.5 GPT-5.4-mini Opus 4.7 Sonnet 4.6 MiMo 2.5 Pro MiMo 2.5
97.8% (6/6) 99.8% (6/6) 98.1% (6/6) 100.0% (6/6) 99.8% (6/6) 99.4% (6/6) 98.9% (6/6) 99.8% (6/6) 99.8% (6/6) – (0/6) 96.1% (6/6) 98.0% (6/6)
minidb
nanoweb
98.7% (6/6) 97.6% (6/6) 99.4% (6/6) 99.4% (6/6) – (0/6) 98.3% (6/6)
99.6% (6/6) 99.4% (6/6) 99.3% (6/6) 98.8% (6/6) 98.9% (6/6) 97.6% (6/6)
cells
High effort
cells
Low effort
GPT-5.5
12/12
GPT-5.5
12/12
GPT-5.4-mini
12/12
GPT-5.4-mini
11/12
Opus 4.7
12/12
Opus 4.7
12/12
Sonnet 4.6
12/12
MiMo 2.5 Pro
6/12
MiMo 2.5
12/12
Sonnet 4.6
12/12
MiMo 2.5 Pro
6/12
MiMo 2.5
11/12
1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
1.0
Total-byte Rb
1.2
1.4
1.6
1.8
2.0
2.2
2.4
Total-byte Rb GPT
Claude
MiMo
Figure 1: Conditional inspection effort, Rb , on all-correct cells. Scores use the audited scoring set; missing cells are failure signals. Under this metric in the compile-pass panel, the GPT-5.5 rows are examples of full-coverage, loweffort artifact recovery: Rb = 1.033 in the high-effort panel and Rb = 1.151 for GPT-5.5-low in the low-effort panel. GPT-5.4-mini-high remains second in the high-effort panel (Rb = 1.320). In the low-effort panel, GPT-5.4-mini-low is nearly tied with GPT-5.5-low after rounding while contributing 11/12 all-correct cells. This is a panel-local artifact comparison: artifacts are compared by how much effort the same finders needed to recover intent in the same finder-task context. Appendix M reports vendor token usage separately. Appendix F also reports AELS, a compact aggregate diagnostic that combines audited recovery, repeatability, contributing-cell coverage, and weakly damped conditional inspection effort. We treat AELS as a tie-breaker summary rather than a replacement for the separated recovery, coverage, stability, and effort views. Coverage remains part of the interpretation. GPT-5.4-mini-low improves from 7/12 to 11/12 contributing low-effort cells after the audit, but still has one missing all-correct cell; the task-level recovery gate in Table 3 keeps such partial-recovery behavior visible. 4.6
Rank stability
The effort score aggregates over finders, so the interpretation depends on whether finders agree about relative artifact effort. Appendix Figure 4 reports Kendall τ across finder pairs on total-byte orderings in all-correct cells. High-effort agreement is moderate and positive on both tasks. Loweffort agreement is task-dependent: weaker on scratch_minidb, stronger on scratch_nanoweb. These diagnostics guide construction of family-diverse finder panels for follow-on evaluations. 4.7
Builder–finder affinity
If generated repositories are communication artifacts for future agents, their legibility may depend on who reads them. The affinity residuals ask whether a builder–finder pair recovers intent better 7
Table 4: Low-prior recovery and evidence-audit summary. Audit rows report supported sampled correct answers. Slice All audited scoring rows Low-prior subset
Question-only
Artifact-conditioned
Lift
94.5% 88.9%
98.9% 97.9%
+4.4 pp +9.0 pp
Audit slice
Supported / sampled
All sampled correct answers Low-prior sampled answers
65 / 72 35 / 36
or worse than expected after accounting for builder and finder marginal effects. The full matrix in Appendix Figure 5 reports pair-specific residuals. At the family level, same-family residuals are positive for OpenAI/Codex (+0.076), Claude (+0.041), and MiMo (+0.027), while most cross-family cells are near zero or negative. Appendix K reports the full 12-agent matrix and family-level table. 4.8
Low-prior and evidence-audit sensitivity
Low-prior subsets and artifact-evidence audits test whether recovery still depends on artifact evidence when generic priors are less helpful. The low-prior subset contains ten questions, five per task, selected by question-only three-trial agreement below 90%. On this subset, question-only accuracy is 88.9% and no question reaches the reliable-question threshold. Compile-pass artifact-conditioned recovery on the audited scoring set is 97.9%, a +9.0 point lift over question-only. The lift is task dependent: scratch_minidb rises from 90.0% to 96.8%, while scratch_nanoweb rises from 87.8% to 98.9%. Table 4 summarizes the prior-sensitivity slice and the post-hoc evidence audit in the main text. The post-hoc artifact-evidence audit samples 72 correct compile-pass answers and checks whether cited files, symbols, and rationales distinguish the selected option from alternatives. The audit provides supporting construct evidence: exact-match recovery in the released records usually has artifact-grounded support. In the supported or partially supported audit rows, cited evidence is dominated by source files (66 of 68 rows), with README/docs or configuration/Cargo files appearing in 7 rows each and tests absent as the primary cited channel. Restricted-view ablations such as README-only, source-only, tests-only, or config-only views require new finder runs and should be reported as a separate evidence layer. Appendix L gives the full low-prior effort, recovery, rank-stability, and affinity diagnostics, and Appendix I gives the sampling rule and labels.
5
Discussion
B UILD - AND -F IND establishes an evaluation contract for artifact-side analysis. Controls quantify prior answerability, audits report builder-side implementation coverage, and recovery and stability act as gates before inspection effort serves as an observable proxy for agent-facing artifact legibility. This separation keeps the benchmark useful in high-recovery settings: when exact-match recovery saturates, inspection effort remains an observable artifact behavior rather than a replacement for correctness. Effort matters only because the protocol first asks whether recovery is correct and repeatable. A repository that is quick to inspect but unstable is not treated as better than a slower but reliable repository. Conversely, among artifacts that expose the same intended decisions, the extra inspection required by the same finder is evidence that the artifact is harder for downstream agents to use as working context. Affinity diagnostics add a second caution: legibility can be reader-dependent, so builder rankings should be interpreted together with finder calibration and family-diverse panels.
6
Scope and limitations
The empirical claims are scoped to the released panel, two task families, finder policies, and strong priors: question-only accuracy reaches 94.5%, so accuracy is a gate while conditional effort carries the artifact-legibility comparison. The scoring audit avoids penalizing finders for unimplemented 8
gold claims, but B UILD - AND -F IND measures agent-facing recovery of specification-traced labels. It does not measure runtime correctness, security, deployment readiness, or human maintainability. The implementation audit is artifact inspection, not a full behavioral conformance test suite. Public task packs may also become contaminated, so private packs remain necessary.
7
Release, safety, and maintenance
This study uses synthetic software tasks and no human-subject data; coding-agent runs should execute in isolated workspaces with resource limits. We release the harness, task specs, question banks, artifacts, records, canonical tables, reports, scripts, claim-evidence map, Croissant-compatible metadata [Akhtar et al., 2024], licenses, and evidence audit. Stored records and deterministic transforms reproduce the reported analysis; live reruns depend on provider models, CLIs, endpoints, and drift. Prior benchmark assets are cited for context and are not redistributed. Model APIs and CLIs listed in Tables 6 and 7 were accessed as provider services under their applicable terms; no model weights or provider CLI code are redistributed.
8
Conclusion
B UILD - AND -F IND treats generated repositories as communication artifacts for future agents. The builder–finder protocol asks whether a downstream agent can recover intended behavior and design choices from the repository alone. In the released high-prior task pack, recovery accuracy is already near saturation, so inspection effort provides the main panel-local comparison after recovery and stability gates. The resulting benchmark does not replace behavioral correctness tests; it adds an artifact-side view of whether generated repositories expose intent clearly enough for later agents to use.
References Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Luca Foschini, Pieter Gijsbers, Joan Giner-Miguelez, Sujata Goswami, Nitisha Jain, Michalis Karamousadakis, Satyapriya Krishna, Michael Kuchnik, Sylvain Lesage, Quentin Lhoest, Pierre Marcenac, Manil Maskey, Peter Mattson, Luis Oala, Hamidah Oderinwale, Pierre Ruyssen, Tim Santos, Rajat Shinde, Elena Simperl, Arjun Suresh, Goeffry Thomas, Slava Tykhonov, Joaquin Vanschoren, Susheel Varma, Jos van der Velde, Steffen Vogler, Carole-Jean Wu, and Luyao Zhang. Croissant: A metadata format for ML-ready datasets. In Advances in Neural Information Processing Systems, volume 37, pages 82133–82148. Curran Associates, Inc., 2024. doi: 10.52202/079017-2610. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ 9547b09b722f2948ff3ddb5d86002bc0-Paper-Datasets_and_Benchmarks_Track.pdf. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Jürgen Börstler, Kwabena E. Bennin, Sara Hooshangi, Johan Jeuring, Hieke Keuning, Carsten Kleiner, Bonnie MacKellar, Rodrigo Duran, Harald Störrle, Daniel Toll, and Jelle van Assema. Developers talking about code quality. Empirical Software Engineering, 28(6):128, 2023. doi: 10.1007/s10664-023-10381-0. Raymond P. L. Buse and Westley Weimer. Learning a metric for code readability. IEEE Transactions on Software Engineering, 36(4):546–558, 2010. doi: 10.1109/TSE.2009.70. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLEbench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, 2025. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 9
Jane Cleland-Huang, Orlena C. Z. Gotel, Jane Huffman Hayes, Patrick Mäder, and Andrea Zisman. Software traceability: Trends and future directions. In Future of Software Engineering, FOSE 2014, pages 55–69, 2014. doi: 10.1145/2593882.2593891. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Advances in Neural Information Processing Systems, 2021. Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng, Qinyun Wu, Pengfei Gao, Xinchen Wang, and Cuiyun Gao. Understanding large language model performance in software engineering: A large-scale question answering benchmark. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2025. doi: 10.1145/3726302.3730262. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination-free evaluation of large language models for code. In International Conference on Learning Representations, 2025. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. Andrew J. Ko and Brad A. Myers. Debugging reinvented: Asking and answering why and why not questions about program behavior. In Proceedings of the 30th International Conference on Software Engineering, pages 301–310. ACM, 2008. doi: 10.1145/1368088.1368130. Andrew J. Ko and Brad A. Myers. Extracting and answering why and why not questions about Java program output. ACM Transactions on Software Engineering and Methodology, 20(2):4:1–4:36, 2010. doi: 10.1145/1824760.1824761. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machine Learning, 2023. Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. InfiBench: Evaluating the question-answering capabilities of code large language models. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024. doi: 10.52202/079017-4087. Tianyang Liu, Canwen Xu, and Julian McAuley. RepoBench: Benchmarking repository-level code auto-completion systems. In International Conference on Learning Representations, 2024a. Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo. Refining ChatGPT-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology, 33(5):1–26, 2024b. doi: 10.1145/3643674. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2021. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. Delano Oliveira, Reydne Bruno, Fernanda Madeiral, and Fernando Castor. Evaluating code readability and legibility: An examination of human-centric studies. In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution, pages 348–359. IEEE, 2020. doi: 10.1109/ICSME46990.2020.00041. 10
OpenAI. Introducing SWE-bench verified. introducing-swe-bench-verified/, 2024.
https://openai.com/index/
David L. Parnas. On the criteria to be used in decomposing systems into modules. Communications of the ACM, 15(12):1053–1058, 1972. doi: 10.1145/361598.361623. Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, and Xiaodong Gu. SWE-QA: Can language models answer repository-level code questions? arXiv preprint arXiv:2509.14635, 2025. Jonathan Sillito, Gail C. Murphy, and Kris De Volder. Questions programmers ask during software evolution tasks. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 23–34. ACM, 2006. doi: 10.1145/1181775.1181779. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research. arXiv preprint arXiv:2504.01848, 2025. Jan Strich, Florian Schneider, Irina Nikishina, and Chris Biemann. On improving repository-level code QA for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 209–244. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-srw.28. Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. Can LLMs replace human evaluators? an empirical study of LLM-as-a-judge in software engineering tasks. Proceedings of the ACM on Software Engineering, 2(ISSTA):1955–1977, 2025. doi: 10.1145/ 3728963. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations, 2025. The appendices follow the paper’s dependency order: protocol and implementation substrate; measurement calibration; validity audits; empirical diagnostics; and release documentation.
A
Benchmark procedure
This appendix states the procedure as pseudocode. It complements the artifact-facing description in Section 3 and the harness notes in Appendix B. Notation follows Section 3.4: b is a builder, f a finder, t a task, and r a trial index. Ab,t,r is the artifact a builder produces, Qt the question bank, and yt,q the gold option for question q. We write Rbuild and Rfind for per-cell trial counts. Three properties are surfaced explicitly. First, the specification st is hidden from finders; only the artifact and the question bank cross the role boundary. Second, every invocation is keyed by a base seed seed0 plus the trial index, so option order and any sampling decisions are reproducible. Third, find runs are gated on a successful build invocation: a build with status = ok produced an artifact directory and is eligible for artifact-conditioned find runs. For the reported primary analysis, compile-probe failure is an exclusion gate; excluded artifacts and their find records are retained for validity-stress diagnostics. Only agent-side failures (timeouts, CLI errors, infrastructure failures) keep an artifact out of the canonical artifact-present release. 11
Algorithm 1 B UILD - AND -F IND benchmark execution. Require: Tasks T (with hidden specs st and banks Qt ), builders B, finders F, trial counts Rbuild , Rfind , base seed seed0 . Ensure: Build records and find records. 1: Rbuild ← ∅; Rfind ← ∅ 2: for all t ∈ T , b ∈ B, r ∈ {0, . . . , Rbuild − 1} do 3: ρ ← B UILDER .RUN(b, st , seed0 + r, r) 4: Rbuild ← Rbuild ∪ {ρ} 5: end for 6: for all ρ ∈ Rbuild with ρ.status = ok and ρ.compile_pass, f ∈ F, r ′ ∈ {0, . . . , Rfind − 1} do 7: ϕ ← F INDER .RUN(f, ρ, Qρ.t , seed0 + r′ , r′ ) 8: Rfind ← Rfind ∪ {ϕ} 9: end for 10: return Rbuild , Rfind
Reported instantiation. The reported panel uses |T | = 2 tasks (scratch_minidb, scratch_nanoweb), |B| = |F| = 12 agent configurations (Section 4, Table 6), Rbuild = 2 build trials per (b, t) cell, and Rfind = 3 find trials per (ρ, f ) cell. The build grid contains |T |·|B|·Rbuild = 48 status-ok artifact-present build runs. Seven fail the release-mode compile probe, leaving 41 compilepass primary artifacts. Each task carries |Qt | = 15 questions, yielding 41 · |F | · Rfind = 1476 primary artifact-conditioned find records and 1476 · 15 = 22140 raw question rows. After this audit, 21312 primary finder-answer rows remain in the scored recovery set. The spec-only and question-only controls (Section 3.3) add 72 + 72 control find records under the same Rfind . The base seed is seed0 = 20260501.
Algorithm 2 Builder trial: synthesize artifact Ab,t,r . Require: Builder b, hidden spec st , seed, trial index r. Ensure: Build record ρ. 1: W ← N EW W ORKSPACE(b, t, r) ▷ isolated; seeded with edit-task starter if any 2: Pb ← R ENDER B UILDER P ROMPT(st ) ▷ deterministic; finder never sees Pb or st 3: u ← I NVOKE(b, Pb , W, τ = st .τbuild ) ▷ records token usage and novel inspection bytes 4: Ab,t,r ← W ▷ the codebase produced inside W 5: v ← VALIDATE(Ab,t,r ) ▷ compile probe and structural metadata 6: ρ ← ⟨b, t, r, Ab,t,r , u, v, status = u.status, hash(st ), hash(Pb ), seed⟩ 7: return ρ
The build record’s status field is taken from the agent invocation result, so it reflects whether the agent finished cleanly within its time budget. The compile-probe outcome lives in the validation payload v and is decoupled from status, so a status-ok artifact may still be retained as diagnostic evidence even when it is excluded from the primary find panel. 12
Algorithm 3 Finder trial: recover intent from artifact. Require: Finder f , build record ρ with artifact Aρ.b,ρ.t,ρ.r , question bank Qt , seed, trial index r′ . Ensure: Find record ϕ with per-question correctness. 1: require ρ.status = ok and ρ.compile_pass ▷ primary analysis gate 2: W ′ ← C LONE R EAD O NLY(Aρ.b,ρ.t,ρ.r ) ▷ strips build outputs; finder cannot mutate the artifact 3: for all q ∈ Qt do 4: σr′ ,q ← S HUFFLE O PTIONS(q, seed ⊕ hash(q)) 5: end for 6: Pf ← R ENDER F INDER P ROMPT(Qt , {σr′ ,q }q ) 7: u ← I NVOKE(f, Pf , W ′ , τ = st .τfind ) 8: Ŷ ← R EADA NSWERS(W ′ /find_answers.json) 9: for all q ∈ Qt do 10: ŷq ← σr−1 ▷ de-shuffle to canonical option key ′ ,q (Ŷ [q]) 11: correctq ← 1{ŷq = yt,q } 12: end for 13: ϕ ← ⟨f, ρ, r ′ , {(ŷq , correctq )}q∈Qt , u, seed, hash(Pf )⟩ 14: return ϕ
The orchestrator drains the build queue before any find work starts. Records on disk are reused when their spec hash and prompt hash match the current configuration; mismatches abort to preserve raw evidence across re-runs. Per-run usage u is the source of the inspection-effort metric in Section 3.4: I NVOKE routes each agent CLI through a local observation layer that records novel retrieval and tool-call bytes alongside vendor-reported tokens.
B
Implementation details
The harness validates YAML specifications and question banks, renders deterministic builder and finder prompts, invokes agent CLIs through adapter registries, and writes append-only run records. Each record contains the task ID, status, validation or answer payload, and usage metrics where applicable; its manifest binds the registry agent ID, adapter family, model identifier, seed, trial ID, prompt hash, specification hash, creation time, B UILD - AND -F IND version, and environment hash. A local observation layer records novel retrieval bytes and novel tool-call bytes at the conversation level, excluding transport framing, tool schemas, system prompts, and replayed context.
C
Additional experimental details
Each task has 15 multiple-choice questions. The reported logical view selects one successful row per planned logical cell and excludes older pilots and preserved infrastructure failures. Build trials use planned seed labels 20260501 and 20260502 as run identifiers. Finder trials use planned seeds 20260501, 20260502, and 20260503 both as run identifiers and as deterministic option-shuffle seeds. When a preserved infrastructure failure requires a rerun, the harness writes a distinct append-only recovery-seed record and the logical view selects the successful replacement. These seeds serve as run identifiers and deterministic shuffle seeds; provider-side sampling seeds are outside this record. Controls use the same finder panel and trial count but replace the artifact context with either the specification or the question bank alone. Some successful replacement build and find records use recovery-seed prefixes such as 202615xx; these are append-only rerun identifiers selected only after preserved infrastructure failures while leaving the logical trial index and option-shuffle seeds unchanged for finder trials. Table 5: Logical panel accounting for the reported experiment. Question rows count graded multiple-choice answers.
13
D
Record type
Logical units
Planned build artifacts Artifact-present generated artifacts Compile-pass primary artifacts Compile-probe-failed artifacts Formal find runs Spec-only controls Question-only controls
2 tasks × 12 builders × 2 trials status-ok release path exists primary analysis gate excluded diagnostic subset 41 artifacts × 12 finders × 3 trials 2 tasks × 12 finders × 3 trials 2 tasks × 12 finders × 3 trials
Records
Question rows
48 48 41 7 1,476 72 72
– – – – 22,140 1,080 1,080
Agent and harness configuration
Table 6: Agent configurations in the final panel.
Label
Harness / provider
Model identifier
Effort
Claude Opus 4.7 Claude Opus 4.7-low Claude Sonnet 4.6 Claude Sonnet 4.6-low GPT-5.5 GPT-5.5-low GPT-5.4 Mini GPT-5.4 Mini-low MiMo 2.5 Pro MiMo 2.5 Pro-low MiMo 2.5 MiMo 2.5-low
Claude Code / Anthropic Claude Code / Anthropic Claude Code / Anthropic Claude Code / Anthropic Codex / OpenAI Codex / OpenAI Codex / OpenAI Codex / OpenAI Claude Code / MiMo Claude Code / MiMo Claude Code / MiMo Claude Code / MiMo
claude-opus-4-7 claude-opus-4-7 claude-sonnet-4-6 claude-sonnet-4-6 gpt-5.5 gpt-5.5 gpt-5.4-mini gpt-5.4-mini mimo-v2.5-pro mimo-v2.5-pro mimo-v2.5 mimo-v2.5
high low high low high low high low high low high low
Harness note. MiMo lacked a native coding-agent harness satisfying the orchestration and logging requirements of this benchmark. In our implementation, the MiMo API format was compatible with the Claude Code harness and incompatible with the Codex harness, so MiMo was invoked through the Claude Code harness. MiMo rows therefore evaluate model behavior together with the Claude-Code-style harness policy.
14
Table 7: Execution stack for the reported harness and analysis environment.
Component
Version or setting
Codex CLI Claude Code CLI B UILD - AND -F IND harness Python uv Rust toolchain Codex effort knob Claude Code effort knob
codex-cli 0.124.0 2.1.114 (Claude Code) 0.1.0 3.13.5 0.9.8 rustc 1.94.0; cargo 1.94.0 model_reasoning_effort=high/low –effort high/low
Run manifests record the exact registry agent ID, adapter family, model identifier, prompt hash, specification hash, seed, trial ID, creation time, B UILD - AND -F IND version, and environment hash for every build and find invocation. The release preserves these manifest fields as columns of the derived tables so that reported numbers can be audited without relying on appendix-level command transcripts. The stored records and deterministic transforms reproduce the reported analysis. Live reruns of builder and finder interactions depend on provider model availability, CLI versions, provider endpoints, and model drift; reruns should therefore be treated as new evidence layers with their own records.
E
Finder calibration
Table 8 reports the per-finder calibration evidence behind the instrument framing in Section 3. Formal recovery is artifact-conditioned exact-match recovery on the audited scoring set. The ∆ columns report lift over the corresponding question-only control: overall, on low-prior questions, and on implementation-signal questions. Spec-only controls are saturated in this task pack and are retained as validity context in the generated calibration summaries. Table 8: Finder calibration summary. Formal is a percentage; ∆ columns are percentage-point lifts over the corresponding question-only control. Mean tokens are per find run, in thousands. Finder
Formal Q-only ∆ Low-prior ∆ Impl.-signal ∆ Mean tok.
Opus 4.7-high Opus 4.7-low Sonnet 4.6-high Sonnet 4.6-low GPT-5.5-low GPT-5.5-high MiMo-v2.5-high MiMo-v2.5-low MiMo-v2.5-pro-low MiMo-v2.5-pro-high GPT-5.4-mini-low GPT-5.4-mini-high
99.8% 99.7% 99.6% 99.5% 99.4% 99.2% 99.0% 98.8% 98.8% 98.7% 97.4% 97.2%
-0.2 -0.3 +4.1 +1.7 -0.6 -0.8 +5.7 +3.3 +34.3 +8.7 -0.4 -2.8
-0.5 -0.7 +12.1 +5.1 -0.3 -0.3 +18.0 +10.4 +39.8 +26.9 +1.1 -3.7
-0.2 -0.4 +6.1 +2.6 -0.2 -0.2 +8.8 +5.3 +38.3 +11.7 +0.1 -2.5
889 470 133 116 390 547 404 397 409 413 331 639
The post-calibration core-finder selector in scripts/select_eandd_core_finders.py applies hard gates for formal recovery, finder-specific question-only lift, low-prior lift, implementation-signal lift, cost, rank-stability, and affinity. It selects Sonnet 4.6-low and Sonnet 4.6-high as the routine future finder panel, and records MiMo-v2.5-pro-low as a prior-stress audit finder. Under these gates, Opus and Codex finders have high formal recovery but saturated finder-specific question-only controls on this task pack, making them less useful for the routine panel’s prior-stress role. The generated selection artifact is runs_release/reports/eandd_final/core_finder_selection/.
F
Metric details
The reported effort unit is novel agent inspection bytes, the sum of input-side novel_retrieval_bytes and output-side novel_tool_call_bytes. Retrieval bytes are distinct tool_result bytes delivered to the model. Tool-call bytes are emitted arguments for search 15
and read actions. System prompts, tool schemas, transport overhead, and replayed context are excluded. All effort scores are computed inside a fixed analysis panel with builders B, finders F, and tasks T . For an effort metric m, let Gb,f,t be the all-correct runs for builder b, finder f , and task t on the audited scoring set. When this set is nonempty, the cell cost is cm (b, f, t) =
1
X
|Gb,f,t |
m(b, f, t, r).
(2)
r∈Gb,f,t
If no all-correct run exists, the cell is excluded from the conditional effort average and counted in coverage. Within a fixed high- or low-effort panel, the within-finder ratio is rm (b, f, t) =
cm (b, f, t) . minb′ ∈B: cm (b′ ,f,t) defined cm (b′ , f, t)
(3)
Let Cb be the finder-task cells where builder b has a defined all-correct cost. The conditional builderconditioned artifact-effort diagnostic is X 1 Rm (b) = exp log rm (b, f, t) . (4) |Cb | (f,t)∈Cb
The tables write this score as Rb for the reported total-byte view and report |Cb | explicitly. For the task-level recovery gate, define ¯ g(b, f, t) = Acc(b, f, t),
(5)
¯ is computed on the audited scoring set and bars denote means over repeated find runs. For where Acc each builder–task table entry, let Db,t be the set of contributing finders for builder b on task t. The table reports X 1 Gtask (b, t) = g(b, f, t), (6) |Db,t | f ∈Db,t
with the corresponding contributing finder-cell count reported in parentheses. Aggregate tie-breaker diagnostic. score
For compact ordering, we define an artifact-evidenced legibility AELSη (b) = Āb S̄b Cb Rb−η .
(7)
Here Āb is mean audited recovery over formal primary cells, S̄b is mean repeat-answer agreement over repeated find runs, Cb = |Cb |/|F × T | is contributing-cell coverage, and Rb is the total-byte conditional effort diagnostic above. We use η = 0.1 so effort acts as a weak tie-breaking discount. AELS is reported only as an aggregate diagnostic; the primary interpretation remains the separated gates and conditional effort views. Table 9: Aggregate artifact-evidenced legibility score (AELS0.1 ). Higher is better. Ā is audited recovery, S̄ is repeatability, C is contributingcell coverage, and Rb is conditional total-byte inspection effort. Builder row
AELS0.1
Ā
S̄
C
Rb
0.970 0.950 0.949 0.948 0.934 0.876 0.767 0.685 0.498 0.456 0.390 0.362
0.988 0.975 0.996 0.991 0.994 0.942 0.915 0.960 0.886 0.989 0.857 0.466
0.985 0.987 0.993 0.983 0.993 0.984 0.976 0.989 0.982 0.978 0.960 0.498
1.000 1.000 1.000 1.000 1.000 1.000 0.917 0.750 0.583 0.500 0.500 0.417
1.033 1.137 1.514 1.320 1.727 1.750 1.927 1.483 1.202 1.799 1.715 1.908
codex_gpt5_5_high codex_gpt5_5_low claude_code_opus_4_7_high codex_gpt5_4_mini_high claude_code_opus_4_7_low claude_code_sonnet_4_6_low claude_code_mimo_v2_5_low claude_code_sonnet_4_6_high codex_gpt5_4_mini_low claude_code_mimo_v2_5_pro_low claude_code_mimo_v2_5_high claude_code_mimo_v2_5_pro_high
16
G
Artifact layout sensitivity audit
The released primary panel uses a root-level compile probe: an artifact must present a complete Cargo project at the artifact root and pass cargo build –release. This gate treats project layout as both a communication artifact and packaging metadata, because downstream finders receive the artifact root as the codebase boundary. To separate layout fidelity from Rust source buildability, we ran a post-hoc sensitivity audit on the seven artifact-present builds excluded by the primary compile gate. The audit used existing artifacts without rerunning builders or modifying outputs. If an excluded artifact had no root Cargo.toml but contained exactly one nested Cargo.toml, we copied that nested crate to a temporary directory and ran cargo build –release with an isolated target directory. Six of the seven excluded builds contained a unique nested crate, and all six built successfully under this relaxed probe. The remaining excluded build contained no Rust source files and no nested Cargo project.
Table 10: Post-hoc sensitivity audit for primary compile-probe exclusions. The relaxed probe supplies a diagnostic view alongside the primary panel. Probe slice
Builds
Primary root-level compile-probe exclusions Unique nested Cargo crate found Nested crate builds under relaxed probe Nested crate build failures Nested crate build timeouts No nested Cargo crate found
7 6 6 0 0 1
This audit shows that the excluded set is dominated by artifact-layout violations: six builds produced buildable source one directory below the expected artifact boundary, while one build contained no Rust source files or nested Cargo project. We therefore keep the primary estimand unchanged, but interpret the compile-probe exclusions as a mixture of layout-contract failures and no-source output diagnostics.
H
Artifact audit and qualitative builder patterns
We manually inspected the 48 generated build artifacts in the canonical artifact-present release: two tasks, twelve builders, and two build trials per builder-task cell. The audit covered source files, tests, README files, configuration files, and Cargo metadata. It looked for direct gaming signals: question IDs, answer-option labels, gold-answer keys, answer-distribution hints, copied question or option phrasing, benchmark requirement or intention IDs embedded in artifacts, and documentation that repeated evaluation wording in place of describing the implementation. We found no evidence of these direct leakage patterns in the generated artifacts. This audit is a canonical-release validity check; adaptive-attack robustness remains a separate stress-test setting. Table 11 reports builder-side implementation coverage over the artifact-present release. Coverage is the fraction of artifact-question pairs audited as implemented-gold. This is a builder-side diagnostic, separate from finder recovery and conditional inspection effort. 17
Table 11: Builder-side implementation coverage over artifact-present builds. Coverage is the fraction of artifact-question pairs audited as implemented-gold. Builder Opus 4.7-high Opus 4.7-low GPT-5.4-mini-high GPT-5.5-high GPT-5.5-low Sonnet 4.6-high MiMo 2.5-low Sonnet 4.6-low MiMo 2.5 Pro-high MiMo 2.5 Pro-low MiMo 2.5-high GPT-5.4-mini-low
Pairs Implemented-gold Coverage Non-gold / ambig. 60 60 60 60 60 60 60 60 60 60 60 60
60 60 60 60 59 58 57 57 56 56 53 51
100.0% 100.0% 100.0% 100.0% 98.3% 96.7% 95.0% 95.0% 93.3% 93.3% 88.3% 85.0%
0/0 0/0 0/0 0/0 1/0 2/0 2/1 3/0 4/0 4/0 4/3 8/1
The same inspection exposed qualitative differences in how builder families communicate design intent. Codex-built artifacts were generally more concise: they tended to use flatter crate layouts, shorter README files, and source organization as the main carrier of design intent. Claude Opus and Sonnet artifacts more often included longer README architecture sections and richer module-level comments, making design choices explicit in prose as well as code. MiMo artifacts were more variable, often with thicker scaffolding and more extensive project shells, but less consistent structure across trials. These patterns provide descriptive context for why artifact legibility can interact with finder family: downstream agents may benefit differently from terse code-structure signals versus explicit prose documentation.
I
Post-hoc artifact-evidence audit
Exact-match grading makes the benchmark reproducible, but exact-match correctness alone cannot show whether a finder used artifact evidence. To connect the deterministic score to the intended construct, we audited a stratified sample of correct finder answers using the structured evidence fields already present in finder records. The formal score remains exact-match; this audit checks alignment between exact-match recovery and artifact-grounded rationales in the released records. We sampled 72 correct answers from the compile-pass artifact-conditioned panel with audit seed 20260601, stratifying by task, builder family, finder family, and low-prior versus high-prior question status. The sample excludes question-only and spec-only controls. For each sampled answer, the auditor inspected the recorded rationale, evidence files, evidence symbols, and the corresponding artifact files. We assign one of four labels. Supported means the cited artifact evidence directly supports the selected answer and distinguishes it from plausible alternatives. Partially supported means the cited evidence is relevant while leaving the gold option underdetermined. Unsupported means the cited evidence is absent, irrelevant, or contradicted by the artifact. Prior-like means the rationale mainly uses generic engineering priors or question wording. Table 12: Post-hoc artifact-evidence audit over sampled correct finder answers. Unsupported and prior-like labels are combined in the final column. Slice
N
Supported
Partial
Unsupported/Prior-like
All sampled correct answers Low-prior questions scratch_minidb scratch_nanoweb
72 36 36 36
65 35 31 34
3 0 2 1
4 1 3 1
This audit evaluates construct alignment for the released exact-match records by checking whether existing correct answers often carry plausible artifact-grounded rationales. Among the 68 supported 18
or partially supported rows, cited evidence is concentrated in source files (66 rows); README/docs and configuration/Cargo files appear in 7 rows each, with tests absent as the primary cited channel in this sample. Restricted-view ablations such as README-only, source-only, tests-only, config-only, and source+tests without README views require new live finder runs and should be released as a separate evidence layer. A future task pack can promote these evidence fields into required, graded outputs.
J
Accuracy and robustness views
Builder downstream accuracy provides validity context for the artifact-effort diagnostics. Figure 2 reports an implementation-aware recovery view: it combines builder-side implementation coverage with downstream recovery on audited scoring rows, while the effort views answer how much work the same finders spend. For builder b, the plotted rate is P P ! P ! + 1{ŷb,f,t,a,s,q = yt,q } f,t,a,s q∈Q+ t,a |Qb,t,a | b,t,a P Db = . (8) P + t,a |Qt | f,t,a,s |Qb,t,a | The first factor is downstream recovery on the audited scoring set; the second factor is builder-side implementation coverage over artifact-present builds. 98.9%
GPT-5.5-high
99.2%
GPT-5.4-mini-high
99.4%
Opus 4.7-high 95.9%
Sonnet 4.6-high 92.9%
MiMo 2.5 Pro-high 86.0%
MiMo 2.5-high
97.7%
GPT-5.5-low GPT-5.4-mini-low 84.0%
99.3%
Opus 4.7-low 94.4%
Sonnet 4.6-low 92.6%
MiMo 2.5 Pro-low
92.6%
MiMo 2.5-low 84%
86%
88%
90%
92%
94%
96%
98%
100%
Implementation-aware recovery and implementation coverage
Recovery × coverage
Implementation coverage
GPT
Claude
MiMo
Figure 2: Implementation-aware builder downstream recovery diagnostic. Filled markers multiply downstream recovery on audited rows by builder-side implementation coverage; open markers show coverage alone. The axis is truncated, and values are diagnostic rather than a leaderboard.
Figure 3 reports pivot views that decouple several effects: which finders are used as instruments, whether builder and finder families are matched, and whether the questions are high-prior or implementation-sensitive. These slices are robustness checks around the reported artifact comparison. They show why the paper uses multiple views: raw recovery, finder accuracy, and conditional artifact-efficiency emphasize complementary parts of the panel. Slice definitions are fixed by the analysis script. The calibrated-instrument row restricts the builder view to the four highest artifact-conditioned recovery finders and the finder view to the four highest downstream-recovery builders. Low-prior-answerability questions are those with question-only threetrial agreement below 90%. Implementation-signal questions are those where artifact-conditioned three-trial agreement exceeds question-only agreement by at least 5 percentage points. Cross-family rows exclude same-family builder–finder pairs. 19
Finder view
All rows
MiMo 2.5 Pro 99.5%
Opus 4.7 99.4%
GPT-5.5 low 99.4%
Opus 4.7 99.8%
Opus 4.7 low 99.7%
Sonnet 4.6 99.6%
Calibrated
Sonnet 4.6 low 100.0%
MiMo 2.5 Pro-low 100.0%
MiMo 2.5 Pro 100.0%
Sonnet 4.6 100.0%
Opus 4.7 low 100.0%
Opus 4.7 100.0%
Cross-family
MiMo 2.5 Pro 99.7%
GPT-5.5 low 99.4%
Opus 4.7 99.2%
Sonnet 4.6 99.6%
Opus 4.7 99.6%
Opus 4.7 low 99.5%
Impl. signal
MiMo 2.5 Pro-low 100.0%
Sonnet 4.6 low 99.5%
Opus 4.7 99.4%
GPT-5.5 low 99.8%
GPT-5.5 99.8%
Opus 4.7 99.8%
Low-prior
MiMo 2.5 Pro-low 100.0%
GPT-5.4 mini-low 99.3%
Sonnet 4.6 low 99.1%
GPT-5.5 low 99.7%
GPT-5.5 99.7%
Opus 4.7 99.5%
#1
#2
#3
#1
#2
#3
100 99 98 97
Recovery
Builder view
96 95
Figure 3: Control-conditioned robustness views. Values are exact-match recovery percentages over audited scoring rows. Table 13: Task-specific calibration rows corresponding to Table 1. Artifact-conditioned rows use the audited scoring set. Slice
Task
Question-only Question-only Spec-only Spec-only Artifact-present, all Artifact-present, all Compile-pass primary Compile-pass primary
scratch_minidb scratch_nanoweb scratch_minidb scratch_nanoweb scratch_minidb scratch_nanoweb scratch_minidb scratch_nanoweb
Build
Find
Accuracy
Perfect
∆ vs. question-only
– – – – 24 24 21 20
– – – – 864 864 756 720
94.6% 94.4% 99.8% 100.0% 98.7% 99.1% 98.7% 99.2%
– – – – 83.2% 89.6% 82.7% 90.4%
– – +5.2 pp +5.6 pp +4.1 pp +4.7 pp +4.1 pp +4.7 pp
Table 14: Task-specific repeatability rows omitted from Table 2.
scratch_minidb, formal scratch_nanoweb, formal
High / minidb
0.60 93.3%+
Build
Find
Answer rows
3-trial agreement
Reliable questions
21 20
756 720
10872 10440
96.9% 98.2%
93.3% 100.0%
0.87 100.0%+
0.47 91.7%+
1.0 0.8
High / nanoweb
0.60 100.0%+
0.60 100.0%+
0.60 100.0%+
Low / minidb
0.60 80.0%+
0.33 100.0%+
0.60 75.0%+
0.6 0.4 0.2
Low / nanoweb
0.60 86.7%+
0.80 66.7%+
0.60 91.7%+
All
Same-family
Cross-family
Median Kendall τ
Slice
0.0
Figure 4: Pairwise finder agreement on total-byte builder-effort orderings in all-correct cells over the audited scoring set. 20
K
Affinity diagnostics
Affinity residuals are diagnostic checks for pair-specific builder–finder effects. They are computed by taking the recovery score used in the matrix view, subtracting the builder mean and finder mean, and adding back the grand mean. Positive values indicate above-marginal recovery for a builder–finder pairing in this panel. These diagnostics have panel-local scope. Figure 5 shows the full 12-agent matrix, and Table 15 aggregates those residuals by model family.
GPT-5.5
+0.07
-0.07
-0.18
+0.00
-0.03
+0.03
+0.11
+0.09
+0.21
-0.10
-0.13
-0.00
GPT-5.4-mini
+0.16
+0.23
-0.24
-0.07
-0.04
-0.02
+0.07
+0.08
-0.46
+0.02
+0.10
+0.18
0.6
Opus 4.7
-0.05
+0.01
+0.06
+0.03
+0.12
+0.02
-0.08
+0.06
-0.08
-0.02
+0.02
-0.09
Sonnet 4.6
+0.01
+0.08
+0.14
+0.00
+0.10
-0.04
-0.03
+0.08
-0.22
-0.03
+0.05
-0.12
MiMo 2.5 Pro
-0.01
+0.12
-0.18
+0.09
+0.21
-0.08
-0.09
+0.14
-0.31
+0.04
+0.23
-0.15
MiMo 2.5
-0.10
-0.12
-0.09
+0.08
+0.02
+0.04
-0.01
-0.09
-0.13
+0.01
+0.18
+0.22
GPT-5.5-low
+0.06
+0.16
-0.14
-0.02
-0.07
-0.02
-0.08
+0.13
+0.18
-0.06
-0.21
+0.07
GPT-5.4-mini-low
+0.10
-0.06
-0.08
-0.01
+0.07
+0.10
+0.17
+0.01
-0.35
+0.05
+0.14
-0.14
Opus 4.7-low
+0.02
-0.00
+0.13
-0.10
-0.17
+0.01
+0.01
-0.09
+0.29
-0.01
-0.11
+0.02
0.2
0.0
Affinity residual
Builder
0.4
−0.2
−0.4 -0.02
-0.04
-0.04
+0.01
-0.11
+0.16
+0.04
-0.09
-0.09
MiMo 2.5 Pro-low
-0.04
+0.02
+0.16
-0.08
-0.15
-0.05
-0.12
-0.09
+0.64
-0.10
-0.17
-0.01
MiMo 2.5-low
-0.12
-0.32
+0.12
+0.10
-0.02
+0.04
+0.03
-0.19
+0.07
+0.14
+0.02
+0.12
w 2. iM o M
5 2. M
iM o
5-
Pr o-
64.
−0.6
lo
lo
w lo
w lo et nn
s pu O
-m .4 PT -5
4.
in
7-
i-l
-lo .5 PT -5 G
G
So
w
5 2. iM o
5 M
iM o
2.
et nn So
M
6 4.
7 s pu O
-m .4 PT -5
4.
in
.5 PT -5 G
G
w
+0.30
ow
-0.03
Pr o
-0.08
i
Sonnet 4.6-low
Finder
Figure 5: Pair-specific builder–finder affinity residuals for the 12-agent compile-pass panel. Scores use the audited scoring set. Red indicates above-marginal recoverability; blue indicates belowmarginal recoverability.
Table 15: Family-level affinity residuals in the compile-pass panel. Positive values are above marginal expectations. Builder family
OpenAI/Codex finder
Anthropic/Claude finder
MiMo finder
0.076 -0.013 -0.063
-0.077 0.041 0.036
0.001 -0.028 0.027
OpenAI/Codex Anthropic/Claude MiMo
L
Low-prior subset sensitivity
This appendix repeats the main effort, recovery, ordering-stability, and affinity diagnostics on the ten low-prior questions: those with question-only three-trial agreement below 90%. The subset contains five questions from each task. Accuracy and all-correct gates are recomputed only on these low-prior 21
questions after the scoring audit. Inspection effort is still recorded at the whole find-run level, so the effort-based plots assign the full run-level inspection bytes to the subset; these figures upper-bound subset-specific effort and serve as conservative sensitivity diagnostics. cells
High effort
cells
Low effort
GPT-5.5
12/12
GPT-5.4-mini
12/12
GPT-5.4-mini
12/12
GPT-5.5
12/12
Opus 4.7
12/12
Opus 4.7
12/12
Sonnet 4.6
12/12
MiMo 2.5 Pro
6/12
MiMo 2.5
12/12
Sonnet 4.6
12/12
6/12
MiMo 2.5
MiMo 2.5 Pro 1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
11/12
1.0
1.2
1.4
Total-byte Rb
1.6
1.8
2.0
2.2
2.4
Total-byte Rb GPT
Claude
MiMo
Figure 6: Low-prior analogue of Figure 1. Conditional inspection effort, Rb , is computed on cells where the finder answered all audited low-prior questions for the task correctly at least once. Lower values mean less conditional inspection after the subset recovery gate; the cells column reports contributing finder–task cells out of 12.
Table 16: Low-prior analogue of Table 3. The recovery gate uses audited low-prior subset accuracy. Entries report task-specific mean answer recovery, with contributing finder cells shown in parentheses.
minidb
GPT-5.5 95.0% (6/6) GPT-5.4-mini 95.6% (6/6) Opus 4.7 99.4% (6/6) Sonnet 4.6 96.7% (6/6) MiMo 2.5 Pro 100.0% (6/6) MiMo 2.5 88.7% (6/6)
High / minidb
Low effort Builder
nanoweb 100.0% (6/6) 100.0% (6/6) 100.0% (6/6) 100.0% (6/6) – (0/6) 97.8% (6/6)
0.60 93.3%+
minidb
GPT-5.5 GPT-5.4-mini Opus 4.7 Sonnet 4.6 MiMo 2.5 Pro MiMo 2.5
0.87 100.0%+
nanoweb
96.4% (6/6) 100.0% (6/6) 98.9% (6/6) 98.3% (6/6) 98.3% (6/6) 97.8% (6/6) 98.1% (6/6) 98.3% (6/6) – (0/6) 100.0% (6/6) 95.4% (6/6) 96.7% (6/6)
0.47 91.7%+
1.0 0.8
High / nanoweb
0.60 100.0%+
0.60 100.0%+
0.60 100.0%+
Low / minidb
0.60 80.0%+
0.60 100.0%+
0.60 75.0%+
0.6 0.4 0.2
Low / nanoweb
0.60 86.7%+
0.80 66.7%+
0.60 91.7%+
All
Same-family
Cross-family
Median Kendall τ
High effort Builder
0.0
Figure 7: Low-prior analogue of Figure 4. Pairwise finder agreement is computed on total-byte orderings in cells where the audited low-prior subset is recovered perfectly at least once. 22
GPT-5.5
+0.02
+0.06
-0.20
-0.00
+0.03
+0.04
+0.06
+0.08
+0.20
-0.11
-0.17
-0.00
GPT-5.4-mini
+0.13
+0.14
-0.24
-0.06
+0.03
+0.07
+0.04
+0.01
-0.45
+0.03
+0.10
+0.20
Opus 4.7
-0.04
+0.06
+0.06
+0.03
+0.12
+0.04
-0.07
-0.01
-0.09
-0.02
+0.01
-0.07
Sonnet 4.6
-0.02
+0.02
+0.14
+0.01
+0.11
+0.04
-0.06
+0.02
-0.19
-0.02
+0.05
-0.10
MiMo 2.5 Pro
-0.05
+0.08
-0.18
+0.09
+0.19
+0.04
-0.13
+0.16
-0.32
+0.04
+0.22
-0.14
MiMo 2.5
+0.03
-0.10
-0.12
+0.07
+0.05
+0.00
+0.04
-0.17
-0.15
-0.02
+0.15
+0.21
GPT-5.5-low
+0.05
+0.02
-0.11
+0.02
-0.12
-0.09
+0.05
+0.13
+0.21
-0.00
-0.17
+0.01
GPT-5.4-mini-low
-0.00
+0.07
-0.00
-0.07
+0.04
+0.05
+0.07
+0.05
-0.34
-0.01
+0.07
+0.08
Opus 4.7-low
+0.02
-0.10
+0.15
-0.06
-0.17
+0.01
+0.01
-0.10
+0.32
+0.03
-0.08
-0.03
Sonnet 4.6-low
-0.07
-0.02
+0.29
-0.01
-0.04
-0.10
-0.02
-0.02
+0.17
+0.05
-0.09
-0.15
MiMo 2.5 Pro-low
-0.10
-0.08
+0.13
-0.10
-0.15
-0.06
-0.04
-0.01
+0.61
-0.11
-0.07
-0.01
MiMo 2.5-low
+0.05
-0.17
+0.08
+0.09
-0.11
-0.03
+0.05
-0.13
+0.05
+0.13
-0.02
+0.01
0.6
0.2
0.0
Affinity residual
Builder
0.4
−0.2
lo 52. iM o M
5 2. M
iM o
−0.6
w
w Pr o-
4. et So
nn
s pu O
lo
w 6-
lo 74.
in G
PT -5
.4
-m
PT -5
lo
w
i-l ow
w .5
-lo
2. G
iM o M
5 iM o
2.
et M
nn So
5
Pr o
6 4.
7 4. s pu O
-m .4 PT -5 G
G
PT -5
in
.5
i
−0.4
Finder
Figure 8: Low-prior analogue of Figure 5. Residuals use the same matrix normalization as the main affinity diagnostic, with recovery scores recomputed on the audited low-prior question subset.
M
Resource usage diagnostics
Vendor token counts are reported as diagnostics. They use provider-reported token totals when available, or the observed vendor-token total in the local efficiency record when provider totals are absent. Table 17 reports the input/output split for the artifact-present formal panel, including compilefailed artifacts and the artifact-conditioned find runs they triggered, because compute diagnostics account for spent model calls across the full artifact-present release. Input and output are split so their sum equals the token total used elsewhere in this appendix; input includes cache-read and cache-creation categories when the provider reports them separately, and output includes reasoning tokens when reported. The 144 control find runs add 19.2M tokens in total (9.8M spec-only and 9.4M question-only). The conditional builder-effort diagnostic remains within-finder relative inspection effort over novel inspection bytes. All reported runs were orchestrated on a MacBookPro18,3 with an Apple M1 Pro CPU, 10 cores, 16 GB RAM, macOS 15.6, and arm64 architecture. Local compute was used for orchestration, Cargo compile probes, and deterministic analysis; model inference ran on provider infrastructure. Run records include wall_clock_seconds; token totals are reported as the stable compute diagnostic because live rerun latency depends on provider scheduling and model serving. 23
Table 17: Vendor token usage diagnostic by role and agent configuration in the artifact-present formal panel. Token columns are totals in millions except mean/run, which is thousands of total tokens per run.
N
Role
Agent
Runs
Input
Output
Total
Mean/run
Build Build Build Build Build Build Build Build Build Build Build Build
Claude Opus 4.7-high Claude Opus 4.7-low Claude Sonnet 4.6-high Claude Sonnet 4.6-low GPT-5.5-high GPT-5.5-low GPT-5.4 Mini-high GPT-5.4 Mini-low MiMo 2.5 Pro-high MiMo 2.5 Pro-low MiMo 2.5-high MiMo 2.5-low
4 4 4 4 4 4 4 4 4 4 4 4
25.2 18.7 30.4 12.9 12.7 7.6 29.5 8.8 57.4 38.2 61.0 42.0
0.43 0.31 0.39 0.19 0.16 0.10 0.36 0.09 0.38 0.34 0.36 0.25
25.7 19.0 30.8 13.1 12.9 7.7 29.9 8.9 57.8 38.6 61.3 42.3
6419 4744 7709 3281 3217 1922 7471 2231 14445 9643 15335 10564
Find Find Find Find Find Find Find Find Find Find Find Find
Claude Opus 4.7-high Claude Opus 4.7-low Claude Sonnet 4.6-high Claude Sonnet 4.6-low GPT-5.5-high GPT-5.5-low GPT-5.4 Mini-high GPT-5.4 Mini-low MiMo 2.5 Pro-high MiMo 2.5 Pro-low MiMo 2.5-high MiMo 2.5-low
144 144 144 144 144 144 144 144 144 144 144 144
131.8 68.6 18.4 16.9 79.5 55.6 94.0 48.2 58.5 59.1 56.8 56.4
1.09 0.44 0.88 0.37 0.96 0.62 1.67 0.65 1.08 1.05 0.98 0.93
132.9 69.0 19.2 17.2 80.5 56.2 95.7 48.9 59.6 60.1 57.8 57.3
923 479 134 120 559 390 664 339 414 417 401 398
Benchmark card
Intended use. The benchmark is intended for comparing generated artifacts within a calibrated finder-task panel by the recoverability of specification-traced design intent, selecting and validating finder panels, and auditing generated repositories for agent-facing legibility. Private task packs are intended for contamination-resistant evaluation under the same protocol. Out-of-scope use. Out-of-scope uses include certifying deployment safety, security, runtime correctness beyond the recorded build-validation fields, human maintainability, public-task-pack leaderboards, and single-axis universal rankings of coding models. Data composition. The released task pack contains scratch_minidb and scratch_nanoweb, each with a hidden specification, canonical intended behaviors and design choices, and 15 specification-traced MCQs. The canonical artifact-present release contains 48 build rows and 1728 artifact-conditioned find rows. The reported compile-pass primary panel contains 41 build rows, 1476 artifact-conditioned find rows, and 144 control find rows in the released tables; after the scoring audit, its scored recovery set contains 21312 finder-answer rows. Seven compile-failed artifacts remain in the release as validity-stress diagnostics. Pilot and infrastructure-failure rows remain under non-reporting panels, and the release includes derived analysis reports. Quality control. Specifications are schema-validated and manually reviewed by the authors. Question banks must reference canonical intended behaviors and design choices and pass duplicate-option checks; authors also inspect the questions, distractors, trace annotations, and gold answers for ambiguity and traceability. Generated artifacts receive build-validation checks during the build phase; the reported primary analysis gates on compile-pass artifacts, audits artifact-question implementation status, and reports excluded artifacts separately. Canonical tables are produced from append-only run records by deterministic transforms and then carried into the release unchanged. Metrics. Metrics are exact-match recovery accuracy as a validity check, within-finder relative inspection effort Rb over novel inspection bytes on all-correct cells, and a co-reported task-level recovery gate that penalizes partial recovery without mixing in effort. Diagnostics include spec-only and question-only controls, prior-lift views, compile-failed stress diagnostics, rank stability, builder– finder affinity views, artifact-evidence audits, low-prior sensitivity views, and token-usage summaries. In the current high-prior task pack, accuracy is a gate and calibration signal, and conditional effort 24
carries the main discriminative signal. Conditional inspection effort is interpreted as a proxy for agentfacing artifact legibility only after recovery and stability gates are satisfied. Affinity is panel-local and diagnostic. Known biases. The current task pack is single-language and covers two task families, while the generated artifacts remain substantial: among source-bearing final build cells, the median artifact contains roughly 4.2k source LOC, and many span 2k–9k LOC. This scale makes repeated builder– finder trials costly; the canonical artifact-present release consumed more than 1.1B vendor-reported tokens across formal and control runs, while the compile-pass primary panel uses roughly 954M. Questions intentionally preserve ordinary software-engineering priors, and controls show that those priors are strong in this release. The intended interpretation is panel-conditioned and prior-conditioned, including the finder policies and CLI harnesses used here. The public task pack has panel-local scope; private task packs are the intended route for contamination resistance and broader coverage. Maintenance. Task packs and tables are versioned. New task packs can be added without changing the record schema. Corrections are recorded as new evidence layers with prior records preserved.
25