Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models
arXiv:2606.31511v1 [cs.SE] 30 Jun 2026
Mehmet İşcan∗ PythaLab, Yıldız Technical University, Istanbul, Turkey
Abstract In deployment settings where retraining is not feasible, small frozen code models are commonly asked to repair a failed program after being shown their own failing output. This practice is often treated as an additional retry mechanism. From a Popperian perspective, however, it raises a deeper epistemic and measurement problem. A generated program can be viewed as a tentative conjecture, while a violation observed during test execution can be viewed as a test-oracle-relative and executable counterexample. The scientific value of feedback should therefore be attributed not to re-exposure to failing code itself, but to whether the candidate conjecture is opened to external, executable, and auditable criticism. Prior work has often bundled failing code, execution evidence, and retry instruction into a single feedback packet, without decomposing packet components against a blind resampling baseline under the same output-generation budget and against content-free placebo controls. As the third stage of a falsification-centered measurement program, this study presents a placebo-controlled evaluation instrument that compares bare failing code, executed facts, code-plus-facts, neither, and template- and bullet-count-matched placebo conditions on the same task–model units. The contribution is not a new repair algorithm, but a reflexive measurement methodology combining packet decomposition, placebo mirroring, matched-budget comparison, same-unit discordant-pair evaluation, fresh-generation confirmation, and provenance-based executable audits, so that both the model’s program conjecture and the researcher’s claim that “feedback content works” are made falsifiable. Across six HumanEval+ and MBPP+ benchmark×model cells with three 0.5B–1.5B frozen code models, 290 dead task–cell units were evaluated, where no candidate in a cached best-of-N pool (N = 8) passed the public tier. The main run produced 7,000 fresh seeded generations, and a preregistered diagnostic follow-up produced 1,400 additional generations. True unlock required passing both the public and prompt-hidden EvalPlus tiers. Blind resampling exceeded bare-code retry by +18 net unlocks (discordant 25/7, Holm-adjusted p = 0.0021). Code-plus-facts produced +18 net recovery relative to bare code (discordant 21/3, Holm-adjusted p = 0.00042), and a +15 advantage over the same-code generic-bullet placebo supported the contribution of executed fact content (Holm-adjusted p = 0.0041). In the diagnostic follow-up, an instruction-only effect was not distinguishable (+3, p = 0.36), which is not evidence of absence. The remaining difference was associated with code-bearing context, although pure code causation was ∗
Corresponding author. PythaLab, Yıldız Technical University, Istanbul, Turkey. E-mail: [email protected]. Preprint.
not claimed because code content remained confounded with prompt length. Code-plus-facts and blind resampling tied at the pooled level with 26 unlocks each, which is not evidence of equivalence, non-inferiority, non-superiority, or unit-level identical behavior, but only zero observed pooled difference in the tested sample. The content-attribution contrast changed direction in a small disjoint sensitivity sample, and surface-form explanations were not fully ruled out. Six prospective external-controller follow-ups on the search-bound stratum further supported this boundary: the two strongest content channels, own near-miss code and runtime execution trace, tied with content-free shape placebo. The findings are limited to observed unlock outcomes and make no direct claim about latent sampling capacity, model distribution, or general program-synthesis ability. Taken together, falsification in the tested frozen-small-model regime was valuable not as Popperian vocabulary, verbal self-critique, or re-exposure to failing code, but as comparison with externally produced, oracle-relative, executable counterexamples. This execution-grounded criticism was observed as a limited signal repairing harmful code-bearing conditioning, not as a content lever exceeding blind resampling. The transferable contribution is therefore a methodology for making self-repair feedback claims falsifiable through content-vs-form, discovery-vs-confirmation, and audit-vs-statistical-verdict distinctions. Keywords: large language models, code generation, self-repair, execution feedback, preregistration, placebo control, falsification, small language models
1
Introduction
Small frozen code models now appear regularly in laptop, local workstation, and edgedevice deployment settings where retraining is not feasible. In such settings, the model is often expected to generate not a single final program, but multiple candidate programs under a fixed generation budget. When a candidate program fails the tests, generation is repeated, and the operational work unit is therefore formed not as a single completion but as a retry loop (J. Wang et al., 2024; Zubair, Al-Hitmi, & Catal, 2025). Within this retry loop, adding the failed program back to the prompt, showing the evidence obtained during execution, and asking the model to produce a corrected solution have been adopted as natural debugging practices. However, the intuitive appeal of this practice often leads three distinct assumptions to be accepted jointly: it is assumed that showing the failing code again is useful, that execution evidence carries an additional signal, and that feedback-assisted retry is more effective than fresh sampling performed under the same output-generation budget. This problem can be formulated not only as a technical prompt-design issue, but also as a Popperian conjectures-and-refutations process (Popper, 1959, 1963). The generated program can be treated as a conjecture whose correctness is only tentatively accepted. The violation observed in the executed test can then be interpreted as an executable counterexample, produced against a specific behavioral consequence of that conjecture and relative to the test oracle. From this perspective, the value of self-repair feedback should arise not from re-exposing the model to its own failed output, but from opening that output to external, execution-grounded, and auditable criticism. The central question is therefore not whether feedback is useful in general. The sharper question is which form of refutation carries operational signal: exposure to failing code, executed counterevidence, generic retry instruction, or blind resampling under the same budget. The expectation that feedback may be useful is supported primarily by findings reported on stronger models or on models specifically trained to use feedback. Iterative self-refinement has been reported to provide improvement across different task families (Madaan et al., 2023; Shinn et al., 2023), and execution-grounded debugging loops have been shown to improve performance when test results and traces are incorporated back into the prompt (X. Chen, Lin, Schärli, & Zhou, 2024; Zhong, Wang, & Shang, 2024). It has also been shown that 33B open models trained for multi-turn execution feedback can reach high pass rates on HumanEval and MBPP (T. Zheng et al., 2024). For this reason, feedback-driven repair is 2
treated as one of the central workflows in large language model (LLM)-based software testing and program-repair reviews (Takallou, Gautam, & Shirolkar, 2026; J. Wang et al., 2024; Zubair et al., 2025). By contrast, it has been shown that intrinsic self-correction can reduce reasoning accuracy when no external signal is available (Huang et al., 2024; Valmeekam, Marquez, & Kambhampati, 2023), that self-critique without a sound checker can degrade symbolic reasoning (Stechly, Valmeekam, & Kambhampati, 2025), and that models cannot reliably verify their own code against natural-language specifications (Jin & Chen, 2025). When the cost of feedback turns is taken into account, self-repair gains have also been reported to be dominated by additional independent samples (Olausson, Inala, Wang, Gao, & Solar-Lezama, 2024). This limitation appears especially pronounced for code models below a few billion parameters. Reliable self-refinement is often obtained only when retraining is performed with the relevant objective (Cho, Kang, Kim, & Lee, 2025; Ding, Min, Kaiser, & Ray, 2024; Jiang et al., 2024). It has also been reported that runtime feedback is “not always helpful” for small open models (Tian et al., 2024), that the generation–verification gap remains non-positive at the 0.5B scale (Song et al., 2025), and that self-generated reasoning can reduce pass rates for lightweight models (Yang et al., 2024). In this regime, where retraining is excluded, the true competitor of feedback-assisted repair is not another prompting recipe but the use of the same output-generation budget for fresh samples. Under repeated sampling, coverage has been shown to scale across broad budget ranges (Brown et al., 2024), and “more samples” has been used as a fundamental baseline for adaptive methods in compute-optimal test-time strategies (Damani, Shenfeld, Peng, Bobu, & Andreas, 2025; Snell, Lee, Xu, & Kumar, 2025). More elaborate verification strategies have been reported to require additional compute even to match plain repeated sampling (Singhi et al., 2025), while multi-turn self-repair can fall below independent sampling under equal output budget (K. Zheng et al., 2025). The decisive operational question is therefore not whether a revision is better than the previous completion. It must be tested directly whether the same generation budget should be allocated to retry with failing context or to blind resampling after discarding that context. This study is positioned as the third stage of a falsification-centered measurement program in which this question has been progressively narrowed. In the first stage of the program, Popperian code-generation skill was decomposed through a labels-only scaffold, a lengthmatched placebo, and an execution oracle (İşcan, 2026a). It was shown that full Popperian procedural content did not provide a separable execution-correctness contribution over the labels-only scaffold, and that the gain observed in the small model was associated more with scaffold structure than with Popperian vocabulary. The same model’s self-selection using a Popperian rubric also did not exceed random selection. In the second stage of the program, falsification was moved from prompt content to post-hoc candidate processing (İşcan, 2026b). Twenty-six semantic post-hoc operators, including selection, verification, repair, elimination, sound vetoes, and generation conditioning, are evaluated against a Best-of-N baseline under a deterministic execution oracle and a leakage-free matched-compute protocol. No semantic output-space operator exceeded held-out accuracy over Best-of-N in the tested cells. Thus, Popperian vocabulary and model-internal post-hoc falsification are found not to carry a separable signal, and the remaining question is directed toward whether externally executed counterevidence can carry signal within iterative regeneration. Four identification problems in the literature prevent this question from being answered directly. First, positive studies often present failing code, execution evidence, and repair instruction inside a single feedback bundle. Therefore, the component from which the reported gain arises cannot be determined. Second, negative studies often remove the bundle entirely. As a result, it cannot be disambiguated whether the observed harm arises from the revision act, failing code, execution evidence, or the surface form of additional context. Third, when feedback-assisted regeneration and blind resampling are not compared under the same output-generation budget, the practitioner’s choice is not directly tested. Fourth, aggregate pass rates over tasks that can already be solved by ordinary sampling provide only limited information about the zero-pass-in-pool regime in which the retry loop is most needed. These identification problems are compounded when the independent effects of prompt surface form are taken into account. In large-scale software-engineering evaluations, the feedback packet is often treated as a single intervention (Z. Zheng et al., 2025). Although 3
prompt-ingredient sweeps have begun to separate certain components, the model’s own failing code is often not evaluated as an independent arm, and same-unit outcomes are not paired through discordant-pair tests (Khojah, de Oliveira Neto, Mohamad, & Leitner, 2025). It is also known that prompt formatting and arbitrary prompt architecture can affect outputs independently of task content (Brucks & Toubia, 2025; Sclar, Choi, Tsvetkov, & Suhr, 2024). Therefore, an ordinary no-feedback control is not sufficient to separate task-relevant fact content from the effects of additional text presence, template, and bullet structure. For all these reasons, the placebo-controlled disambiguation protocol of the program is transferred in this study to the axis of self-repair feedback content, and a five-arm controlled decomposition is presented. Bare failing code (P), blind resampling (B), executed facts only (TC), code plus facts (CF-SF), and code plus generic placebo bullets (GR-SHAPE) are compared on the same task–model units and under the same output-generation budget. Through the mirror chain, source code, failure instruction, template, and bullet count are held fixed between CF-SF and GR-SHAPE, while only task-relevant fact content is changed. This comparison constitutes the most tightly controlled content contrast in the design. By contrast, exact token length, lexical distribution, numeric-literal density, literal input/output (I/O) syntax, and information density are not held fixed. The B–P and CF-SF–P comparisons are defined as packet-level effects. Since these arms follow independent trajectories, they are not interpreted as same-code minimal pairs. The independent role of failure instruction is additionally tested through a preregistered sixth arm. The primary estimand is defined on dead task–cell units for which no candidate in a cached best-of-N pool (N = 8) passed the public tier. The term “dead” does not mean that no correct program exists in the model distribution. It denotes only zero-pass status with respect to the finite cached pool. Across six benchmark×model cells constructed from HumanEval+ and MBPP+ with three 0.5B–1.5B frozen code models, 290 dead units are evaluated. The same output-generation budget is allocated to each arm for each unit. Because input-token compute is not equalized, the term matched output-generation budget is used throughout the study. Correctness is determined by execution on public EvalPlus tests and prompt-hidden EvalPlus tests that are not shown in the prompt (Liu, Xia, Wang, & Zhang, 2023). This design controls the advantage of producing more output samples and allows the content, form, and exposure components of the feedback packet to be evaluated within the same operational decision problem. The measurement logic of the study is not limited to arm decomposition. Discovery and confirmation are separated. Unplanned contrasts observed in the preceding preregistered stage are not used as confirmatory evidence, but are treated only as discovery signals that generate new hypotheses. Each effect is promoted only once to a new preregistered family, is re-measured on the same fixed units with fresh seeds and fresh generations, and discovery outputs are not included in the confirmation analysis (Hofman, Chatzimparmpas, Sharma, Watts, & Hullman, 2023; Mayo & Spanos, 2006; Popper, 1959, 1963). Thus, fresh-output confirmation is provided, but no independent-task replication claim is made. This distinction is built into the measurement design to prevent exploratory evidence from being mixed with confirmatory evidence. This procedure also establishes Popperian falsification not only as an object-level test of programs but also as a reflexive evaluation architecture. At the object level, generated programs are treated as conjectures, and executed test violations are treated as counterexamples. At the meta-methodological level, the study’s own hypotheses, contrasts, and evaluation pipeline are kept open to criticism through preregistration, placebo controls, fresh-generation confirmation, and executable audit invariants. When an audit failure occurs, the original verdict is not deleted and the amendment is reported separately, in order to prevent negative evidence from being neutralized by post hoc reinterpretation. Thus, falsification is operated not only as a test directed at model outputs, but also as a critical-control discipline to which the research system that produces claims about those outputs is itself subjected. No new repair method is proposed in this study. Instead, a component-level measurement methodology is presented for measuring in what form the falsification signal is carried within the self-repair feedback packet. The originality of this methodology is established through the decomposition of the feedback packet into its components, the testing of surface-form explanations through content-free placebo controls, the comparison against blind resampling under a matched output-generation budget, the use of same-unit discordant-pair evaluation, and the separation of statistical verdict from audit verdict. In this way, claims about 4
self-repair feedback are made measurable not only through aggregate pass rates, but also through the layers of content, form, sampling budget, confirmation status, and executable provenance. In this framework, the study makes five original contributions. The first is philosophical and conceptual: self-repair feedback is reformulated not merely as an iterative retry practice but as a measurable falsification problem grounded in the Popperian conjecture– counterexample relation, and by treating generated programs as conjectures and executed test violations as oracle-relative executable counterexamples, the distinction between exposure and criticism is made explicit, and falsification is applied not only to object-level program outputs but also to the researcher’s claims about feedback content. The second is a measurement and evaluation contribution: a five-arm controlled decomposition is developed to separate the components inside the feedback packet, in which bare failing code, blind resampling, executed facts, code-plus-facts, and shape-matched placebo arms are compared on the same task–model units and under the same output-generation budget, so that packet decomposition, placebo mirroring, a matched-budget baseline, same-unit discordant-pair evaluation, fresh-generation confirmation, and provenance-based executable audits are combined within a single reflexive measurement instrument. The third is an identification contribution: failing code, execution evidence, and retry instruction, which are often presented jointly in prior work, are decomposed, so that the samecode mirror contrast separates executed fact content from generic bullet-shaped companion text and packet-level contrasts evaluate bare-code retry, code-plus-facts retry, and blind resampling within the same operational decision problem, leaving the measured recovery less confounded with exposure, surface form, instruction, or an additional sampling opportunity. The fourth concerns statistical rigor and confirmation: discovery outputs are not mixed with confirmation evidence, promoted hypotheses are retested with fresh seeds and fresh generations, and exact McNemar tests, Holm correction, same-unit paired evaluation, and a preregistered family structure keep claims bounded by observed contrasts, while nonsignificant results, pooled ties, and descriptive guards are not converted into equivalence or no-effect claims. The fifth is a reproducibility and audit contribution: the evaluation pipeline is made auditable through immutable unit keys, seed-namespace isolation, frozen prompt rendering, mirror SHA equality, forbidden-token checks, executable audit invariants, forensic audit trails, and dated amendments, and statistical verdicts are kept separate from audit verdicts, so that not only model outputs but also the evaluation system that produces claims about those outputs is left open to falsification. The claims are limited to 0.5B–1.5B frozen code models, Python HumanEval+/MBPP+ tasks, the zero-pass-in-pool regime, a single prompt scaffold, and a matched output-generation budget. No direct inference is made about stronger models, non-dead tasks, other programming languages, or repository-level coding settings. The comparison between code-plus-facts and blind resampling is interpreted outside the superiority family and is not used to produce an equivalence or non-inferiority claim. The content-attribution interpretation is limited to the dimensions held fixed inside the mirror chain and does not assume equality of exact token length or lexical distribution. The remainder of the paper is organized as follows. The measurement instrument, population definition, regeneration arms, mirror chain, preregistered statistical family, generation protocol, and audit invariants are presented in §2. Primary effects, sixth-arm diagnostic decomposition, content-attribution analyses, sensitivity checks, and audit outcomes are reported in §3. Mechanism interpretations, prior-work comparison, practical implications, and validity boundaries are discussed in §4. The prospective external-controller follow-ups conducted after the primary analysis are given in §4.1. Finally, the bounded conclusion and transferable methodological contribution of the study are summarized in §5.
2
Methods
The method used in this study was structured as the third stage of a falsification-centered measurement program applied to feedback content. The placebo-controlled disambiguation protocol and the leakage-free, matched-budget execution protocol developed in the previous stages of the program were applied here to the components of an iterative self-repair packet (İşcan, 2026a, 2026b). At the object level, the generated program is treated as a conjecture. The violation produced by public test execution is interpreted as an execution-grounded 5
1 UNIT SELECTION
2 PARALLEL CONDITIONING ARMS
3 SCORING AND ENDPOINT
Matched output-generation budget: 4 rounds × 1 candidate per arm identical decoding | dedicated fresh seeds
DEAD TASKCELL UNIT benchmark × model × task
P
Bare-code retry
own failing code + failure instruction
1
2
3
4
B
Blind resampling
original task only; failure context discarded
1
2
3
4
≤ 6 violated public I/O facts; source code withheld
1
2
3
4
1
2
3
4
1
2
3
4
1.
TC
Facts only
full public test tier if passed
facts fixed; add code + instruction
cached best-of-N pool
zero true pass
EXECUTION-BASED SCORING
trajectory-pinned mirror chain
CF-SF
Code + same facts
TC facts + source code they were run on + instruction
2.
full prompt-hidden tier
facts → placebo; code/template/count fixed
GR-SHAPE
Code + placebo
same code, template and bullet count; generic bullets
TRUE UNLOCK ≥ 1 of 4 candidates passes both tiers
CF-SF vs GR-SHAPE is the only same-code content contrast; all other comparisons are packet-level effects.
Figure 1: Falsification-centered measurement instrument. Each dead unit for which no candidate in the cached pool passed the public tier was evaluated under five regeneration arms with the same output-generation budget. Four rounds were run for each arm, one candidate was generated in each round, and a fresh seed from a dedicated namespace was used. The mirror chain links the CF-SF packet to the TC trajectory and the GR-SHAPE packet to the CF-SF trajectory. Only in the CF-SF–GR-SHAPE comparison were source code, failure instruction, template, and bullet count held fixed while task-relevant fact content was changed. The endpoint is true unlock. At least one candidate generated under the same arm must pass both the public and prompt-hidden tiers.
and oracle-relative counterexample to that conjecture (Popper, 1959, 1963). At the metamethodological level, the strength of the executed counterexample is taken to depend on assumptions about the test oracle, sandbox, and harness. Because of this Duhem–Quine boundary, the arm contrasts, statistical claims, and the evaluation pipeline that produced those claims are kept open to critical testing through preregistration, fresh-generation confirmation, and executable audit invariants. A five-arm controlled decomposition was used to measure the component-level effects of the self-repair feedback packet. Each dead task–cell unit was evaluated under five parallel regeneration arms with a matched output-generation budget. The outcome was defined by whether at least one candidate generated under the same arm passed both the public and prompt-hidden test tiers. The measurement instrument is summarized in Figure 1. This section describes the population and endpoint, arm definitions, mirror chain, preregistered statistical family, generation protocol, audit invariants, and discovery-toconfirmation procedure in sequence. Procedure 1 is given not as a new repair algorithm, but as the executable evaluation procedure of the measurement instrument. 2.1
Measurement instrument
2.1.1 Population, endpoint, and unit of analysis Definition 1 (Dead task–cell unit). A cell is defined as a benchmark–model pair. A task–cell unit is the benchmark task evaluated under a specific cell and is immutably keyed by ⟨benchmark | model | task⟩. A unit is classified as dead when none of the candidates in the cached best-of-N pool for that unit (N = 8, fixed by the K_POOL code constant) passes the public test tier under standard and robust extraction. The per-task candidate cache files are an uncommitted upstream artifact. This definition is made with respect to the finite cached pool and does not imply that no correct program exists in the model distribution. Definition 2 (True unlock). A dead unit is counted as unlocked by a given arm only if at least one regenerated candidate from that arm passes the full public tier and the full prompt-hidden tier. Prompt-hidden test cases are not shown to any arm. However, because 6
Table 1: Per benchmark×model cell sample flow. Dead units were first capped at cap = 60 per cell, and pre-run token exclusion was then applied. Primary = capped − exclusions. The capped total is 291, and the single exclusion (humanevalplus__115) reduces the primary sample to 290. Initial counts were computed from the deterministic EvalPlus loader (MBPP+ 378 nominal minus 13 loader exclusions = 365, HumanEval+ 164). All counts were recomputed from committed ANTIDOTE prerun manifests with scripts/reconstruct_sample_selection.py and reproduced under the seeded selection rule.
Benchmark
Model
HumanEval+ HumanEval+ HumanEval+ MBPP+ MBPP+ MBPP+
deepseek-coder:1.3b qwen2.5-coder:0.5b qwen2.5-coder:1.5b deepseek-coder:1.3b qwen2.5-coder:0.5b qwen2.5-coder:1.5b
Total
Initial Dead Capped Excl. Primary Disjoint 164 164 164 365 365 365
119 40 24 171 111 47
60 40 24 60 60 47
1 0 0 0 0 0
59 40 24 60 60 47
15 0 0 30 15 0
164+365
512
291
1
290
60
the dead population is defined only by public-tier pass/fail outcomes in the cached pool, and because the prompt-hidden tier defines the final true-unlock scoring endpoint, this tier is interpreted not as an untouched test set but as a prompt-hidden evaluation tier. The population was constructed from 512 dead units obtained by crossing the HumanEval+ and MBPP+ benchmarks (Austin et al., 2021; M. Chen et al., 2021; Liu et al., 2023) with three frozen models served through Ollama. The models were fixed as qwen2.5-coder:1.5b, qwen2.5-coder:0.5b-instruct (fp16), and deepseek-coder:1.3b (Guo et al., 2024; Hui et al., 2024). The seeded and capped fixed sample used in the predecessor experiment was inherited unchanged. After the inherited pre-run token exclusion, the primary confirmation sample was fixed at n = 290. Thus, the promoted hypotheses are retested on the same task–cell sample in which the discovery effects were produced, but with entirely fresh generations. A separate disjoint sample consisting of 60 dead units that had not previously been included in the primary sample was constructed for external-direction sensitivity. This sample contains 15 HumanEval+/deepseek units, 30 MBPP+/deepseek units, and 15 MBPP+/qwen0.5b units. The per-cell sample flow is summarized in Table 1. In each cell, dead units were first capped at cap = 60 per cell, and the pre-run token exclusion was then applied. The only exclusion was the humanevalplus__115 unit in the HumanEval+/deepseek-coder:1.3b cell. Therefore, the capped 60 units in that cell were reduced to 59 primary units, yielding a total primary sample of 290 and a capped total of 291. The disjoint sample was drawn from the unselected units of each cell. A 32-bit seed derived by sha256 from the payload antidote-disjoint|benchmark|model and random.Random.sample were used for this selection. The cached best-of-N pool was fixed at N = 8 (K_POOL code constant). Per-task candidate cache files were retained as an uncommitted upstream artifact. The provenance of the public and prompt-hidden tiers is given in Table 2. The public tier consists of EvalPlus base tests. This tier is used by the evaluation harness, but it is not rendered verbatim in any arm prompt. In the TC and CF-SF arms, only up to six public assertions violated by the incumbent are shown as literal input/output (I/O) facts. No test-derived content is provided to the B and P arms. The GR-SHAPE arm is instead given count-matched generic placebo bullets. The prompt-hidden tier consists of EvalPlus plus tests and is deterministically capped at the first 300 tests in EvalPlus source order. This tier is not shown to any arm. Dead-status selection is performed only on the public tier, whereas the prompt-hidden tier is used only for final true-unlock scoring. Therefore, the prompt-hidden tier is referred to as the prompt-hidden evaluation tier that defines the final scoring endpoint and is not presented as an untouched held-out test set. Units are selected by seeded hash from the eligible pool remaining outside the fixed sample (§3.2). The disjoint sample is not included in the primary statistical family and is used only for directional sensitivity. 7
Table 2: Test-tier provenance. The public tier (Ibase , EvalPlus base input suite) is available to the harness but is not shown verbatim in any arm. It appears only in the TC/CF-SF arms as ≤ 6 violated input/output (I/O) facts. The prompt-hidden tier (Iplus , EvalPlus plus input suite, first 300 in source order) is not shown to any arm. Dead-status selection depends only on public-tier outcomes. The prompt-hidden tier is not used for selection and is evaluated only during final true-unlock scoring. Therefore, it is not an untouched held-out set. All tiers use EvalPlus 0.3.1, pinned in uv.lock. Benchmark
Tier
HumanEval+ HumanEval+ MBPP+ MBPP+
public prompt-hidden public prompt-hidden
Source
Prompt exposure
Dead-status
Scoring
EvalPlus Ibase EvalPlus Iplus (first 300) EvalPlus Ibase EvalPlus Iplus (first 300)
≤ 6 facts (TC/CF-SF) none ≤ 6 facts (TC/CF-SF) none
✓ — ✓ —
✓ ✓ ✓ ✓
Table 3: Prompt-hidden test-count summary. The deterministic first-300 cap binds only on HumanEval+ (126 tasks truncated, pre-cap median 972, max 1000). On MBPP+ the cap never binds (max 147). The cap binds the prompt-hidden tier only. Dead-status selection uses public-tier outcomes only, so the cap affects final true-unlock scoring rather than dead selection.
Benchmark
Tasks
Median plus
Max plus
>300
HumanEval+ MBPP+
164 365
972 105
1000 147
126 0
Subset rule first 300 (source order) first 300 (source order)
Cell-level dead-task counts, model digests, and round-0 prompts are recorded as SHApinned version-controlled manifests before any generation is initiated. This prevents sample identity and prompt rendering from being changed after the run. The matched budget was operationalized as an equal output-sample count per unit. Four regeneration rounds were run for each arm, one candidate was produced in each round, and identical decoding parameters were used. Thus, four output samples were allocated to each arm for each unit. Input-token compute was not equalized. Code-bearing prompts reached approximately 1,290–1,480 characters depending on the arm, compared with a mean length of approximately 406 characters for B prompts. Therefore, strict end-to-end floating-point operation (FLOP) matching was not provided. Instead, a matched output-generation budget was provided. Outcome differences cannot be attributed to generating more candidates. Prompt length and prefill cost remain residual design factors (Brown et al., 2024; Olausson et al., 2024; K. Zheng et al., 2025). 2.1.2 Regeneration arms and mirror chain The five arms shown in Figure 1 are defined in Table 4. In the P condition, the model was shown its own round-specific best-so-far failing code and was asked to produce a corrected solution. In each round, the arm-specific incumbent was selected from the union of the cached best-of-N pool and the candidates generated in the previous rounds of that arm. Selection was based on the highest public-assert pass count, namely the number of public assertions passed. When pass counts were tied, ties were resolved deterministically in favor of the most recently generated candidate. Thus, under a tie, the latest round output rather than a pool entry was accepted as the incumbent. In the B condition, no failure context other than the original task was provided, and each round was run as an independent blind resample. These two arms represent the bare-code retry practice and the discard-the-context baseline, respectively. In the TC condition, only execution-grounded counterevidence was presented. Public assertions violated by the incumbent solution in the corresponding trajectory were shown as at most six literal I/O facts through a frozen no-ellipsis renderer. These facts were used as observable counterexamples that refuted a specific behavioral consequence of the candidate’s correctness conjecture in an oracle-relative manner. The source code on which the facts had been executed was withheld from the prompt. If the incumbent violates no public assertion, that is, if all public tests are passed and failure occurs only on the prompt-hidden tier, the fact block is rendered as an empty block containing only the header line and no bullets. This empty-violation behavior is shared by the TC, CF-SF, and GR-SHAPE arms. Since the 8
Table 4: Five regeneration arms. Four output generations were used per unit for each arm. In the CF-SF–GR-SHAPE mirror pair, source code, failure instruction, template, and bullet count were held fixed, and only task-relevant fact content was changed. The TC–CF-SF transition adds source code and the failure instruction together. P comparisons involve independent arm trajectories and are therefore not interpreted as same-code minimal pairs. Arm
Conditioning packet
P B TC CF-SF GR-SHAPE
bare own failing code (incumbent) none (blind resample of the task) violated public I/O facts only (≤ 6) TC’s facts + the code they were run on CF-SF’s code + shape-matched generic bullets
Own code
Executed facts
Companion text
✓ — — ✓ ✓
— — ✓ ✓ —
— — — — ✓
GR-SHAPE placebo bullet count mirrors the true fact count in TC, zero true facts correspond to zero placebo bullets. In this case, the CONFIRM-local renderer inserts no degenerate fallback sentence. The three code-bearing arms, P, CF-SF, and GR-SHAPE, were terminated with the following failure instruction: “Your solution failed the tests. Write a corrected, different solution.” The two code-free arms, B and TC, did not include this instruction. Therefore, in the main five-arm design, the failure instruction remained coupled with code-bearing context. This coupling was tested by a separate preregistered sixth-arm follow-up. The CF-SF and GR-SHAPE arms were constructed as mirror conditions. In the CF-SF condition, the source code from which the TC fact block had been produced was shown in full, and the failure instruction was added because of the code-bearing scaffold. The rendered fact block was copied verbatim from the TC packet, and equality was verified by per-round SHA-256 audit. Thus, the TC–CF-SF transition added source code and the failure instruction together while evidence was held fixed. This comparison is not interpreted as a code-only effect. The GR-SHAPE condition was constructed as a shape-matched placebo. In this arm, the CF-SF source-code block was preserved exactly and verified by per-round SHA-256 equality. The failure instruction, template, and bullet count were also held fixed. Executed facts were replaced with bullets selected from a frozen list of generic requirement statements. Placebo bullets were constructed so that they contained no function name, assert literal, or task identifier, and an executable forbidden-token audit was applied to all rendered prompts. Thus, in the CF-SF–GR-SHAPE contrast, source code, failure instruction, template, and bullet count are held fixed while task-relevant fact content is changed. This comparison is designed to separate the effect of executed counterevidence from the presence and shape effects of generic companion text. Nevertheless, because exact token length and lexical distribution are not fully equalized, it is not claimed that all surface-form differences are removed. It has been shown that meaning-preserving changes in prompt form can produce large differences in accuracy (Sclar et al., 2024), and that arbitrary prompt architecture can bias outputs independently of task content (Brucks & Toubia, 2025). Therefore, the placebo arm is used as the load-bearing control for feedback-content attribution. To test the code–instruction coupling in the main design, a preregistered sixth arm named BI (blind plus instruction) was run as a sequential follow-up. BI was constructed by adding only the failure instruction used in the code-bearing arms to the B scaffold. No code, facts, or bullets were added. Four rounds were run on the same fixed and disjoint units, a separate seed namespace was used, and outcomes were paired at the unit level with the frozen B and P outcomes. Thus, the instruction component and the code-bearing-context component were measured through separate contrasts. Code content and the intrinsic prompt length introduced by code are not decomposed in this follow-up. 2.2 Statistical analysis, generation protocol, and disclosure 2.2.1 Preregistered statistical family The frozen statistical family consisted of three one-sided claims evaluated on pooled discordant pairs: • C1′ (co-primary, trap hypothesis): B > P. • C2′ (co-primary, antidote hypothesis): CF-SF > P. 9
Procedure 1 Five-arm evaluation procedure. The procedure is not a repair algorithm, but an evaluation instrument that measures the arm-level true-unlock outcome for each dead unit under a matched output-generation budget. Input: Dead unit set U, arms A = {P, B, TC, CF-SF, GR-SHAPE}, rounds R = 4, frozen packet renderer, frozen decoding parameters Output: Per-unit and per-arm outcome unlock(u, a) 1: for all dead units u ∈ U do 2: for r = 1 to R do 3: retrieve the frozen round-r trajectory state for each arm 4: render πu,P,r from the P failing incumbent 5: render πu,B,r from the original task only 6: render πu,TC,r from at most six violated public I/O facts 7: render πu,CF-SF,r by copying the TC fact block and adding the source code on which those facts were executed render πu,GR-SHAPE,r by copying the CF-SF code block and replacing the fact block 8: with count-matched placebo bullets 9: verify mirror SHA equality and run the forbidden-token audit 10: for all arms a ∈ A do 11: derive fresh seed su,a,r from the dedicated namespace 12: ĉu,a,r ← decode(πu,a,r , su,a,r ) 13: execute ĉu,a,r on the public tier 14: if ĉu,a,r passes the public tier then 15: execute ĉu,a,r on the prompt-hidden tier 16: end if 17: record the candidate, scores and frozen transition state 18: end for 19: end for 20: for all arms a ∈ A do 21: unlock(u, a) ← 1[∃r ≤ R : ĉu,a,r passes both tiers] 22: end for 23: end for • C3′ (confirmatory secondary, content attribution): CF-SF > GR-SHAPE. Each claim was evaluated using an exact one-sided McNemar test (Fagerland, Lydersen, & Laake, 2013; McNemar, 1947). For the favored-only discordant count b01 and the reverse-only count b10 , the exact probability was computed as pexact =
b01 +b10 X k=b01
b01 + b10 k
1 b01 +b10 . 2
(2.1)
Here, b01 denotes units unlocked by the favored arm but not unlocked by the comparison arm. b10 denotes units unlocked only by the comparison arm. Concordant pairs do not contribute to the test statistic. Same-unit paired-binary evaluation allows the unique unlock pattern of each contrast to be measured instead of an aggregate pass-rate difference. Familywise error for the three p-values was controlled by Holm correction (Holm, 1979). For ordered values p(1) ≤ · · · ≤ p(m) , the first threshold violation is defined as o n 0.05 . (2.2) k = min i : p(i) > m−i+1 H(1) , . . . , H(k−1) are rejected, and all hypotheses are rejected if no violation is found. Here, m = 3 and the familywise significance level was fixed at 0.05. Under the frozen statistical rule, a positive family verdict required both co-primary claims to satisfy the Holm-adjusted p < 0.05 condition. If only one co-primary claim passed the threshold, the verdict was defined as NOT-CERTIFIED with a named split, and partial certification was not applied. Passing the threshold for C3′ allowed the antidote result to be described as “content-attributed.” Failure of C3′ alone did not block the co-primary statistical verdict. 10
The term “certified” was used only as the operational label of the pre-committed statistical decision rule. Preregistration was internal and version-control-timestamped. No registered report using OSF, AsPredicted, or in-principle acceptance was used. The statistical verdict and audit verdict were kept separate. If the frozen audit rule produced a failure, the original audit status was preserved as AUDIT-FAILURE even if the statistical thresholds had been passed. The amended verdict computed after the dated amendment was reported separately, and the original verdict was not deleted. Each contrast was paired through immutable task–cell keys. The evaluator executable asserted that the pooled n, b01 , and b10 values were equal to the sums of the six per-cell tables. This invariant was added after the pooling-collapse bug detected in the discovery stage. To make the effect direction visible beyond pooled counts, the Haldane–Anscombesmoothed pooled matched-pair odds ratio is reported as P6 b01,s + 0.5 d ORpool = Ps=1 . (2.3) 6 s=1 b10,s + 0.5 Here, b01,s and b10,s are the favored- and reverse-direction discordant counts within stratum s. This quantity is not a Cochran–Mantel–Haenszel (CMH) estimator. It is used as a pooled discordant odds ratio for descriptive effect-size summary, together with per-cell signs and guard checks. Two additional preregistered descriptive guards are applied to constrain claim wording to observed effects. The material-rescue guard requires the GR-SHAPE−P rescue to be less than half of the CF-SF − P rescue for content-attributed wording. The observed deployment-harm gate applies the condition b10 ≤ 0.02 (2.4) n separately to each co-primary contrast. This gate is based on observed reverse-harm share. It does not provide a confidence bound, non-inferiority proof, or deployment safety guarantee for the population-level harm rate. Therefore, the gate is interpreted only as a preregistered wording constraint. Power is computed before the run as exact-binomial conditional power based on discoverystage discordant counts: 1 − β = Pr[pexact (X, D − X) ≤ αHolm ] , X ∼ Binomial(D, π), (2.5) where D is the expected discordant-pair count, X is the favorable-direction discordant count, and π is the favorable share. The conservative first-step Holm threshold αHolm = 0.05/3 is used. Pre-run power is computed as 0.72 for C1′ , using D ≈ 38 and π = 0.737. For C2′ , power is computed as 0.83, using D ≈ 36 and π = 0.75. Power drops to the 0.16–0.38 range at half the discovery effect. Therefore, it is pre-committed that a non-significant but direction-preserving result is interpreted as “failed to confirm” rather than as “refuted.” The comparison of whether the tested arms exceeded blind resampling is not included in the frozen superiority family. Since equality or non-superiority cannot be certified as a positive result in a one-sided superiority test, the blind-resampling bound is reported with descriptive discordant counts. 2.2.2 Generation protocol and audit invariants Generation parameters were frozen before the run, as shown in Table 5. Each generation used 1,024 completion tokens, a 4,096-token context, one candidate per round, and temperature 0.8. Sampler settings were inherited unchanged from the discovery stage. The in-run prompt-token guard capped prompts at 3,072 tokens. The pre-run preflight excluded the unit whose round-0 prompt might exceed the effective 2,944-token bound. The primary sample preserved the single inherited exclusion from the predecessor experiment, and no additional unit was excluded during the current main run. Each generation is seeded through a dedicated namespace: s = sha256 antidote|⟨cell⟩|⟨task⟩|⟨arm⟩|⟨round⟩ mod 232 . (2.6) Bar-delimited fields keyed a single generation. The full seed payload was stored in each record. An executable audit verified that the 7,000 payloads in the main five-arm run were 11
Table 5: Frozen experimental configuration. Decoding parameters, arm set, sample sizes, and seed namespace were committed to version control before the main run began. The disjoint-60 sample was used only for sensitivity analysis and was not included in the primary statistical family. Parameter
Value
Cells Models
6 deepseek-coder:1.3b qwen2.5-coder:0.5b-instruct-fp16 qwen2.5-coder:1.5b 2 (HumanEval+, MBPP+) 4 5 (P, B, TC, CF-SF, GR-SHAPE) 290 1024 (protocol) 4096 (protocol) antidote| 290 × 5 × 4 = 5800 60 × 5 × 4 = 1200 7000 1,400
Benchmarks Rounds Arms n (fixed sample) num_predict num_ctx Seed namespace Total fixed-sample generations Disjoint-60 generations Main-run total generations Sixth-arm follow-up generations
unique and contained no foreign namespace prefix. A separate namespace was used for the sixth-arm follow-up. Generated candidate code was executed inside a sandbox with a 2 GiB address-space limit. An exclusive serving lock restricted generation requests to a single job at a time and reduced concurrent-serving interference. The audit layer was implemented not as an implementation detail, but as the falsification layer of the method. During the pre-run stage, per-cell render manifests, SHA-pinned round-0 prompts, constraint-render audits, and model digests were committed to version control. During evaluation, the following invariants were applied executably: transition completeness, exactly 5 arms × 4 rounds × included units, mirror equality, per-round source-code and constraint-block SHA equality, forbidden-token audit on all GR-SHAPE prompts, seed uniqueness and namespace isolation, provenance-based anti-reuse checks against discovery records, and the pooled-equals-cell-sum invariant. The full implementation was gated before the first token was produced by a falsification suite containing eight executable counterexamples. The suite covered mirror integrity, seed isolation, sample identity, prompt-hidden-test isolation, and hostile-output resilience conditions. Under the frozen rule, any audit failure assigns the original verdict as AUDIT-FAILURE. One audit episode was triggered during the main run and one during the sixth-arm follow-up. Original verdicts, forensic analyses, dated amendments, and amended verdicts are reported separately (§3.3, §3.2). Generation outputs, unit identities, and scores are not changed during amendments. Thus, failures of the pipeline itself are not deleted post hoc, and the claim-producing system is also subjected to critical testing. 2.2.3 Discovery and single-promotion rule The promoted hypotheses were derived from a previously run preregistered family in the same research program, named CONFIRM-TC. This family tested whether facts-only conditioning (TC) exceeded bare code (P) and produced a mixed result. The first claim passed the threshold with a +15 effect (Holm p = 0.0301). The corpse-mechanism claim reversed with −3 (p = 0.78), and the family did not receive a positive verdict. In the same run, two unplanned contrasts produced larger descriptive effects: B − P = +18 (p = 0.0025) and CF-SF − P = +18 (p = 0.0020). Of the ten possible undirected pairwise contrasts among the five arms, five were inspected. Under a worst-case Bonferroni correction applied to all ten contrasts, the values would remain approximately 0.025 and 0.020. Nevertheless, these observations are not counted as confirmatory evidence and are used only as screening signals that generate new hypotheses. The promotion rule was fixed before the new run began. Each descriptive effect was allowed to be promoted to a fresh preregistered family only once. Discovery outputs were 12
not re-included in the confirmation analysis. If the fresh test failed, second promotion was prohibited. This rule was applied to prevent a failed family from being protected from refutation through post hoc reinterpretation or repeated testing (Hofman et al., 2023; Mayo & Spanos, 2006; Popper, 1959, 1963). To keep the estimand on the same task population, task lists, cached pools, and incumbents were preserved. Seeds and generated tokens were completely renewed. Therefore, the procedure provided fresh-generation confirmation. It did not provide independent-unit or independent-population replication. 2.3
Post-primary external-controller follow-up instruments In addition to the primary preregistered pipeline, six external controllers that guide the frozen small model at inference time on the search-bound stratum are evaluated as prompt-only follow-up instruments. These instruments are not part of the primary frozen evaluation pipeline, are not preregistered, and are not subjected to that pipeline’s audit invariants. Their mechanisms are summarized here for method completeness, whereas their results are reported in Table 15. Each controller is compared with blind resampling (B) and its own content-free shape placebo under a matched output-generation budget (R=4). The shared suffixes in the names refer to joint-embedding predictive architecture (JEPA) and reinforcement learning (RL) components. The deployed arms are mostly deterministic fixed-rule variants rather than learned controllers. The learned stack is deferred. Only FJR-JEPA-RL deploys learned LinUCB selection online. Below, the deployed mechanism of each controller is summarized in a compact mathematical form derived from the source code. Components that were designed but not run in this study are marked as deferred. Common notation is defined as follows. A unit u = (b, m, τ ) denotes benchmark b, frozen model m, and task τ . Each arm runs R=4 rounds. The incumbent cu,r at round r is defined as the candidate with the highest public-pass count up to that point. For public test j ∈ {1, . . . , Ju }, with pass flag fj ∈ {0, 1}, the cached incumbent’s public-pass fraction and the search-bound stratum are defined as su q0 (u) = , SSB = u : 13 ≤ q0 (u) < 1 . (2.7) Ju P Here, su = j fj is the number of passed public assertions. True unlock Ya (u) is defined as at least one candidate generated under arm a passing both the public and prompt-hidden tiers in a round (Definition 2). FJR-, RIFT-, and DCH-JEPA-RL share a falsification-severity primitive: 1−p , p = P (e | b). (2.8) 1+p Here, e is an executed public counterexample. T (e) ∈ [0, 1] denotes the oracle-trust value associated with the evidence kind. T (e) = 1.0 was used for public assertions, T (e) = 0.95 for metamorphic evidence, and T (e) = 0 for model-generated evidence. p is the expectedness of the evidence under the public background. Surprising refutations (p → 0) receive maximum severity. All controllers operate on public-only state and read the hidden tier only during final true-unlock scoring. σ(e) = T (e) · 1[e refutes h] ·
FJR-JEPA-RL. FJR-JEPA-RL (Falsification-JEPA Reinforced Resynthesis) is used as a deployed learned controller. On public-only context x, one LinUCB ridge estimator is maintained for each arm in the action set A = {B, CE, MR, SPEC, DIV} and updated with the observed public reward r: Aa ← Aa + xx⊤ ,
ba ← ba + r x,
θ̂a = A−1 a ba .
(2.9)
The per-round action is selected as the argmax of an additive score. Ties are resolved with a deterministic sha256 tie-break: p c Q(s, a) = wq V (F (z, a)) − V (z) + α x⊤ A−1 a x + η ∆κ(a) (2.10) + wN N (a) − wanc AnchorRisk(a) − wC c(a). Here, F denotes the JEPA latent transition predictor, V the public-trained value proxy, c z the public-only latent state, and s the policy state. κ is the corroboration score. ∆κ(a) 13
defines the expected one-step corroboration gain of action a through the severity in Eq. (2.8). N denotes behavioural novelty, α the LinUCB exploration constant, and wq , η, wN , wanc , wC fixed score weights. The last two terms are anchoring-risk and compute-cost penalties. wanc corresponds to the rho field in the code. When the value head is unfit, V ≡ 0.5 is returned, so the JEPA term is exactly 0 at cold start. The hidden-label value head is offline-only and is not used in the online run. RIFT-JEPA-RL. RIFT-JEPA-RL (Refutation-Inversion Falsification Thought loop) is used as a deployed fixed-rule conservative gate. For ten symbolic assumption-inversion axes, severity is aggregated per axis from fired refuters, and a conservative threshold rule is applied: ρa =
max
σ(er ),
r: axis(r)=a
a⋆ = arg max ρa ,
fire a⋆ ⇐⇒ ρa⋆ > δ, otherwise B.
a
(2.11)
In the deployed M1 configuration, δ=0 is used. The recipe for a fired axis is produced by closed-form inversion of the refuted claim: H̄ = H ⊕ flip(c⋆ ). The M2 path, which extends the same gate to the form of a pessimistic Q− = Q̂ − β σQ with learned advantage and epistemic uncertainty, is left deferred. DCH-JEPA-RL. DCH-JEPA-RL (Dialectical Counter-Hypothesis JEPA-RL) is defined as a designed-only and unrealizable instrument. Falsified assumptions are designed to be converted into a version-space bank Ht consisting of K competing counter-hypothesis cards and are formulated to be updated by Popperian exponential elimination: X µt+1 (i) ∝ µt (i) exp −λ ℓ(Hi , e) , ℓ(Hi , e) = T (e) 1[Hi ̸|= e]. (2.12) e∈Et
Action selection is defined by a pessimistic lower-confidence advantage computed through an offline group-disjoint JEPA world model Fψ and hidden-transfer value head Vω : Q− (s, a) = Q̂(s, a) − β σ̂Q (s, a), at = arg max LCB(a) if max LCB(a) > δt , otherwise B.
(2.13)
a
a
b Here, LCB(a) = ∆(a) − β U (a) is defined as the lower-confidence bound. λ is the Popperian elimination temperature, β is the pessimism or lower-confidence coefficient, and U is total epistemic uncertainty. ℓ is the per-card elimination penalty and is distinct from RIFT’s ρa severity aggregate. This stack is not run. Because the required offline transition corpus is not available on disk, the controller cannot be instantiated. Therefore, Eq. (2.12)–(2.13) is classified entirely as designed-but-deferred. ECK-JEPA-RL. ECK-JEPA-RL (Executable Constraint Kernel JEPA-RL) is used as a deployed fixed-rule kernel. For a search-bound unit, the public assertions violated by the incumbent are accepted as an executable constraint kernel K only when they can be safely serialized into a Python literal and parsed with ast.parse: fired(K) = ∃j : ¬fj ∧ serializeOK(j) ∧ astParse block(K) , ( (2.14) ECK2 , fired(K) at = B, otherwise. Failing witnesses are ranked with the fixed 7-axis severity w⊤ ζj . Here, w = (1, 0.8, 0.7, 0.6, 0.4, 0.3, 0.3) is fixed. This code-free interface corresponds to the TC arm. The learned world-model/value/LCB stack is left deferred. Self-repair pilot. The self-repair pilot is defined as an instrument with no learned component. This instrument does not inject a new signal. The shared CONFIRM/antidote arms, namely B, CF-SF, GR-SHAPE, and TC, are reapplied in a live-confirmed search-bound sub-stratum. A unit is kept only if, when the cached incumbent is rerun, the public-pass fraction still remains inside the band: keep u ⇐⇒ m ∈ M ∧ 31 ≤ q̂0 (u) < 1, 14
q̂0 (u) =
npassed . ntotal
(2.15)
P For each arm, true unlock Ya (u) and pooled count Na = u Ya (u) are computed. Comparisons are made using the one-sided exact McNemar test in Eq. (2.1). In addition, a 50,000-replicate bootstrap risk-difference confidence interval clustered by (benchmark, task) is used. EOT-JEPA-RL. EOT-JEPA-RL (Execution-Outcome Trace JEPA-RL) is used as a deployed fixed-rule outcome kernel. For each public test, the candidate’s observed value oj is compared with the expected value yj , and a mechanical outcome trace is constructed: oj − y j , float (|oj − yj | ≤ atolj ⇒ pass) (missing, extra, first_diff, ∆ ), container len δj = d(oj , yj ) = (2.16) 0, o = y j j reprdiff(oj , yj ), otherwise. In the container case, the tuple returns the missing/extra/first_diff elements and the length difference ∆len = len(oj ) − len(yj ). atolj denotes the per-test absolute tolerance, and reprdiff denotes the difference at the repr-string level. Each failing outcome is scored with fixed weights: sevj = wcrash 1[exception ∨ timeout] + wtype 1[type(oj ) ̸= type(yj )] + wsmall 1[small literal] + wanchor 1[anchor].
(2.17)
Here, w = (4, 2, 1, 0.5) is used. sevj is an EOT-local score independent of the σ primitive in Eq. (2.8). The type term is evaluated only when an observed value is available. The most severe kF =3 failures (Ft ) and kA =2 anchors (At ) form a kernel. The deployed NOLEARN rule is |Ft | > 0 ∧ nexc /|Ft | ≥ 21 EXC, at = OBS_ANCHOR, At = ̸ ∅ OBS, otherwise,
at = B if fire(Kt ) = 0.
(2.18)
The rationale for EOT is defined as a placebo-distinctness hypothesis. According to this hypothesis, the observed content O carries repair-class information beyond the shape summary S(O), whereas the content-free placebo does not: I(Cu ; O | x, S(O)) > 0 and I(Cu ; Oshape | x, S(O)) = 0. This statement is used not as proof, but as a falsifiable hypothesis. The learned LCB/JEPA policy is left deferred. Each of these instruments is paired with its own content-free shape placebo: FJR-SHAPE, RIFT-SHAPE, ECK-SHAPE, GR-SHAPE and EOT-SHAPE. For EOT, an additional EOTDECOY control that corrupts the observed value is used. The corresponding verdicts are reported in Table 15.
3
Results
The experiments were conducted on HumanEval+ and MBPP+ dead task–cell units using three frozen open-weight code models served through Ollama (qwen2.5-coder:1.5b, qwen2.5-coder:0.5b-instruct, and deepseek-coder:1.3b). Each unit was evaluated under five regeneration arms with the same output-generation budget. Four rounds were run for each arm, and one candidate was generated in each round. Candidates were first scored by execution against the public test tier and, if this tier was passed, against the prompt-hidden EvalPlus tier. True unlock was defined as at least one candidate generated under the same arm passing both tiers together (Definition 2). Primary analyses were performed on the fixed sample of n = 290. Sensitivity analyses were conducted on 60 disjoint units that did not overlap with the primary sample. The main five-arm run was conducted on 350 units, formed by the union of the 290 primary units and 60 disjoint units. In total, 350 units × 5 arms × 4 rounds produced 7,000 generations, and 1,750 transition files were recorded. The primary statistical family is 15
Table 6: Preregistered family on n = 290 dead units. Pooled exact one-sided McNemar tests were evaluated within a three-claim Holm family. b01 denotes units unlocked only by the favored arm. b10 denotes units unlocked only by the comparison arm. Net effect was computed as b01 − b10 . The share column reports b01 /(b01 + b10 ) with the exact Clopper–Pearson 95% interval. The final column reports the Haldane–Anscombe-smoothed pooled matched-pair odds ratio. c pool Contrast b01 b10 Net p (raw) p (Holm) Verdict Share [95% CI] OR C1′ C2′ C3′
B>P CF-SF > P CF-SF > GR-SHAPE
7 +18 0.001051 0.002102 PASS 3 +18 0.0001386 0.0004157 PASS 7 +15 0.004065 0.004065 PASS
25 21 22
0.78 [0.60, 0.91] 0.88 [0.68, 0.97] 0.76 [0.56, 0.90]
3.40 6.14 3.00
Per-cell net effects (6 cells each) C1' B > P
C2' CF-SF > P
C3' CF-SF > GR-SHAPE
HE+ / qwen-1.5b
+2
+1
MBPP+ / qwen-1.5b
+2
+5
HE+ / qwen-0.5b-inst
+2
+5
MBPP+ / qwen-0.5b-inst
+4
HE+ / deepseek-1.3b
+4
-1
4
2
0
2
Net (McNemar b01-b10)
4
+5
+1
+2 +2
+0
+4
MBPP+ / deepseek-1.3b
+4
+4 6
4
2
0
2
Net (McNemar b01-b10)
4
+5 6
6
4
2
0
2
Net (McNemar b01-b10)
4
6
Figure 2: Per-cell net discordant effects for the three family claims across the six benchmark×model cells. The dashed line denotes zero. C1′ and C2′ are in the positive direction in all six cells. C3′ is positive in four of six cells, with a worst cell of −1.
based on 5,800 generations from the fixed sample. The remaining 1,200 generations were produced for the disjoint sensitivity sample. An additional 1,400 fresh generations were obtained in the preregistered sixth-arm follow-up. All pre-run manifests were committed before generation began. The anti-reuse audit episode triggered in the main run is reported in §3.3, and the lookup-scope episode identified during the sixth-arm follow-up is reported in §3.2. Generation outputs, unit identities, and scores were not changed during amendments. Original and amended audit verdicts were preserved separately. Unless the disjoint sample is explicitly specified, the findings below are reported on the fixed sample of n = 290. 3.1
Primary effects and observed blind baseline
All three preregistered claims passed the frozen statistical thresholds (Table 6 and Figure 2). For the trap hypothesis C1′ , blind resampling unlocked 25 units that were not unlocked by the bare-code arm. Seven units were observed in the reverse direction. Thus, a +18 net effect was obtained in favor of blind resampling (exact one-sided p = 0.00105, Holm-adjusted p = 0.0021). For the antidote hypothesis C2′ , a 21/3 discordant split and a +18 net effect were obtained in favor of the code-plus-facts packet (p = 0.000139, Holm-adjusted p = 0.00042). Because the two arms followed independent trajectories, this comparison is reported as an arm-level packet effect rather than as a same-source-code fact-addition effect. For the content-attribution claim C3′ , a 22/7 discordant split and a +15 net effect were observed (Holm-adjusted p = 0.0041). In this mirror comparison, source code, failure instruction, template, and bullet count were held fixed, while the fact block was changed between task-relevant facts and generic bullets. Because exact token length and lexical distribution were not equalized, this contrast is reported only as content-favoring componentlevel evidence under the matched dimensions. Under the frozen statistical decision rule, the family label was assigned as CERTIFIED, content-attributed. The audit verdict is reported separately from the statistical label in §3.3. The smoothed pooled matched-pair odds ratios computed with Equation (2.3) were 3.4 for C1′ , 6.1 for C2′ , and 3.0 for C3′ . The direction of the co-primary effects was preserved in all 16
Net effect reproduces at +18 on 100% fresh generations Discovery (CONFIRM) Confirmation (ANTIDOTE)
Net effect (b01 b10)
20
+18
+18
+18
+18
28/10
25/7
27/9
21/3
15
10
5
0
B P (trap)
CF-SF P (antidote)
Figure 3: Discovery and fresh-generation confirmation. Descriptive contrasts observed with discovery seeds and preregistered estimates obtained with 100% fresh confirmation seeds are shown on the same n = 290 units. Net effects were reproduced as +18 for both contrasts, although discordant compositions changed.
six cells. The preregistered cell guard was defined for C1′ and C2′ in each benchmark×model cell by the following predicate: a cell fails the guard if the net discordant difference is ≤ −2 or if the one-sided binomial p < 0.05 in the base-only harm direction. No cell triggered this guard. This result indicates that no conflict was found between the pooled direction and the cell-level signs. It does not imply that cell-level effect magnitudes were homogeneous or precisely estimated. Discovery-stage contrasts were reproduced with the same net magnitude on 100% fresh seeds and fresh generations. In the discovery run, B − P = +18 and CF-SF − P = +18 were observed. In the confirmation run, +18 was again obtained for both contrasts (Figure 3). However, discordant compositions changed from 28/10 to 25/7 for the trap contrast and from 27/9 to 21/3 for the antidote contrast. Thus, fresh-generation confirmation shows net-effect agreement on the same fixed sample, not reproduction of individual task outcomes (Gorman & Bedrick, 2019). The arm-level performance profile was summarized by pooled true-unlock counts (Figure 4 and Table 8). Blind resampling (B) and code-plus-facts (CF-SF) produced 26 unlocks each. Facts only (TC) produced 15, placebo (GR-SHAPE) produced 11, and bare code (P) produced 8 unlocks. The observed ordering was B = 26 = CF-SF > TC = 15 > GR-SHAPE = 11 > P = 8. The corresponding true-unlock rates were 9.0% for B and CF-SF, 5.2% for TC, 3.8% for GR-SHAPE, and 2.8% for P. These marginal counts summarize the general arm profile and are not used as substitutes for paired discordant contrasts. The generation-level public-pass profile was broadly consistent with the same ordering (Table 7). Of the 1,160 candidates generated per arm, 4.0% passed the public tier under CF-SF, 3.9% under B, 2.3% under TC, 1.9% under GR-SHAPE, and 1.8% under P. Public-to-true conversion was 26/42 = 61.9% for B, 26/35 = 74.3% for CF-SF, and 8/13 = 61.5% for P. Thus, although fewer units passed the public tier under CF-SF than 17
Table 7: Per-arm generation profile on the fixed sample (n = 290 units and 1,160 generations per arm). Prompt length is the cell-size-weighted pooled mean. Public-pass rate is reported as a generation-level proportion. Public and true unlocks are unit counts. The final column reports the first true-unlock round. Arm
Mean prompt chars Pass rate (%) Pub unlocks True unlocks (% of 290) Rounds r1/r2/r3/r4
P B TC CF-SF GR-SHAPE
1289.9 406.4 655.3 1482.9 1475.9
1.8 3.9 2.3 4.0 1.9
13 42 23 35 15
8 (2.8) 26 (9.0) 15 (5.2) 26 (9.0) 11 (3.8)
2/3/2/1 4/4/7/11 2/5/3/5 7/10/4/5 3/4/3/1
Pooled TRUE unlocks per arm (n=290 dead tasks) 30
26
26
TRUE unlocks
25 20
15
15
11 10
8
5 0
P
B
TC
CF-SF
Arm
GR-SHAPE
Figure 4: Pooled true unlocks across 290 dead units. Blind resampling (B) and code-plus-facts (CF-SF) tied with 26 unlocks. Facts only (TC) produced 15, placebo (GR-SHAPE) produced 11, and bare code (P) produced 8 unlocks.
Table 8: Per-cell true unlocks by arm and per-cell net discordants for the three family claims. The pooled row is exactly equal to the sums of the cell-level columns. The pooled-equals-cell-sum audit invariant was satisfied with zero mismatch.
Cell
n P B TC CF-SF GR-SHAPE C1′ C2′ C3′
HE+ / qwen-1.5b MBPP+ / qwen-1.5b HE+ / qwen-0.5b-inst MBPP+ / qwen-0.5b-inst HE+ / deepseek-1.3b MBPP+ / deepseek-1.3b
24 47 40 60 59 60
2 1 2 1 0 2
4 3 4 5 4 6
1 2 3 2 6 1
3 6 7 2 2 6
4 2 2 0 2 1
+2 +2 +2 +4 +4 +4
+1 +5 +5 +1 +2 +4
−1 +4 +5 +2 0 +5
under B, the true-unlock count preserved on the prompt-hidden tier was the same for the two arms. First true unlocks were distributed across all four rounds. Round counts were 7/10/4/5 under CF-SF and 4/4/7/11 under B. Because per-round success rates were low and counts were limited, these profiles are reported descriptively. In the observed comparison with blind resampling, the code-plus-facts arm tied at the pooled level. Both arms unlocked 26 of the 290 units. However, the unlocked unit sets were not the same. A symmetric 20/20 discordant split was observed for the CF-SF-only and 18
CF-SF B = 0: blind ceiling replicates (20/20 discordant split) 8 7
Discordant split CF-SF only b01=20 B only b10=20 pooled 20/20 cells on diagonal
MBPP+ / qwen-1.5b
6
CF-SF unlocks
HE+ / qwen-0.5b-inst MBPP+ / deepseek-1.3b
5 4 3
HE+ / qwen-1.5b
2
MBPP+ / qwen-0.5b-inst HE+ / deepseek-1.3b
1 0 0
1
2
3
4
5
B (blind resample) unlocks
6
7
8
Figure 5: Observed comparison with blind resampling. Per-cell true-unlock counts for CF-SF and B are heterogeneous around the identity line. Pooled over all cells, the two arms tied with 26 unlocks and produced a symmetric 20/20 discordant split. This result does not imply equivalence.
B-only directions (Figure 5). Thus, the observed pooled difference was zero, but unit-level agreement was not obtained. This result does not show equivalence, population-level non-inferiority, or identical behavior. It only shows that the pooled net difference was zero under the current sample and output-generation budget. Per-cell counts were heterogeneous. Blind resampling exceeded code-plus-facts by 1, 2, and 3 unlocks in three cells. Code-plus-facts exceeded blind resampling by 3 unlocks in two cells, and one cell tied. Thus, cell-level differences cancelled within the pooled average. The facts-only arm also remained below blind resampling: TC − B = −11 was obtained (12/23, descriptive one-sided p = 0.98). This descriptive comparison shows that facts-only conditioning did not produce additional pooled gain over blind resampling in the tested setting. 3.2
Attribution and sensitivity analyses
In the main trap comparison, failing code, failure instruction, and prompt length changed together. To test the independent contribution of the failure instruction, a sixth arm, BI (blind plus instruction), was run under a separate frozen preregistration. BI was constructed by adding only the failure instruction used in P to the B scaffold. No code, facts, or bullets were added. Mean prompt length was 406 characters for B, 478 for BI, and 1,290 for P. BI was run for four rounds on the same 350 units, producing 1,400 fresh generations under a dedicated namespace. Outcomes were paired at the unit level with the B and P outcomes frozen from the main run. Equality audits were applied to model digests, decoding parameters, and unit lists. The net-count decomposition was 19
Table 9: Standing descriptives and preregistered wording guards (n = 290 fixed sample). Descriptive rows are not included in the certified family. The blind-resampling row shows the observed pooled tie. The material-rescue guard compares placebo rescue with one half of antidote recovery. The observed-harm gate is a point-estimate wording rule applying the b10 /n ≤ 0.02 threshold and does not provide a safety proof or probabilistic guarantee. Item
b01
b10
Net
p (raw)
Verdict
CF-SF − B (observed pooled tie) TC − B GR-SHAPE − P Rescue guard: GR-SHAPE-P vs (CF-SF-P)/2 Deployment gate C1’ Deployment gate C2’
20 12 11 – – –
20 23 8 – 7 3
0 −11 +3 +3 vs 9 7/290 3/290
0.5627 0.9795 0.3238 – 0.0241 0.0103
– – – PASS FAIL PASS
Table 10: Sixth-arm family on the same n = 290 units. Pooled exact one-sided McNemar tests were evaluated within a two-claim Holm family. BI generations are fresh, whereas B and P outcomes are frozen from the main run. By construction, the net differences sum to +3 + 15 = +18.
Contrast ′
D1 D2′
B > BI BI > P
b01 b10 18 19
Net p (raw) p (Holm) Verdict
15 +3 4 +15
0.3642 0.0013
0.3642 FAIL 0.002599 PASS
c pool Share [95% CI] OR 0.55 [0.36, 0.72] 0.83 [0.61, 0.95]
1.19 4.33
(B − BI) + (BI − P) = (B − P) = +18. This equality is reported not as a substantive finding, but as an algebraic bookkeeping identity showing that the decomposition was correctly constructed on the same units. No statistically detectable effect was found for the failure instruction. B exceeded BI by +3 (D1′ : 18/15, p = 0.36). This result does not prove that the failure instruction has no effect. It only shows that no detectable instruction-only effect was observed under the current sample and design. BI exceeded P by +15 (D2′ : 19/4, Holm-adjusted p = 0.0026). The direction was positive in all six cells, and the smoothed pooled matched-pair odds ratio was computed as 4.33. BI produced 23 total true unlocks and fell between B and P in the pooled ordering. The preregistered diagnostic follow-up did not detect a statistically distinguishable instruction-only effect in this sample. The BI–P contrast is reported as the observed diagnostic difference between the code-bearing P arm and the code-free instruction arm. Because code content remained confounded with additional prompt length, no pure code-content causation claim is made (§4.2). In this follow-up, only BI generations were freshly produced, whereas B and P outcomes were reused in frozen form from the main run. Therefore, this analysis is a diagnostic decomposition of the bare-code contrast, not a fully fresh three-arm confirmatory experiment. The main and follow-up families were evaluated under separate Holm corrections. When a conservative Bonferroni correction was applied to all five confirmatory tests across the two families, C1′ , C2′ , C3′ , and D2′ remained significant, whereas the intended-null D1′ remained non-significant. In the content-attribution and sensitivity analysis, the CF-SF–GR-SHAPE mirror contrast used the same source code, failure instruction, template, and bullet count. Executed facts were replaced with generic requirement bullets. On the primary sample, CF-SF−GR-SHAPE = +15 was obtained (Holm-adjusted p = 0.0041). The descriptive rescue of the placebo over the bare-code arm was +3 (11/8, p = 0.32). This value was below the preregistered materialrescue threshold of +9. This result shows that observed recovery on the primary sample was in the direction of task-relevant fact content under the matched dimensions. Because exact token length and lexical distribution were not equalized, attribution is limited to the matched dimensions defined in the Methods section. The empty-fact exposure audit showed that, for TC and CF-SF in the primary sample, round-0 per-unit fact counts were distributed between 1 and 6, and that all 290 units received at least one genuine executed fact at round-0 entry (Table 11). The rendered violation block is round-specific and is derived from the incumbent trajectory of each round. Per-round 20
Table 11: Empty-fact exposure. Round-0 rendered violation fact counts for the facts-only (TC) and code-plus-facts (CF-SF) arms across the 290 primary units. The rendered violation block is round-specific, derived from each round’s incumbent trajectory. Per-round SHA-256 equality holds across all four rounds for 250 of 290 units and varies for the remaining 40 (Round-identical column). No primary unit received zero facts at round-0 entry, although a later round may render zero facts once a candidate stops violating the public tier.
Arm
0
1
2
3
4
5
6
Zero-fact
Round-identical
TC CF-SF
0 0
43 43
44 44
122 122
23 23
15 15
43 43
0 (0.0%) 0 (0.0%)
250/290 250/290
SHA-256 equality was preserved across all four rounds for 250 of the 290 units and varied for the remaining 40 units. Because no unit received a zero-fact prompt at round-0 entry, the post hoc C3′ sensitivity on units receiving at least one genuine fact is identical to the primary C3′ sample. This analysis is post hoc and does not generate a new confirmatory claim. The disjoint 60-unit sensitivity sample produced a weaker and mixed pattern. Co-primary directions were preserved, but net magnitudes fell to +1. For C1′ , a 2/1 discordant split was observed, and for C2′ , a 3/2 discordant split was observed. The content-attribution contrast reversed with −2 (1/3). On the same sample, sixth-arm comparisons produced +4 (6/2) for the code-bearingcontext component and −3 (2/5) for the instruction comparison. The disjoint sample was small, drawn by seeded random sampling from unselected eligible units separate from the primary sample, and underpowered by design. It is therefore not interpreted as independent confirmation. The direction change in C3′ is nevertheless recorded as a sensitivity result. The observed-harm gate was applied separately to the two co-primary comparisons. For C2′ , the reverse-harm share was 3/290 = 0.0103 ≤ 0.02. For C1′ , it was 7/290 = 0.0241 > 0.02. Accordingly, the preregistered wording rule permitted bounded wording for the codeplus-facts packet relative to the bare-code packet. Wording that universally prefers blind resampling to bare-code retry was not supported. This gate is based on a point estimate. It does not provide a confidence bound showing that the population-level harm rate is below 2%, nor does it provide a non-inferiority result or a deployment safety guarantee. Practical wording is limited to the tested sample, observed contrasts, and preregistered threshold. Because the same benchmark task could appear in multiple model cells, a post hoc clustered sensitivity analysis was conducted on the primary sample. This analysis does not modify the preregistered exact McNemar family and does not define a new confirmatory family. The cluster key was taken as benchmark|task, and the model was not included in the cluster key. Thus, observations of the same task under different models were kept within the same cluster, and the 290 primary units corresponded to 202 task clusters. This valid count of 202 task clusters is numerically identical to, but conceptually different from, the erroneous 202-row key collapse reported in the audit trail, where 290 units were merged across cells by bare task keys. For each contrast, the paired absolute risk difference was computed in percentage points. Benchmark composition was preserved by resampling clusters within benchmark, and a 50,000-replicate cluster bootstrap was used to produce percentile 95% confidence intervals (CIs), with deterministic seed sha256(task-cluster-bootstrap-v1). Results are given in Table 12. The 95% CIs for the B−P, CF-SF−P, and CF-SF−GR-SHAPE contrasts excluded zero. The interval for the CF-SF−B (blind) contrast included zero. Because discordant pairs did not cluster jointly across model cells, the clustered intervals resembled the unit-level pattern and did not change the qualitative ordering of the four contrasts. Since this analysis is post hoc, it does not re-establish the nominal error rate of the preregistered tests. 21
Table 12: Post hoc task-cluster sensitivity analysis (does not modify the preregistered family). Cluster key: benchmark|task. 50,000 cluster-bootstrap replicates were resampled within benchmark. Deterministic seed: sha256(task-cluster-bootstrap-v1). Risk difference is reported in percentage points. Values were computed from committed primary result manifests using analysis/task_cluster_effects.py and independently verified as float-identical using analysis/verify_clustered_effects.py.
Contrast
Net
RD (pp)
95% cluster CI
B−P CF-SF − P CF-SF − GR-SHAPE CF-SF − B
+18 +18 +15 +0
+6.21 +6.21 +5.17 +0.00
[+2.48, +10.00] [+3.06, +9.57] [+1.68, +8.81] [−4.40, +4.39]
In a post hoc q0 stratification, the dead pool was divided into four bins according to q0 , the ratio of the number of public assertions passed by the cached incumbent to the total number of public assertions: q0 =0 (n=159), 0<q0 <1/3 (n=22), 1/3≤q0 <2/3 (n=51), and 2/3≤q0 <1 (n=58). The total remained n = 290. This stratification was not preregistered and does not generate a new confirmatory claim. In the q0 =0 stratum, the descriptive difference between CF-SF and B was −8. Discordant counts are available in the committed manifests. In the search-bound stratum (1/3≤q0 <1, n=109), the cached CF-SF−B difference was +10 (p=0.0154). The prospective test of this post hoc discovery slice is reported in §4.1. These values were derived from committed result manifests, and no generation output or score was changed. 3.3
Audit outcomes The sixth-arm evaluator initially produced AUDIT-FAILURE. A total of 120 missingbaseline errors corresponded to the crossing of 60 disjoint units with two baseline arms. Before any code change was made, the forensic audit showed that baseline records were present but that the loader read only the fixed-sample collection. A dated amendment expanded the lookup scope and preserved the fatal predicate for genuinely absent units. The amendment was gated by two executable counterexamples requiring that the disjoint baseline be resolved and that an intentionally deleted baseline produce a fatal failure. Original and amended verdicts were reported together. The as-frozen evaluator for the main run initially produced AUDIT-FAILURE. The anti-reuse invariant defined to detect accidental reuse of discovery outputs was triggered on 28 distinct (output SHA-256, prompt SHA-256) pairs that were also present in discovery records. These pairs were observed in a total of 34 confirmation records. Before any code change was made, the forensic audit found no seed-payload collision. Records with output equality had distinct namespaces and distinct seeds. Prompt equality was confirmed to be expected by design. Because the same task lists, cached pools, and incumbents were preserved, some round-0 prompts were byte-identical. Collision incidence was 34 records out of 7,000, approximately 0.5%. The arm distribution of colliding records was P = 17, CF-SF = 15, TC = 2, B = 0, and GR-SHAPE = 0. This distribution was reported descriptively and was not used for causal or directional interpretation. Because provenance records were distinct, output-hash equality alone was not accepted as evidence of cross-run reuse. With the dated amendment, output-hash equality under intentionally identical prompts was removed as a fatal predicate and retained as a counted diagnostic. Seed-payload collision, namespace reuse, and provenance inconsistency were retained as fatal predicates. The amendment was gated by five executable checks, including a counterexample requiring that a doctored seed-copy remain a fatal failure and a full real-data re-evaluation. The official amended verdict was computed only after all checks had been completed. Generation data, outputs, and scores were not changed. Thus, the assumptions of the evaluation pipeline itself were also left open to executable criticism, and the original failure was not removed post hoc from the record.
22
Table 13: Verdict status by layer. Statistical verdicts and audit verdicts are reported separately. Original as-frozen AUDIT-FAILURE statuses are preserved. Dated amendments do not have the same evidential weight as untouched preregistration.
Layer
As-frozen sta- Forensic finding tus
Primary statisti- CERTIFIED cal family (C1′ , C2′ , C3′ ) Main-run audit
4
AUDITFAILURE
Dated amend- Postment amendment status
Co-primary direction None required held in all six cells. No cell guard triggered
CERTIFIED (preregistered exact McNemar)
Anti-reuse invariant Output-pair CERTIFIED fired on 34 records predicate (suite 5/5) / 28 distinct (output, reclassified prompt) SHA pairs. as diagnostic. Forensic audit found Seed-payload zero seed-payload col- collision kept lision fatal
Diagnostic follow- CERTIFIED up (statistical)
D1′ instruction-only None required CERTIFIED +3, p = 0.36. D2′ (two-claim Holm) +15, p = 0.0026
Diagnostic follow- AUDITup (audit) FAILURE
120 missing-baseline errors (loader read only the fixed-sample collection)
Lookup CERTIFIED widened; (two added missingcounterexambaseline kept ples) fatal for genuinely absent units
Discussion
The results obtained in this study answer a narrower and more fundamental question than the extent to which self-repair feedback is useful for small frozen code models: which kind of information inside the feedback packet carries operational signal, and whether that signal produces value beyond blind resampling. Three main patterns emerged jointly. First, the bare-code retry arm remained below blind resampling. Second, the code-plus-facts packet recovered the bare-code deficit at the pooled level, and the same-code placebo contrast favored task-relevant fact content on the primary sample. Third, this recovery did not exceed blind resampling, and code-plus-facts and blind resampling produced zero observed pooled difference in the tested sample. Therefore, in the tested regime, the observed function of feedback design was not to produce additional pooled unlocks over blind resampling, but to close the pooled unlock deficit induced by the code-bearing packet. This pattern shows that self-repair cannot be interpreted simply as “the model can correct itself if it sees its own error.” Re-showing bare failing code does not provide new external information about the candidate program. Instead, the previous failed solution is moved back into the conditioning context. This operation may pull the model’s sampling distribution toward the local trajectory of the failed solution and may narrow the exploration of the alternative solution space. The preregistered sixth-arm follow-up narrows this interpretation. Adding the failure instruction without code did not produce a statistically detectable difference. The remaining difference was associated with code-bearing context. However, because code content was not separated from intrinsic prompt length, it is not claimed that failing code alone produced harm. The more limited inference supported by the data is that the bare code-bearing packet produced lower outcomes than blind resampling and that this difference was not explained by an instruction-only manipulation. One plausible explanation for this pattern is anchoring. A concrete but wrong artifact may pull the generation distribution toward the local neighborhood of the failed solution. Concrete references have long been known to produce anchoring in human judgment (Tversky & Kahneman, 1974). In code models, buggy context has been shown to reduce pass rates (Dinh et al., 2023), and a related-but-wrong function has been shown to anchor generated 23
outputs in a way that cannot be explained only by verbatim copying (Jones & Steinhardt, 2022). Self-preference effects observed in large language model (LLM) evaluators are also consistent with the possibility that a model may assign disproportionate weight to its own outputs (Panickssery, Bowman, & Feng, 2024). Nevertheless, the present experiment does not measure token-level or activation-level mechanisms. Irrelevant-context distraction, effectiveattention reduction, prompt-length burden, and trajectory persistence could produce the same arm-level pattern. Anchoring is therefore presented not as a confirmed mechanism, but as a parsimonious explanation consistent with the measured contrasts. Executed facts have a different epistemic status. Violation facts are derived not from the model’s verbal self-critique or from its introspective judgment about its own program, but from the execution of candidate code on public tests. Thus, checker-grounded information, rather than a model-internal assertion, is added to the feedback packet (Huang et al., 2024; Olausson et al., 2024). It has also been reported that verified test-case content can be useful for code generation (Rizqullah & Albassam, 2026) and that usefulness is associated more with verification status than with mere text presence (Zhang, Wang, Xia, Wang, & Li, 2023). The same-code placebo contrast on the primary sample supports this distinction as well. Executed facts produce higher outcomes than generic companion text. The safest interpretation of this result is that the recovery is not explained only by template, bullet count, or companion-text presence. Rather, under the matched dimensions, recovery is associated with task-relevant and execution-grounded content. Because exact lexical distribution and token length are not equalized, all surface-form explanations are not ruled out. In addition, the direction change of the content-attribution contrast in the disjoint sensitivity sample limits the external stability of this interpretation. This distinction carries a direct philosophical consequence for self-repair in LLMs. The production of a new natural-language critique by an LLM about its own output does not by itself imply that a strong Popperian refutation has been produced. Such a critique may be another text artifact generated from the linguistic distribution of the same model. By contrast, an execution-derived violation is an external comparison that tests the behavioral claim of the program under an oracle independent of the model. In this study, the generated program is treated as a tentative conjecture, and the executed test violation is treated as an oracle-relative and executable counterexample that refutes a specific behavioral consequence of that conjecture (Popper, 1959, 1963). The useful signal is obtained not from the model’s use of Popperian vocabulary, not from the model’s verbal critique of its own output, and not from re-exposure to failing code, but from comparing the conjecture with failure facts revealed by execution. Thus, in the tested regime, falsification functions not as exposure, but as execution-grounded criticism. At the same time, the Popperian framing should not be interpreted here as an instruction that improves model reasoning through vocabulary alone. The observed effect belongs to executed facts. The Popperian contribution appears more at the level of measurement architecture. Hypotheses are bounded in advance, the verdict implied by unfavorable outcomes is specified, discovery outputs are not reused as confirmation evidence, and the evaluation pipeline itself is kept open to executable audit invariants. In this sense, the Popperian method functions not as a reasoning capability possessed by the model, but as a critical-control discipline applied to model outputs and to the researcher claims produced about those outputs. The methodological originality lies precisely in this point. Not only the program conjecture, but also the researcher conjecture that feedback content is useful, is made falsifiable through placebo controls, a matched-budget baseline, fresh-generation confirmation, and audit counterexamples. This two-level structure has direct consequences for LLM evaluation studies. Since LLM outputs are stochastic, prompt-sensitive, and sensitive to surface form, apparent improvement may often arise not from content but from prompt length, template, formatting, retry instruction, or an additional sampling opportunity. Therefore, the apparent success of a feedback mechanism does not show that it carries epistemically meaningful content. Falsification is not reduced here to the question of whether the model’s answer passed the test. The questions of which component the improvement can be attributed to, and under which counterfactual controls the researcher’s attribution claim survives, are also made part of the measurement. For this reason, packet decomposition, placebo mirroring, and matched output-generation budget are not merely engineering controls. They function as
24
methodological falsification tools that reduce the distance between claim and evidence in LLM evaluation. The observed comparison with blind resampling further bounds this interpretation. Codeplus-facts and blind resampling reached the same pooled unlock total, but they succeeded on different units. The observed tie showed zero observed pooled difference in the tested sample. It does not imply equivalence, population-level non-inferiority, or identical behavior. Nevertheless, none of the tested feedback arms exceeded blind resampling under the current sample and matched output-generation budget. Code-plus-facts closed the pooled unlock deficit relative to the bare-code packet, but did not produce additional pooled gain over blind resampling. This result shows that improvement over the previous completion does not imply superiority over fresh sampling under the same budget. In the dead-task regime, the experiment directly measured outcome differences between conditioning packets. By contrast, the latent generative mechanism that produces these differences was not identified. In the present design, the tested feedback packets did not produce observed pooled superiority over blind resampling (Brown et al., 2024; Snell et al., 2025). This result makes no claim about the model distribution or latent support. Whether correct programs exist within the same model distribution, or which arms would have an advantage under larger sampling budgets, is not the estimand of this study. What is measured is the paired effect of specific feedback packets on observed true-unlock outcomes under a finite zero-pass-in-pool regime and a fixed output-generation budget. This finding has a broader methodological implication for how inference-time improvement claims should be evaluated. Methods that exceed a blind baseline may need to modify the search or generation process more substantively, rather than merely reconditioning on a spent candidate. Search over diverse plans (E. Wang et al., 2025), functional decomposition (J. Chen et al., 2024), decoding guidance with live execution traces (Lavon, Katz, & Wolf, 2025), and inference-aware retraining (Chow et al., 2025) are examples in this direction. However, prompt-only variants of this class are prospectively tested on the search-bound stratum and remain null (Table 15). Therefore, under the current evidence, the operational implication is that every richer feedback claim should be evaluated against blind resampling under an equal output-generation budget before it is advanced. When read together with the previous two stages of the falsification-centered measurement program, a broader signal boundary emerges. In the first stage of the program, Popperian procedural content was shown not to provide a separable execution-correctness contribution over a labels-only scaffold, and the measured improvement was associated with scaffold structure (İşcan, 2026a). In the same study, a same-model Popperian self-judge did not exceed random selection. In the second stage of the program, semantic post-hoc falsification operators, including selection, verification, repair, elimination, and generation conditioning, were compared against Best-of-N under a deterministic execution oracle and a leakage-free matched-budget protocol (İşcan, 2026b). No semantic output-space operator exceeded held-out accuracy over Best-of-N in the tested cells. Measured gains were obtained along axes outside the semantic reasoning space, such as expression-layer recovery and compute scheduling. The present study examined the falsification signal within iterative feedback content. This result does not merely repeat the previous negative findings. Execution-grounded facts produced measurable recovery relative to a harmful code-bearing packet. However, this recovery did not produce additional pooled unlocks over blind resampling in the tested sample. Together, the three stages of the program support the following distinction: Popperian vocabulary, model-internal self-judgment, and semantic post-hoc reasoning did not carry a separable accuracy signal in the tested settings. Externally executed counterevidence closed the pooled unlock gap associated with the code-bearing packet, but did not produce additional pooled unlocks over blind resampling. This distinction is central for the epistemology of feedback in LLMs. A critique asserted by the model and a counterexample produced by an execution oracle are not the same type of signal. A model-internal critique may be linguistically plausible, but it can remain bound to the distributional regularities of the same model. An execution-derived counterexample compares the model’s claim with a computational procedure outside the model. Therefore, epistemic provenance is not merely metadata. The source from which feedback is produced, the oracle under which it is generated, the auxiliary assumptions on which it depends, and the placebo controls through which it survives are part of the interpretation of the outcome. 25
In the tested regime, the useful channel is not model-internal reflection, but externally grounded error information. Table 14 positions this contribution against prior work through experimental controls rather than terminology. The self-repair literature has shown that execution feedback can be useful on capable or specially trained models (X. Chen et al., 2024; Madaan et al., 2023; T. Zheng et al., 2024; Zhong et al., 2024). However, because failing code, evidence, and instruction are usually bundled into a single packet, exposure and evidence cannot be separated. Intrinsic self-correction critiques have shown that revision can reduce performance when no external signal is available (Huang et al., 2024; Stechly et al., 2025; Valmeekam et al., 2023). In these studies, however, the feedback packet is often removed entirely, so the distinction between the revision act and prompt content remains unresolved. In the present study, blind re-generation and bare-code retry are evaluated as separate arms, and lower performance is shown to be associated more with the code-bearing packet than with the revision instruction. The matched-budget literature has defined the correct operational baseline and has shown that repair can be dominated by repeated sampling (Brown et al., 2024; Olausson et al., 2024; Snell et al., 2025; K. Zheng et al., 2025). However, packet ingredients are usually not decomposed with a placebo control within the same experiment. In this study, same-unit pairing, equal output-sample counts, prompt-hidden scoring, fresh-generation confirmation, and mirror placebo are combined within a single instrument. The breadth of the design is deliberately limited: HumanEval+ and MBPP+, three small models, and a single prompt scaffold are used. In return, packet components, the operational baseline, and the content placebo are compared under controlled conditions. This trade-off strengthens internal attribution while limiting external generalization. The closest numerical precedents are directionally consistent with the present findings. The prior result that buggy-context completions fall below no-context completions (Dinh et al., 2023) supports the B > P direction. In the present design, the model’s own failing code was evaluated against a blind operational baseline on the same units. The result that expert-human feedback improves repair under a matched budget (Olausson et al., 2024) is consistent with the direction that external evidence can carry value. Findings that multi-turn feedback can produce small gains or negative outcomes (K. Zheng et al., 2025) indicate that feedback quantity and useful content must be separated. The distinct contribution of the present study is the attempt to separate task-relevant content from matched packet structure through a same-code placebo contrast. Three bounded practical implications follow from the findings. First, bare failing code should not be treated as a reliable default feedback packet. In the primary sample, P produced the lowest pooled unlock count among the tested arms and was exceeded by blind resampling by +18 net unlocks. However, because the observed-harm gate is not satisfied for the blind resampling comparison, a universal rule of “always discard the code” is not supported. The supported conclusion is that bare-code retry does not provide reliable superiority over blind resampling and produces lower pooled outcomes in the current sample. Second, if failing-code retention is unavoidable, adding executed public-test facts appeared to be the strongest feedback intervention among the tested options. The code-plus-facts arm recovered the bare-code deficit at the pooled level and satisfied the observed-harm gate. The same-code placebo contrast showed that generic diagnostic bullets did not substitute for executed evidence. This recommendation should be bounded to settings in which source code is already included in the packet, a public checker is available, and the current model-task regime applies. Third, every feedback design should be benchmarked against blind resampling under an equal output-generation budget. Improvement over the previous completion does not imply superiority over fresh sampling. In the present experiment, executed facts recovered the bare-code deficit, but did not exceed the blind baseline. These implications should not be interpreted as a deployment safety guarantee. The observed-harm gate is based on a point estimate and does not include a confidence-bound or non-inferiority analysis. In addition, the CF-SF–P comparison is a packet-level contrast involving independent trajectories. Therefore, practical wording should not state that facts are definitively safe. It should instead state that, in the tested sample, the code-plus-facts packet produces higher outcomes than the bare-code packet and ties with blind resampling at the pooled level.
26
Table 14: Principal experimental controls in the closest prior work and in this study. “Matched output budget” means that the same output-generation count was allocated to feedback and sampling conditions. “Placebo” denotes a template-matched packet that contains no task identifiers or executed facts. “Fresh generations” means that newly seeded outputs were used outside discovery outputs. Decomp. Matched Hidden Fresh feedback output budget Placebo test generations Advantage / limitation
Work Self-Refine (Madaan et al., 2023) Self-Debugging (X. Chen et al., 2024) LDB (Zhong et al., 2024)
—
—
—
—
—
Broad task transfer, feedback fully bundled
partial
—
—
—
—
—
—
—
—
—
OpenCodeInterpreter (T. Zheng et al., 2024) CYCLE (Ding et al., 2024)
—
—
—
—
—
Execution traces help, no placebo or budget match Step-level debugging, capable models and bundled packet Multi-turn gains, trained model and undecomposed packet
—
—
—
—
—
DebugBench (Tian et al., 2024)
—
—
—
—
—
Huang et al. (2024)
—
✓
—
—
—
partial
✓
—
—
—
—
✓
—
✓
—
İşcan (2026a)
partial
✓
✓
✓
—
İşcan (2026b)
—
✓
partial
✓
—
This study
✓
✓
✓
✓
✓
Olausson et al. (2024) Brown et al. (2024)
4.1
Improves small models by training, no inference-time isolation Broad debugging benchmark, feedback not decomposed Matched-budget critique, packet removed rather than decomposed Honest budget accounting, bundle compared with baseline Coverage-scaling baseline, no feedback decomposition Labels-only and placebo disambiguation, procedural content does not separate Leakage-free post-hoc evaluation, no semantic operator beats BoN Mirror content contrast and fresh-output confirmation, limited scale, language and scaffold
Post-primary external-controller follow-ups
After completion of the primary analysis, prospective external-controller experiments were evaluated on the search-bound stratum to examine whether the primary finding would break under a more favorable subset and richer feedback channels. These experiments involve prompt-only controllers carrying joint-embedding predictive architecture (JEPA), reinforcement learning (RL), and linear upper-confidence-bound (LinUCB) components. The mechanisms of the instruments are defined in §2.3. The post hoc q0 stratification (§3.2) showed that the code-bearing packet produced its highest relative recovery in the search-bound regime. This observation raises the question of whether external controllers operating on the frozen small model could exceed blind resampling in that stratum. Six controller experiments were conducted after the primary 27
Table 15: Six external-controller experiments conducted on the search-bound stratum (1/3≤q0 <1) after completion of the primary preregistered analysis. All use matched output budget R=4 and frozen 0.5–1.5B Ollama models. The endpoint is a fresh candidate passing both the public and prompt-hidden tiers (true unlock). These experiments were not preregistered and were not subjected to the audit invariants of the primary frozen evaluation pipeline. n denotes the number of liveconfirmed search-bound units or the specified cell count. Abbreviations: RD, risk difference; pp, percentage points. Controller
Mechanism / injected signal
n
vs B (net, p)
vs placebo (net, Verdict / kill-cause p)
FJRJEPA-RL
LinUCB + JEPA 46 over prompt subactions (learned sub-action selection)
+3 (p=0.274)
−1 (p=0.773)
NULL — shape placebo reproduced the movement; gain is prompt FORM
RIFTJEPA-RL
Symbolic assump- — tion inversion (10 axes)
—
—
NULL — detector fired on 0/24 live failing candidates (≡B); green fixtures ̸= real coverage
DCHJEPA-RL
Counter— hypothesis + offline groupdisjoint warm-start
—
—
UNREALIZABLE — required offline transition corpus absent on disk
ECKJEPA-RL
Executable con- 109 TC>B straint kernel (p=0.2905) (code-free interface ≈TC)
+3 —
UNDERPOWERED — n<160 (own power bar); learned stack deferred
Self-repair pilot
CF-SF: own near- 69 miss code + executable public failing assertions
+6 (p=0.073); vs GR-SHAPE: +1 NULL — cached RD +8.7 pp (p=0.50); 0/69 +10/ + 11 did not [−1.4, +18.5] leak replicate prospectively
EOTJEPA-RL
Execution69 outcome trace (runtime observedvs-expected, exceptions, deltas)
+3 (p=0.274)
vs SHAPE: +1 NULL — near(p=0.50) identical to the self-repair pilot
study to test this question (Table 15). None produced a controls-surviving gain. These experiments are non-preregistered internal follow-ups and are not cited as a separate reference. Two experiments failed at the level of realizability before effect measurement. RIFTJEPA-RL used a detector operating over ten symbolic inversion axes. Although 19 modules and 136 test green fixtures were produced, no axis was triggered in any of 24 distinct failing candidates in the live data (fire-rate 0/24). Therefore, the controller remained observationally equivalent to B. This result shows that green fixtures do not guarantee real live-data coverage. DCH-JEPA-RL could not be instantiated because the required offline transition corpus was not available on disk. Thus, it was shown that offline warm-start dependencies must be verified before the run. Two experiments produced null or underpowered results. FJR-JEPA-RL (LinUCB + JEPA over prompt sub-actions) produced FJR>B = +3 (p=0.274) in two cells (HumanEval+/qwen2.5-coder-1.5b and 0.5b, n=46). However, the shape placebo reproduced the movement with FJR>FJR-SHAPE = −1 (p=0.773). This pattern suggests that the observed movement may be associated more with prompt form than with learned content. The executable constraint kernel of ECK-JEPA-RL showed interface liveness with a 1.0 fire-rate. However, the search-bound cached stratum (n=109) remained below the n≥160 power bar from the controller’s own pre-run power calculation. Since the code-free interface corresponds to TC, search-bound TC>B was limited to +3 (p=0.2905), and the learned stack was deferred. 28
The content channels carrying the model’s own near-miss code and its own runtime behaviour were tested directly and prospectively. The self-repair pilot (CF-SF: the model’s own near-miss code + executable public failing assertions) was run on 69 live-confirmed search-bound units with 1,104 fresh generations. CF-SF>B was found to be +6 (p=0.073, cluster-bootstrap risk difference (RD) +8.7 percentage points (pp), 95% confidence interval (CI) [−1.4, +18.5] pp, including zero). However, the decisive control CF-SF>GR-SHAPE was only +1 (p=0.50). The placebo was verified to be content-free (0/69 leakage). The cached post hoc +10/ + 11 discovery was not prospectively replicated. This result indicates that the discovery may have been a small-n discovery artifact. EOT-JEPA-RL injected the candidate’s actual runtime behaviour as an executable trace. This channel includes observed-versus-expected values, exceptions, and deltas, and constitutes a richer content source beyond raw exposure and static tests. On the same 69 units and 1,104 generations, EOT-NOLEARN>B was +3 (p=0.274), and the decisive contrast EOT-NOLEARN>EOT-SHAPE was +1 (p=0.50). At the full-public-pass level, the two arms remained equal (13=13, n=69). This pattern produced a result almost identical to the self-repair pilot. These follow-ups strengthen the philosophical scope of the primary conclusion but do not change its evidential status. The two strongest content channels in the program, own near-miss code and own runtime behaviour, prospectively tie with content-free shape placebo in the most favorable search-bound stratum. Together with the capacity wall in the q0 =0 stratum, this result points to a practical boundary for gains obtainable through inference-time content manipulation on a frozen small model. These prospective null results are consistent with the primary finding: the observed CF-SF=B tie and the B>P trap in the primary family appear consistent with a form-not-content ceiling. Exceeding this boundary may require inference-aware retraining or search-space restructuring rather than frozen-model inference-time manipulation (Chow et al., 2025; Ding et al., 2024). 4.2
Validity boundaries and future work
The validity boundaries of the study should be evaluated across construct, conclusion, internal, and external validity. Construct validity is strengthened by the execution endpoint requiring both public and prompt-hidden tests. However, dead-status selection depends only on the public-tier pass/fail outcomes of the cached pool. The prompt-hidden tier is used only at the final true-unlock scoring endpoint. Test cases are not shown in any prompt. Nevertheless, because the prompt-hidden tier defines the final scoring endpoint, it is not called an untouched external test set. This distinction should be preserved for all claims. Conclusion validity is supported by exact paired tests, Holm correction, immutable unit d pool is a pooled discordant ratio and should keys, and pre-run power analysis. In contrast, OR not be interpreted as a Cochran–Mantel–Haenszel (CMH) estimator. No equivalence test is performed for the blind tie. No confidence-bound is computed for the observed-harm gate. These quantities provide descriptive boundaries. Internal validity is not equal across contrasts. Since the CF-SF–GR-SHAPE contrast holds source code, failure instruction, template, and bullet count fixed, it is the most tightly controlled content contrast in the design. However, exact token length and lexical distribution are not equal. Therefore, the result provides evidence in favor of task-relevant content, but does not eliminate all surface-form alternatives. In the B–P comparison, failing code, failure instruction, and prompt length change together. The sixth arm narrows the instruction factor, but does not separate code content from intrinsic length. A content-free, syntax-matched, and length-matched code-like padding arm is needed to test the remaining confound. The CF-SF–P comparison is also not a same-code single-ingredient contrast. Whereas CF-SF is pinned to the TC trajectory, P follows its own trajectory. Therefore, C2′ measures the arm-level effect of the code-plus-facts packet relative to the bare-code packet. It does not measure the isolated effect of adding facts to identical source code. Same-code content attribution is based primarily on the CF-SF–GR-SHAPE contrast. Future work should include a direct P+facts arm that mirrors the P source code. The term matched compute is also bounded. Output generation count is equalized. However, input tokens, wall-clock time, and total floating-point operations (FLOPs) are not equalized. Longer code-bearing prompts carry higher prefill cost. The design controls the additional-output advantage, but does not provide strict compute equivalence. Tokenmatched and FLOP-matched comparisons would provide a stronger compute claim. 29
Fresh-generation confirmation is conducted on the same units, cached pools, and incumbents. This design prevents reuse of discovery outputs and provides seed-level robustness. It does not provide independent-task replication or benchmark-level generalization. The disjoint sample preserves the co-primary directions, but the content-attribution contrast reverses. Therefore, primary C3′ should be considered internally strong but externally unstable. Preregistration is based on an internal version-control timestamp and is not a third-party registry. Frozen audit rules produce initial failures during the main and follow-up runs, and post-run dated amendments are applied. Outputs are not changed, original verdicts are preserved, and amendments are gated by executable counterexamples. Nevertheless, amended audit status does not have the same evidential weight as untouched external preregistration. Independent and externally registered replication would reduce this limitation. External validity is limited along four main dimensions. Model scale. Only 0.5B–1.5B frozen models are evaluated. On larger models, the barecode trap may weaken, disappear, or produce a different pattern (Dinh et al., 2023; Lam, Wang, Huang, & Lyu, 2025; Olausson et al., 2024). Task regime. The estimand applies to tasks for which no candidate in the finite cached pool passed the public tier. Effects may shrink or change direction on easier tasks that can be solved by ordinary sampling. Benchmark and language. Only Python HumanEval+ and MBPP+ tasks are used. Repository-level, multi-file, interactive, or different-language coding tasks are not evaluated. Prompt scaffold. A single prompt template is used for each arm. Robustness over alternative prompt families and surface forms is not tested. Future work should directly target the identified boundaries. Priority extensions include a direct mirror arm that adds executed facts to identical source code from the P trajectory, syntax- and length-matched irrelevant-code controls, token- and FLOP-matched comparisons, externally registered independent-task replication, larger model families, non-dead tasks, and repository-level benchmarks. Token-level or activation-level analyses are also needed to distinguish the anchoring account from general distraction, effective-attention reduction, and trajectory-persistence explanations. This boundary is further supported by the six prospective external-controller experiments conducted on the most favorable search-bound stratum (Table 15). These experiments suggest that, in frozen small models, inference-time content manipulation may need to be replaced by inference-aware retraining or search-space restructuring. Under these boundaries, the central conclusion is bounded as follows. In the primary dead-task sample, bare code-bearing retry remains below blind resampling. The code-plusfacts packet recovers this deficit, and the same-code placebo comparison favors executed fact content. Nevertheless, none of the tested feedback arms exceeds blind resampling, content attribution is not stable in the disjoint sensitivity sample, and deployment wording is bounded only by the descriptive observed-harm gate. The program-level interpretation is broader, but still bounded. In the tested frozen-smallmodel settings, the falsification signal does not produce separable accuracy gain as Popperian vocabulary, model-internal self-judgment, or semantic post-hoc selection. Execution-grounded counterevidence is able to repair the deficit produced by harmful conditioning. Therefore, the value carried by falsification here arises not from the model evaluating its own conjecture more effectively, but from that conjecture being exposed to externally produced and executable criticism. From the perspective of scientific methodology, the main contribution is that this critical relation is not left as a philosophical metaphor, but is embedded into the measurement design. The model conjecture is tested by the execution oracle, whereas the researcher conjecture is tested through preregistration, placebo controls, a matchedbudget baseline, fresh-generation confirmation, and executable audits. Thus, falsification is operated in LLM evaluation not merely as a test that filters failed programs, but as a reflexive methodological discipline that audits the research arrangement producing evidence, attribution, and claim calibration.
5
Conclusion
In this study, the self-repair feedback packet used for small frozen code models in deployment settings where retraining is not feasible was examined at the component level. Across six benchmark×model cells constructed from HumanEval+ and MBPP+ with three 30
0.5B–1.5B frozen code models, 290 zero-pass-in-pool task–cell units were evaluated under a five-arm controlled decomposition. In the main run, 7,000 fresh seeded generations were produced, and in the preregistered sixth-arm follow-up, 1,400 additional fresh seeded generations were produced. The output-generation count was equalized across arms, and correctness was determined by execution on public tests and prompt-hidden EvalPlus tests. Thus, it was tested directly not only whether self-repair feedback produces more text, a longer prompt, or an additional retry opportunity, but also which feedback component carries execution-grounded signal. The obtained results indicate that bare failing code should not be assumed to be a reliable feedback object. Blind resampling exceeded the bare-code arm by +18 net unlocks (discordant 25/7, Holm-adjusted p = 0.0021). In the preregistered sixth-arm diagnostic follow-up, no statistically detectable effect was found for the failure instruction, and the remaining +15 difference remained associated with code-bearing context (Holm-adjusted p = 0.0026). Nevertheless, because code content was not separated from intrinsic prompt length, pure code-causation is not claimed. The code-plus-facts packet produced +18 net recovery relative to the bare-code packet (discordant 21/3, Holm-adjusted p = 0.00042). In the same-code placebo contrast, a +15 difference was obtained in favor of executed facts (Holm-adjusted p = 0.0041). This pattern shows that, on the primary sample, recovery was associated not merely with generic feedback shape, but with task-relevant and executiongrounded content under the matched dimensions. However, the direction change of the content-attribution contrast in the disjoint sensitivity sample forms a clear boundary for external stability. In addition, because exact token length and lexical distribution were not equalized, all surface-form explanations were not ruled out. None of the tested feedback arms exceeded blind resampling under the matched outputgeneration budget. Code-plus-facts and blind resampling each produced 26 unlocks, but did so over different unit sets and showed a symmetric 20/20 discordant split. This outcome is not evidence of equivalence, non-inferiority, or identical behavior. It only showed an observed pooled tie under the current sample and output-generation budget. Therefore, the main observed function of the code-plus-facts packet was to close the pooled unlock deficit relative to the bare-code packet, not to produce additional pooled unlocks over blind resampling. The preregistered observed-harm gate is satisfied for the code-plus-facts versus bare-code comparison, but is not satisfied for the blind resampling versus bare-code comparison. This gate should be interpreted only as a point-estimate wording constraint. If failing code must remain inside the packet, the addition of executed public-test facts is supported by the current evidence. By contrast, it is not claimed that failing code should always be discarded, that the code-plus-facts packet is deployment-safe, or that it is superior to blind resampling. The philosophical significance of the results is that falsification carries value in LLMs not as a prompting vocabulary or as a model-internal self-critique capability, but as external and executable criticism. In this study, the generated program was treated as a tentative conjecture. The violation observed during public execution was evaluated as an oracle-relative counterexample to a specific behavioral consequence of that conjecture. The findings sharpen when read together with the two preceding studies, in which Popperian vocabulary, modelinternal self-judgment, and semantic post-hoc falsification did not produce separable accuracy gain in the tested small-model settings. Externally executed counterevidence was able to repair the loss associated with harmful conditioning. However, this signal was not observed as a content lever that exceeded blind resampling. The value carried by falsification here is therefore not the creation of a new capability in the model, but the opening of the model conjecture to external, auditable, and execution-grounded criticism. This boundary is also supported by six prospective external-controller follow-ups conducted on the search-bound stratum. The two strongest content channels, own near-miss code and runtime execution trace, tie with content-free shape placebo (§4.1). These follow-ups should not be treated as preregistered primary evidence, but as post-primary prospective evidence examining the boundary of the primary finding in a more favorable stratum. Nevertheless, the observed pattern is consistent with the interpretation that inference-time content manipulation remains limited in frozen small models, and that robust gains beyond blind resampling may require inference-aware retraining, stronger search procedures, or search-space restructuring. The transferable contribution of the study is not a new repair algorithm. The central contribution is a reflexive measurement and evaluation methodology that makes claims 31
about self-repair feedback falsifiable. Packet decomposition, mirror placebo, matched outputgeneration budget, same-unit discordant analysis, fresh-generation confirmation, immutable provenance, and executable audits are combined within a single instrument. Thus, not only model outputs, but also the evaluation system that produces claims about those outputs, is left open to critical scrutiny. Preserving original audit failures and dated amendments in the record prevents negative evidence from being neutralized post hoc and applies the logic of falsification back onto the research pipeline itself. For this reason, the final contribution of the study is not that self-repair is improved by a better prompt, but that LLM feedback claims are transformed into measurable, auditable, and falsifiable scientific objects under the distinctions of content-vs-form, discovery-vs-confirmation, and statistical-verdict-vs-audit-verdict.
Reproducibility and Data Availability The research data, result manifests, generated outputs, prompt renders, seed payloads, audit records, pre-registration and amendment records, generation and evaluation code, falsification suites, and figure/table generation scripts produced and analyzed in this study have not been released through a public repository, DOI, or external deposit. These materials are available from the corresponding author upon reasonable request for academic verification and replication. Sharing of the requested materials may be limited by file size, technical transfer constraints, third-party benchmark/model license conditions, and institutional policy constraints. The open-weight models and EvalPlus benchmarks used in the study are already available from their respective public sources.
Ethics and Generative-AI Use Disclosure The study evaluates publicly released open-weight code models on public benchmarks. No human subjects or personal data are involved. Large language models were used in two declared roles: as the objects of the study, namely the three tested frozen models, and as manuscript-preparation assistants for drafting and editing under the author’s direction and review. All experimental design decisions, pre-registrations, and verdicts belong to the author. All reported numbers are produced by the executable pipeline described above.
Funding This work was supported by the Scientific and Technological Research Council of Türkiye (TÜBİTAK) under the 1001 programme, project no. 225M316, “A Tilt-Trirotor Vertical Take-Off and Landing Controller Enabling Task-Oriented Transfer and Rapid Adaptation: Hardware-in-the-Loop and Real Validation of a Meta-Learning–Based Reinforcement Learning Architecture.” The meta-learning code developed in that project was integrated into the test harness together with the frozen language models studied here and was used during the agent-design phase to optimize the algorithms designed by the author.
Conflict of Interest The author declares that there is no conflict of interest.
References Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., & Sutton, C. (2021). Program synthesis with large language models. Retrieved from https://arxiv.org/abs/2108.07732 Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024). Large language monkeys: Scaling inference compute with repeated sampling. Retrieved from https://arxiv.org/abs/2407.21787 Brucks, M., & Toubia, O. (2025). Prompt architecture induces methodological artifacts in large language models. PLOS ONE, 20 (4), e0319159. doi: 10.1371/journal.pone.0319159 Chen, J., Tang, H., Chu, Z., Chen, Q., Wang, Z., Liu, M., & Qin, B. (2024). Divideand-conquer meets consensus: Unleashing the power of functions in code generation. In Advances in neural information processing systems 37 (neurips). Retrieved from https://arxiv.org/abs/2405.20092 (arXiv:2405.20092) 32
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., & Zaremba, W. (2021). Evaluating large language models trained on code. Retrieved from https://arxiv.org/abs/2107.03374 Chen, X., Lin, M., Schärli, N., & Zhou, D. (2024). Teaching large language models to selfdebug. In The twelfth international conference on learning representations. Retrieved from https://arxiv.org/abs/2304.05128 (ICLR 2024; arXiv:2304.05128) Cho, J., Kang, D., Kim, H., & Lee, G. G. (2025). Self-correcting code generation using small language models. In Findings of the association for computational linguistics: Emnlp 2025 (pp. 2345–2368). Suzhou, China: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2025.findings-emnlp.127/ doi: 10.18653/v1/2025.findings-emnlp.127 Chow, Y., Tennenholtz, G., Gur, I., Zhuang, V., Dai, B., Thiagarajan, S., Boutilier, C., Agarwal, R., Kumar, A., & Faust, A. (2025). Inference-aware fine-tuning for bestof-n sampling in large language models. In International conference on learning representations (iclr). Retrieved from https://arxiv.org/abs/2412.15287 (ICLR 2025; arXiv:2412.15287) Damani, M., Shenfeld, I., Peng, A., Bobu, A., & Andreas, J. (2025). Learning how hard to think: Input-adaptive allocation of LM computation. In International conference on learning representations (iclr). Retrieved from https://arxiv.org/abs/2410.04707 (arXiv:2410.04707) Ding, Y., Min, M. J., Kaiser, G., & Ray, B. (2024). CYCLE: Learning to self-refine the code generation. In Proceedings of the acm on programming languages (oopsla). Retrieved from https://arxiv.org/abs/2403.18746 (OOPSLA 2024; arXiv:2403.18746) Dinh, T., Zhao, J., Tan, S., Negrinho, R., Lausen, L., Zha, S., & Karypis, G. (2023). Large language models of code fail at completing code with potential bugs. In Advances in neural information processing systems 36 (neurips). Retrieved from https:// arxiv.org/abs/2306.03438 (arXiv:2306.03438) Fagerland, M. W., Lydersen, S., & Laake, P. (2013). The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. BMC Medical Research Methodology, 13 (1), 91. Retrieved from https://doi.org/10.1186/ 1471-2288-13-91 (Open Access; 8 pages) doi: 10.1186/1471-2288-13-91 Gorman, K., & Bedrick, S. (2019). We need to talk about standard splits. In Proceedings of the 57th annual meeting of the association for computational linguistics (acl) (pp. 2786– 2791). Retrieved from https://aclanthology.org/P19-1267/ doi: 10.18653/v1/P191267 Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y. K., Luo, F., Xiong, Y., & Liang, W. (2024). DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence. Retrieved from https://arxiv.org/abs/2401.14196 Hofman, J. M., Chatzimparmpas, A., Sharma, A., Watts, D. J., & Hullman, J. (2023). Pre-registration for predictive modeling. Retrieved from https://arxiv.org/abs/ 2311.18807 Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6 (2), 65–70. Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2024). Large language models cannot self-correct reasoning yet. In The twelfth international conference on learning representations. Retrieved from https://openreview.net/ forum?id=IkmD3fKBPQ (Published as a conference paper at ICLR 2024) Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., Dang, K., Fan, Y., Zhang, Y., Yang, A., Men, R., Huang, F., Zheng, B., Miao, Y., Quan, S., Feng, Y., Ren, X., Ren, X., Zhou, J., & Lin, J. (2024). Qwen2.5-Coder technical report. Retrieved from https://arxiv.org/abs/2409.12186 33
İşcan, M. (2026a). Scaffold, not vocabulary? a controlled, two-tier, pre-registered study of a popperian code-generation skill. Retrieved from https://arxiv.org/abs/2606 .06454 İşcan, M. (2026b). Selection without signal, recovery through expression: A measurement study of post-hoc falsification operators for frozen small code models. Retrieved from https://arxiv.org/abs/2606.16999 Jiang, N., Li, X., Wang, S., Zhou, Q., Hossain, S. B., Ray, B., Kumar, V., Ma, X., & Deoras, A. (2024). LeDex: Training LLMs to better self-debug and explain code. In Advances in neural information processing systems 37 (neurips). Retrieved from https://arxiv.org/abs/2405.18649 (arXiv:2405.18649) Jin, H., & Chen, H. (2025). Uncovering systematic failures of LLMs in verifying code against natural language specifications. In 2025 40th IEEE/ACM international conference on automated software engineering (ASE) (pp. 3819–3823). IEEE. doi: 10.1109/ASE63991.2025.00323 Jones, E., & Steinhardt, J. (2022). Capturing failures of large language models via human cognitive biases. In Advances in neural information processing systems 35 (neurips). Retrieved from https://arxiv.org/abs/2202.12299 Khojah, R., de Oliveira Neto, F. G., Mohamad, M., & Leitner, P. (2025). The impact of prompt programming on function-level code generation. IEEE Transactions on Software Engineering, 51 (8), 2381–2401. doi: 10.1109/TSE.2025.3587794 Lam, M. H., Wang, C., Huang, J.-t., & Lyu, M. R. (2025). CodeCrash: Stress testing LLM reasoning under structural and semantic perturbations. Retrieved from https:// arxiv.org/abs/2504.14119 Lavon, B., Katz, S., & Wolf, L. (2025). Execution guided line-by-line code generation. Retrieved from https://arxiv.org/abs/2506.10948 Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023). Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Advances in neural information processing systems (Vol. 36, pp. 21558–21572). Retrieved from https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html (37th Conference on Neural Information Processing Systems (NeurIPS 2023)) Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-Refine: Iterative refinement with selffeedback. In Advances in neural information processing systems (neurips). Retrieved from https://arxiv.org/abs/2303.17651 (arXiv:2303.17651) Mayo, D. G., & Spanos, A. (2006). Severe testing as a basic concept in a Neyman–Pearson philosophy of induction. The British Journal for the Philosophy of Science, 57 (2), 323–357. (Advance Access published April 11, 2006) doi: 10.1093/bjps/axl003 McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12 (2), 153–157. doi: 10.1007/BF02295996 Olausson, T. X., Inala, J. P., Wang, C., Gao, J., & Solar-Lezama, A. (2024). Is self-repair a silver bullet for code generation? In Proceedings of the twelfth international conference on learning representations (iclr). Retrieved from https://openreview.net/forum ?id=y0GJXRungR Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. In Advances in neural information processing systems 37 (neurips 2024). Retrieved from https://papers.nips.cc/paper_files/paper/ 2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html doi: 10.52202/079017-2197 Popper, K. R. (1959). The logic of scientific discovery. London: Hutchinson. (English translation of Logik der Forschung (1934); degree of corroboration C(h,e) in Appendix *ix) Popper, K. R. (1963). Conjectures and refutations: The growth of scientific knowledge. London: Routledge and Kegan Paul. (Verisimilitude/truthlikeness in ch. 10 and Addenda) Rizqullah, M., & Albassam, E. (2026). Model-agnostic empirical evaluation of test-driven prompt engineering on improving accuracy and efficiency in large language models Python code generation. IEEE Access, 14 , 22801–22821. doi: 10.1109/AC34
CESS.2026.3662817 Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024). Quantifying language models’ sensitivity to spurious features in prompt design, or: How I learned to start worrying about prompt formatting. In The twelfth international conference on learning representations (iclr). Retrieved from https://arxiv.org/abs/2310.11324 (arXiv:2310.11324) Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. In Advances in neural information processing systems (neurips). Retrieved from https://arxiv.org/abs/ 2303.11366 (arXiv:2303.11366) Singhi, N., Bansal, H., Hosseini, A., Grover, A., Chang, K.-W., Rohrbach, M., & Rohrbach, A. (2025). When to solve, when to verify: Compute-optimal problem solving and generative verification for LLM reasoning. In Conference on language modeling (colm). Retrieved from https://arxiv.org/abs/2504.01005 (arXiv:2504.01005) Snell, C., Lee, J., Xu, K., & Kumar, A. (2025). Scaling LLM test-time compute optimally can be more effective than scaling model parameters. In International conference on learning representations (iclr). Retrieved from https://openreview.net/forum?id= 4FWAwZtd2n (ICLR 2025 (Oral); arXiv:2408.03314) Song, Y., Zhang, H., Eisenach, C., Kakade, S. M., Foster, D., & Ghai, U. (2025). Mind the gap: Examining the self-improvement capabilities of large language models. In International conference on learning representations (iclr). Retrieved from https:// arxiv.org/abs/2412.02674 (ICLR 2025; arXiv:2412.02674) Stechly, K., Valmeekam, K., & Kambhampati, S. (2025). On the self-verification limitations of large language models on reasoning and planning tasks. In International conference on learning representations (iclr). Retrieved from https://arxiv.org/abs/2402.08115 (ICLR 2025; arXiv:2402.08115) Takallou, M. A., Gautam, A., & Shirolkar, P. (2026). LLMs for end-to-end machine learning: A comprehensive review. IEEE Access, 14 , 47717–47738. doi: 10.1109/ACCESS.2026.3678429 Tian, R., Ye, Y., Qin, Y., Cong, X., Lin, Y., Pan, Y., Wu, Y., Hui, H., Liu, W., Liu, Z., & Sun, M. (2024). DebugBench: Evaluating debugging capability of large language models. In Findings of the association for computational linguistics: Acl 2024. Retrieved from https://arxiv.org/abs/2401.04621 (arXiv:2401.04621) Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185 (4157), 1124–1131. doi: 10.1126/science.185.4157.1124 Valmeekam, K., Marquez, M., & Kambhampati, S. (2023). Can Large Language Models Really Improve by Self-critiquing Their Own Plans? Retrieved from https://arxiv.org/ abs/2310.08118 (Preprint. arXiv:2310.08118, 12 Oct 2023.) Wang, E., Cassano, F., Wu, C., Bai, Y., Song, W., Nath, V., Han, Z., Hendryx, S., Yue, S., & Zhang, H. (2025). Planning in natural language improves LLM search for code generation. In International conference on learning representations (iclr). Retrieved from https://arxiv.org/abs/2409.03733 (ICLR 2025; arXiv:2409.03733) Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024). Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 50 (4), 911–936. doi: 10.1109/TSE.2024.3368208 Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024). Chain-of-thought in neural code generation: From and for lightweight language models. IEEE Transactions on Software Engineering, 50 (9), 2437–2457. doi: 10.1109/TSE.2024.3440503 Zhang, K., Wang, D., Xia, J., Wang, W. Y., & Li, L. (2023). ALGO: Synthesizing algorithmic programs with LLM-generated oracle verifiers. In Advances in neural information processing systems 36 (neurips). Retrieved from https://arxiv.org/abs/2305.14591 (arXiv:2305.14591) Zheng, K., Decugis, J., Gehring, J., Cohen, T., Negrevergne, B., & Synnaeve, G. (2025). What makes large language models reason in (multi-turn) code generation? In International conference on learning representations (iclr). Retrieved from https://arxiv.org/ abs/2410.08105 (ICLR 2025; arXiv:2410.08105) Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., & Yue, X. (2024). OpenCodeInterpreter: Integrating code generation with execution and refinement. In Findings of the association for computational linguistics: Acl 2024 (pp. 12834–12859). Bangkok, Thailand: Association for Computational Linguis35
tics. Retrieved from https://aclanthology.org/2024.findings-acl.762/ doi: 10.18653/v1/2024.findings-acl.762 Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2025). Towards an understanding of large language models in software engineering tasks. Empirical Software Engineering, 30 (2), 50. doi: 10.1007/s10664-024-10602-0 Zhong, L., Wang, Z., & Shang, J. (2024). Debug like a human: A large language model debugger via verifying runtime execution step by step. In Findings of the association for computational linguistics: Acl 2024 (pp. 851–870). Bangkok, Thailand: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2024 .findings-acl.49/ doi: 10.18653/v1/2024.findings-acl.49 Zubair, F., Al-Hitmi, M., & Catal, C. (2025). The use of large language models for program repair. Computer Standards & Interfaces, 93 , 103951. doi: 10.1016/j.csi.2024.103951
A
Audit and amendment trail
This appendix records the audit events in the order in which they occurred, together with their dispositions and evidential status. The purpose is not only to report the final amended verdict, but also to make the failure modes of the evaluation pipeline itself reproducible and critically visible. For this reason, the original audit failures were not deleted, dated amendments were reported separately, and post-amendment verdicts were not substituted for the original as-frozen verdicts. (i) Pre-run audit. Before any generation was initiated, the implementation was tested with a falsification suite containing eight executable counterexamples. The suite covered mirror-equality violation, seed-namespace violation, foreign-prefix seeds, sample-identity violation, hidden-test isolation, token-bound violation, resume integrity, and hostile model outputs. The suite result was recorded as 8/8 passed. Six per-cell pre-run manifests were committed before the first token was generated. Thus, sample identity, model digest, prompt rendering, and initial constraints were made version-controlled so that they could not be changed after the run. (ii) Discovery-stage inheritance. The pooled-equals-cell-sum invariant and immutable unit keys were inherited from the corrected evaluator of the discovery stage. In that stage, a pooling-collapse bug was caught by adversarial review. The bug caused bare task keys to merge units across model cells and collapsed 290 units into 202 rows. This issue was fixed before the discovery-stage verdict was issued. In the current study, exact equality between pooled counts and per-cell sums was preserved as an executable invariant to prevent the same class of error from recurring. (iii) Main post-run anti-reuse audit. The as-frozen anti-reuse invariant of the main run returned AUDIT-FAILURE. The failure was triggered on 28 distinct (output, prompt) hash pairs shared with discovery records. These pairs appeared in 34 confirmation records in total. Before any code change was made, a forensic audit was conducted. This audit showed that no seed-payload collision was present, that the records exhibiting equality had distinct namespaces and distinct seeds, that prompt equality was expected by design, and that certain round-0 prompts could remain byte-identical because the same task lists, cached pools, and incumbents were preserved. The record-level collision incidence was computed as 34/7,000 ≈ 0.5%. The arm distribution of colliding records was found to be P = 17, CF-SF = 15, TC = 2, B = 0, and GR-SHAPE = 0. This distribution was recorded descriptively and was not used for causal or directional interpretation. Because provenance records were distinct, output-hash equality alone was not accepted as evidence of cross-run reuse. With the dated amendment, output-pair equality under intentionally identical prompts was reclassified from a fatal predicate to a counted diagnostic. Seed-payload collision, namespace reuse, and provenance inconsistency were retained as fatal predicates. The amendment was gated by a five-counterexample falsification suite, including a counterexample requiring a doctored seed-copy to remain fatal. The official amended evaluation was recomputed only after the suite passed 5/5. Both the as-frozen AUDIT-FAILURE verdict and the post-amendment CERTIFIED verdicts are preserved in the main results documents and in this appendix. 36
(iv) Deconfounding follow-up audit. The as-frozen evaluation of the sixth-arm diagnostic follow-up also returned AUDIT-FAILURE. The failure consisted of exactly 120 missingbaseline errors, corresponding to 60 disjoint units crossed with two baseline arms. Before any code change was made, the forensic audit showed that baseline records were present under the disjoint collection of the main manifest, but that the loader was reading only the fixed-sample collection. The lookup scope was expanded by a dated amendment. The missing-baseline predicate was retained as fatal for genuinely absent units. Before the diagnostic follow-up verdict was computed, the amendment was gated by two added counterexamples requiring the disjoint baseline to resolve and an intentionally deleted baseline to produce a fatal failure. Original and amended verdicts are reported separately in §3.2. (v) Post hoc q0 stratification. The primary sample was divided post hoc into four bins according to the cached incumbent’s public-pass fraction q0 : q0 =0 (n=159), 0<q0 <1/3 (n=22), 1/3≤q0 <2/3 (n=51), and 2/3≤q0 <1 (n=58). The total was preserved as n = 290. This stratification was not preregistered and does not generate a new confirmatory claim. The calculations were derived from committed result manifests, and no generation output or score was changed. The corresponding results are reported as post hoc sensitivity evidence in §3.2. (vi) Prospective controller follow-ups. Six external-controller experiments were conducted on the search-bound stratum. Two pilot runs, namely self-repair and EOT-JEPA-RL, were run on 69 live-confirmed units using separate namespaces and fresh seeds, with 1,104 generations each. No audit failure was triggered in either pilot run. These controller experiments were conducted outside the frozen evaluation pipeline of the primary study. Therefore, they were not subject to the audit invariants of the primary pipeline. The results and this limitation are reported in §4.1. This audit trail was kept to show that the statistical verdict and the audit verdict have different epistemic statuses. Satisfaction of statistical thresholds does not imply that pipelinelevel assumptions are automatically valid. Similarly, the amended verdict obtained after a dated amendment does not mean that the original as-frozen failure has been removed from the record. This distinction was preserved so that the evaluation system’s own conjectures would also remain open to executable counterexamples.
B
Prompt scaffolds
The five arms share a single general scaffold family and differ only in the conditioning block. The templates below are reproduced verbatim from the repository’s frozen prompt builders. <TASK PROMPT> denotes the unmodified benchmark task, <INCUMBENT CODE> denotes the trajectory’s best-so-far failing solution, and <FACT BULLETS> denotes at most six violated public assertions. Fact bullets were produced by the frozen no-ellipsis renderer as literal input and expected-output equality statements and were not elided. The incumbent’s observed output was not additionally shown in the prompt. <GENERIC BULLETS> denotes placebo bullets drawn in order from the frozen generic list and matched in count to the fact bullets. B (blind resample). Write a Python solution for this task. <TASK PROMPT> Return your answer in a ‘‘‘python code block.
BI (blind + instruction, the deconfounding sixth arm). Write a Python solution for this task. <TASK PROMPT> Your solution failed the tests. Write a corrected, different solution. Return your answer in a ‘‘‘python code block.
37
P (bare own code). Write a Python solution for this task. <TASK PROMPT> A previous solution is shown below. It did not pass the public tests. Previous solution: ‘‘‘python <INCUMBENT CODE> ‘‘‘ Your solution failed the tests. Write a corrected, different solution. Return your answer in a ‘‘‘python code block.
TC (facts only). Write a Python solution for this task. <TASK PROMPT> Your implementation must satisfy these input/output requirements: - <FACT BULLETS> Return your answer in a ‘‘‘python code block.
CF-SF (own code + the same facts). Write a Python solution for this task. <TASK PROMPT> A previous solution is shown below. It did not pass the public tests. Previous solution: ‘‘‘python <INCUMBENT CODE> ‘‘‘ Your implementation must satisfy these input/output requirements: - <FACT BULLETS> Your solution failed the tests. Write a corrected, different solution. Return your answer in a ‘‘‘python code block.
GR-SHAPE (own code + shape-matched generic bullets). The GR-SHAPE template is identical to the CF-SF template. Only <FACT BULLETS> is replaced with <GENERIC BULLETS>. The same source-code block was preserved and verified against CF-SF through per-round SHA-256 equality. The bullet count was also kept identical to CF-SF. Bullets were drawn in order from the frozen generic-requirement list: - the function returns a value of the documented return type - the implementation handles the documented input domain - the result is deterministic for the same inputs - the function preserves the documented ordering requirements - edge cases described by the task are handled explicitly - the implementation follows the task’s stated behavior
An executable forbidden-token audit was applied to every rendered GR-SHAPE prompt. This audit verified that no generic bullet contained the task’s function name, any public-assert literal or constant, or any identifier extracted from the task prompt. Within the same (task, round), the fact bullets shown to the TC and CF-SF arms were kept byte-identical. This equality was enforced by a per-round constraint-block SHA-256 check. 38
C
Per-cell unlock rates
Table 16 re-expresses the per-cell true-unlock counts in Table 8 as rates for readers who wish to compare cells with unequal sizes. The pooled row repeats the endpoint percentages reported in §3.1. This table does not define a new measurement or a new statistical family. Its purpose is to make clearer how the per-cell counts should be read under different denominators. Table 16: Per-cell true-unlock rates by arm (percentage of cell units) and the pooled rate over all 290 units in the final row. Rates are a re-expression of the counts in Table 8. No new measurement is reported. The pooled ordering is not driven by a single cell. The per-cell maximum belongs to B or CF-SF in five of the six cells. The only exception is the HE+/deepseek-1.3b cell, where TC leads with 10.2%. P does not reach the cell maximum in any cell.
Cell
P
B
TC
CF-SF
GR-SHAPE
HE+ / qwen-1.5b MBPP+ / qwen-1.5b HE+ / qwen-0.5b-inst MBPP+ / qwen-0.5b-inst HE+ / deepseek-1.3b MBPP+ / deepseek-1.3b Pooled
8.3 2.1 5.0 1.7 0.0 3.3 2.8
16.7 6.4 10.0 8.3 6.8 10.0 9.0
4.2 4.3 7.5 3.3 10.2 1.7 5.2
12.5 12.8 17.5 3.3 3.4 10.0 9.0
16.7 4.3 5.0 0.0 3.4 1.7 3.8
39