A Cost That Practice Accuracy Hides: Worked Solutions in Self-Generated Training Data and the Compositional Use of New Rules

A Cost That Practice Accuracy Hides: Worked Solutions in Self-Generated Training Data and the Compositional Use of New Rules

arXiv:2611.12225v1 [cs.SE] 9 Nov 2026

Jiantong Xu∗ Independent Researcher

Abstract A worked solution can make self-generated training targets shorter and cheaper, but accuracy on the practiced operation may hide how well a model can reuse newly learned rules. We study this choice in a 2B-parameter reasoning model learning two invented quantitative rules. The model generates targets from a rule sheet alone or from the sheet plus a program-written worked solution; targets are accepted for final-answer correctness, and the support is removed for fine-tuning and evaluation. Training questions request forward use in one segment; composition requires sequential reuse of the same rule across two segments. After six passes, using a program-checked numerical signature, sheet-only targets lead by 24.3 percentage points (95% item-bootstrap CI 14.6–34.0) on composition in a prospective test. On unseen argument values, observed forward accuracy is identical (87.5% each), but composition accuracy is 58.3% versus 23.6%, a 34.7-point gap (25.7–43.8). An independently sampled second target-bank pair repeats the direction (+14.6 points, 2.1–26.0). Ten passes on the shorter worked-solution targets, approximately matching the domain tokens of six sheet-only passes while changing other training quantities, narrow the gap to 16.3 points (6.3–26.7) in the first bank pair and 6.2 points (−6.3 to 18.8) in the second, where the pre-specified test does not hold. At six passes each, the worked-solution policy uses fewer capture tokens and less update time, yet the trained models use 2.6 times as many generated tokens per composition success under the numerical signature (2.1 times in the second bank pair). Choosing a capture context therefore requires testing how the new rules are reused, alongside the inference cost of those uses, rather than relying on forward accuracy alone.

1

Introduction

Teams that deploy language models often need them to apply rules the models have never seen: the equations of a new battery-cell model, a revised pricing formula, a newly issued design standard. Putting the rules in every prompt costs context and latency and fails when the rules are long or the model must work without them, so the rules are put into the weights. One established recipe lets the model write its own training data [14, 15, 20, 23]: the model answers practice questions with the rules in its context, outputs with a correct final answer are kept, and the model is fine-tuned on them without the rules in the input. We call the material in the context while the model writes a training target the capture context. What the capture context should contain is an active design choice, and supplying a worked solution or a verified answer alongside the rules is a well-motivated strategy. Several self-training methods show the target writer an answer or a solution that the trained model will not see: STaR ∗

ORCID: https://orcid.org/0009-0005-7470-9236.

1

uses the correct answer as a hint for rationales of problems that the model failed [25], SCORE includes a correct solution of the same question when a small model writes critiques of its own wrong solutions [27], and on-policy self-distillation (OPSD) conditions the teacher on a verified reference solution while the student sees only the question [28]. Most directly, in the knowledge-acquisition ablation of Self-Distillation Fine-Tuning (SDFT), an on-policy self-teacher given both a source article and a worked answer reached 89% strict accuracy on new facts, compared with 75% given the article alone [18, App. A.2]. A solver, a reference program or a stronger model can often supply a worked solution, and the targets written with it can be shorter and cheaper to train on [9, Sec. 4]. Candidate contexts are then judged by what a data pipeline can measure cheaply: the cost of capture and training, and accuracy on held-out questions like the practice questions. The question of this paper is whether that criterion is enough. Yet the benefits of solution-assisted supervision are not universal. Fine-tuning offline on answerconditioned [10] or solution-guided [9, Sec. 4] reasoning traces can degrade mathematical reasoning even when every training trace passes a correctness filter; in on-policy self-distillation, the best privileged context changes with model scale and task [22], privileged solution views add little beyond reference-free training [26], and a solution of another problem can do better than the correct reference [19]. These findings point to a tension: a training target that is easy to generate and verify is not necessarily one that teaches the model to use the underlying knowledge on its own. The studies differ in procedure, in whether the knowledge is new, and in what they measure. Work on composable knowledge editing and synthetic function acquisition already connects context-conditioned learning with the use of new knowledge [11, 12]. Our question is narrower: when the rule sheet and training procedure are shared, can adding a worked solution during offline target generation leave forward accuracy similar but change sequential reuse of the new rules? Machine learning has a name for the general risk: a pipeline is underspecified when it can return predictors that are equivalent on held-out data from the training distribution but behave differently elsewhere [3]. We ask whether one common choice in self-generated training data, a worked solution in the capture context, is such a case, and how large the hidden difference is. In a setting built to measure it, it is: two capture contexts with nearly the same forward accuracy give models whose composition of the new rules differs by 24 to 35 points, and the context that is cheaper to train uses more generated tokens per composition success under the numerical value signature. We study the question in a domain built for it (Section 3): a fictional battery cell with two newly assigned quantitative rules and exact program keys. A 2B-parameter reasoning model writes its own targets for 128 forward questions, each about one segment of constant current, under one of two capture contexts: the rule sheet (Gen), or the same sheet plus a worked solution that a short program computes from the sheet and the question (Inst). The worked solution adds no rule specification that the sheet lacks, the model can answer the practice questions from the sheet alone, and the instruction is the same in both contexts. Both policies share the acceptance rule, the target selection, the training recipe and the evaluation conditions; what differs is the targets that each context produces, and with them the target lengths and the numbers of supervised tokens and updates per pass. The trained models answer without any context, on the practiced forward operation and on two operations that no training question requests: inversion, which recovers a rule parameter from an observed value, and composition, which applies a rule over two consecutive segments, so that the state at the end of the first segment is the start of the second. Composition here is the sequential reuse of one learned rule, not a combination of the two rules. We compare the two complete policies; we do not isolate which property of the targets is responsible (Section 3.3). Unless labeled otherwise, the results below use the value signature, a program check for the keyed numerical values (Section 3).

2

(a) 1. Capture (base model)

2. Train (LoRA+) input: the question only target: the kept answer 6 or 10 passes over 128 questions

context: one of the two below question: a forward practice question kept if the final answer is correct

3. Test (no context) forward use practiced inversion not requested composition not requested

Gen: the rule sheet Inst: the sheet + a program-written worked solution

(b) Forward use (practiced; 24 items)

Composition (not requested; 48 items) recorded margin

pp (rates, %)

Six passes each

pp (rates, %)

+2.8 (86 vs 83)

+24.3 (59 vs 35)

0.0 (88 vs 88)

+34.7 (58 vs 24)

+8.3 (88 vs 79)

+14.6 (46 vs 31)

+5.6

+19.4

fresh items, bank pair 1 (TB-2)

+0.7 (86 vs 85)

+16.3 (59 vs 43)

fresh items, bank pair 2 (TB-2R)

-2.1 (88 vs 90)

+6.2 (46 vs 40)

-0.7

+11.3

fresh items, bank pair 1 (TB-1) new arguments, bank pair 1 (OS-2) fresh items, bank pair 2 (RB-1) fresh items, pooled pairs Worked solution: ten passes

fresh items, pooled pairs

−20

0

20

40

sheet only minus worked solution (pp) pre-specified test: difference and 95% item-bootstrap interval

−20

0

20

40

sheet only minus worked solution (pp) pooled over the two bank pairs (descriptive)

Figure 1: Forward accuracy does not reveal the capture context; composition does. (a) The two capture policies differ only in the capture context. Completions accepted for finalanswer correctness become training targets, and the model is trained on them with the context removed. (b) Success without any context, sheet-only (Gen) minus worked-solution (Inst) policy, in percentage points with 95% item-bootstrap intervals (value signature; rounded success rates in parentheses). Forward differences are small and their intervals contain zero; composition differences are positive in every row. Training the worked-solution policy to about the same number of domain tokens (ten passes) shrinks the composition difference, and with the second bank pair the recorded test (TB-2R) does not hold. Bank pair 2 was not evaluated on new argument values. Pooled rows average the two bank pairs per item and are descriptive. Findings. Each experiment answers one question about the hidden difference (Figure 1). Does practice-like accuracy show how well each policy can use the rules? No: after six passes both policies reach high forward accuracy, but their composition differs. We found the difference after the fact on the first evaluation panel and confirmed it prospectively on new items with a new training seed, +24.3 points [14.6, 34.0] (Section 4.2). Is the difference recall of the trained argument values? No: on items whose rule arguments occur in no training question, forward accuracy stayed identical (87.5% each), while composition success was 58.3% for sheet-only and 23.6% for worked-solution targets, a difference of +34.7 points [25.7, 43.8], as the configuration-bound form of our account had predicted before the test (Section 6.1). How much of the gap is explained by training amount? Part of it. Worked-solution targets are shorter, so we trained that policy for ten passes, which brings its domain tokens close to those of six sheet-only passes but also adds replay, updates and schedule cycles; with the original target banks it narrowed the gap but still trailed by 16.3 points

3

[6.3, 26.7] (Section 4.3). Does the effect depend on a single sample of training targets? A second, independently captured bank pair repeated the direction at six passes, +14.6 points [2.1, 26.0]; at matched dose its difference was +6.2 points [-6.3, 18.8], and the pre-specified test did not hold, so the size of the gap at matched dose is uncertain (Section 6.2). Does a supplied final answer alone produce the cost? We cannot tell: an answer-only context failed its forward-accuracy gate twice, and the reading at its fixed second look is weak (Section 6.3). What this means for practice. In the numbers that a data pipeline usually logs, the worked solution looks efficient: with the first bank pair, 42% fewer generated tokens per accepted target and 28% less update time for six passes, with nearly the same acceptance rate and forward accuracy. At deployment, its six-pass models have fewer value-signature successes on composition and spend 2.6 times as many generated tokens per success (2.1 times with the second bank pair; Section 8). The ten-pass extension narrows the accuracy difference but removes the measured update-time saving. Capture contexts should therefore be compared on operations that the training data do not request before one is chosen. An account with a prospective test. An idealized gradient-flow surrogate illustrates how equal practice accuracy and divergent transfer can coexist: the training targets set the direction of parameter updates, whereas practice accuracy tracks progress along that direction rather than the direction itself (Section 5). It rests on stated assumptions, including observation links shared by the two policies, predicts orderings rather than ratios, and does not identify a mechanism of the network. Before the new-argument test ran, we recorded what each form of the account and a value-association alternative predicted; the configuration-bound form was the one that held. Two later predictions failed: the dose-matched second bank pair did not keep the recorded margin, and the answer-only policy did not lie near the sheet-only policy (Section 6). Contributions. The paper makes two main contributions. (i) A controlled demonstration that similar observed forward accuracy can conceal large differences in the sequential reuse of newly assigned rules. Prospective tests, new argument values, independently sampled targets and dose extensions establish both the separation and its training-dose boundary. (ii) A measurement of the associated cost trade-off: at six passes per policy, savings in capture tokens and update time accompany 2.6 times (second bank pair: 2.1 times) as many generated tokens per composition success under the value signature. The ten-pass extension reduces the accuracy gap but costs more update time than six sheet-only passes. The evaluation construction and the gradient-flow surrogate support these findings. Exact keys and separate forward, inversion and composition items make the comparison reusable; the surrogate supplies an explicit account with its full prediction record, including the failed predictions, in Sections 5 and 6. The comparison is between complete policies in one invented domain, with one main model and one training recipe; which property of the targets carries the difference, and how fully each policy has mastered the rules, remain open (Section 9).

2

Related work and what is new here

Answers and worked solutions when training targets are written. Learning with privileged information gives a teacher information that the student will not have at test time; generalized distillation trains the student on the outputs of such a teacher [13]. Context distillation is the language-model form: a model with extra context in its prompt writes targets for itself without that context [20]. Our capture context is privileged information of this kind, available only while targets 4

are written. Answers and solutions are common forms of this information. STaR’s rationalization step uses the correct answer as a hint for a rationale that is then trained without it [25]; SCORE puts a correct solution of the same question into the prompt as a hint when a small model critiques its own wrong solutions, and removes the hint for fine-tuning [27, Sec. 3.1]; and OPSD conditions the teacher on a verified reference solution while the student, which samples its own trajectories, sees only the question [28, Sec. 3.2]. In the knowledge-acquisition ablation of SDFT, an on-policy teacher that sees a source article and a worked answer gives 89% strict accuracy on new facts, against 75% with the article alone [18, App. A.2]. When the added information teaches less. Lee et al. [10] hold the generator, problems, correctness filter and fine-tuning recipe fixed and vary only whether the generator sees the gold answer with a request to reach it: training Qwen3-8B on its own answer-conditioned chains instead of answer-blind ones lowers MATH-500 accuracy by 16.2 points, chain length does not explain the loss, and with the answer visible but a neutral instruction the loss falls to about one point (their Sections 3 and 4.1). They also report a train–test length inversion: the answer-conditioned training chains are shorter, yet the trained student writes longer chains at test time while scoring worse (their Section 4.1, measured in characters). Kim et al. [9, Sec. 4] fine-tune offline on 800 correct responses written with or without a full solution in the context; the solution-guided responses are about 2k instead of 12k tokens long, and training on them degrades every mathematics benchmark they report, which they attribute to suppressed expressions of uncertainty in targets that presuppose information unavailable at inference. Tian et al. [22] compare five granularities of privileged context for an on-policy self-teacher, in domain and on four transfer benchmarks, and find that the preferred context changes with scale and task and that the teacher–student discrepancy at initialization does not order the contexts by utility (their Section 4.3). Zhang et al. [26] hold the verified answer fixed across six privileged views, from the answer alone to the full solution trace, and find that reference-free on-policy self-distillation accounts for much of the improvement; the added value of a view is small and does not grow with how much of the solution it reveals (their Sections 2.3 and 3.1). Penaloza et al. [16] distil privileged information into agent policies, relate success to the divergence that the information induces and to its utility (their Section 7), and detect references to the privileged information in teacher traces with a keyword list (their Appendix D.1). Shrestha and Tessier [19] find that in on-policy self-distillation the correct reference does not give a consistent benefit and that a solution of another problem can do better. Other methods replace full solutions with structural guidance or skill cards [8, 29]. Together, these studies establish that what the target writer sees matters and that solution-guided offline targets can harm reasoning. New knowledge, its use and its composition. For new knowledge, Padmanabhan et al. [14] distil entity definitions, Akyürek et al. [1] train on self-generated implications, Park et al. [15] compare self-generated practice formats for hypothetical news, including new mathematical operations, and show that keeping the news in the training input impairs learning (contextual shadowing, which our kept-sheet control replicates), and Yang et al. [23] distil a textual rule and test a new operator on larger operands. HPSE is a close precedent: it uses self-distillation from a privileged in-context state to improve the composability of unstructured knowledge edits, including multi-hop use [12]. Its intervention is a hybrid rollout that supplies missing facts where the student’s trajectory lacks coverage (their Sections 2 and 3). MixSD constructs supervision by mixing tokens from knowledge-conditioned and prior-conditioned distributions of the base model [11]. Its KGFunc evaluation tests synthetic arithmetic operations on new inputs; a separate unseen-operation split probes retained arithmetic capabilities, with few-shot input–output examples still supplied 5

(their Sections 3–5). These studies already connect context-conditioned learning with the use and generalization of new knowledge. Press et al. [17] measure how often a model answers every sub-question but not the composed question; Yuan et al. [24] separate atomic string functions from their composition and study what training makes composition generalize; Tang and Yang [21] find that on-policy distillation transfers compositional skill but little new factual knowledge; and Allen-Zhu and Li [2] show that stored knowledge need not be extractable in another format. Gekhman et al. [5] relate fine-tuning on new knowledge to hallucination; we observe a related loss of abstention. Similar on held-out practice, different elsewhere. An underspecified pipeline returns predictors with equivalent held-out performance in the training domain that behave differently in deployment [3], and shortcut learning names decision rules that do well on standard benchmarks but fail to transfer to harder test conditions [4]. Our comparison makes this evaluation concern concrete: changing the capture context produces trained states with similar observed forward accuracy but different compositional performance. At six passes, every paired training seed on fresh items favors the sheet-only policy. The question is what an observed forward score reveals, not whether the policies have identical latent mastery. What is new here. Our contribution is a controlled test of a particular offline data-generation choice: whether adding a worked solution to the same rule sheet changes the model’s later sequential reuse of newly assigned rules, even when forward evaluation reports similar accuracy. HPSE changes rollout coverage, and MixSD changes token-level mixing; here the intervention is the presence of a worked solution while a fixed bank of training targets is generated. The instruction, acceptance rule, target selection, training recipe and evaluation conditions are shared, while the generated targets and their associated training exposure can differ. Unlike the answer-conditioned prompts of Lee et al. [10], our instruction does not ask the model to reach a shown answer; this is a different prompting condition, not evidence of a different mechanism. The experiment separates practiced forward use from unrequested inversion and sequential reuse, including reuse at new rule-argument values. The composition gap holds in a prospective test, at new argument values and with an independently sampled bank pair; longer training narrows it, and the matched-dose test holds for the original banks but not the second pair. We quantify the accompanying capture-token, update-time and inference-token trade-offs, with inference cost measured per value-signature success. The train–test length inversion reported by Lee et al. [10] appears here on composition but not on the practiced forward operation (Section 7). Because our study examines offline target capture, it complements rather than adjudicates on-policy self-distillation studies. Our literature search was focused, not exhaustive (Appendix H).

3

Experimental design

3.1

New rules with exact keys

The Kestrel cell is a fictional battery element with two rules (Appendix D gives the full sheet). T1 gives the voltage v of a “Varn layer” a time h after the start of a constant-current segment: v = r vstart + (1 − r) gI with r = (1 + h/τ )−2 , where τ is a time constant and g a gain; v is continuous p at a current step. T2 gives the usable charge at a discharge current I: Quse = Qnom (1 − k I/Iref ); a segment lowers the state of charge (SOC) by Ih/Quse . Both rules resemble familiar battery models but differ from them: first-order relaxation is exponential, and a simple charge model has a constant

6

Forward (practiced), A-T2. The cell has Qnom = 15 Ah, k = 1/6, and Iref = 4 A. At time 0, its SOC is 4/5. A constant discharge current I = 16 A flows from time 0 to time 900 s. Calculate the p usable charge Quse at this current. Calculate the SOC at the end of the discharge. Key: Quse = 15(1 − 61 16/4) = 10 Ah; final SOC = 4/5 − 4/10 = 2/5. (Constant capacity would give 0.533.) Composition (not requested in training), B-T2-comp. The cell has Qnom = 20 Ah, k = 1/6, and Iref = 4 A. At time 0, its SOC is 7/10. A constant discharge current I = 9 A flows from time 0 to time 2100 s. Then a constant discharge current I = 16 A flows from time 2100 s to time 3000 s. Calculate the SOC at the end of the second discharge. Key: each segment has its own usable charge (15 Ah, then 40/3 Ah); final SOC = 7/10 − 7/20 − 3/10 = 1/20. (Constant capacity: 0.2375; the first current’s Quse for both segments: 0.083.)

Figure 2: The same rule, used once or twice. Two fresh test items (abridged). The composition item needs T2 at two currents; no training question contains a second segment. capacity. The rules are newly assigned to a fictional cell, and the base model never produces a Kestrel value without the sheet (Section 4.1), although the mathematical forms are ordinary. A seeded program generates every item and computes the exact Kestrel answer and the answers that the familiar alternatives imply; it keeps an item only if these values are separated by a fixed margin. Item families. A (forward use, one segment): compute v and the terminal voltage (A-T1), or Quse and the final SOC (A-T2). B-inv (inversion): recover τ from an observed voltage, or k from an observed SOC drop. B-comp (composition): the same rule over two consecutive segments, where the state at the end of the first segment starts the second: a current segment followed by a zero-current rest under the same relaxation law (B-T1-comp), or two discharge segments at different currents, each with its own Quse (B-T2-comp; Figure 2). Composition is therefore the sequential reuse of one rule, not a combination of T1 with T2. C-abst (abstention): a question about a layer whose law is not given; the correct answer says that the value is not determined. The 128 training questions are A items, 64 per rule. Throughout, “unpracticed” means not requested by the training questions: no training question asks for an inversion, a rest segment or a second current. We do not claim that no training target contains such a step or one of its substeps; the targets were scanned with patterns, but a complete semantic reading was not done (Section 9). Item sets and endpoints. The sealed set (124 items: 48 A, 24 B-inv, 24 B-comp, 12 C-abst and 16 recall and concept probes) was generated with the training items before any training. The fresh set (120 items: 24 A, 48 B-inv, 48 B-comp) was generated later from the same seeded stream for a prospective test; no fresh world repeats an earlier world of the same rule. Sealed and fresh items use rule arguments that also occur in training questions: h/τ ∈ {1/4, 1/3, 1/2, 1} and p I/Iref ∈ {1, 3/2, 2}. The new-argument set (120 items with the same family counts) was generated before any of its requests; every T1 segment ratio p and every T2 current ratio in it is new, and 39 of its 60 T2 items also have a new product k I/Iref . The composition endpoint of each 120-item panel is its 48 B-comp items. An arm with three training seeds gives 144 composition answers and an arm with two seeds gives 96; these answers are repeated measurements of 48 items, not independent items.

3.2

Capture policies, selection and training

Capture. With Thinking enabled, the base model answers each training question twice under each capture context (Figure 1a). In Gen, the system message contains the rule sheet; in Inst, it 7

contains the same sheet and a program-written worked solution of that question, which states the factor, the intermediate values and the final answer. An automated judge (Claude Opus 5.5 at high effort, in a separate context) accepts an answer if its final answer is correct against the program key; acceptance does not certify every intermediate statement. The training target is the whole completion, including the Thinking text. Gen has accepted first-sample targets for all 128 questions and Inst for 127. For the replication (Section 6.2), each policy answered every training question twice more with new request seeds, and the same judge and rule produced a second, independent pair of target banks. An answer-only policy (Ans; Section 6.3) was captured in the same way, with the sheet and only the final-answer line of the worked solution. A kept-sheet control (Kept) trains on the Gen targets with the sheet left in the training input. Training. Each pass presents one accepted target per question with the context removed; passes alternate between the two accepted samples. The recipe is the same for every policy: LoRA+ [6, 7] of rank 16 on all seven projection matrices of every layer (25.1M trainable parameters), learning rates 10−5 and 4 × 10−5 for the two factors, warm-up, constant phase and cosine decay within each pass, token cross-entropy over the whole completion, and a KL penalty (weight 0.05) to the base model on replay answers. An update holds at least 4,096 supervised domain tokens, and each pass continues from the adapter and optimizer state of the previous pass with a new learning-rate schedule. Weights and the key–value cache are BF16, the adapter and optimizer are FP32, and nothing is quantized. Models. MiniCPM5-2B (revision 12a3808a) is the main model. Qwen3-4B-Thinking-2507 (revision 768f209d) repeats the Gen acquisition result only. Before training, each model had to pass a gate: with the sheet in its context it must solve each item family, and without the sheet it must fail. Qwen3-1.7B failed this gate (0 of 8 rest-segment items with the sheet) and was not trained.

3.3

What the comparison identifies

The worked solution adds no rule specification beyond the sheet: a deterministic program computes it from the sheet and the question. It does change the distribution of the generated targets and, with it, the training exposure: target length, content (including any incidental steps of unrequested operations), references to the solution and the number of supervised tokens per pass. The experiment measures the downstream contrast between two complete captureand-training policies; it does not identify which of these changes mediates the contrast. Where we report equal forward rates, they are observed rates, not evidence of equal mastery.

3.4

Measures and statistics

Each trained state answers each item once without any context, with up to 32,768 new tokens and a fixed seed per item that is shared across states. Sampling follows each model’s published configuration with a repetition penalty of 1.05 (MiniCPM5-2B: temperature 1.0, top-p 0.95); capture uses the same settings. Value signature (pre-specified primary measure). A program searches the final answer for each keyed quantity; a number matches if it lies within a quarter of the generator’s separation margin of the Kestrel value. A quantity whose Kestrel value also appears among the numbers of the question is set aside as ambiguous. An answer succeeds if at least one keyed quantity is unambiguous and

8

every unambiguous quantity has its Kestrel value. The signature checks numbers, not derivations, so a correct number in a wrong role can pass. Official endpoint (secondary layer). The same judge model reads the final answer with the item key and a rubric; success requires a normal completion, an English final answer, a correct conclusion, correct supporting values and no material contradiction. Each judge packet, read in its own context, holds the answers of several states to the same item under opaque identifiers, with four answers of known verdict as sentinels. An answer without an assessment (for example, without a final answer) is a failure in every denominator. The two measures read the same answers and agree on 92.5–98.6% of assessed answers per population (Appendix F); this agreement is not an independent replication. All judgments in this study are automated; no answer was graded by a human. Statistics. For each item, we average success over the training seeds of an arm and compare arms by the paired per-item difference: its mean, a 95% percentile interval from 10,000 item-bootstrap draws (seed 20261011), and an exact two-sided sign-flip test, with Holm correction within each pre-specified family. A pre-specified test holds if the difference reaches its recorded margin, the interval excludes zero and the (Holm-adjusted) p is below 0.05. The intervals describe variation over items given the trained states. They do not cover variation between training runs or target banks, which we examine separately with the seed-level results (Appendix B), nor variation between models or domains. The sign-flip test assumes sign-symmetric item contrasts; policies were not randomized to items. We label every result pre-specified (test and margin fixed before the data), prospective confirmation (a hypothesis from earlier data tested on new items with a fixed rule), sensitivity, descriptive or exploratory (chosen after the data). Appendix E lists every pre-specified test with its outcome and the dates on which it was recorded.

4

Equal forward accuracy, unequal composition

4.1

Self-generated targets put the new rules into the weights

The base model never produces a Kestrel value without the sheet (0 of 48 sealed forward answers in each of two draws, and 0% on the fresh and new-argument forward items). With Gen targets, official forward success on the sealed items is 8.3% after two passes, 63.5% after four and 82.3% after six (descriptive contrast recorded in the protocol: six minus two passes +74.0 pp [63.9, 83.3]), and inversion and composition rise with it (Appendix A, Table 7). With the sheet kept in the training input (Kept), the same targets give 0% forward success in each of two seeds, a local replication of contextual shadowing [15, 23]. On Qwen3-4B-Thinking-2507, six Gen passes raise forward success from 16.7% after two passes to 81.2% (+64.6 pp [53.1, 76.0]), but its transfer to unpracticed operations misses its pre-specified margin (+19.8 pp against 20), so this model supports the acquisition result only, and we did not run the policy comparison on it. Use training also has side effects: it removed correct abstention on the 12 tested questions about a layer whose law is not given (base model 4 of 12 in each draw; six Gen passes 0 of 12 in each seed), and one six-pass seed reached the pre-specified damage threshold on a 56-item retention panel (Appendix A).

4.2

Forward accuracy does not reveal the capture context; composition does

Discovery. We trained Inst to test whether a worked solution in the capture context changes forward acquisition. The pre-specified superiority test did not reject: after six passes both policies have 82.3% official forward success on the sealed items (0.0 pp [-11.5, 11.5]), an interval too wide to 9

Table 1: The composition gap holds prospectively, and ten passes of the worked-solution policy do not close it. Fresh items, bank pair 1 (MiniCPM5-2B, no context). Rates are peritem means over training seeds (number of seeds in brackets); differences are Gen minus Inst in percentage points, with 95% item-bootstrap intervals. TB-1 and TB-2 are pre-specified (value signature, 10-point margin, Holm over the two); TB-2 is a sensitivity comparison (Section 4.3). Official: semantic judge. Items

Measure

Composition (48) Composition (48) Inversion (48) Forward (24)

signature official signature signature

Gen 6 [3] Inst 6 [3] Inst 10 [2] 59.0 56.9 66.7 86.1

34.7 31.2 50.0 83.3

42.7 38.5 54.2 85.4

Gen 6 − Inst 6

Gen 6 − Inst 10

TB-1: +24.3 [14.6, 34.0] TB-2: +16.3 [6.3, 26.7] +25.7 [16.0, 35.4] +18.4 [8.0, 28.8] +16.7 [4.9, 28.5] +12.5 [0.3, 25.3] +2.8 [-9.7, 15.3] +0.7 [-15.3, 17.4]

show equal acquisition. On the same states, composition success differed by +33.3 pp [16.7, 50.0]. Because we found this difference after seeing the data, we treated it as a hypothesis and tested it prospectively. Prospective confirmation. Before any new request, we recorded two tests on 120 newly generated items and a third training seed of each policy (Appendix E): TB-1 (six Gen passes minus six Inst passes) and TB-2 (six Gen passes minus ten Inst passes; Section 4.3), both on the 48 fresh composition items, with the value signature, a 10-point margin and Holm correction over the two. TB-1 holds: +24.3 pp [14.6, 34.0], Holm p < 0.001; the sheet-only policy is ahead on 29 items and behind on 5 (Table 1). The official judge, applied to all 288 composition answers, agrees in direction: 82 of 144 Gen answers succeed against 45 of 144 Inst answers, +25.7 pp [16.0, 35.4]. The lead is positive in each of the three seed pairs (+14.6, +33.3 and +25.0; Appendix B) and on both rules; it is largest on the two-current items (+30.6 pp [20.8, 40.3]) and smallest on the rest-segment items (+18.1 pp [1.4, 34.7]). Inversion, the other unpracticed operation, shows a lead of +16.7 pp [4.9, 28.5], while the fresh forward rates are 86.1% and 83.3% (+2.8 pp [-9.7, 15.3]). An exploratory contrast of contrasts (composition minus forward difference, per item) gives +21.5 pp [5.6, 36.8]; it does not control latent mastery, and the item sets were not randomized to operations.

4.3

Not only less training

The most direct objection is dose, and this section asks how much of the gap it explains. Workedsolution targets are shorter (the median capture response has 1,868 generated tokens with the worked solution and 3,105 without), so six Inst passes supervise 44% fewer domain tokens than six Gen passes (Appendix A, Table 8). Ten Inst passes come within 6.6% of the domain tokens of six Gen passes and reach nearly the same fresh forward rate (85.4% against 86.1%). The composition lead persists: TB-2 holds (+16.3 pp [6.3, 26.7], Holm p = 0.005; official +18.4 pp [8.0, 28.8], 82 of 144 against 37 of 96 answers), and on the two training seeds that both arms share the lead is +16.7 pp [4.2, 29.2]. Dose matters: four more passes raise Inst composition from 34.7% to 42.7%. Within the range we tested, it does not close the gap. Ten passes also add replay tokens, target visits, updates and learning-rate cycles, so TB-2 is a sensitivity comparison: the lead persists at about matched domain tokens and observed forward rate, but no single dose quantity is isolated. The same picture holds along the sealed training ladders (Figure 3, left): the Inst states lie below the Gen curve at every forward rate that both reach, and ten passes move Inst up its own curve rather than onto the Gen curve; over forward rates of 50–80%, the mean vertical distance between the curves is +18.7 pp [2.4, 34.9] (exploratory; the statistic was defined after the 10

answers existed). Section 5 explains why this pattern can arise and what it predicts; Section 6.2 tests the dose result with a second bank pair.

5

Why equal practice accuracy can hide different transfer

We wrote this account after the results of Section 4 were known, so its agreement with them is retrospective. Its predictions for the tests of Section 6, and those of rival accounts, were recorded before any answer of those tests existed (Appendix E). It is an idealized surrogate: it shows that one simple mechanism produces the pattern, not that the network uses it.

5.1

Two routes, one direction

Idealize the trained model’s answer to a forward question as a linear readout of two routes to the requested value. A general route g(x) evaluates the rule from the arguments and can be reused when the configuration changes. A support-bound route h(x) gives the correct value only on practice-like questions, for example a forward procedure that works only in the trained configuration. On every training question both routes give the target value, scaled to g = h = 1, and the output is f = u g + v h. The targets supervise the two routes with gains α and β, so that u = αθ1 and v = βθ2 for trained parameters θ whose effective update starts at zero, as a LoRA update does. With a squared surrogate loss on the practiced output and the replay penalty idealized as an isotropic quadratic penalty with weight λ > 0, L(θ) = 12 (αθ1 + βθ2 − 1)2 + λ2 ∥θ∥2 .

(1)

Proposition 1 (the targets fix the direction). Let a = (α, β) with ρ2 = α2 + β 2 > 0, and let θ follow 2 the gradient flow θ̇ = −∇L(θ) from θ(0) = 0. Then θ(t) = θ∗ (1 − e−(ρ +λ)t ) with θ∗ = a/(ρ2 + λ), so α2 v(t) = β 2 u(t) for all t, and v(t)/u(t) = β 2 /α2 for all t > 0 when α = ̸ 0. The practiced output is m(t) = u(t) + v(t), and where h does not apply the output is c u(t) = k m(t), with k = c α2 /ρ2 and an efficiency c ≤ 1 of the general route in that use. Proof. ∇L(θ) = a (a⊤ θ − 1) + λθ, so the Hessian aa⊤ + λI has eigenvalue ρ2 + λ along a and λ orthogonal to it, and the minimizer is θ∗ = a/(ρ2 + λ). The deviation θ(t) − θ∗ starts at −θ∗ , which 2 lies along a, and decays as e−(ρ +λ)t along that eigenvector, which gives θ(t). Then u = αθ1 = α2 s(t) 2 and v = βθ2 = β 2 s(t) with s(t) = (1 − e−(ρ +λ)t )/(ρ2 + λ), so m = ρ2 s(t) and c u = c (α2 /ρ2 ) m. Training follows a direction that the targets set, and more training only moves along it. Two policies can therefore reach the same practiced output with different splits between the routes, and different outputs where the support-bound route does not apply. The surrogate idealizes a great deal: the LoRA update is bilinear in its factors, not linear in θ, and LoRA+ learning-rate asymmetry, adaptive optimization, token-level cross-entropy and a KL replay penalty do not reduce to this flow; the routes are not identified in the network.

5.2

From outputs to accuracy: orderings, not ratios

The surrogate describes continuous outputs, while we measure success rates. Reading a ratio of success rates as an estimate of k would be invalid, because success is a threshold or link function of the output: with practiced output 0.9, transfer output 0.45 and a success threshold of 0.4, the output ratio is 0.5 and the success ratio is 1. A monotone observation model connects the two without estimating anything. Let forward accuracy be F = ΦF (m) and composition accuracy 11

C = ΦC (k m). We assume policy-invariant, strictly increasing observation links ΦF and ΦC over the range considered. This is an additional modeling assumption, not a consequence of using the same evaluation protocol: the items, decoding and scoring are shared, but a policy can change answer format or error types, and with them the links. Proposition 2 (orderings). Under this observation model, the states of a policy lie on the curve 



C = hk (F ) = ΦC k Φ−1 F (F ) ,

(2)

which is strictly increasing in k at every F reached with m > 0. The proof is immediate: Φ−1 F exists on the forward accuracies reached because ΦF is strictly increasing, m = Φ−1 (F ) > 0, and ΦC is strictly increasing. Three predictions follow, none of which F needs the form of the links: R1 Ordered curves. Each policy traces one curve of composition against forward accuracy, and curves with larger k lie above at every forward accuracy that both reach; they do not cross. R2 Dose moves along the curve. More passes on the same targets move a state along its own curve, so a policy with a lower curve stays lower at matched forward accuracy, whatever its dose. R3 Speed alone collapses. Policies that differ only in learning speed (ρ, at fixed α/ρ) share k and lie on one curve. R2 and R3 turn the dose objection into a test. If the worked solution only slowed learning, a worked-solution state that reaches the forward accuracy of a sheet-only state would also reach its composition accuracy. TB-2 compared such states (86.1% and 85.4% forward success) and found a composition lead of 16.3 points, as R2 requires and R3 does not; because TB-2 was recorded before the account existed, this agreement is retrospective. The common-link assumption could fail if one policy changed answer format or effort; the worked-solution models write longer composition answers, not shorter ones (Section 7).

5.3

What the support-bound route is bound to

The surrogate does not say what h is bound to, and the forms that it leaves open predict different results at argument values that no training question used. If h holds values or templates of the trained arguments (argument-bound), forward use fails at new arguments in the policy that relies more on h, so a forward gap appears. If h is a forward procedure that works at any arguments but only in the trained configuration (configuration-bound), forward use survives at new arguments in both policies while the composition gap persists. In a value-association alternative, neither policy holds a rule that works at p new values: both fall to or near the floor at new arguments, and within T2, items whose product k I/Iref occurs in training do better than items with a new product. We recorded these predictions with probabilities before any answer of the new-argument test existed (Table 4): a forward gap of at least 10 points (argument-bound, 0.35); a forward gap within ±10 points (configuration-bound, 0.55); a composition gap larger than the forward gap (0.60); and trained products at least 15 points easier than new ones (value association, 0.35). The prediction was not obvious: worked-solution targets assert values at the training arguments, so an argument-bound shortfall was a live possibility. Later we recorded predictions for a second bank pair (the direction of the gap should recur), for that pair at matched dose (R2: the gap should persist, recorded with probability 0.30 that the test would reach its margin), and for an answer-only context. For the answer-only context, the account as we then stated it located the cost in values that targets confirm against supplied intermediate results, so its most probable recorded reading (0.40) was a position near the sheet-only 12

policy: without a supplied method, the targets would have to compute the rule themselves. An absent-reference account predicted the opposite, a position near the worked-solution policy, because targets that rely on a supplied result teach the model to rely on a result that is absent later. The strongest alternative that none of these tests can exclude is a package: the worked-solution targets are shorter, deliberate and check less, cite an absent solution and supply fewer tokens per pass, and any combination of these could produce the gap.

6

Three tests recorded before their data

6.1

New argument values preserve forward accuracy and the composition gap

Every sealed and fresh item reuses rule arguments of the training questions, so a model could answer with values or templates bound to those arguments, and the two policies could differ in how much they do so. This test asks whether the gap is a difference in recall of the trained argument values. Before any request, we generated the new-argument set (Section 3) and recorded two tests with their reading rules and the predictions of Section 5.3: OS-1, six Gen passes minus the base model on the 24 forward items, with a 30-point margin; and OS-2, six Gen passes minus six Inst passes on the 48 composition items, with a 10-point margin (value signature, Holm over the two). The six-pass states of the original banks, three seeds per policy, answered without any context. Both tests hold (Table 2). Forward success is 87.5% for Gen against 0% for the base model (OS-1: +87.5 pp [80.6, 93.1]; official Gen 86.1% and base model 0%, the only two states that the official forward layer covers). The observed forward rates of the two policies are identical, 87.5% each (0.0 pp [-9.7, 9.7]); by rule they differ by +8.3 (T1) and −8.3 points (T2), so these comparisons do not establish equivalent mastery. On composition, sheet-only targets lead by 34.7 points (OS-2: [25.7, 43.8], Holm p < 0.001; official +34.7 pp [25.7, 43.8]), in each of the three seed pairs (+29.2, +35.4 and +39.6) and on both rules (+38.9 on rest-segment and +30.6 on two-current items). With the sheet in its context, the base model answers 98.3% of these items (fresh items: 92.5%): the new-argument items remain highly solvable when the p rules are supplied. Within T2, Gen answers are not more successful on items whose product k I/Iref occurs in training (55.6% of 21 items) than on items with a new product (59.8% of 39). On the second model, six Gen passes also answer 87.5% of the new-argument forward items (base model 0%; with the sheet, 100%). The configuration-bound form predicted this pattern; the argument-bound form (a forward gap of at least 10 points) and the value-association form (trained products at least 15 points easier) did not. This is the account’s one prospective confirmation that discriminates between explanations. It rules against pure lookup of the original argument grid as the source of the gap; it does not exclude mixed template-and-value mechanisms, and it does not show general symbolic extrapolation. The gap is larger here than on fresh items because the worked-solution policy is lower on these items (23.6% against 34.7%), while the sheet-only policy is not (58.3% against 59.0%); the item sets differ, so this comparison is descriptive.

6.2

A second bank pair: the direction recurs; the matched-dose test fails

Training seeds vary the order and sampling of training, not the capture, so the gap could belong to one pair of sampled banks. We recorded this replication before any of its requests (amendments 7 and 8): we captured two new samples per question for each policy, with the same questions, contexts and settings and new request seeds, accepted and selected them by the same rules, and trained

13

Table 2: At argument values that no training question used, forward use survives in both policies and the composition gap remains. New-argument items (MiniCPM5-2B, no context; three seeds per policy, per-item means). OS-1 (30-point margin) and OS-2 (10-point margin) are pre-specified. The official layer covers the forward answers of Gen and the base model and all composition answers. Items

Measure

Gen 6 Inst 6 Base model

Forward (24) Forward (24) Inversion (48) Composition (48) Composition (48)

signature official signature signature official

87.5 86.1 62.5 58.3 54.9

87.5 – 47.9 23.6 20.1

0.0 0.0 2.1 2.1 –

Gen 6 − Inst 6 0.0 [-9.7, 9.7] – +14.6 [2.8, 26.4] OS-2: +34.7 [25.7, 43.8] +34.7 [25.7, 43.8]

six passes with the first two training seeds of the original arms. The test RB-1 (value signature, 10-point margin) compares the new banks on the 48 fresh composition items. RB-1 holds (Table 3): sheet-only targets lead by 14.6 points [2.1, 26.0] (p = 0.033; official +16.7 pp [5.2, 28.1], 42 of 96 against 26 of 96 answers), and, as recorded in advance, the lead is larger on two-current (+22.9) than on rest-segment items (+6.2). The observed forward rates differ by 8.3 points (87.5% against 79.2%, [-10.4, 27.1]); the recorded ±10-point gate is a point-estimate rule, not an equivalence test, so at six passes this replication comes with a forward shortfall of the worked-solution bank. To remove that shortfall, we recorded a dose extension before any of its requests: the new worked-solution bank trained to ten passes (97% of the domain tokens of the new sheet-only bank), with the TB-2 rule (test TB-2R). Forward success was then 89.6% against 87.5%, and composition 39.6% against 45.8%: +6.2 points [-6.3, 18.8] (p = 0.43; official +7.3 pp [-5.2, 19.8]). TB-2R does not hold. For this bank pair, R2 is not supported at the recorded margin, and a difference in learning speed alone (R3) is not excluded. Does the second bank pair carry a smaller cost, or is it the same cost measured with noise? Our data cannot decide (Appendix B; exploratory). The gaps of the two bank pairs differ by +9.7 pp [-6.6, 27.1] at six passes and +10.1 pp [-5.6, 25.7] at matched dose, and seed pairs within one bank pair differ by as much: from +14.6 to +33.3 points within bank pair 1 at six passes, and +16.7 and −4.2 points within bank pair 2 at matched dose. The two sheet-only banks reach 59.0% and 45.8% composition success, but trained with the same seed they differ by +2.1 points [-14.6, 18.8] for one seed and +25.0 points [6.3, 43.8] for the other. With two banks per policy and two or three seeds per bank, variation between banks cannot be separated from variation between training runs. Pooled over both bank pairs, with each pair weighted equally per item (a descriptive estimate named in advance; the equal weighting was fixed when it was first computed, after the six-pass results), the lead is 19.4 points [12.3, 26.2] at six passes and 11.3 points [2.4, 19.8] at matched dose (official +21.2 and +12.8); these intervals are conditional on the four sampled banks and contain neither bank nor training-run variation. The direction of the gap recurs in both bank pairs and at both doses; at matched dose its size is uncertain, 6.2 and 16.3 points for the two bank pairs.

6.3

An answer-only context: the mediator stays open

Does the cost need the supplied method, or does a supplied final answer suffice? A third policy (Ans) captured targets with the rule sheet plus the line “Checked final answer of this question:” and the final-answer line of the worked solution, with no method and no intermediate value; capture, acceptance, selection and training followed the second bank pair (two seeds). Before any answer, we recorded a reading table and a gate: the comparisons AS-1 (Gen 6 minus Ans 6) and AS-2 14

Table 3: An independently sampled bank pair repeats the direction of the gap at a smaller size; at matched dose its own gap is not distinguishable from zero. Fresh items, value signature unless stated; per-item means over training seeds (number of seeds in brackets). Bank 1 is the bank of Table 1. Lower block: the worked-solution policy of each bank trained for ten passes (about the domain tokens of six sheet-only passes); the bank-2 composition row is the test TB-2R. Pooled: mean of the two bank-pair differences per item, with a 95% item-bootstrap interval (descriptive, conditional on the four banks). Bank 1 Items

Gen [3] Inst [3] Gen [2] Inst [2] Bank 1 Bank 2

Forward (24) Inversion (48) Composition (48) Composition, official

86.1 66.7 59.0 56.9

83.3 50.0 34.7 31.2

87.5 60.4 45.8 43.8

+8.3 +27.1 +14.6 +16.7

+5.6 [-5.6, 17.4] +21.9 [13.9, 30.0] +19.4 [12.3, 26.2] +21.2 [14.6, 27.6]

Matched dose: worked-solution policy trained for ten passes [2] Forward (24) 86.1 85.4 87.5 89.6 +0.7 Inversion (48) 66.7 54.2 60.4 58.3 +12.5 Composition (48) 59.0 42.7 45.8 39.6 +16.3 Composition, official 56.9 38.5 43.8 36.5 +18.4

−2.1 +2.1 +6.2 +7.3

−0.7 [-10.8, 10.8] +7.3 [-0.3, 15.1] +11.3 [2.4, 19.8] +12.8 [4.2, 21.2]

80

sheet only sheet + worked solution

70

79.2 33.3 31.2 27.1

Pooled

+2.8 +16.7 +24.3 +25.7

Sealed items: training ladders (numbers: passes)

composition success (%)

Gen − Inst

Bank 2

Fresh items: evaluated states (untrained model at 0, 0)

6

bank pair 1

answer only

bank pair 2

more passes GEN-T 6

60 50

ANS-T 8

10 4

40

ANS-T 10 INST-T 10

GEN-R 6

6

INST-R 10 ANS-T 6

30

INST-T 6 INST-R 6

20 10

4

2

0

2

0

20

40 60 forward success (%)

80

100 60

65

70

75 80 85 forward success (%)

90

95

Figure 3: More training moves the worked-solution policy along its own curve; with bank pair 1 that curve stays below the sheet-only policy, and with bank pair 2 it comes close. Composition against forward success (value signature). Left: sealed items along the training ladders of bank pair 1 (means of training seeds 1 and 2; the untrained model is the first point; numbers are passes); the mean vertical distance between the curves over forward rates of 50–80% is exploratory. Right: fresh items at every evaluated state; arrows join states of one policy and bank after more passes (worked-solution policy: six to ten passes; answer-only policy: six, eight and ten passes).

15

(Ans 6 minus Inst 6), on the original banks’ states, would be read only if Ans forward success was within 15 points of Gen. It was not: the answer-only policy reached 70.8% against 86.1%, 15.3 points below, so no reading applies. Descriptively, its composition success was 31.2% (Inst 34.7%, Gen 59.0%; AS-1 +27.8 pp [17.7, 37.8], AS-2 −3.5 pp [-14.9, 8.3]). A lower forward rate puts a state lower on its policy’s curve, so the failed gate made the result uninformative about the account’s prediction in either direction. We then recorded, before any request, a dose extension of Ans to eight passes (103% of the domain tokens of six Gen passes), read with the unchanged table, and a fixed second look at ten passes (129%) if eight passes failed the gate. Eight passes failed the gate again: forward success 68.8%, 17.4 points below, while composition rose to 44.8%. Ten passes passed it (77.1%, 9.0 points below) and gave the “graded” row of the table: Ans 10 composition (46.9%) is 12.2 points [0.7, 23.6] below Gen 6 and 12.2 points [0.7, 23.6] above Inst 6. The support for this reading is weak. Both lower bounds are 0.7 points after two looks without adjustment. The official judge places Ans 10 nearer Inst 6 (+18.4 pp [7.3, 29.5] and +7.3 pp [-4.9, 19.8]), the pattern of the row in which the answer alone gives most of the cost. The recorded comparators are states of the original banks, and against the second bank pair, captured in the same campaign as Ans, its composition rate is within 7.3 points of both policies (exploratory, chosen after the result). From six to eight passes, Ans composition rose by 13.5 points [-1.0, 27.1] while its forward rate did not rise (70.8% and 68.8%); movement along one curve does not produce that, so either the common link fails for this policy or the change is noise. The account’s most probable reading, a position near the sheet-only policy, did not occur, and the data do not show whether a supplied final answer alone produces most of the cost.

6.4

Scorecard

Table 4 lists every recorded prediction with its probability and outcome. The binding test is the account’s one discriminating prospective confirmation. The bank replication held as recorded at six passes, but the dose ordering (R2) held only for bank pair 1, where it was retrospective, and failed its margin for bank pair 2. The predictions about the likelihood diagnostic D held, but their sign is close to guaranteed by construction (Section 7), so they are a weak check. The answer-only prediction did not hold. These outcomes are correlated readings of a few experiments, not a combined statistical validation of the account.

7

What the targets and the trained models write

The analyses in this section were defined after the results were known. They are program counts and readings of saved text, not tests, and they describe text, not internal mechanisms; they indicate which properties of the targets an edit experiment would have to separate. Both policies write and evaluate the rules; they differ in what surrounds the computation. Table 5 describes every accepted training target of the original banks. Both policies write each rule symbolically in every target and evaluate it with the question’s numbers in 84–96% of targets, so the worked solution does not remove the computation from the targets. Inst targets name the worked solution in 81–96% of cases, state in 42–47% that a computed value matches it, and contain fewer markers of the model’s own deliberation and checking; the reduction is strongest for T2, the rule with the larger composition gap. The second bank pair repeats the pattern (median deliberation markers 18 and 24 in the sheet-only targets and 9 and 4 in the worked-solution targets,

16

Table 4: The binding prediction held; the dose-matched second bank pair and the answer-only policy did not follow the account. Predictions recorded before each test, with their probabilities and outcomes (value signature; “gap”: Gen minus Inst after six passes unless stated). Answer-only readings were recorded with probabilities 0.40 (near Gen: the cost needs the method), 0.20 (graded), 0.15 (near Inst: the answer alone gives the cost) and 0.25 (other or not comparable) before the six-pass test, and 0.25, 0.05, 0.35 and 0.35 before the dose extension. Test

Prediction

New arguments

forward gap at least +10 pp (argument-bound) forward gap within ±10 pp (configuration-bound)

Likelihood D

Second banks

Answer only

Matched dose

Prob. Outcome

0.35 no: 0.0 [-9.7, 9.7] 0.55 yes (observed rates; not equivalence) composition gap minus forward gap positive 0.60 yes: +34.7 T2: trained product at least +15 pp above new 0.35 no: −4.2 product (value association) larger for Inst than for Gen targets, each rule 0.90 yes; sign close to guaranteed by construction arm difference larger for T2 than for T1 0.65 yes (two rules; weak) RB-1 holds 0.55 yes: +14.6 [2.1, 26.0] forward gap within ±10 pp 0.70 yes: +8.3 (point-estimate gate) composition gap larger on T2 than on T1 0.65 yes: +22.9, +6.2 pooled composition gap at least +15 pp 0.60 yes: +19.4 [12.3, 26.2] forward success within 15 pp of Gen 0.75 no: 15.3 below reading of AS-1, AS-2 (see caption) not comparable deliberation markers between the two policies 0.70 yes second-bank Inst 10 forward within ±10 pp of Gen 0.80 yes: −2.1 6 TB-2R holds (gap at least +10 pp, interval above 0.30 no: +6.2 [-6.3, 18.8] 0) TB-2R gap positive 0.85 yes answer-only, eight passes, forward within 15 pp 0.80 no: 17.4 below answer-only reading (see caption) graded, at ten passes answer-only composition rises from six to eight 0.80 yes: +13.5 passes

for T1 and T2). In Gen targets, marker count and length are almost collinear (r = 0.93 and 0.96), so these data cannot separate deliberation from length, and none of these counts is shown to mediate the gap. The likelihood diagnostic D records where a target came from, not what the student learned. Adding the worked solution to the sheet raises the base model’s log-likelihood of every Inst target and lowers that of every Gen target (Table 5), as we had predicted with probability 0.90. This sign pattern is close to guaranteed by construction. Let qs and qR be the base model’s target distributions with and without the solution, and D(y) = log qs (y)/qR (y). Then Eqs [D] = KL(qs ∥ qR ) ≥ 0 and EqR [D] = −KL(qR ∥ qs ) ≤ 0, so targets sampled with the solution tend to have positive D and targets sampled without it negative D; sampling temperature, acceptance filtering and per-token normalization keep this from being exact. D therefore measures how far the solution moves the base model’s distribution of targets. It does not measure the supervision that reaches any route of the trained student, and because the solution is a deterministic function of the sheet and the question, it is not an estimate of a conditional mutual information. Tian et al. [22, Sec. 4.3] likewise find that teacher–student discrepancy does not order privileged contexts by utility.

17

Table 5: Both policies compute with the rules; the worked-solution targets deliberate and check less and confirm against a solution that will be absent. Accepted training targets of bank pair 1, both samples (program counts defined after all results). “Deliberation markers”: words such as wait, hmm and maybe; “self-check phrases”: verify, double-check, recompute and similar. D: mean of log p0 (y | x, R, s) − log p0 (y | x, R) per token under the base model p0 , one target per question, where x is the question, R the sheet and s the worked solution. Targets Rule written symbolically (%) Rule evaluated with the question’s numbers (%) Worked solution named (%) A value stated to match the solution (%) Median length (tokens) Median deliberation markers Median self-check phrases D (nats per token)

Gen T1

Inst T1

Gen T2

Inst T2

121 100 91.7 0 0 2,746 18 1 −0.091

119 100 95.8 80.7 42.0 2,268 10 0 +0.139

128 100 84.4 0 0 3,339 26 2 −0.090

127 100 85.8 96.1 46.5 1,501 4 0 +0.198

The worked-solution models expend more test-time tokens on composition, yet still fail more often. A natural suspicion is that training on shorter, solution-guided targets teaches the student model to generate shorter responses at test time. On fresh compositional items, the opposite occurs: the median six-pass Inst response spans 8,837 generated tokens compared to 4,744 for Gen, whereas on practiced forward items the ordering reverses (2,748 versus 3,588 tokens). The second bank pair exhibits the same pattern on composition (8,700 versus 5,349 median tokens). Lee et al. [10, Sec. 4.1] report a similar inversion of training and test lengths for answer-conditioned chains in mathematics; here the longer answers appear on the unrequested operation, not on the practiced one. Under composition, both policies often fall back to the familiar law. On fresh two-current items, about two fifths of the answers of each six-pass policy carry the SOC that a constant capacity would give (40% and 42%), although both apply the rate-dependent usable charge in 89% of six-pass forward T2 answers; the policies differ mainly in answers that match no known value (18% against 43%; Appendix C). Worked-solution training leaves references to the absent solution. After training, Inst models refer to a worked solution or a sheet in the Thinking of 76–85% of their answers, although none is present (six-pass Gen: 1–6%; base model 2%). In joint teacher–student training, references to privileged information in student outputs did not grow with training [16, App. D.1]; offline training on targets that cite the solution does pass the citations on. This is a pattern count; it does not show that the references cause the composition gap. The rules appear in solutions, not on request. After six passes, both policies write the T1 factor in 86–100% of fresh forward T1 answers, but none of nine answers of the six-pass Gen seeds to direct recall probes states either relation (Appendix C); we read this as elicitation behavior [2], not as absent knowledge.

18

Table 6: The worked solution saves at capture and training and spends at inference. Measured from the saved run records. Capture: base model, two samples per training question. Update time: sum of the measured optimizer-step times of all passes, mean over two seeds (model loading excluded). Inference: the fresh composition items, no context; “tokens per success” divides all generated composition tokens by the value-signature successes. The ten-pass states use the targets of the six-pass state of the same bank. Bank pair 1 Gen 6

Inst 6 Inst 10

Capture: accepted / judged answers 249/256 246/256 Capture: generated tokens per accepted target 3,568 2,057 Training: supervised domain tokens (M) 2.67 1.49 Training: update time per seed (s) 1,894 1,357 Inference: median tokens per composition answer 4,744 8,837 Inference: mean tokens per composition answer 6,025 9,198 Inference: tokens per composition success 10,206 26,491

8

Bank pair 2 Gen 6

Inst 6 Inst 10

(same) 242/256 247/256 (same) 3,533 2,023 2.49 2.57 1.50 2,245 1,664 1,224 6,526 5,349 8,700 7,071 6,264 9,073 16,556 13,666 29,032

(same) (same) 2.49 2,076 6,710 7,658 19,346

What the choice costs in practice

Capture and training costs describe only part of the policy trade-off; we also measure inference on unrequested composition. Table 6 gives all three for each policy and bank pair, measured from the saved run records of one A40 GPU with one container image. Measured. Against the sheet-only policy of the same bank pair, the worked-solution policy needs 42% fewer generated tokens per accepted target at capture and 28% less update time for six passes (bank pair 1; 43% and 26% for bank pair 2), with nearly the same acceptance rate. At inference, its models generate 53% more tokens per composition answer (means of 9,198 against 6,025) and 2.6 times as many per composition success under the value signature (26,491 against 10,206); with bank pair 2 the factors are 1.45 and 2.1. Ten worked-solution passes reduce the inference cost per composition success under the value signature (16,556 and 19,346 tokens) but take more update time than six sheet-only passes (2,245 against 1,894 seconds and 2,076 against 1,664) and remain less accurate on composition (Table 3). Across the seed-aggregated bank-by-dose conditions in Table 6, the worked-solution policy has neither a higher composition success rate nor fewer generated tokens per value-signature success than the sheet-only policy of its bank pair. Estimated. To compare the savings with the extra inference, we convert update time into generated-token equivalents at the measured generation throughput of the same GPU (1,191 tokens per second in the evaluation job of the second bank pair, with 64 concurrent requests). On this conversion, the worked-solution policy of bank pair 1 saves 382,000 capture tokens and about 639,000 token equivalents of update time, and spends about 3,170 more tokens per composition query. Under this conversion, after roughly 320 composition queries the worked-solution policy has used more compute in total, while it has fewer value-signature successes (about 310 queries for bank pair 2, computed the same way). This estimate assumes that capture, training and inference compute are interchangeable at that rate; it ignores prefill and the cost of producing the worked solutions (a program here, but a solver or a stronger model in practice), and it holds only for this model, task mix and hardware.

19

9

Interpretation, scope and limitations

What the comparison shows. Two capture contexts that a practice-accuracy criterion would treat as equivalent produced models that composed the new rules differently, and the context that was cheaper to capture and train was the one that used more generated tokens per composition success under the value signature at inference. The surrogate of Section 5 offers one account of why the practice metric misses this: in the surrogate, practice accuracy measures progress along the direction that the targets set, not the direction itself. The evaluation that exposed the difference needs no further training: unrequested-operation items with exact keys, answered without context (48 fresh composition items per state here). Equal forward rates, not equal mastery. Equal observed forward rates do not establish equal mastery of the rules: the forward comparisons are superiority tests that did not reject, with intervals of about ±10 to ±19 points. The claim concerns what a practitioner observes: at the forward rates that a held-out evaluation reports, identical on the new-argument panel (87.5% each) and within 3 points on the fresh items, composition differed by 24 to 35 points. Where forward rates differ more, we say so (bank pair 2 at six passes: 8.3 points; the answer-only policy: 15.3 points). The trajectory view addresses a stronger form of the concern that the worked-solution models know the rules less well: along the sealed ladders, the worked-solution states lie lower at every forward rate that both policies reach (Figure 3; exploratory). It does not identify latent mastery. How much of the cost goes with training dose. Worked-solution targets are shorter, so six passes supervise fewer domain tokens. Trained for ten passes, close to the domain tokens of six sheet-only passes, the worked-solution policy gains composition in both bank pairs, and the pooled gap falls from 19.4 to 11.3 points. With bank pair 1 a gap of 16.3 points remains and the pre-specified test holds; with bank pair 2 the gap is 6.2 points and the test fails, so for that pair a gap at matched dose is not established. Ten passes also change replay, target visits, updates and learning-rate cycles, so neither result isolates one dose quantity. Values of the training arguments. Memorized argument values do not explain the gap: at argument values that no training question used, both policies keep 87.5% forward success, the composition gap is +34.7 points, and trained products are not easier than new ones (Section 6.1). Mixed template-and-value mechanisms are not excluded. Target banks and the judge. A second, independently captured bank pair gives the same direction at both doses, and every one of the five six-pass seed pairs on fresh items favors the sheet-only policy (+14.6 to +33.3 points); how much of the remaining variation comes from banks rather than training runs cannot be told from two banks (Section 6.2). The pre-specified measure is the program-checked signature, not the judge, and the judge agrees with it in direction on every test; because both read the same answers, this agreement does not validate the judge (Appendix F). What the targets contain. “Unpracticed” means not requested by the training questions. A pattern scan of the targets found no inversion and a few passing mentions of a zero-current segment; a complete semantic reading was not done, because the reading model’s content filter blocked it and we did not repeat it with another model. This limits claims about what the models were never exposed to. The complete-policy comparison includes any differences in the generated targets; the

20

present audit does not determine whether incidental exposure to compositional steps contributes to the observed gap. Domain and model. The invented domain is what makes the measurement possible: the rules are new to the model, the keys are exact, and practiced and unpracticed uses are separated by construction. Its cost is generality, which we do not claim. The second model repeats the acquisition result, including at new argument values, but its unpracticed transfer was too weak for the policy comparison to be worth running. Limitations. • Mediator. The worked solution changes target length, content, references and supervised tokens together, and we do not identify which change produces the gap. The surrogate of Section 5 is one account; the absent-reference account and the package alternative are not excluded. • Extensions are cells, not a grid. New argument values were tested with bank pair 1 and six-pass states; bank pair 2 was tested on fresh items only. • Variation. Two target banks per main policy and one for the answer-only policy; two or three training seeds per bank. Item-bootstrap intervals do not contain variation between banks or training runs, and our data cannot separate these two sources. Comparisons across bank pairs and with the answer-only policy cross evaluation jobs that used the same request seeds and device type but different batching. • Measures. The signature checks numbers, not derivations; the official judge is a language model with sentinel checks, and the ten-pass verdicts of bank pair 1 come from a later batch than its six-pass verdicts. One numeric-parser repair was made after outputs existed (Appendix E); it changed no earlier signature. • Scope. One invented domain with two rules, the policy comparison on one model, one training recipe, and up to ten passes. • Training records. The run records of fourteen training passes (the third Gen and Inst seeds and the second Kept seed) were not copied from the training storage during the study; the companion repository takes them from the author’s archive of that storage and checks them against their training jobs under the trainer’s rules, a file listing made on the storage before it was archived, and the evaluated checkpoints. Future work, in order of what it would change. (1) Several independently sampled banks per policy, each trained to matched dose with at least two seeds, to estimate how much the size of the cost varies between banks and training runs; the second bank pair on new argument values belongs here. (2) An edit test of the mediator: remove the references to the solution from worked-solution targets, against an edit-matched control at a fixed budget. (3) The policy comparison on a model whose unpracticed transfer is strong enough for adequate power, and in a second domain. (4) A masked human audit of a preselected answer panel and a semantic audit of the training targets. (5) For the answer-only policy, comparators captured in the same campaign.

10

Conclusion

Similar forward accuracy can conceal a substantial difference in the sequential reuse of newly learned rules. In this controlled study, a worked solution made accepted self-generated targets shorter and reduced capture tokens and update time, yet the resulting models composed the rules less well. At new argument values, both policies reached 87.5% observed forward accuracy, while composition 21

accuracy was 58.3% with sheet-only targets and 23.6% with worked-solution targets. The gap was also confirmed prospectively on fresh items and repeated in direction with independently sampled targets. Longer training reduced the difference. At approximately matched domain-token dose, the original bank pair retained a 16.3-point gap, while the second pair’s 6.2-point gap did not meet its pre-specified test. Training amount therefore explains part of the observed separation; the remaining gap at matched dose is uncertain. At six passes per policy, the worked-solution models used 2.6 times as many generated tokens per composition success under the value signature (2.1 times with the second bank pair), despite their lower capture-token and update-time requirements. The practical lesson is to evaluate capture contexts by the uses they teach, not just the targets they make easier to produce. For self-generated rule learning, test reuse beyond the practiced operation and count the inference cost of that reuse before selecting a context on forward accuracy alone.

Statements Author contribution and AI use. Jiantong Xu conceived the research program, its ideas and the questions of this study, designed the experiments, set their methodological direction, made the scientific decisions as the work developed, and is responsible for all content. AI systems assisted under the author’s direction: Claude (Anthropic; mainly Claude Opus 5.5 in Claude Code) and Codex (OpenAI) in experimental implementation and execution; Claude, Codex and ChatGPT (OpenAI) in scientific discussion and critical review of the design, analyses and manuscript; and Claude and ChatGPT in the analysis, the literature search and the drafting and revision of the manuscript. Claude Opus 5.5 also made the automated judgments described in Section 3. The Kestrel items, keys and worked solutions are generated by the study’s programs; the rule sheets and rubrics were written with AI assistance, and the battery-domain teaching texts of Appendix G come from an earlier question bank of the project that was written with AI assistance and partly adapted from outputs of DeepSeek and Monica. Funding.

This research was self-funded and received no external funding or other support.

Competing interests.

The author declares no competing interests.

Compute. All training and inference used one NVIDIA A40 (48 GB) at a time. The service queue records 34 GPU hours of task time for the project’s two studies of this question, including the familiar-domain study (Appendix G) and a second-model candidate that failed its gate; billed machine time, including setup, was at least 36 hours. The new-argument evaluation, the likelihood scoring, the second bank pair and the answer-only policy added 6.9 hours of task time (7.25 hours billed), and the dose extension added 3.0 hours of task time (3.1 hours billed). Availability. The companion repository github.com/JetXu-LLM/kestrel-capture-context (release v1.0.0) contains the item generator and all item sets, the protocols and amendments with a record of when each was first written, the training jobs and target selections, every model answer and judgment, the analysis code and derived results, and a script that recomputes on a CPU, from the saved answers and verdicts, every success rate, contrast, interval and test outcome in Sections 4 and 6 and Appendix B, every plotted value, and the answer lengths and outcome types of Sections 7 and 8; the statistics of the training targets and the capture and training costs come from the 22

study’s own analysis scripts, which the repository also contains. Every trained adapter, including the intermediate passes and the controls, is a release asset of the repository with its configuration, base-model revision and checksums. Code and adapters are released under the Apache License 2.0, and data, materials and documentation under CC BY 4.0.

References [1] Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, and Jacob Andreas. Deductive closure training of language models for coherence, accuracy, and updatability. arXiv preprint arXiv:2401.08574, 2024. Version 2. [2] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14316, 2023. Version 3. [3] Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03395, 2020. Version 2. [4] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07780, 2020. Version 5. [5] Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning LLMs on new knowledge encourage hallucinations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. arXiv:2405.05904. [6] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024. Version 2. [7] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. arXiv:2106.09685. [8] Yubo Jiang, Fengying Xie, Zhiguo Jiang, and Haopeng Zhang. Distill skills into weights, not prompts: Abstract skills as privileged signals for on-policy self-distillation. arXiv preprint arXiv:2608.09826, 2026. Version 1. [9] Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of LLMs? arXiv preprint arXiv:2603.24472, 2026. Version 4. [10] Jungseob Lee, Seungyoon Lee, Suhyune Son, Dongyub Jude Lee, Sungbin Han, Sugyeong Eo, and Heuiseok Lim. Answer-conditioned chains of thought degrade verifiable-reasoning distillation in large language models. arXiv preprint arXiv:2607.14552, 2026. Version 1. [11] Jiarui Liu, Lechen Zhang, Yongjin Yang, Yinghui He, Yingheng Wang, Weihao Xuan, Zhijing Jin, and Mona T. Diab. MixSD: Mixed contextual self-distillation for knowledge injection. arXiv preprint arXiv:2605.16865, 2026. Version 3.

23

[12] Tianci Liu, Zihan Dong, Tianchun Li, Yi-Chung Chen, Qiming Cao, Xingchen Wang, Shiyang Wang, Zichen Miao, Linjun Zhang, Haoyu Wang, and Jing Gao. Hybrid-policy self-editing for composable unstructured knowledge editing. arXiv preprint arXiv:2608.11660, 2026. Version 1. [13] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643, 2015. Version 3. [14] Shankar Padmanabhan, Yasumasa Onoe, Michael J. Q. Zhang, Greg Durrett, and Eunsol Choi. Propagating knowledge updates to LMs through distillation. In Advances in Neural Information Processing Systems, 2023. arXiv:2306.09306. [15] Core Francisco Park, Zechen Zhang, and Hidenori Tanaka. New news: System-2 fine-tuning for robust integration of new knowledge. arXiv preprint arXiv:2505.01812, 2025. Version 3. [16] Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942, 2026. Version 3. [17] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. Version 3. [18] Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026. Version 2. [19] Samyak Shrestha and Alexander Tessier. Rethinking privileged information in on-policy self-distillation. arXiv preprint arXiv:2608.18271, 2026. Version 1. [20] Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint arXiv:2209.15189, 2022. Version 1. [21] Yixuan Tang and Yi Yang. On-policy distillation teaches new skills but not new knowledge. arXiv preprint arXiv:2610.09639, 2026. Version 1. [22] Kanghui Tian, Siyuan Liu, Tianxiang Jiang, Shuai Dong, Yizhuo Li, Tian Ding, Yuan Guo, Songze Li, Haowen Hou, Congcong Wang, and Yi Wang. What should a self-teacher see? privileged context design for on-policy self-distillation. arXiv preprint arXiv:2609.25623, 2026. Version 2. [23] Wenkai Yang, Yankai Lin, Jie Zhou, and Ji-Rong Wen. Distilling rule-based knowledge into large language models. arXiv preprint arXiv:2311.08883, 2023. Version 3. [24] Lifan Yuan, Weize Chen, Yuchen Zhang, Ganqu Cui, Hanbin Wang, Ziming You, Ning Ding, Zhiyuan Liu, Maosong Sun, and Hao Peng. From f (x) and g(x) to f (g(x)): LLMs learn new skills in RL by composing old ones. arXiv preprint arXiv:2509.25123, 2025. Version 3. [25] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, 2022. arXiv:2203.14465. [26] XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, and Tat-Seng Chua. What does privileged information add to on-policy self-distillation? arXiv preprint arXiv:2609.20612, 2026. Version 1. 24

[27] Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15637–15653, 2024. [28] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. Version 3. [29] Xuyang Zhao, Liting Zhang, Zichen Xu, Zhihu Wang, Xu Caiyue, Shiwan Zhao, and Qicheng Li. Is more privileged information better? from solution traces to problem-solving structure in self-distilled reasoning. arXiv preprint arXiv:2608.01589, 2026. Version 1.

A

Acquisition, controls and side effects

Table 7: Six passes make the rules usable; at six passes the two policies have the same observed forward rate and different composition. Official success (%) of MiniCPM5-2B on the 124 sealed items without context, mean over seeds (number in brackets). s : value-signature rates of states without an official reading (the protocol judged an arm’s A and B items only once one of its states reached 20% forward signature success, and assigned Inst after four passes to the signature). State Base model, no sheet [2 draws] Gen [3] Gen [2] Gen [2] Inst [2] Inst [2] Inst [2] Inst [2] Kept [2]

Passes

Forward A (48)

Inversion (24)

Composition (24)

– 2 4 6 2 4 6 10 6

0.0 8.3 63.5 82.3 4.2 63.5s 82.3 89.6 0.0s

0.0 2.8 43.8 72.9 0.0 22.9s 60.4 56.2 0.0s

0.0 2.8 29.2 68.8 0.0 14.6s 35.4 50.0 6.2s

The dose contrasts, recorded in the protocol as descriptive (without a margin), are six minus two passes +74.0 pp [63.9, 83.3] and six minus four passes +18.8 pp [6.3, 31.3]; a third six-pass Gen seed reaches 91.7% forward signature success. The pre-specified tests that the capture context changes forward acquisition did not reject: six Inst passes against six Gen passes 0.0 pp [-11.5, 11.5], and ten Inst passes −7.3 pp [-17.7, 2.1]. On Qwen3-4B-Thinking-2507, six Gen passes raise forward signature success from 16.7% (two passes) to 81.2% (QT-1: +64.6 pp [53.1, 76.0]); the test of unpracticed operations against the base model gives +19.8 pp [11.5, 29.2] and misses its 20-point margin. That Kept fails to produce the rules does not show that nothing was learned internally. Abstention. On the 12 tested questions about a layer whose law is not given, the base model correctly says that the value is undetermined in 4 of 12 cases in each of two draws; after six Gen passes, both seeds answer 0 of 12 correctly (−33.3 pp [-50.0, -16.7]), usually by applying a learned or familiar law to the unknown layer. This is a result on the tested panel, not a general calibration or safety evaluation. 25

Table 8: Ten worked-solution passes match the domain tokens of six sheet-only passes but also change every other dose quantity. Training quantities of the evaluated states of bank pair 1 (MiniCPM5-2B), from the frozen training plans and the logged updates (the update counts of the third seed are planned counts; Section 9). State

Domain tokens Replay tokens Target visits Updates (per seed) LR cycles

Gen 6 passes Inst 6 passes Inst 10 passes

2,667,150 1,494,891 2,491,485

469,206 469,206 782,010

768 762 1,270

429 / 428 / 432 284 / 292 / 281 476 / 487

6 6 10

Retention. On a 56-item retention panel, one six-pass Gen seed scores 18 of 24 professional answers against 22 for the base model, which reaches the pre-specified damage threshold; the other seed scores 21. The panel cannot establish general non-inferiority. Declarative training first. Two passes on the model’s own declarative explanations of the rules, followed by two passes of use practice, give 47.9% forward success, below four passes of use practice (−15.6 pp [-29.2, -1.0]); about 37% of the accepted declarative texts contained a wrong explanation, so this is not a test of correct declarative teaching.

B

Sources of variation

The item-bootstrap intervals of the main text describe variation over items given the trained states. This appendix shows the variation between training runs and between target banks that those intervals do not contain. Each Gen seed is paired with the Inst seed of the same index; the two policies used the same pass seeds, and the second bank pair used the first two training seeds of the first. The bank-pair differences and the same-seed comparisons across banks are exploratory analyses that we defined after all results existed. Bank pair 1

fresh composition success (%)

80

Bank pair 2

Answer only

70 60 50 40 30 20 10

training seed 1

training seed 2

training seed 3

mean over seeds

0 GEN 6

INST 6

INST 10

GEN 6

INST 6

INST 10

ANS 6

ANS 8

ANS 10

Figure 4: Training runs of one bank vary about as much as the two banks do. Valuesignature success on the 48 fresh composition items for every trained state, one mark per training seed (hollow: ten passes), with the mean over seeds as a bar.

26

Table 9: Every six-pass seed pair favors the sheet-only policy; at matched dose three of four do, and the two bank pairs do not differ distinguishably. Composition (value signature, 48 items per seed pair), sheet-only minus worked-solution policy in percentage points, with 95% item-bootstrap intervals for the comparisons in the lower blocks. Comparison

Seeds or contrast

Six passes, fresh items, bank pair 1 Six passes, fresh items, bank pair 2 Six passes, new arguments, bank pair 1 Matched dose, fresh items, bank pair 1 Matched dose, fresh items, bank pair 2

seeds 1, 2, 3 seeds 1, 2 seeds 1, 2, 3 seeds 1, 2 seeds 1, 2

Same seed, two banks: Gen 6, bank 1 minus bank 2

seed 1 seed 2 seed 1 seed 2

+2.1 [-14.6, 18.8] +25.0 [6.3, 43.8] +10.4 [-6.3, 27.1] −4.2 [-22.9, 14.6]

six passes matched dose six passes, official matched dose, official

+9.7 [-6.6, 27.1] +10.1 [-5.6, 25.7] +9.0 [-7.6, 26.7] +11.1 [-5.6, 27.1]

Same seed, two banks: Inst 10, bank 1 minus bank 2 Gap of bank pair 1 minus gap of bank pair 2

C

Difference +14.6, +33.3, +25.0 +14.6, +14.6 +29.2, +35.4, +39.6 +8.3, +25.0 +16.7, −4.2

Additional diagnostics

Outcome types under composition. On fresh two-current items, about two fifths of the answers of each six-pass policy carry the SOC that a constant capacity would give (Table 10), although both apply the rate-dependent usable charge in 89% of six-pass forward T2 answers. The policies differ mainly in answers that match no known value. These are descriptive categories of visible numbers; they do not identify a hidden route. Table 10: Under composition the familiar law returns in both policies; the workedsolution policy adds answers that match no known value. Value-signature outcome types on the fresh composition items of bank pair 1 (percent of answers). “Familiar law”: a keyed quantity carries the value of a constant capacity or a familiar relaxation law; “one current only”: the usable charge of one current used for both segments; “partial”: some keyed quantities correct, none familiar; “other”: numbers that match no keyed value. One Inst 10 two-current answer (2.1%) had no final answer. Items (answers)

State

Two-current (72/72/48)

Gen 6 Inst 6 Inst 10 Gen 6 Inst 6 Inst 10

Rest-segment (72/72/48)

Correct

Familiar law

One current only

Partial

Other

38.9 8.3 16.7 79.2 61.1 68.8

40.3 41.7 60.4 1.4 2.8 2.1

2.8 6.9 2.1 – – –

0.0 0.0 0.0 4.2 22.2 18.8

18.1 43.1 18.8 15.3 13.9 10.4

The rules in solutions and on request. After six passes, both policies write the T1 factor (1 + h/τ )−2 in 86–100% of fresh forward T1 answers (2–14% after two passes; Kept 0%). Asked directly for the rules, the same models do not state them: in the nine answers of the three six-pass Gen seeds to the recall probes, none states either Kestrel relation. We read this as elicitation 27

behavior, in line with the format dependence of knowledge extraction [2]; it does not show that the knowledge is absent or inaccessible under every prompt.

D

The Kestrel rule sheet

The Gen capture context appends the following text to the system message (“Answer in English. Give the requested result and the scientific reason that supports it.”): You know these relations. Use the relations that apply to the question. Write each relation that you use in full. For a Kestrel cell with a source U, a series resistor R0, and a Varn layer in series, V = U - I R0 - v, where v is the Varn voltage. A constant-current segment starts at a current step and lasts until the next current step. At a time h after the start of a segment with current I, v = r v_start + (1 - r) g I, with r = (1 + h/tau)ˆ(-2), where v_start is v at the start of the segment. Here tau > 0 is the Varn time constant and g is the Varn gain. At an ideal current step, v is continuous. For a constant discharge current I > 0, the usable charge is Q_use = Q_nom (1 - k (I/I_ref)ˆ(1/2)), with k > 0 and k (I/I_ref)ˆ(1/2) < 1. During a constant-current discharge segment, the SOC decrease is the outward charge I h divided by Q_use. One Ah is 3600 C.

The Inst context adds a heading and a program-written worked solution of the question, which states the factor, the intermediate values and the final answer for that question only. The Ans context adds the heading “Checked final answer of this question:” and the final-answer line of that worked solution.

E

Pre-specification record

The study ran under two written protocols, of 26 October 2026 (the captures of bank pair 1, the first two training passes and their evaluation) and of 28 October (the training ladders and their sealed evaluation), and nine amendments to the second; each was written before the requests that it governs. Amendments 1 and 2 (28 October) added the second model and the Inst dose ladder; amendment 3 (31 October) recorded the prospective tests TB-1 and TB-2; amendment 4 (1 November) recorded the semantic judging of the ten-pass composition answers and the new-argument tests OS-1 and OS-2; amendment 5 (2 November, before any request of that evaluation) fixed their reading rules; amendment 6 (2 November) recorded the predictions of the three binding forms; amendment 7 (5 November, before any of its requests) recorded the second bank pair, test RB-1 and the likelihood predictions; amendment 8 (5 November, 07:11 UTC) set two training seeds per new bank and added the answer-only policy with its gate and reading table; and amendment 9 (6 November, before any of its requests) recorded the dose extensions with test TB-2R and the answer-only second look. The recorded campaign events place the write of amendment 3, including the TB-1 and TB-2 rules, at 00:52:36 UTC on 31 October 2026, the generation of the fresh items at 00:52:50 and the first evaluation request at 02:06:45; the heading of the amendment gives 01:30 UTC, which is not the recorded write time. These dates come from the study’s own records and execution logs, not from an external registry. For each protocol and amendment, the companion repository gives its write time, every later change with its time, and the first queue or request record of every run that it governs (protocol/CHRONOLOGY.md). Two kinds of departure from the records occurred. First, the new-argument evaluation recorded on 1 November ran only on 5 November: no GPU was available in the data center that held the trained adapters during two bounded attempts, and a budget rule then stopped it until the author replaced the rule; its items and tests were unchanged, and its reading rules were revised (amendment 5) before any of its requests. Second, one scoring repair followed the outputs: on 5 November, after the new-argument answers existed but before any reading of them, 28

the value-signature parser was changed so that a product such as 35 × 10800, which it had read as 35 × 1010800 and which made a conversion overflow, becomes an infinite number that matches no key. The repair changed no earlier signature, and such misreadings can only make a number fail to match. Table 11: Every pre-specified test, with its recorded probability and outcome, including the ones that failed. Test or prediction

Recorded

Probability

Outcome

TI-1, TI-2 (the capture context changes forward acquisition) QT-1 (second model, forward dose) QT-2 (second model, unpracticed operations, 20 pp margin) Kept six passes, forward signature at most 5% (first; second seed) TB-1 (fresh composition, same passes) TB-2 (fresh composition, about the same domain tokens) Official TB-2 at least +10 pp OS-1 (new arguments, 30 pp margin) OS-2 (new arguments, 10 pp margin) RB-1 (second bank pair; two seeds per arm, replacing the three-seed version of A7, 0.60) Answer-only policy, forward within 15 pp of Gen TB-2R (second bank pair, ten worked-solution passes) Answer-only policy after eight passes, forward gate Answer-only reading at the second look (ten passes) Familiar battery domain: T-D, T-S, T-V (Appendix G) Predictions of Table 4

A2

0.35, 0.20

not held

A1, A2 A1, A2

0.60 0.50

held not held (+19.8)

protocol; A3

0.80; 0.85

held (0%)

A3 A3

0.60 0.40

held held

A4 A4 A4 A8

0.65 0.75 0.45 0.55

held (+18.4) held (+87.5) held (+34.7) held (+14.6)

A8 A9

0.75 0.30

not held (−15.3) not held (+6.2)

A9 A9 earlier study

0.80 0.05 (graded) 0.65, 0.55, 0.45

not held (−17.4) graded not held

A6 to A9

0.30–0.90

see that table

F

Agreement of the two measures

Table 12 compares the value signature with the official endpoint on every answer of the study’s second phase that has a semantic assessment. The new-argument rows exclude the 24 forward answers of the base model (all failures on both measures); the row for the later banks covers every state of the second bank pair (six and ten passes) and of the answer-only policy (six, eight and ten passes). Answers that failed the automatic checks of the endpoint, and judge dispositions without an assessment (for example, without a final answer), are endpoint failures and are excluded here. The two measures read the same final answers, so their agreement is not an independent replication. In the column “Sig. T, off. fail”, the final answer contains every Kestrel value but the judge rejected its conclusion or support or found a contradiction. On MiniCPM5-2B, all 13 sealed inversion answers in the last column belong to two items whose Kestrel value the signature classifies as ambiguous (counted as not T by the pre-specified rule), and for 8 of the 9 forward answers in that column the signature found, for at least one keyed quantity, the value of a familiar law and no Kestrel value.

29

Table 12: The two measures agree on 92.5–98.6% of assessed answers per population. Value signature against official endpoint, answers with a semantic verdict. Model

Items

MiniCPM5-2B MiniCPM5-2B MiniCPM5-2B MiniCPM5-2B MiniCPM5-2B MiniCPM5-2B MiniCPM5-2B Qwen3-4B-Thinking Qwen3-4B-Thinking Qwen3-4B-Thinking

sealed A sealed B-inv sealed B-comp fresh B-comp new-argument A new-argument B-comp fresh B-comp (later banks) sealed A sealed B-inv sealed B-comp

G

Answers Agree (%) Sig. T, off. fail Sig. not T, off. success 479 239 240 383 72 288 572 144 72 72

94.8 92.5 93.8 96.9 98.6 93.8 95.6 96.5 98.6 94.4

16 5 15 12 1 14 25 5 1 4

9 13 0 0 0 4 0 0 0 0

A familiar-domain comparison that did not survive a dose control

The question of this paper arose in an earlier study on familiar battery questions (256 training questions; the same model and recipe). There, captures with a general relation sheet beat captures with a worked solution on 48 sealed closed-book problems by +21.5 pp. A recalibration of the judge reduced the gap to +17.4 pp, and a run that trained the worked-solution arm to about the same cumulative domain tokens left +10.4 pp [-0.7, 21.5], which did not meet its pre-specified criterion. Two further pre-specified tests (targets with fewer uncertainty markers, and a context with only the checked final answer) also did not hold. The primary claim for the familiar domain was not confirmed, and the residual difference is unresolved. Unlike the Kestrel items, these problems use relations that the base model already knows, so they measure improved use of existing knowledge rather than the use of new rules.

H

Literature reading record

We read the full texts (or, where marked, the abstract or named parts) of the closest prior work, at these versions: Shenfeld et al. [18] (2601.19897v2, Sections 1 and 4 and Appendix A.2); Kim et al. [9] (2603.24472v4, Sections 1–7); Lee et al. [10] (2607.14552v1, abstract, Sections 1, 3, 4.1, 6 and 7, Appendices A and G); Zhang et al. [27] (Findings of ACL 2024, abstract, the opening of Section 3 and Section 3.1); Zhao et al. [28] (2601.18734v3, abstract and Sections 3.2 and 4.2); Tian et al. [22] (2609.25623v2, Sections 1–5, including Section 4.3); Zhang et al. [26] (2609.20612v1, abstract, Sections 1, 2 and 3.1 and the opening of Section 3.2); Penaloza et al. [16] (2602.04942v3, abstract, Sections 1, 3, 4.1–4.3 and 5–10, Appendix D.1); Shrestha and Tessier [19] (2608.18271v1, abstract, Sections 1–2 and 4, Appendices A.1, A.2 and A.4); Liu et al. [12] (2608.11660v1, abstract, introduction and method); Liu et al. [11] (2605.16865v3, Sections 3–6); Park et al. [15] (2505.01812v3, Sections 3–5); Yang et al. [23] (2311.08883v3, Sections 4–5.1); Yuan et al. [24] (2509.25123v3, Sections 3–4.3); Tang and Yang [21] (2610.09639v1, Sections 3–4.1); Padmanabhan et al. [14] (2306.09306v2, Sections 1–4 and 8); Snell et al. [20] (2209.15189v1, abstract and Section 3); Zelikman et al. [25] (2203.14465v2, Sections 3.1–3.2); Lopez-Paz et al. [13] (1511.03643v3, abstract and the openings of Sections 3 and 4); D’Amour et al. [3] (2011.03395v2, abstract and Sections 2.1–2.2); Geirhos et al. [4] (2004.07780v5, abstract); Press et al. [17] (2210.03350v3, abstract); Allen-Zhu and Li [2] (2309.14316v3, abstract); Gekhman et al. [5] (2405.05904v3, abstract); Akyürek et al. 30

[1] (2401.08574v2, abstract); Zhao et al. [29] (2608.01589v1, abstract and an earlier full reading); Jiang et al. [8] (2608.09826v1, abstract and main results). Focused searches covered offline rejectionsampling fine-tuning with hint-conditioned generation; comparisons of general principles and worked examples in context distillation or fine-tuning; references to an absent reference solution after self-distillation; and installing invented rules or operations and testing untrained operations. The search was not exhaustive; arXiv search coverage is incomplete, and several on-policy papers were read at the abstract level only.

31

Related documents

• Learning Reusable Agent Subroutines from Experience#1344605
• Measuring Weak-to-Strong Legibility of Reasoning Models#1624603
• Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Composition#1011232
• PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces#1014930
• Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking#1342976