arXiv CSopen access

Sibyl-AutoResearch: Autonomous Research Needs Self-Evolving Trial-and-Error Harnesses, Not Paper Generators

Sibyl-AutoResearch: Autonomous Research Needs Self-Evolving Trial-and-Error Harnesses, Not Paper Generators Chengcheng Wang1* Qinhua Xie2* Wei He3 Jianyuan Guo4 Shiqi Wang4 Chang Xu1 University of Sydney 2 East China Normal University 3 TokenRhythm AI 4 City University of Hong Kong [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

arXiv:2605.22343v1 [cs.MA] 21 May 2026

1

Abstract Autonomous research systems increasingly make the scientific workflow executable: agents can propose ideas, run code, inspect results, and draft papers. But executable workflows do not by themselves produce research judgment. We analyze where current systems lose trial experience: weak evidence becomes prose, pilot signals become broad claims, memory remains textual, and recurring process failures do not change later behavior. We introduce Sibyl-AutoResearch, a self-evolving AutoResearch framework built around Scientific Trial-and-Error Harnesses. A harness lets agents run bounded trials, preserve positive and negative outcomes, and route lessons into later planning, validation, claim scope, scheduling, critique, writing, and harness repair. We formalize this through two auditable conversion units: trial-to-behavior conversion, which links trial signals to later research actions, and trial-to-harness-behavior conversion, which links recurring process failures to system updates. We implement the framework in S IBYL, a file-backed autonomous research system that exposes the state, roles, memory, gates, and artifact traces needed to inspect these conversion paths. A retrospective audit identifies eight high-confidence conversion events, with a median latency of one iteration and a maximum latency of three iterations. A recovered-failure registry further shows how five naturally occurring failure classes, including duplicate results, stale numbers, and unsupported statistics, were blocked, downgraded, or routed into later repair. These traces do not establish a comparative performance claim; they show that the proposed conversion units are recoverable from realistic autonomous-research workspaces. The S IBYL framework and system are available at https://github.com/Sibyl-Research-Team/AutoResearch-SibylSystem.

1

Introduction

Autonomous research is becoming a concrete systems problem. LLM-driven agents [33, 19, 4] can already read papers, write code, call tools, run experiments, revise drafts, and keep long contexts. The harder question is no longer only whether these agents can assist a human researcher. It is whether an agent can improve as a researcher across attempts: form better priors, notice fragile evidence earlier, stop bad stories before they become claims, and carry hard lessons into the next project. This is also how human research practice develops. A researcher who has spent months with a benchmark often knows which metric is brittle, which baseline result is suspicious, which negative * Equal contribution.

Preprint.

Trace Path Across Iterations (b) Closed Adaptive Loop

(a) Trial-to-Behavior Conversion (Audit Unit) Operationalize research judgment as an auditable link from trial signals to future behavior.

1

1. Trial Signal

update inform future trials

Observed outcome, failure, measurement, or insight ✓

Success

× Failure

!

6

Convert experience into updates to plans, checks, claims, and harness rules

5

Evidence

Decision

Update

reviewer objection

2

process failure

Trial

Run experiments and collect data

Auditable link (artifacts, validation) from trial signal to future decisions/actions

audit unit in the loop

Autonomous Research Harness

Memory Store lessons, failures, and negative results

Artifacts, results, and validation 4

validation insertion

task plan

claim downgrade

review

scheduler policy

evolution

harness repair

+

trace path

+

update at t + k

t

t+k

(trial time)

(future time)

Reflection

Evidence Maturity: Boundary Between Trial-and-Error and Paper Generation

4

Analyze and critique Claims

plan mutation

reflection

Counted only if signal, trace, and update are linked signal at t

Updated plan, new check, claim revision, or harness fix

Checks

Updates (to t+k iteration)

decision

3 Evidence

3. Behavior Change

Plan

Trace path

evidence failure

Hypothesis / Plan Propose ideas and plan trials

Behavior / Harness Update

Anomaly

2. Trace Path

Trial

Signals (From t iteration)

Trials generate experience, which is converted into better science and a better harness.

Policy

execution completion

pilot signal

analysis-ready evidence

paper-ready evidence

audited claim

(a) Trial-to-behavior conversion and the closed adaptive

(b) Auditable conversion events and the evidence matu-

harness loop.

rity boundary.

Figure 1: Two views of Scientific Trial-and-Error Harnesses. (a) An auditable conversion event links a trial signal, a trace path, and a later behavior change inside a closed adaptive harness loop. (b) A conversion is counted only when a signal at iteration t, a trace path, and an update at t + k are linked; the maturity ladder separates execution completion, pilot signals, analysis-ready evidence, paper-ready evidence, and audited claims.

result is useful, and which pilot result is not ready for a paper. This ability is often called intuition or taste. We call it research judgment: experience-backed behavior that changes what the researcher does next. The point is not that judgment is mysterious, but that it is produced by many concrete trials, mistakes, repairs, and reviews. Recent autonomous research systems automate many parts of the research loop. For example, end-toend AI scientist frameworks proceed from idea generation to draft papers [19, 35], while research assistant and co-scientist systems support hypothesis formation, literature synthesis, and candidate ranking [29, 9, 8]. Metric-driven loops can further search over code or methods when an explicit score is available [16]. Together, these systems show that agents can participate in scientific work, but they also expose a deeper design gap: completing research stages is not the same as accumulating research judgment. The gap becomes visible in the failures that occur after useful signals have already been observed. A pilot may expose a broken metric, but the next plan still relies on it. A reviewer objection may identify an unsupported claim, but the writer only polishes the claim. A failed GPU run may reveal a wasteful experiment order, but the scheduler repeats it. A reflection file may record the right lesson, but the planner, critic, or supervisor never receives it. These are not just bugs; they are missing update paths from trial history to later action. We address this missing-update-path problem with Sibyl-AutoResearch, a self-evolving AutoResearch framework built around Scientific Trial-and-Error Harnesses, and we instantiate it in S IBYL, a file-backed autonomous research system. By harness, we mean the research environment around the agent: its state, tools, roles, memory, gates, artifact contracts, compute policies, and repair mechanisms. It lets agents try ideas under bounded controls, preserve outcomes, and convert trial history into later action. In a strong harness, past trials change future research behavior, while recurring process failures change the harness itself. These two feedback loops form agent-harness co-evolution. The S IBYL system is not only a motivating example. It is the concrete implementation through which the framework was refined and audited. In S IBYL, research state, plans, role outputs, experiment artifacts, reviews, reflections, and writing products are stored as inspectable files. Early system runs repeatedly preserved useful signals without routing them into the next plan, claim boundary, validation gate, or scheduling policy. Those failures forced the framework to become more explicit about conversion units, role-specific routing, and harness self-evolution. We operationalize this argument with two auditable conversion units. Trial-to-behavior conversion asks whether a trial signal at iteration t changes an action at iteration t + k, such as a plan, validation, claim boundary, schedule, critique, or writing scope. Trial-to-harness-behavior conversion asks whether repeated process failures change gates, prompt overlays, telemetry requirements, scheduler policies, repair tasks, or protected constraints. The unit is small on purpose: a signal, a trace path, and a later behavior change. Figure 1a shows the audit unit inside the adaptive harness loop. The contributions are summarized as follows: 2

1. A failure analysis for autonomous research. We identify six recurring ways existing systems lose trial experience before it becomes later research behavior. 2. Two auditable conversion units. We define trial-to-behavior and trial-to-harness-behavior conversion as inspectable links from trial signals to later behavior and harness updates. 3. The Sibyl-AutoResearch framework. We distill seven harness functions and observable commitments for preserving evidence, routing memory, separating perspectives, managing compute, and repairing recurring failure paths. 4. A concrete S IBYL system. We describe S IBYL, a file-backed autonomous research system that implements the framework and preserves traces for stress-testing whether the proposed conversion units are observable in realistic autonomous-research workflows.

2

Failure modes: where autonomous research loses experience

Autonomous research systems already make many research actions executable. The core failure is subtler. A system can propose hypotheses, run code, optimize a metric, preserve logs, and write a draft while still losing the experience that those actions should have produced. We call the missing route an update path: the path by which a trial signal becomes a later constraint on planning, validation, claim scope, resource allocation, critique, writing, or harness behavior. F1: Paper completion hides evidence immaturity. A pipeline can finish a paper even when the evidence is weak or corrupted. The needed update path is from weak evidence to a claim boundary: the writer should downgrade or remove the claim, and the planner should receive a validation task. F2: Pilot signals collapse into paper claims. A cheap pilot can be useful because it points to a direction. It does not by itself support a broad scientific claim. The needed update path is from pilot signal to maturity state: the system should mark the result as exploratory and require a stronger full-scale gate before writing general conclusions. F3: Visible objectives become bad objectives. Metric-driven loops work well when the objective is trusted. Research metrics often are not. A broken metric should trigger measurement critique, control design, and baseline audit. If the system simply optimizes the same metric again, the trial did not become research judgment. F4: Memory stays textual instead of routed. Long context can preserve a lesson in text while failing to route it to the role that needs it. The needed update path is from lesson to role-specific behavior: planner checks, critic objections, supervisor gates, scheduler policy, or writer restrictions. F5: More trials do not improve trial policy. Many-trial systems can increase experiment volume without improving the order, cost, stopping rules, or sanity checks. A failed or wasteful run should change resource allocation, early-stop policy, or cheap-check ordering. F6: Process failures recur because the harness does not change. Missing artifacts, stale tables, incomplete telemetry, and paper/evidence synchronization errors are not only project-local mistakes. Repeated process failures should change the harness: new gates, prompt overlays, artifact contracts, telemetry requirements, repair tasks, or protected constraints. These failure modes are distilled from recurring patterns in S IBYL workspace traces and from comparison with prior autonomous-research designs. They are not presented as exhaustive. Their role is to identify where paper-completion systems lose experience and where a harness must expose an update path. For example, in one sparse-autoencoder replication run, a strong writing score coexisted with duplicate result files, a feature-count mismatch, a missing sparsity-matched control, and many inactive sparse features. In an image-augmentation pilot, a CIFAR-10 direction looked promising, but full-scale claims were blocked because the larger follow-up experiments were still missing. In diffusion-language-model acceleration and dynamic weight-decay projects, unsupported statistics, metric failures, and control problems became later claim downgrades, validation tasks, and harness repairs rather than polished prose. Table 1 restates the six failure modes as missing update paths. Each row points to a positive harness function in Section 4. 3

Table 1: Common autoresearch failure modes restated as missing update paths. Failure mode

Missing update path

F1: Paper completion

Evidence signals do not constrain Polished writing can absorb uncertainty without sending claims weak claims back to validation. Pilot signals do not change evidence A direction worth testing is mistaken for a result worth maturity stating. Measurement failures do not revise Optimizing an easy proxy can move the system away from the objective understanding. Lessons do not reach the role that The system can restate a failure without changing the next needs them plan, critique, or gate. Negative or wasteful trials do not re- Trial volume increases without better stopping rules, vise resource policy ordering, or scope. Process failures do not repair the har- The same artifact, telemetry, or synchronization failure ness can recur across projects.

F2: Pilot collapse F3: Bad objective F4: Unrouted memory F5: Trial-volume drift F6: Static harness

3

Why it matters

Existing systems and remaining gaps

End-to-end autonomous research systems. AI scientist systems show that LLM agents can generate ideas, run experiments, interpret results, and draft manuscripts [19, 35]. Agent laboratory and coscientist systems emphasize human collaboration, literature synthesis, hypothesis generation, and candidate ranking [29, 9, 8]. Domain-specific scientific agents further show that language-model systems can combine literature search, code execution, tool use, and laboratory or chemistry-specific automation [5, 4]. More recent systems extend this trajectory toward longer-horizon and more domain-grounded discovery: cmbagent uses a planning-and-control multi-agent architecture for an autonomous cosmology analysis task, Kosmos coordinates data analysis and literature search through a structured world model, SAGA evolves scientific objective functions rather than treating objectives as fixed, and Aster accelerates iterative program-improvement loops across scientific and engineering tasks [34, 21, 6, 3]. MLAgentBench makes the experimentation loop itself an evaluation target by asking agents to improve machine-learning systems through file edits, execution, and result inspection [12]. General-purpose multi-agent and software-agent systems also show how roles, messages, tools, human interaction, and agent-computer interfaces shape task execution [33, 11, 36, 15]. These systems make more of the scientific workflow executable, but the failure modes above show why execution alone is not enough: trial signals must change later research behavior and the harness that hosts later trials. Metric-driven search and verifier-rich discovery. Autoresearch loops, FARS-style systems, AlphaEvolve-style systems, Aster-style program improvement, CORAL-style multi-agent evolution, and PaperBench-like evaluations show the value of repeated trials when feedback is clear [16, 2, 22, 3, 27, 31]. This line of work is closest to classical AutoML, neural architecture search, and black-box optimization: the system searches over candidates under an explicit objective [13, 39]. The difference is that open-ended research often lacks a single trusted objective. A harness must therefore route measurement failures and negative results into revised metrics, controls, and claim boundaries, not only into another search step. Agent memory, reflection, and harness infrastructure. Tool-using agents establish the basic pattern of interleaving language reasoning with external actions, while reflection and memory systems show that lessons from failed trials can improve later behavior [37, 28, 30, 25, 20, 32, 26]. Broad agent benchmarks and software or research benchmarks expose the importance of execution, long-context state, and artifact inspection [18, 15, 12, 31]. Long-running agents require harnesses, tools, tracing, and guardrails to make progress inspectable and safe [38, 1, 24, 23]. Scientific provenance and reproducibility work gives complementary tools for connecting claims to artifacts [17, 10]. Work on weak evidence and publication incentives warns that polished claims can outrun the evidence base [14]. We connect these threads by treating a harness as a memory-bearing research environment with explicit evidence boundaries and by requiring claim-relevant behavior changes to be visible in traces. Expertise and research judgment. The idea that expertise grows from repeated, feedback-rich experience is old [7]. Research judgment in this paper is the system-level analogue of that process. We do not claim that agents acquire human expertise in the psychological sense. We make a narrower systems claim: if an autonomous research system has learned from a trial, later behavior should expose that learning. 4

4

The Sibyl-AutoResearch framework

The diagnosis above changes the unit of analysis and motivates the Sibyl-AutoResearch framework. The central unit of autonomous research is the trial: a bounded encounter with a real research environment that produces a signal about a hypothesis, method, measurement, baseline, validation check, resource policy, or process. A trial is valuable when the signal changes later behavior. A failed trial can be especially useful because it rules out a tempting story, exposes a fragile metric, or reveals a weakness in the harness. Sibyl-AutoResearch treats a Scientific Trial-and-Error Harness as the environment that makes those updates possible. It is a set of harness functions around the agent: state, tools, roles, memory, gates, artifact contracts, compute control, and repair mechanisms. These functions do not guarantee good science. They make research behavior inspectable at the places where judgment should appear. Although the framework is presented abstractly in this section, it was refined through system-building pressure from S IBYL. Whenever a S IBYL run preserved a useful signal without changing later behavior, we treated the failure as evidence that the framework needed a more explicit update path. The seven functions below are therefore design commitments for AutoResearch systems, not claims that any current harness fully solves autonomous research. H1: Trial orchestration. Each trial should have a question, expected evidence, dependencies, outputs, and stop conditions. The observable commitment is that earlier evidence changes the next trial plan, branch priority, or task dependency. H2: Evidence maturity. Execution completion, pilot signal, analysis-ready evidence, paper-ready evidence, and audited claim are different states. The observable commitment is that a claim advances only after validation, scope control, and artifact links. Negative evidence can increase maturity by ruling out a false story. H3: Traceability. A behavior update should be tied to artifacts: plans, configs, logs, tables, reviews, negative evidence, and writing changes. The observable commitment is that a reader can reconstruct why a later action changed. H4: Routed research memory. Reflection should not remain as a free-form note. Lessons must be routed to planners, experimenters, critics, supervisors, schedulers, or writers. The observable commitment is that a past lesson changes a later role-specific check or decision. H5: Perspective separation. Optimistic, skeptical, methodological, supervisory, and writing roles should have different authority. The observable commitment is that objections become validation tasks, plan mutations, stopped branches, or claim downgrades rather than free-form disagreement. H6: Resource-aware trial policy. Research trial-and-error is bounded by GPU time, token budget, and human review attention. The observable commitment is that failed or wasteful trials change sanity-check policy, allocation, monitoring, or recovery behavior. H7: Harness self-evolution with protected constraints. Some trial signals are about the research question. Others are about the harness that produced the research. Repeated missing artifacts, stale outputs, telemetry gaps, or gate failures should change prompt overlays, validation gates, artifact contracts, scheduler policies, or repair tasks. The observable commitment is that self-evolution strengthens evidence integrity rather than optimizing polish, pass rate, or reviewer gaming. Trial-to-behavior conversion is the agent-side audit unit. A signal at iteration t must alter an action at iteration t + k: which direction to try, which evidence to distrust, which validation to run earlier, how strongly to state a claim, when to stop a branch, how to allocate GPU budget, or when to delay writing. A conversion links three artifacts: a trial signal, a trace path, and a later behavior change. Figure 1b shows this unit. Trial-to-harness-behavior conversion is the harness-side audit unit. A recurring process failure must alter a harness function: a gate, prompt overlay, telemetry requirement, scheduler policy, repair task, artifact contract, or protected constraint. These two conversion units define agent-harness co-evolution: the agent accumulates research judgment, and the harness learns how to make later research loops safer, cheaper, and more informative. Table 2 maps the six failure modes to the seven positive commitments above. 5

Table 2: Observable commitments in the Sibyl-AutoResearch framework. Harness function

Primary failure mode ad- What must be observable dressed

H1 Trial orchestration

F5 Trial-volume drift

H2 Evidence maturity

H3 Traceability

H4 Routed memory

H5 Perspective separation

H6 Resource policy

H7 Harness self-evolution

5

Failure it rules out

Earlier evidence changes later plans, de- More activity without pendencies, branch priority, or stopping better trial policy. decisions. F1 Paper completion; F2 Claims move through explicit maturity Completed workflows Pilot collapse states with validation and scope labels. that overclaim weak, stale, or corrupted evidence. F1 Paper completion Claims and behavior updates link to Well-written claims scripts, configs, logs, results, reviews, whose evidence path and negative evidence. cannot be reconstructed. F4 Unrouted memory Lessons change checks or priors of the Long context that roles that need them. remembers text but does not change action. F3 Bad objective; F4 Un- Objections become validation tasks, plan Critique that remains routed memory mutations, stopped branches, or claim theater. downgrades. F5 Trial-volume drift Failed or wasteful trials change sanity- Trial volume that wastes check order, allocation, monitoring, or compute and review recovery. budget. Recurring process failures change gates, The same infrastructure F6 Static harness overlays, telemetry, scheduler policy, re- failure recurring across pair tasks, or protected constraints. projects.

The S IBYL system

S IBYL is the concrete system realization of Sibyl-AutoResearch. It is a file-backed autonomous research system in which research state, plans, roles, memory, gates, experiment artifacts, reviews, and writing outputs are preserved as inspectable files rather than hidden runtime state. Early versions of the system exposed the same failure pattern repeatedly: useful signals were preserved as files but did not reliably change planning, validation, claim scope, scheduling, critique, or writing authority. The framework made those failures nameable, and later S IBYL mechanisms were instrumented to preserve the traces needed to audit them. We do not present S IBYL as a controlled benchmark against prior systems or as comparative performance evidence. Its role in this paper is both architectural and methodological: it shows how the proposed AutoResearch framework can be implemented in a real autonomous-research environment, and it provides the trace substrate used to audit the proposed conversion units. The current implementation reflects the seven harness functions in Section 4. Trial orchestration uses an artifact-backed state machine and task plans. Evidence maturity is enforced through decisions, quality gates, validation requirements, and writing restrictions. Traceability comes from workspace artifacts, event logs, reviews, experiment state, and writing outputs. Routed memory uses reflection postprocessing, evolution records, issue categories, and role-specific lesson overlays. Separate planner, experimenter, critic, supervisor, skeptic, methodologist, writer, and editor roles keep objections from being silently absorbed into prose. GPU scheduling, dependency layers, monitoring, recovery, repair tasks, self-heal mechanisms, and protected constraints provide resource policy and harness self-evolution. Two implementation boundaries are especially important. First, reflection outputs are normalized into issue categories, converted into evolution records, and injected as role-specific lesson overlays rather than left as long-context text. Second, writing agents consume a validated claim registry with maturity labels, artifact links, and validation status; pilot signals remain usable as pilot signals but cannot be upgraded into paper-ready claims by prose alone. Table 3 maps each framework commitment to the current S IBYL mechanism that makes the corresponding update path inspectable. The mapping is descriptive, not a claim of completeness. Appendix A expands these mechanisms and supporting diagrams. Section 6 then asks whether the preserved traces contain actual conversion events. 6

Table 3: How the current S IBYL system operationalizes the Sibyl-AutoResearch commitments. Harness function

S IBYL mechanism

H1 Trial orchestration

State machine and task plans

H2 Evidence maturity H3 Traceability H4 Routed memory H5 Perspective separation H6 Resource policy H7 Harness self-evolution

Observable update enabled

Earlier evidence changes plans, dependencies, pilot/full transitions, or branch priority. Decision gates and validation require- Weak evidence triggers a refinement decision, ments pivot, claim downgrade, or writing restriction. Workspace artifacts and event logs Claims and updates trace to plans, logs, results, reviews, and negative evidence. Reflection, evolution records, and role Recurring issues become checks for planners, overlays experimenters, critics, supervisors, or writers. Role-separated agents and debates Objections become validation tasks, plan mutations, or scoped claims. GPU scheduler, dependency graph, and Failed or wasteful trials change sanity-check recovery policy, allocation, or monitoring. Evolution records, self-heal tasks, and Process failures become gates, telemetry protected constraints requirements, overlays, or repair tasks.

This implementation detail matters because writing and research roles do not have the same authority. A writer can synthesize validated claims but cannot upgrade a pilot result. A supervisor can allow advancement only with scoped risks. A critic’s objection must become a task or boundary condition. A scheduler can run expensive experiments, but low evidence value should trigger cheap checks first. These boundaries are scientific integrity mechanisms, not only engineering choices.

6

Evidence from the S IBYL system

We use preserved S IBYL workspaces to ask a narrower question than system performance: can the proposed conversion units be recovered from realistic autonomous-research traces? This is a retrospective process audit, not a controlled comparison and not an estimate of average harness effectiveness. We mark 8 high-confidence conversion events across the workspace set and estimate a median latency of 1 iteration, with a maximum visible latency of 3 iterations, from signal to behavior update. The count is hand-audited and conservative. It is useful only as evidence of inspectability; it should not be read as a benchmark score or a claim that S IBYL converts every useful signal. For readability, the paper names cases by the research problem or failure type rather than by internal workspace directories. Appendix B gives additional case summaries and artifact categories without exposing internal file paths. The evidence is organized in three layers. First, hand-audited conversion events test the mechanism directly: did a signal change a later plan, validation task, claim boundary, schedule, critique, or writing restriction? Table 4 summarizes this conservative event sample. Second, the recovered-failure registry asks whether naturally occurring evidence-boundary failures were blocked, downgraded, or routed into repair. Third, an aggregate review-to-action audit asks whether reviewer-like objections become later experiments, validation, or harness changes rather than another score to optimize. The paper’s claim is not that S IBYL drafts receive high review scores; it is that objections and failures convert into later research or harness behavior. Patterns across the audited cases. The detailed traces behind these conversion events are preserved in Appendix B. Across cases, the same pattern repeats and explains why the framework emphasizes conversion rather than memory volume or paper quality: a weak, stale, or corrupted signal matters only when it becomes a later change in algorithm design, validation, claim boundary, writing permission, or harness policy. Figure 2 shows one concrete path. In the dynamic weight-decay case, paper errors, budget concerns, corrupted controls, and hidden negative results for an auxiliary baseline trigger a refinement decision; the next iteration adds controller repair, budget assertions, 9-of-9 stability tests, raw-log checks, and a scoped advancement decision. Recovered-failure registry. The strongest controlled experiment would inject known failures into a held-out workspace. We do not report such a new run here. Instead, we report a recovered-failure registry from naturally occurring traces. This is weaker than an injected benchmark but stronger than anecdote because each row contains a concrete failure class, an audit artifact type, a catch mechanism, and a later update. Table 7 gives the failure rows without exposing internal file paths. 7

Table 4: Audited conversion-event sample. Latency is measured in iterations when a later update iteration is visible.

Case

Trial signal

Later behavior update

Dynamic weight-decay con- Controller instability, compute- Repaired controller, epochtrol budget confounds, and hidden budget assertions, 9/9 stanegative controls. bility tests, and scoped advancement. Diffusion-language-model Unsupported statistics, accept- Speedup story reframed as acceleration rate mismatch, and one proposed an interference taxonomy; accelerator behaving as a func- full-scale replication pretional no-op. requisites added. Sparse-autoencoder absorp- Writing stagnation and repeated Experiment-first planning tion missing source-to-paper valida- and validation-first gate in tion. later iterations. Sparse-autoencoder absorp- Confidence-interval inversion, Source-to-paper validation tion stale headline ratios, and incom- script, corrected aggregapatible first-letter rates. tion, and headline ratio reduced from 4.1× to 2.7×. Failed sparse-autoencoder Writing score rose while du- Duplicate detection, replication plicate replicates, feature-count feature-count verification, mismatch, and inactive features sparsity-matched controls, broke the evidence base. and single-source analysis became prerequisites. Image-augmentation pilot Pilot produced a go signal but Claims blocked at the full-scale evidence was absent. pilot/full boundary and larger follow-up experiments required. Diffusion-language-model Caching pilot spent 54 minutes Later policy recorded a 10caching pilot to find 15.2× overhead. minute throughput sanity check to catch similar failures earlier. Missing-review gate stress Review score missing at quality Iteration hard-blocked and test gate. rolled back to review.

Latency Harness function 1

H1, H2, H3

1

H2, H5

3

H1, H4

1

H2, H3

1

H2, H3

0

H2

0

H6, H7

0

H2, H7

Aggregate review-to-action audit. We also audited a generated-review archive containing reviewerlike artifacts over 51 project-iteration snapshots from 11 workspaces, with three review surfaces per snapshot. We include this audit as a stress test for the framework’s premise, not as a validation of generated review scores. These are pressure tests, not peer-review outcomes. The most important lesson is negative: review scores are poor progress metrics. The three surfaces disagree in schema and calibration, and within each surface scores barely move across iterations even when objections persist. The objections themselves are more useful. They cluster around validation strength, claim scope, baselines, controls, reproducibility, and artifact synchronization, which are the same evidenceboundary risks the workspace traces already flag. Detailed counts and per-surface calibration are in Appendix C. To check whether such objections become later research items, we align internal S IBYL reviews, reflections, and next-iteration plans (the post-hoc external reviews are not used as inputs here). In this hand-parsed diagnostic sample, across 12 audited traces and 37 parseable structured action-plan rows, score-drop rows carry roughly twice as many high-severity issues as score-up rows (8.7 vs. 4.0 per row) and a heavier corrective load. When a score-drop row has a visible next plan, the next iteration is dominated by experiments and controls (about two thirds of tasks), with the rest split between validation/artifact repair and harness changes. Figure 3 summarizes this review-to-action path; raw transition counts and supervisor-score calibration are in Appendix C. Harness-side evidence. The S IBYL evolution-memory records show trial-to-harness-behavior conversion. The central digest contains 416 recurring issue patterns: 212 experiment, 89 writing, 84 analysis, 20 system, 4 ideation, 3 pipeline, 3 planning, and 1 efficiency. Routing is explicit: experiment issues are assigned to experiment-running and planning roles, analysis issues to supervisory and critique roles, and writing issues to drafting and editing roles. The prompt loader then injects selected 8

hidden negative auxiliarybaseline result

raw-log mismatch

budget confound

controller instability

Case: Dynamic-WD (iteration 13 → 14)

weakened / unsupported hypothesis

corrupted control

Signals observed (iteration 13) Iter 013

REFINE GATE

weak evidence + sync, validation, and claim-scope risks

Iter 014

scoped action / next step

ADVANCE GATE

repair, validation, and claim narrowing

Actions & outcomes (iteration 14)

9/9 unit tests pass

controller stability repair

budget assertions added

raw-log cross-checks

auxiliary-baseline / control audit

residual risks scoped

narrowed 3-tier claim

Figure 2: Dynamic weight-decay gate-to-action flow. Controller instability, budget confounds, raw-log mismatches, hidden negative auxiliary-baseline results, corrupted controls, and a weakened or unsupported hypothesis at iteration 13 trigger a refinement decision. Iteration 14 then performs controller stability repair, 9-of-9 unit tests, budget assertions, raw-log cross-checks, auxiliary-baseline/control audit, and a narrowed three-tier claim before issuing a scoped advancement decision. (a) Score drops are useful when they route work

iters 13 -> 14

(b) Workload by score movement

(c) Action-plan focus

(d) Next-iteration task focus

High-severity issues Recommended focus items

Mean count

20

Next-plan tasks (visible plans)

15

10

12.6 8.7 8.4

11.0

9.6 6.5

62

20

57

score down n=88

56

17

13

flat n=50

13

7

20

6

flat n=48

28

12

7

score up n=98

32

13

41

11

score up n=55

31

13

9

5.9

5.8

5

9.9

7.5

score down n=154

4.0

3.8

no prior n=81

20

14

no prior n=79

37

57

10

10

0 score down flat score up no prior rows=10 rows=6 rows=10 rows=11 plans=7 plans=5 plans=5 plans=8

Experiment/control

0

20

40

60

80

100

0

Share (%) Validation/artifact

Harness/system

20

40

60

80

100

Share (%) Claim/writing

Other

Figure 3: Internal review scores as issue-to-action signals. (A) Two concrete score-drop rows show that a lower score is useful when it routes work: a sparse-autoencoder absorption drop becomes a validation-first plan and source-to-paper validation script; a dynamic weight-decay drop becomes added controls, 9/9 unit tests, and a narrowed claim. (B) Mean issue and focus load per parsed row; next-plan tasks are averaged only over rows with a visible next-iteration task plan, and the x-axis reports both parsed rows and visible-plan rows. (C–D) Heuristic multi-label classifications of structured action-plan recommendations and next-iteration task-plan entries. The post-hoc generated reviews are not causal inputs to these loops; they are pressure tests for whether similar objections would be actionable.

lessons as role-specific overlays. The diffusion-language-model caching case also exposes a resourcepolicy update: the reflection records that a 54-minute pilot revealed a 15.2× overhead failure that a 10-minute throughput sanity check could have caught earlier.

7

Limitations and alternative views

The evidence is retrospective and concentrated in one author-built harness for computational AI/ML workspaces. The framework and S IBYL co-developed, so the traces are not an independent validation set for the theory. They are better understood as theory-building and stress-test evidence. The 9

conversion count is hand-marked, the recovered-failure registry uses natural failures rather than a new injected benchmark, and the ablations are process diagnostics rather than repeated statistical experiments. The reviewer-like artifacts are generated reviews, not human peer-review decisions, and should be used only as process signals about possible evidence-boundary failures. The transition counts are raw event-log counts and may include resume or checkpoint repetition, so they support process-shape claims rather than exact execution counts. The paper supports a design framework and an existence proof, not a comparative performance claim about S IBYL. Future work should evaluate the conversion units on held-out harnesses, with independent annotators, prospective injected failures, and public artifact bundles. One alternative view is that final manuscript quality is the only outcome that matters. We disagree because a manuscript is an expression of an evidence state: in the failed sparse-autoencoder replication, writing quality improved while the evidence base collapsed. The generated-review audit makes the same point at scale: reviewer-like tools disagree enough in schema and calibration that optimizing their scores would create another brittle objective. A second view is that better verifiers and benchmarks will solve the problem. We agree where objectives are trusted, but open-ended research often discovers that the objective itself is broken; in the diffusion-language-model acceleration case, unsupported statistics and a no-op accelerator forced a change in the objective. A third view is that human researchers should supply judgment, while agents need only provenance. Human responsibility remains central, but scalable oversight still needs traces showing why an autonomous system changed its claims and plans. A final concern is that gates and routed memory may make systems optimize process compliance. This is why the commitments emphasize protected evidence integrity, negative results, and hidden injected failures rather than a single process score.

8

Conclusion

Autonomous research should not be judged mainly by whether a system can generate a complete paper. A paper is only an expression of an evidence state; it does not by itself show whether the system has learned from the trials that produced it. The harder capability is whether a system can turn trial history into research judgment: better plans, stronger validation, narrower claims, safer resource policies, and a harness that becomes harder to fool over time. This paper argues that such judgment requires Sibyl-AutoResearch: a self-evolving AutoResearch framework built around Scientific Trial-and-Error Harnesses. Our experience building S IBYL suggests that the framework and the system cannot be cleanly separated. Better traces reveal missing update paths, and better update-path theory tells later system versions what to preserve, route, block, and audit. Scientific Trial-and-Error Harnesses make this agenda auditable: they ask not only what an agent produced, but what it learned, where that lesson traveled, and how it changed the next research action.

References [1] Ken Aizawa. Writing effective tools for agents — with agents. https://www.anthropic. com/engineering/writing-tools-for-agents, 2025. Anthropic Engineering. Published 2025-09-11. Accessed 2026-05-05. [2] Analemma Team. Introducing FARS. https://analemma.ai/blog/introducing-fars/, 2026. Analemma AI blog. Published 2026-02-11. Accessed 2026-05-05. [3] Emmett Bicker. Aster: Autonomous scientific discovery over 20x faster than existing methods, 2026. URL https://arxiv.org/abs/2602.07040. [4] Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023. doi: 10.1038/ s41586-023-06792-0. URL https://doi.org/10.1038/s41586-023-06792-0. [5] Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. ChemCrow: Augmenting large-language models with chemistry tools, 2023. URL https://arxiv.org/abs/2304.05376. 10

[6] Yuanqi Du, Botao Yu, Tianyu Liu, Tony Shen, Junwu Chen, Jan G. Rittig, Kunyang Sun, Yikun Zhang, Aarti Krishnan, Yu Zhang, Daniel Rosen, Rosali Pirone, Zhangde Song, Bo Zhou, Cassandra Masschelein, Yingze Wang, Haorui Wang, Haojun Jia, Chao Zhang, Hongyu Zhao, Martin Ester, Nir Hacohen, Teresa Head-Gordon, Carla P. Gomes, Huan Sun, Chenru Duan, Philippe Schwaller, and Wengong Jin. Accelerating scientific discovery with autonomous goal-evolving agents, 2025. URL https://arxiv.org/abs/2512.21782. [7] K. A. Ericsson and A. C. Lehmann. Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47(1):273–305, February 1996. ISSN 1545-2085. doi: 10.1146/annurev.psych.47.1.273. URL http://dx.doi.org/10. 1146/annurev.psych.47.1.273. [8] Ali Essam Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn J. Szostkiewicz, Jon M. Laurent, Muhammed T. Razzak, Andrew D. White, Michaela M. Hinks, and Samuel G. Rodriques. Robin: A multi-agent system for automating scientific discovery, 2025. URL https://arxiv.org/abs/2505.13400. [9] Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R D Costa, José R Penadés, Gary Peltz, Yunhan Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Towards an ai co-scientist, 2025. URL https://arxiv.org/abs/2502.18864. [10] Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelligence. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. ISSN 2159-5399. doi: 10.1609/aaai.v32i1.11503. URL http://dx.doi.org/10.1609/ aaai.v32i1.11503. [11] Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework, 2023. URL https://arxiv.org/abs/2308.00352. [12] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation, 2023. URL https://arxiv.org/abs/2310. 03302. [13] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors. Automated Machine Learning Methods, Systems, Challenges. Springer, 2019. [14] John P. A. Ioannidis. Why most published research findings are false. PLoS Medicine, 2 (8):e124, August 2005. ISSN 1549-1676. doi: 10.1371/journal.pmed.0020124. URL http: //dx.doi.org/10.1371/journal.pmed.0020124. [15] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?, 2023. URL https://arxiv.org/abs/2310.06770. [16] Andrej Karpathy. autoresearch. https://github.com/karpathy/autoresearch, 2026. GitHub repository. Accessed 2026-05-05. [17] Timothy Lebo, Satya Sahoo, Deborah McGuinness, Khalid Belhajjame, James Cheney, David Corsar, Daniel Garijo, Stian Soiland-Reyes, Stephan Zednik, and Jun Zhao. PROV-O: The PROV Ontology. W3C Recommendation. World Wide Web Consortium, United States, April 2013. [18] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents, 2023. URL https://arxiv.org/abs/2308. 03688. 11

[19] Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery, 2024. URL https: //arxiv.org/abs/2408.06292. [20] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL https://arxiv.org/abs/2303.17651. [21] Ludovico Mitchener, Angela Yiu, Benjamin Chang, Mathieu Bourdenx, Tyler Nadolski, Arvis Sulovari, Eric C. Landsness, Daniel L. Barabasi, Siddharth Narayanan, Nicky Evans, Shriya Reddy, Martha Foiani, Aizad Kamal, Leah P. Shriver, Fang Cao, Asmamaw T. Wassie, Jon M. Laurent, Edwin Melville-Green, Mayk Caldas, Albert Bou, Kaleigh F. Roberts, Sladjana Zagorac, Timothy C. Orr, Miranda E. Orr, Kevin J. Zwezdaryk, Ali E. Ghareeb, Laurie McCoy, Bruna Gomes, Euan A. Ashley, Karen E. Duff, Tonio Buonassisi, Tom Rainforth, Randall J. Bateman, Michael Skarlinski, Samuel G. Rodriques, Michaela M. Hinks, and Andrew D. White. Kosmos: An AI scientist for autonomous discovery, 2025. URL https://arxiv.org/abs/ 2511.02824. [22] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. Alphaevolve: A coding agent for scientific and algorithmic discovery, 2025. URL https://arxiv.org/abs/2506.13131. [23] OpenAI. Guardrails. https://openai.github.io/openai-agents-python/ guardrails/, 2026. OpenAI Agents SDK documentation. Accessed 2026-05-05. [24] OpenAI. Tracing. https://openai.github.io/openai-agents-python/tracing/, 2026. OpenAI Agents SDK documentation. Accessed 2026-05-05. [25] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024. URL https: //arxiv.org/abs/2310.08560. [26] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. URL https://arxiv.org/abs/2304.03442. [27] Ao Qu, Han Zheng, Zijian Zhou, Yihao Yan, Yihong Tang, Shao Yong Ong, Fenglu Hong, Kaichen Zhou, Chonghe Jiang, Minwei Kong, Jiacheng Zhu, Xuan Jiang, Sirui Li, Cathy Wu, Bryan Kian Hsiang Low, Jinhua Zhao, and Paul Pu Liang. CORAL: Towards autonomous multi-agent evolution for open-ended discovery, 2026. URL https://arxiv.org/abs/2604. 01658. [28] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL https://arxiv.org/abs/2302.04761. [29] Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants, 2025. URL https://arxiv.org/abs/2501.04227. [30] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366. [31] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate ai research, 2025. URL https://arxiv.org/abs/2504.01848. 12

[32] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. URL https://arxiv.org/abs/2305.16291. [33] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation, 2023. URL https://arxiv.org/abs/2308.08155. [34] Licong Xu, Milind Sarkar, Anto I. Lonappan, Inigo Zubeldia, Pablo Villanueva-Domingo, Santiago Casas, Christian Fidler, Chetana Amancharla, Ujjwal Tiwari, Adrian Bayer, Chadi Ait Ekioui, Miles Cranmer, Adrian Dimitrov, James Fergusson, Kahaan Gandhi, Sven Krippendorf, Andrew Laverick, Julien Lesgourgues, Antony Lewis, Thomas Meier, Blake Sherwin, Kristen Surrao, Francisco Villaescusa-Navarro, Chi Wang, Xueqing Xu, and Boris Bolliet. Open source planning & control system with language agents for autonomous scientific discovery, 2025. URL https://arxiv.org/abs/2507.07257. [35] Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search, 2025. URL https://arxiv.org/abs/2504.08066. [36] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793. [37] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2022. URL https: //arxiv.org/abs/2210.03629. [38] Justin Young. Effective harnesses for long-running agents. https://www.anthropic.com/ engineering/effective-harnesses-for-long-running-agents, 2025. Anthropic Engineering. Published 2025-11-26. Accessed 2026-05-05. [39] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning, 2017. URL https://arxiv.org/abs/1611.01578.

13

Appendix The appendix follows the same order as the main argument. Appendix A expands the S IBYL mechanisms and supporting diagrams. Appendix B gives the workspace traces and the recoveredfailure registry behind the conversion-event audit. Appendix C reports the generated-review and event-log diagnostics. Appendix D gives the system-comparison details, and Appendices E and F collect evaluation protocols and governance details.

A

Sibyl implementation and harness diagrams

This appendix expands the implementation sketch from the main text. It is implementation evidence for how the co-developed framework was operationalized, not a full software manual or an independent proof that one harness solves autonomous research. The relevant unit is the research behavior update or harness behavior update that the mechanism makes possible. A.1

Reflection and evolution memory

The memory layer starts from reflection artifacts, but it should not end there. Reflection outputs are normalized into issue categories such as system, experiment, writing, analysis, planning, pipeline, ideation, and efficiency. The evolution layer maintains digests and lessons, computes relevance, and injects selected lessons into later prompts. This is the mechanism behind the paper’s distinction between long context and research memory. Long context can expose past text; routed memory can change which checks a role performs. Memory effectiveness is measured by behavior, not volume. A lesson informs the system when it reduces repeated failures, changes a plan, adds a validation step, changes a claim boundary, or repairs the harness path that allowed the failure. The memory layer therefore stores source links, uncertainty, decay, and re-opening criteria alongside the lesson text. A.2

Decision gates, self-heal, and writing boundaries

The current system exposes idea validation, experiment decisions, quality gates, review stages, and a structured self-heal substrate. The self-heal layer includes error collection, routing, protectedfile constraints, deterministic fixers for recurring failure classes, and repair-task generation. These mechanisms improve workflow reliability and create places where failures become tasks before they recur in the next research loop. The natural boundary mechanism is a validated claim registry. Writing agents consume claims with maturity labels, artifact links, and validation status. Pilot signals remain usable as pilot signals. A paper is generated from the claim registry, not from whatever narrative is most polished after the latest experiment. A.3

Additional mechanism diagrams

Figure 4 expands H2 from Section 4 by showing the evidence maturity states and the claim-evidence check used to allow, downgrade, or block claims. Figure 5 complements the maturity diagram by showing the memory routing and claim-evidence substrates that support those checks.

B

Extended workspace case notes

The following case notes provide the detailed artifacts behind the process-evidence audit. Each case is written around the same structure: trial signal, harness mechanism, and behavior update. These traces are theory-building and stress-test evidence, not held-out validation of a completed theory. 14

Evidence Maturity States

Claim–Evidence Boundary Incoming claim

M4 Audited claim audit gate

• Artifacts

M3 Paper-ready evidence

• Validation checks

Claim–Evidence Check

• Negative evidence

claim gate

• Reviewer objections

M2 Analysis-ready evidence

validated

scoped risk

bad evidence

Allow

Downgrade

Block

Scoped claim

Refine claim

Return to validation

validation gate

M1 Pilot signal pilot gate

M0 Execution completion

Figure 4: Evidence maturity states and the claim-evidence boundary. Execution completion, pilot signal, analysis-ready evidence, paper-ready evidence, and audited claims are separate states. The claim-evidence check allows, downgrades, or blocks claims based on artifacts, validation checks, negative evidence, and reviewer objections.

Trial signal

Issue category

Normalized lesson

Role routing contract

Overlay update

Allow

Downgrade

Block

revise / de-weight

next-trial criteria recurrence

Paper claim scope / maturity / risks

Effectiveness tracking score / recurrence / decay

contradicts

Negative evidence

routed, not broadcast

behavior evidence

plan overlay

objection check

claim downgrade

validation contract

Planner

Critic

Supervisor

Experimenter

Observed role behavior in later trials

Table

Figure

Reviewer note

Aggregated JSON

Analysis script + config

Validation tests

Raw logs

Seeds + checkpoints

Configs

(b) Claim-evidence graph

(a) Trial signal to routed institutional memory

Figure 5: Memory routing and claim-evidence substrates. (a) Trial signals are normalized into issue categories and lesson records, routed to specific roles via overlay updates, and observed as later plan overlays, objection checks, claim downgrades, or validation contracts. (b) The claim-evidence graph links each prose claim to artifacts (tables, figures, reviewer notes, aggregated JSON), analysis scripts and configs, validation tests, raw logs, seeds, checkpoints, and negative evidence.

B.1

Evolution-memory examples

The harness-side evidence is stored in global and per-project evolution-memory records. The important property is role routing: a repeated issue is not left as a free-form reflection note, but is assigned to affected roles with severity, frequency, suggested action, and success patterns. Table 5 gives representative records where recurring signals become harness behavior updates. B.2

Complete project traces

The traces below are organized temporally because the claim is temporal: a signal in one iteration must change a later action. Table 5 above gives cross-project memory evidence. The following complete traces show how a reader should inspect one project end to end. Each trace follows the same chain: project goal, iteration state, trial signal, harness decision, next behavior, and claim effect. 15

Table 5: Examples of trial-to-harness-behavior conversion in evolution memory. Evolution-memory record

Recurring signal

Supervisory lesson record

Post-hoc power analysis appears 5 times with high sever- Supervisor prompts inherit checks ity; conditional mutual information correlations reverse against post-hoc justification, sign across dimensions; the paper’s mean value 0.687 con- cherry-picking, dimension instability, flicts with the source value 0.6492, and Mann–Whitney and source-data mismatch. U = 41.0 conflicts with source U = 28.0. Random sparse-autoencoder baselines, sparsity mis- Planner-side lessons emphasize matches, tautological co-occurrence measurements, and matched controls, baseline unexecuted validation experiments recur across absorp- comparisons, ablations, and execution tion cases; one inhibition graph has precision@20=0.0 of gatekeeper experiments before across 520 predictions, a sparsity level is unmatched in a paper writing. baseline comparison, and 0/12 tests survive Bonferroni or BH-FDR correction. Contradictory empirical results and causal language in Critic-side lessons route contradiction observational analyses recur, including original posi- checks, causal-language audits, tive steering correlations later contradicted by controlled correction for multiple comparisons, matched designs. and demand for reconciled methodology changes. Success patterns are recorded together with failures: hon- The harness preserves positive est negative results recur for six iterations, stable infras- operating habits as prompt overlays tructure completes repeated task batches with zero ex- rather than only accumulating periment failures, and specific numbers are consistently warnings. preferred. The same memory stores probe-quality confounds such as absorption rate versus probe accuracy ρ = −0.67, p < 0.001.

Planning lesson record

Critique lesson record

Global lesson record

B.2.1

Harness behavior update

Complete trace I: Sparse-autoencoder absorption, from writing stagnation to validation-first research

Project goal. The workspace studies sparse-autoencoder feature absorption: when a parent feature absorbs child-feature behavior, how that effect should be measured, and which claims survive across layers, domains, and interventions. This is the best long-horizon case because it contains 11 iteration directories and full role artifacts across planning, experiments, supervision, writing, and reflection. Iteration map. 1. Iterations 1–5: initial measurement and narrative formation. The system builds an initial absorption story, writes drafts, runs targeted probes, and accumulates reflection artifacts. These iterations establish the central research object but also start the pattern that later becomes important: the paper can become smoother while source-to-paper numeric consistency remains fragile. 2. Iterations 6–8: writing stagnation and missing validation. The quality trajectory stalls around 6.5. Reflection repeatedly asks for a source-to-paper validation script, but the recommendation remains a lesson rather than a hard writing gate. Iteration 8 records the ninth recommendation of the script and finds a fabricated 12.3% hedging number where raw data gives 0.0%. The action plan turns this into Gate 0: a 1.5-hour, zero-GPU source-to-paper cross-check must run before further writing. 3. Iteration 9: experiment-first break from polishing. The project breaks the writing-only loop by executing new empirical checks: activation patching, tightened hedging analysis, conditionalmutual-information replication, and threshold sensitivity. The score rises from 6.5 to 7.0 because the system has produced new evidence rather than only a cleaner narrative. 4. Iteration 10: scientific progress plus evidence-boundary failure. The iteration produces a strong probe-degradation result (R2 = 0.777, ρ = −1.0, p = 0.009), decoder-magnitude evidence (6.16 nats for first-letter and 3.98 nats for city-continent), and rate-distortion rejection across 131 pairs. The score still regresses from 7.0 to 6.5 because the paper imports new integrity errors: Table 3 confidence-interval inversion, three incompatible first-letter rates, stale 4.1× headline language, layer-multiplier mismatch, and an unverified patching sign reversal. 5. Iteration 11: data integrity becomes the iteration objective. The next plan explicitly makes the iteration about data integrity and verification. The source-to-paper validation script is implemented, 51/53 checks pass, CI inversions are fixed, per-token aggregation becomes canonical, the headline changes from 4.1× to 2.7×, and the 21.6%/27.1%/34.5% first-letter rates are traced to distinct experimental conditions. Probe degradation becomes contribution #1, and a 20-entity city16

continent spot-check confirms 62.7% mean recovery versus 61.9% expected (d = 2.04, p < 0.001). What the trace demonstrates. The sparse-autoencoder absorption case shows the full agent-harness loop. Trial signals first change the agent’s research behavior: writing gives way to experiments, then experiments give way to validation-first planning. They also change the harness boundary: source-to-paper numeric validation becomes a gate rather than an optional reflection note. The final contribution hierarchy is not the one the early drafts wanted; it is the one that survived repeated trial, critique, and validation failure mode. B.2.2

Complete trace II: Dynamic weight decay, from unstable control law to scoped advancement

Project goal. The workspace studies dynamic weight decay: automatically changing the regularization strength during neural-network training. It is the cleanest positive trace because a concrete algorithmic defect becomes a repaired method, explicit tests, a mutated task plan, and scoped advancement. Iteration map. 1. Iterations 0–7: idea formation, early pilots, and repeated evidence gaps. The project builds a dynamic weight-decay story and accumulates experiments across small and medium settings. Quality moves upward and downward rather than monotonically: the quality log includes 5.5, 7.0, 5.0, 6.5, 6.75, 7.0, and later 6.5. This volatility is useful because it exposes the harness’s claim-boundary function: better prose or a promising pilot does not erase unresolved controls. 2. Iterations 8–12: recurring control and generalization failure mode. Reflection and evolution records keep surfacing missing ImageNet evidence, equivalence-test weakness, budget confounds, and control reliability. The evolution outcome marks missing ImageNet evidence as recurring for 7+ iterations, records equivalence tests passing only 6/12 comparisons, and proposes a preliminary smoke test before a 9-run ImageNet-100 replication plan. 3. Iteration 13: refinement becomes unavoidable. Reflection records raw-log mismatches, hidden negative auxiliary-baseline results, corrupted controls, higher-regularization control gaps, and a 90-epoch ImageNet need. The important system behavior is that these signals do not get absorbed as prose caveats. They become blockers for broad advancement. 4. Iteration 14: repair, validation, and scoped advancement. The supervisor path introduces a repaired controller with floor clipping, moving-average smoothing, and epoch-budget assertions. The fix passes 9/9 stability tests; the single-parameter controller budget changes from 0.0 to 90.61; ImageNet control-signal informativeness reaches 0.987; one hypothesis is no longer supported; and the ImageNet budget confound remains explicitly recorded. The plan mutates into a 14-task refinement and full-experiment program including controller repair, diagnostic CIFAR10, unification fitting, CIFAR-100 ablations, batch-size sweeps, temporal-gate tests, alignment informativeness, ImageNet main runs, and budget-matched ImageNet controls. What the trace demonstrates. The dynamic weight-decay case shows trial-to-behavior conversion in its most direct form. A failed or unstable trial changes the algorithm, then the task plan, then the validation order, then the claim boundary. The final advancement decision is stronger because it is scoped: one hypothesis is narrowed to a three-tier taxonomy, another is treated as unsupported or uncertain, budget confounds remain visible, and the full-run plan inherits the repair obligations. B.2.3

Complete trace III: Diffusion-language-model acceleration, from speedup narrative to interference taxonomy

Project goal. The workspace studies acceleration methods for diffusion language models and asks whether methods compose multiplicatively. It is shorter than the previous two projects, but it is the sharpest negative-evidence trace: the harness converts unsupported speedup claims into metric repair, full-scale gates, and a new thesis about interference. Iteration map. 1. Iteration 1: paper and metric errors become experiment requirements. Reflection fixes fabricated Wilcoxon claims, a tau= 0.0 paradox, failure-atlas number mismatches, a quality17

adjusted-speed formula inconsistency, a 6-pair overclaim where only 3 pairs are feasible, novelty overclaiming, and a speed-report mismatch for one proposed accelerator. The same action plan finds a new accept-rate error: the draft claims α = 0.52, while raw results report average accept rate 0.881 on GSM8K and 0.830 combined. Pairwise-composition evidence is only a 2-seed, 200/1319-sample pilot with per-seed range [1.292, 1.478], so the plan calls for full benchmark replication on 1319 GSM8K plus 500 MATH500 with 3 seeds and bootstrap confidence intervals. 2. Iteration 2: the scientific story flips from multiplication to interference. Result debate reports 15 experiment groups, one proposed accelerator as a functional no-op around 1.16×, destructive interference between two accelerators, partial interference between another accelerator pair, and an autoregressive baseline comparison where Qwen2.5-7B reaches 96% GSM8K at 70.9–471.1 tokens per second. The old claim that speedups compose multiplicatively no longer matches the evidence. 3. Iteration 3: pilot/full maturity rules become explicit. Later lessons encode the rule the earlier paper lacked: pairwise-composition evidence with N < 500 must be labeled a pilot estimate with bootstrap intervals; core pairwise claims require full-scale N ≥ 1319 and 3 seeds. The system also separates per-token speed from output-length effects and standardizes baseline throughput before additional comparisons. What the trace demonstrates. The diffusion-language-model acceleration case shows that negative results are not dead ends. They are belief-calibration events. The harness forces unsupported statistics out of the paper, splits metric definitions, blocks overbroad pairwise claims, and changes the research question from “how do we multiply speedups?” to “which mechanisms interfere, and under what evidence maturity?” B.2.4

Boundary trace: failed sparse-autoencoder replication, when writing quality rises as evidence collapses

The failed sparse-autoencoder replication is not used as a clean success story. Its value is a boundary failure that exposes why data-validation gates must sit before narrative generation. Iteration 1 marks the first-letter proxy as degenerate: 26/27 GPT-2 checkpoints in one experiment and 9/10 in another return exactly 0.0, and the task-agnostic metric is negatively correlated with the first-letter benchmark (r = −0.592, p = 0.12). The pilot quality is marked as not ready to proceed, and the action plan marks the pilot/full escalation error as requiring a system change: if any pilot rating is not ready to proceed, the next stage must be metric or code repair rather than scale-up. Later, the project shows the danger of writing improvement without evidence improvement. A full component run covers 7 variants by 5 replicates, but reflection finds that writing reaches 8/10 while supervisor and critic scores fall to 4.5/10 and 5/10. The evidence base contains byte-identical replicates across nine metrics, a run manifest with 1,024 features while the paper claims 16,384 features, a missing sparsity-matched ablation, 81.6% inactive sparse features, negative explained variance, and a canonical summary with only 3/7 variants. This trace is the cleanest warning that polished writing can move in the opposite direction from scientific maturity. B.2.5

Reversal trace: sparse-autoencoder hypothesis reversal, from falsified hypothesis to new framing

The sparse-autoencoder hypothesis-reversal case shows how the system handles a result that contradicts the original story. Iteration 1 reverses one hypothesis: high-absorption features are more steerable, not less, with reported r = +0.3548 and p = 2.92e−04. The paper reframes around “Absorption as Steering Signature” instead of discarding the result as a failure. Later validation complicates the new story: a controlled matched design gives p = 0.299, one reversal is uncorrected at p = 0.015, a steering metric saturates at layer 8, and later steering-protocol summaries report all effects at 0.0. The valuable behavior is the sequence: falsification becomes a new framing, and later validation is still allowed to downgrade that framing and create pivot signal. B.2.6

Pilot/full boundary trace: image-augmentation case

The image-augmentation workspace runs a CIFAR-10/ResNet18 small pilot on a 5k subset for 10 epochs and observes a 2.68 percentage-point spread, but reflection assigns only 4.0/10 because the full-scale transition is blocked. The experiment state records 20 tracked tasks, 18 completed and 18

2 failed. The useful signal is not the augmentation result itself; it is the separation between task completion, pilot direction, and claim-ready evidence. A second small case, the generalization pilot, reaches a go decision with 0.88 confidence and 48.8s training while the workspace status records that execution was stopped: the experiment state tracks 8 tasks, 6 completed and 2 still running, and later synthesis artifacts still carry a pilot-mode label rather than a full 200-epoch, 7-seed, 27-combination result. Together these cases show that compute state and evidence maturity are inseparable. B.3

Diagnostic workspace observations

The diagnostic workspaces are stress tests for update paths rather than clean component ablations. A no-debate setting has a clean configuration knob and reaches a pilot go decision while exposing metric sensitivity. A memory-positive setting is best read as an evolution-follow-up case rather than a clean memory ablation: a recurring activation-patching lesson is later answered by a pilot with 9/9 patching checks and 67.3% mean recovery. A memory-negative setting is strongest as a boundary case because the quality gate hard-blocks a missing review score and rolls the workspace back. A validationremoval setting is config-only in this audit, and a no-revision setting is a stagnation/escalation case rather than a clean revision-off component test. Details used in the main audit. The no-debate setting ran a synthetic absorption pilot to a go decision; the trained sparse autoencoder showed higher absorption than the random baseline under the overlap method (0.50 versus 0.25), while the ablation method gave 1.00 for both, exposing a measurement-method sensitivity that debate should expose earlier. The memory-positive setting records explicit validation against evolution lessons: activation patching succeeds on 9/9 checks with 67.3% mean recovery, but later reflection still records multiple-comparison, baseline, figure, and circularity issues. The memory-negative setting contains a runtime boundary artifact: the quality gate hard-blocks iteration 5 because it has no review score and rolls the system back to review. System-evolution details. The evolution-memory records normalize reflection outputs into issue categories, severities, affected roles, suggestions, statuses, and success patterns. In the audited workspace set, 12 of 13 workspaces have non-empty outcome records, and the central ledger contains 173 outcome records (the issue-pattern category mix is reported in Section 6 and not repeated here). Representative process failures include missing rendered figures, placeholder references, paperlength overflow, absent code-release plans, underpowered experiments, stale artifacts, paper/LaTeX desynchronization, incomplete GPU telemetry with empty timing fields, non-self-contained evidence bundles with absolute paths, and external synchronization failures. Those failures become routed lesson records for planning, experiment-running, supervision, critique, skepticism, writing, and editing roles. Table 6 summarizes how the diagnostic workspace settings are used in this audit. Table 6: Appendix diagnostic-workspace status. Setting

Use in this draft

No-debate setting

Perspective diagnostic.

Trace value

Clean debate-disabled setting plus metric-sensitivity trace: 0.50 vs. 0.25 under one metric, 1.00 vs. 1.00 under another. Memory-positive set- Evolution-follow-up case. Recurring activation-patching lesson becomes a ting validation block with 9/9 checks and 67.3% mean recovery. Validation-removal Validation-removal setting. Config-only stress-test setting in this audit; useful setting for protocol design, not outcome evidence. No-revision setting Stagnation/escalation case. Records score stagnation, recurring issues, topic drift, and action plans that were not enforced. Memory-negative or Boundary and memory diag- Hard quality-gate rollback and repeated weakened settings nostics. proxy/control failures expose what must be routed into gates and planner prerequisites.

B.4

Recovered-failure registry

Table 7 lists the natural failure classes used for the recovered-failure audit referenced in Section 6. These are existing failures that the harness blocked, downgraded, or routed into repair, not newly injected tests. 19

Table 7: Recovered failures from natural workspace traces. These are existing failures that the harness blocked, downgraded, or routed into repair, not newly injected tests. Failure class Signal Audit artifact type Harness catch Later update Duplicate result 4 of 5 component repli- Reflection note from failed Supervisor and critic Duplicate files cates byte-identical to sparse-autoencoder replica- downgraded evidence detection and another method across tion despite high writing single-source nine metrics. score. analysis prerequisites. CI inversion 5 of 7 confidence inter- Reflection note from Reflection marked the Source-to-paper vals did not contain point sparse-autoencoder absorp- error critical and tied it validation script; estimates. tion case to missing validation. CI fixes in the next iteration. Stale headline 4.1× ratio persisted after Reflection note and revised Source-to-paper vali- Headline number data changed. experiment section from dation exposed stale reduced to 2.7× sparse-autoencoder absorp- prose. under tion case quality-gated aggregation. Feature-count Run used 1024 features Reflection note from failed Feature-count verifica- Claim mismatch while paper claimed sparse-autoencoder replica- tion became a prereq- generation 16,384. tion uisite. blocked until manifest/run alignment. Unsupported Fabricated Wilcoxon val- Action plan and result- Reflection and result Paper reframed statistics ues and α = 0.52 debate record from debate rejected unsup- around vs. measured accept rate diffusion-language-model ported statistics. interference and 0.881. acceleration case full-scale gates.

C

Review-artifact and transition statistics

The generated-review archive contains reviewer-like artifacts for workspace paper drafts. We use them as process pressure tests only. They are not human peer-review decisions and do not validate the domain claims of the drafts. Their value is to expose whether external objections would become claim-boundary or validation work. We also align these generated reviews with available internal S IBYL supervisor reviews when a structured review record exists for the same project iteration. Table 8 separates score calibration from transition structure because neither score should be optimized directly. Internal supervisor reviews are not directly comparable to generated reviewer outputs because they use a 10-point process-review scale and often include role-specific evidence gates. They are still useful as a calibration reference. Among the 51 generated-review snapshots, 39 had a score-bearing S IBYL supervisor review; the mean internal score was 6.13/10, with available dimension-score means of 6.74/10 for novelty, 5.38/10 for experiments, 5.41/10 for soundness, and 5.66/10 for reproducibility across 34 reviews. For the Figure 6 distribution panel, we divide these scores by two and round them, giving 1 snapshot at score 2, 31 at score 3, and 7 at score 4. Using the rough conversion internal/2 to compare against 5-point generated scores, the paired difference internal/2 minus the conservative reviewer averaged +0.14 over 27 pairs, while internal/2 minus the rubric reviewer averaged -0.83 over 12 pairs. This supports the calibration claim only; it should not be read as a unified quality metric. For Figure 3, we separately parse internal reflection-derived action plans and the next iteration’s task plans. Table 9 reports the parsed action-plan rows and visible next-plan task mix. Recommendation categories are heuristic and multi-label because one focus item can ask for both a new experiment and a paper change; next-plan task categories use one primary category per task. The statistic is therefore a process diagnostic rather than an exact causal effect estimate. The artifact coverage itself is an audit signal. The review set contains 51 reviewed project-iteration pairs, while the local collected-paper tree used for this paper contains 46 project-iteration artifact folders. Nine reviewed iterations are absent from that local collection and four local collected iterations are not reviewed. In the image-augmentation case, the latest collected Markdown draft makes full-scale 200-epoch claims, while the reviewed source still describes pilot 10-epoch, 100sample, single-seed results. This is not evidence about reviewer accuracy; it is evidence that a research 20

Table 8: Review-artifact parsing summary. Numeric review scores are treated as calibration diagnostics and audit signals, not as scientific-quality targets.

Statistic

Conservative reviewer

Rubric reviewer

Long-form reviewer

Artifact count

51 generated reviews over 51 generated reviews plus 51 MHTML reviews plus the same project-iteration 51 metadata JSON files, all 51 PDFs; MHTML set. marked completed. contains the review text. Coverage 11 workspaces and 51 11 workspaces and 51 11 workspaces and 51 project-iteration snapshots. project-iteration snapshots. project-iteration snapshots. Complete numeric reviews Rubric subscores for 51/51 Long-form assessment Native score schema for 35/51 snapshots; desk- snapshots; numeric overall with no native numeric only outputs for 16/51. for 18/51. score; automated text parsing assigns coarse ordinal labels to 31/51 assessments and leaves 20/51 unclear. Overall-score distribution 31/35 overall scores are 3; 12/18 numeric overall Text-inferred labels: 10 4/35 are 2; 0/35 are 4 or scores are 4 or higher; reject, 4 major revision, higher. mean 3.94. 14 borderline/workshop, 3 positive, 20 unclear; mapped to scores 2/3/4 only for Figure 6. Parsed subscore calibra- Mean 2.31 across 140 Mean 2.65 across 204 No native subscore tion parsed dimension scores. parsed dimension scores; schema; inferred ordinal paired subscores average mean is 2.65 over the 31 0.34 points higher than the classified text conservative reviewer. assessments. Adjacent scored transi- 20/22 consecutive overall- Only 3 consecutive Inferred ordinal tions score transitions are flat; 1 numeric-overall pairs are transitions are rises and 1 falls. available: 1 flat, 1 rise, 1 non-monotone across 14 fall. adjacent classified pairs: 3 flat, 5 rise, 6 fall. Interpretation Conservative gate; use- More positive calibration; Useful for detailed ful for desk-reject and useful as a different review qualitative objections; evidence-boundary sig- surface, not as a replace- inferred ordinal trend nals. ment for artifact audit. claims require caution.

harness needs artifact synchronization checks before interpreting any review score. Table 10 reports the raw stage-transition counts used to interpret the process shape.

D

System comparison details

Figure 7 and Table 11 compare what different lines of work make observable under the H1–H7 commitments. The comparison is not a leaderboard. It separates two distinct evidence levels on purpose: prior systems are scored from their published descriptions (paper-described: the system text exposes the relevant ingredient) versus our own S IBYL audit (trace-evidenced: the workspace artifacts contain a recoverable signal-to-update path). The two levels are not directly comparable, and a paper-described entry should not be read as weaker engineering. The point is that auditing trace-level update paths requires access to internal artifacts that we have only for S IBYL.

E

Evaluation protocols

These six protocols are how the framework would be tested at scale. Only the first protocol is exercised in this paper; the remaining five are the evaluation contract this framework invites future work to fulfill, with held-out harnesses, independent annotators, prospective injected failures, and public artifact bundles. 21

Table 9: Internal review-to-action parsing summary across 37 parseable action-plan rows in the 12 audited traces. Score movement uses a ±0.25-point threshold on the 10-point internal review scale when a prior score exists. Visible next-plan task mix Score movement Rows Mean delta High-severity Focus issues/row items/row Down

10

-0.82

8.7

8.4

Flat

6

0.00

6.5

5.8

Up

10

+0.70

4.0

7.5

No prior score

11

–

3.8

5.9

88 tasks in 7 visible next plans: 56 experiment/control, 17 validation/artifact, 13 harness/system, 1 claim/writing, 1 other. 48 tasks in 5 visible next plans: 28 experiment/control, 12 validation/artifact, 7 harness/system, 1 claim/writing. 55 tasks in 5 visible next plans: 31 experiment/control, 13 validation/artifact, 9 harness/system, 1 claim/writing, 1 other. 79 tasks in 8 visible next plans: 57 experiment/control, 10 validation/artifact, 10 harness/system, 1 claim/writing, 1 other.

Table 10: Raw stage-transition counts in workspace event logs. Counts are from 1,853 stage-end records across 12 audited traces and may include resume/checkpoint repetition. Transition group

Count

Interpretation

Writing → writing

405

Writing and revision can absorb many loops without necessarily improving evidence maturity. Pilot/full runs, retries, and parallel experiment batches dominate execution volume. Review and debate frequently return control to planning, reflection, or other harness stages. Planning and reflection often route lessons back into experimental work. Idea and result debates are frequent harness-side review surfaces. Draft production commonly triggers final review or quality assessment. Planning/reflection can route work into explicit decision or quality gates. Trial outputs are often checked by validation or decision stages. Validation gates can return immature evidence to planning, reflection, or other harness stages. Gates can force additional experiments instead of allowing narrative advancement. Review/debate objections often require new or repaired experiments.

Experiment → experiment 271 Review → harness

217

Harness → experiment

124

Harness → review

109

Writing → review

90

Harness → validation

90

Experiment → validation Validation → harness

85 85

Validation → experiment

78

Review → experiment

76

Retrospective trial-to-behavior audit (used in this paper). Given a completed workspace, reconstruct the project goal, trial signals, harness mechanisms triggered, behavior updates, maturity labels, negative evidence, and evidence paths. The audit fails if the claimed behavior update cannot be tied to artifacts. Prospective fixed-budget study. Give several harness designs the same research question, compute budget, and human review budget. Compare maturity gain, trace fidelity, unsupported claims, negative-result handling, compute reliability, and human audit burden. 22

(a) Overall-score / ordinal distributions CSPaper native

4

OpenReviewer native

6

31

7

16

5

14

PaperReview text

0

10

7

20

2.65 (n=31)

PaperReview text

20

31

2.89 (n=35)

CSPaper

33

14

Sibyl internal /2

(b) Calibration against internal review

12

30

3.06 (n=39)

Sibyl internal / 2

3.94 (n=18)

OpenReviewer

40

50

1

2

Snapshots out of 51 1

2

3

Score 4

5

400

Current stage group

Consecutive pairs

PaperReview text (n=14) Sibyl internal (n=24)

15 11 10

8 6

5

5

5

3 1

1 1

17

Harness

OpenReviewer (n=3)

20

5

(d) Full stage-transition matrix (n=1853)

CSPaper (n=22)

20

4

no/unclear

(c) Adjacent scored transitions

25

3

Mean score on rough 1-5 scale

1 1

Experiment

124

90

109

4

271

85

68

76

23

3

3

8

90

405

350 300 250

Validation

85

78

Review

217

76

Writing

5

200 150

2

100

0

50 0

flat

up

down

ne ar

H

t

ss

Ex

n

en

rim

id

l Va

pe

io at

w

vie Re

i rit W

ng

Next stage group

Figure 6: Review-artifact calibration and stage-transition counts (appendix view of the data discussed in Section 6). (A) Two native 1–5 generated-review scores, long-form-reviewer text-inferred ordinal labels, and internal S IBYL supervisor scores divided by two and rounded. Gray no/unclear regions mark missing numeric overall scores or unclassified text assessments. (B) Internal S IBYL scores sit closer to the conservative reviewer than to the rubric reviewer; the long-form reviewer is text-inferred only. (C) Adjacent score transitions differ by review surface; no single review score should be optimized directly. (D) The full stage-transition matrix shows writing self-loops, and review or validation stages routing work back to harness and experiment stages.

Table 11: Table form of Figure 7. TE = trace-evidenced (audited workspace artifacts; available only for S IBYL). PD = paper-described (the cited system description exposes the ingredient). ND = not described. The two levels are not directly comparable; S IBYL’s entries reflect what the audit could recover from preserved traces, not a comparative performance claim. System

H1

H2

H3

H4

H5

H6

H7

AI Scientist-style systems Agent Laboratory / co-scientist AlphaEvolve-style verifier systems Reflexion / memory agents Long-running agent harnesses S IBYL traces (this paper)

PD PD PD PD PD TE

PD PD PD ND PD TE

PD PD PD ND PD TE

ND PD PD PD PD TE

PD PD PD ND ND TE

PD ND PD ND PD TE

ND ND ND ND PD TE

Injected-failure stress test. Insert controlled failures such as duplicate result files, missing outputs, stale tables, inconsistent feature counts, unsupported statistics, or pilot/full mismatches. A strong harness should block or downgrade narrative generation and point to the offending artifact. Cross-project memory test. Let one project expose a failure mode, then start another project where the same failure could recur. Measure whether the lesson is retrieved, routed to the right role, and converted into a changed plan or validation check. 23

H1 Trial orchestration

H2 Evidence maturity

H3 Traceability

H4 Routed memory

H5 Perspective separation

H6 Resource policy

H7 Selfevolution

AI Scientist-style systems Agent Laboratory / co-scientist AlphaEvolve-style verifier systems Reflexion / memory agents Long-running agent harnesses Sibyl traces

Trace-evidenced (Sibyl audit only)

Paper-described in cited system

Not described

Figure 7: Documented support for the H1–H7 commitments under two non-comparable evidence levels. Traceevidenced cells (S IBYL only) come from the audited workspace artifacts in this paper. Paper-described cells come from cited system descriptions and indicate that the published text exposes the ingredient, not that the trace-level update path was audited. Not described indicates the cited description does not expose the ingredient.

Harness-evolution test. Let one project expose a process failure such as missing telemetry, stale figure assets, absolute artifact paths, or paper/LaTeX desynchronization. Start later projects where the same failure could recur. Measure whether the harness adds a gate, sentinel, repair task, artifact contract, or scheduler policy, and whether recurrence rate falls. Perspective and efficiency ablations. Remove or weaken skeptic, methodologist, supervisor, validation, memory, or scheduling components under controlled budgets. The outcome should not be only task completion. It should include overclaim rate, missed validation failures, disagreement-toaction conversion, maturity gain, and audit burden.

F

Governance details

A research harness makes autonomous trial-and-error safer and more auditable while preserving human responsibility. If optimized poorly, it becomes a more efficient machine for plausible weak science. The main risks are scientific spam, rhetorical overclaiming, data integrity failures, metric gaming, unsafe trial-and-error in high-risk domains, negative memory suppressing valid exploration, perspective theater, and self-evolution drift. The corresponding mitigations are claim-evidence gates, AI-use disclosure, human accountability, negative-evidence reporting, domain-specific safety gates, hidden injected failures, protected integrity constraints, and periodic audit of memory overlays. Table 12 summarizes the risk-mitigation pairs. The most important policy point is simple: human authors must be able to understand, defend, and rewrite the final submission. Autonomous systems can help create evidence traces and drafts; they should not be treated as accountable authors. Table 12: Governance risks and mitigations. Each row reads as: when the harness creates the failure mode in column 2, the mitigation in column 3 keeps it auditable. Risk

When it arises

Mitigation

Scientific spam

Paper completion is treated as suc- Claim-evidence gates and cess. validation-first writing. Rhetorical overclaiming Writer holds full claim authority. Maturity-labeled claim registry consumed by writers. Metric gaming A single harness score becomes the Multi-dimensional labels and hidden proxy. injected failures. Negative-memory suppres- Lessons are over-generalized into Time decay, source links, and sion bans. re-opening criteria. Self-evolution drift Overlays or gates optimize polish Periodic memory audits against an over integrity. integrity baseline. Unsafe self-modification Repair tasks weaken protected con- Human-reviewed protected files, straints. regression tests, and rollback logs.

24

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500