AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration

arXiv:2605.20025v1 [cs.AI] 19 May 2026

: Self-Reinforcing Autonomous Research with Human-AI Collaboration Jiaqi Liu∗1 , Shi Qiu∗1 , Mairui Li1 , Bingzhou Li1 , Haonian Ji1 , Siwei Han1 , Xinyu Ye1 , Peng Xia1 , Zihan Dong6 , Congyu Zhang1 , Letian Zhang2 , Guiming Chen2 , Haoqin Tu2 , Xinyu Yang3 , Lu Feng1 , Xujiang Zhao7 , Haifeng Chen7 , Jiawei Zhou1 , Xiao Wang8 , Weitong Zhang1 , Hongtu Zhu1 , Yun Li1 , Jieru Mei10 , Hongliang Fei10 , Jiaheng Zhang4 , Linjie Li11 , Linjun Zhang6 , Yuyin Zhou2 , Sheng Wang11 , Caiming Xiong12 , James Zou9 , Zeyu Zheng5 , Cihang Xie2 , Mingyu Ding1 , Huaxiu Yao1 1

UNC-Chapel Hill, 2 UC Santa Cruz, 3 Carnegie Mellon University, 4 NUS, 5 UC Berkeley, Rutgers University, 7 NEC Labs America, 8 Meta, 9 Stanford University, 10 Google, 11 University of Washington, 12 Recrusive.com, ∗ Equal contribution. Contact: {jqliu,shiqiu,huaxiu}@cs.unc.edu 6

Automating scientific discovery requires more than generating papers from ideas. Real research is iterative: hypotheses are challenged from multiple perspectives, experiments fail and inform the next attempt, and lessons accumulate across cycles. Existing autonomous research systems often model this process as a linear pipeline: they rely on single-agent reasoning, stop when execution fails, and do not carry experience across runs. We present AutoResearchClaw, a multi-agent autonomous research pipeline built on five mechanisms: structured multi-agent debate for hypothesis generation and result analysis, a self-healing executor with a Pivot/Refine decision loop that transforms failures into information, verifiable result reporting that prevents fabricated numbers and hallucinated citations, human-in-the-loop collaboration with seven intervention modes spanning full autonomy to step-by-step oversight, and cross-run evolution that converts past mistakes into future safeguards. On ARC-Bench, a 25-topic experiment-stage benchmark, AutoResearchClaw outperforms AI Scientist v2 by 54.7%. A human-in-the-loop ablation across seven intervention modes reveals that precise, targeted collaboration at high-leverage decision points consistently outperforms both full autonomy and exhaustive step-by-step oversight. We position AutoResearchClaw as a research amplifier that augments rather than replaces human scientific judgment. Github: https://github.com/aiming-lab/AutoResearchClaw

1

Introduction

Automating scientific discovery is a major goal of artificial intelligence. Recent LLM-based systems have shown that agents can generate hypotheses, run experiments, and draft papers (Lu et al., 2025; Yamada et al., 2025; Schmidgall et al., 2025a; Tang et al., 2025). Real research, however, does not proceed in a straight line from idea to paper. A researcher proposes a hypothesis, designs an experiment, observes what fails, revises the plan based on that failure, and tries again iteratively. This loop depends on three capabilities: challenging one’s own hypotheses from multiple angles, recovering from failed experiments without losing partial progress, and carrying lessons from past attempts into future ones. Existing systems handle each of these capabilities poorly. On hypothesis quality, single-agent systems such as AI Scientist (Lu et al., 2025; Yamada et al., 2025) use the same model to generate and evaluate hypotheses, which makes it harder to surface weak assumptions or overly easy directions. On execution robustness, systems such as AIDE ML (Jiang et al., 2025) stop after an execution failure and discard partial results that could still be informative. On experience accumulation, multi-agent systems such as Agent Laboratory (Schmidgall et al., 2025a) allow collaboration within a single run but do not carry lessons across runs, so each attempt starts from scratch. The result is that research is treated as a one-off process rather than an iterative cycle.

1

Figure 1 Overview of the AutoResearchClaw pipeline. Given a research idea, the system progresses through three stages: Discovery (scoping, literature search, multi-agent debate for hypothesis generation), Experimentation (self-healing code execution, result analysis with a second debate panel, and Pivot/Refine decisions), and Writing (drafting, review, revision, four-layer citation verification). Optional human-in-the-loop gates (orange) allow oversight at key checkpoints. The cross-run evolution system (bottom) injects time-decayed lessons from prior runs into all phases.

Our key observation is that these three challenges are not independent. Better hypotheses reduce the need for major revisions during execution. More robust execution preserves intermediate results that can inform analysis. Lessons from past runs can improve both hypothesis generation and experiment design in later attempts. Improving one challenge therefore helps the others, which means they need to be addressed together in a unified framework. We present AutoResearchClaw, a multi-agent research pipeline built around five mechanisms that address these challenges jointly. Structured multi-agent debate assigns agents roles such as innovator, pragmatist, and contrarian, and has them critique each other during hypothesis generation and result analysis; a synthesizer then integrates their outputs into a single structured artifact. A self-healing executor uses a Pivot/Refine decision loop to treat failures as information rather than stopping points: after a failure, the system diagnoses the cause, then either adjusts the current experiment and retries (Refine) or moves to a new direction based on what the failure revealed (Pivot). Verifiable result reporting ties all reported numbers to a registry of executed outputs and checks every citation through a four-layer verification pipeline before anything appears in a draft. Human-in-the-loop collaboration provides seven intervention modes spanning full autonomy to step-by-step approval, with a confidence-driven SmartPause mechanism that routes decisions to the researcher only when system uncertainty is high. Cross-run evolution stores structured lessons from previous runs and injects them as guidance in future attempts through a time-decayed weighting scheme. These mechanisms interact: past lessons inform debate, debate improves experiment choices, self-healing keeps the pipeline moving, and verification ensures outputs are grounded in actual results. In summary, our main contribution is AutoResearchClaw, an open-source multi-agent system for autonomous research that addresses hypothesis quality, execution robustness, and experience accumulation together. We introduce ARC-Bench, a 25-topic benchmark focused on the experiment stage, evaluated with a rubric-assisted LLM judge. On this benchmark, AutoResearchClaw outperforms AI Scientist v2 by 54.7%. A human-in-theloop ablation across seven intervention modes shows that targeted human input at high-leverage decision points consistently outperforms both full autonomy and dense step-by-step oversight. Further analysis shows that the modular design of AutoResearchClaw can connect to domain-specific scientific experiments, including high-energy theory. We discuss safeguards for responsible use, including citation verification, claim grounding, and transparency requirements, in Appendix J.

2

Table 1 Feature comparison of autonomous research systems. ✓ = supported, ✗ = not supported, ∼ = partial. Only AI Scientist v2 and AutoResearchClaw provide end-to-end autonomous execution from idea to paper; Agent Laboratory and ResearchAgent stop short.

Feature

AI Sci. v2

AI Co-Sci.

Agent Lab

MLRCopilot

ResearchAgent

Auto Research Claw

✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓

∼ ✗ ✓ ✗ ∼ ✓ ✗ ✗ ✗ ✗ ✗

✗ ∼ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓

∼ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

End-to-end pipeline Real experiment exec. Multi-agent debate Self-healing repair Pivot/Refine loop Cross-run evolution Citation verification Result verification HITL gates Sandbox security Open-source

2

Related Work

Autonomous research systems. LLMs have been applied to autonomous experiment execution (Boiko et al., 2023) and algorithmic discovery (Romera-Paredes et al., 2024; Novikov et al., 2025). End-to-end research systems vary in scope and capability. The AI Scientist (Lu et al., 2025) and its successor (Yamada et al., 2025) generate complete papers from ideas but rely on single-agent reasoning, abort on execution failures, and start each run from scratch. AI Co-Scientist (Gottweis et al., 2025b,a) introduces multi-agent debate for hypothesis validation but does not execute experiments. Agent Laboratory (Schmidgall et al., 2025a) and AI-Researcher (Tang et al., 2025) automate portions of the pipeline but neither verifies results against ground-truth measurements nor accumulates knowledge across runs. MLR-Copilot (Li et al., 2024) targets machine learning research with explicit human feedback at the execution stage. AgentRxiv (Schmidgall et al., 2025b) explores inter-agent collaboration through shared preprint servers. On the evaluation side, ScienceAgentBench (Tian et al., 2025), MLE-bench (Chan et al., 2024), and DISCOVERYWORLD (Jansen et al., 2024) reveal that even the best systems solve fewer than 40% of tasks. As summarized in Table 1, no prior system combines end-to-end execution with multi-agent debate, self-healing, anti-fabrication verification, and cross-run evolution. Multi-agent debate and cross-run learning. Multi-agent debate improves factual accuracy and divergent thinking (Du et al., 2024; Liang et al., 2023; Tran et al., 2025). Role-assigned frameworks such as ChatDev (Qian et al., 2024), MetaGPT (Hong et al., 2024), and AutoGen (Wu et al., 2024) demonstrate effective collaboration in software engineering. For learning from experience, Reflexion (Shinn et al., 2023) and Self-Refine (Madaan et al., 2023) operate within a single episode; SkillRL (Xia et al., 2026) and EvolveR (Wang et al., 2025) extend this to persistent skill libraries across tasks. OmniScientist (Shao et al., 2025a) argues that science is inherently collaborative and proposes protocols for multi-agent research ecosystems. AutoResearchClaw applies debate with domain-specific epistemic roles at two pipeline stages and accumulates lessons across runs through a persistent time-decayed store, combining both mechanisms in a single system. Human-AI collaboration in research automation. The degree of human involvement in autonomous research remains an open design question. At one extreme, the AI Scientist pursues full automation with minimal human oversight. At the other, SciSciGPT (Shao et al., 2025b) positions AI as an assistant under continuous human direction. Between these extremes, Agent Laboratory (Schmidgall et al., 2025a) allows user-defined feedback frequency and reports that human participation at each stage improves quality. AIssistant (Gaddipati et al., 2025) demonstrates 65.7% time savings through strategic human oversight in review writing. Natarajan et al. (2025) provide a theoretical analysis arguing that the optimal level of human intervention depends on how well-defined the task is. Our HITL ablation contributes empirical evidence to this debate: across seven intervention regimes, we find that targeted intervention at high-leverage decision points consistently outperforms both full autonomy and exhaustive step-by-step oversight.

3

3

AutoResearchClaw

3.1

Overview

AutoResearchClaw is organized as a 23-stage pipeline across three phases (Figure 1): Discovery (scoping, literature search, multi-agent hypothesis generation), Experimentation (self-healing code execution, result analysis, autonomous Pivot/Refine decisions), and Writing (drafting, multi-agent review, revision, citation verification). Five mechanisms span all three phases. Multi-agent debate stress-tests hypotheses and conclusions from complementary perspectives. Self-healing execution treats experiment failures as diagnostic information rather than termination signals. Verifiable result reporting enforces that only grounded numbers and verified citations reach the final output. Human-in-the-loop (HITL) collaboration allows researchers to intervene at high-leverage decision points without managing the full pipeline. Cross-run evolution converts past failures into reusable safeguards through a persistent, time-decayed lesson store. Each stage declares a formal input/output contract and supports checkpoint-based resumption; full stage definitions and hardware adaptation details are in Appendix A.

3.2

Multi-Agent Debate

A single LLM agent naturally tends to confirm the hypotheses it generates, because the same model that proposes an idea has no structural incentive to disconfirm it. AutoResearchClaw addresses this by instantiating structured debate at two pipeline stages. Each debate panel uses K=3 agents with complementary epistemic roles and a synthesizer that integrates their outputs into a single structured artifact. Hypothesis-stage debate. During hypothesis formulation, an Innovator proposes high-risk hypotheses that challenge conventional assumptions, a Pragmatist evaluates feasibility given hardware and time budgets, and a Contrarian actively seeks weaknesses and confounds. The synthesizer distills these perspectives into 2–4 falsifiable hypotheses, each annotated with testability criteria and required baselines. Result-stage debate. After experiments complete, a second panel evaluates the results. An Optimist surfaces strong findings, a Skeptic challenges statistical significance and flags potential confounds, and a Methodologist evaluates reproducibility and checks for data leakage. The synthesizer produces a structured assessment that distinguishes supported claims from unsupported ones before any writing stage begins.

3.3

Self-Healing Execution

Experiment failure is common in real research. Existing autonomous systems treat failure as a termination condition and discard all intermediate progress. AutoResearchClaw instead treats failure as diagnostic information: the system identifies what went wrong, decides whether to fix the current approach or change direction, and preserves all recoverable artifacts. Cascading code generation. Research experiments range from single-file scripts to multi-file systems with custom architectures. A scoring function rates each experiment plan along six dimensions: architectural depth, file count, domain difficulty, dependency chains, historical failure rate, and control-flow complexity, and produces a complexity scalar c ∈ [0, 1]. Experiments above a fixed threshold τ (set to 0.6 in all experiments) are dispatched to an external AI coding agent. Experiments below τ are handled by a built-in multi-phase code agent that first emits a per-file blueprint, then generates files in dependency order using AST-derived summaries to maintain cross-file consistency. Static validation gates check for detectable defects including identical ablation implementations and hardcoded metric values before any execution budget is spent. A dedicated benchmark agent handles dataset and baseline discovery; a figure agent produces publication-quality visualizations. Sandboxed execution. All generated code runs in Docker containers under a three-phase network policy. Phase 0 enables network access for dependency installation. Phase 1 enables network access for data acquisition. Phase 2 disables network access entirely during experiment execution, preventing both result exfiltration and pre-computed-result downloading. Metric reporting is handled exclusively through a read-only evaluation harness, so generated code cannot redefine its own measurement infrastructure (Appendix C). Pivot/Refine decisions. When an experiment fails or produces degenerate results, an automated repair loop captures the failure signature and generates targeted fixes. The system then makes one of three decisions: Proceed when evidence supports the hypothesis, Refine when results are weak but the experimental direction is sound, or Pivot when the approach is fundamentally flawed, returning to hypothesis generation with the 4

failure recorded as new evidence. Systems that terminate on any failure avoid ambitious experiments by design. By making failure recoverable, AutoResearchClaw can pursue higher-risk hypotheses that would be abandoned under a brittle execution model.

3.4

Verifiable Result Reporting

LLM-generated papers face two integrity problems: fabricated experimental results and hallucinated citations. Both arise from the same behavior—the model produces plausible-looking content with no grounding in actual evidence. AutoResearchClaw addresses both through deterministic verification gates applied at two granularities. Numeric registry. During execution, the system constructs a verified registry: a whitelist of every value produced by experiment runs, storing per-condition means, standard deviations, and individual seed measurements. At drafting time, pre-built LATEX tables populated exclusively from the registry are injected into the generation prompt. After generation, a post-hoc verifier re-extracts every numeric claim and checks it against the registry, scoped per condition to prevent cross-condition false positives. Claims in strict sections (Abstract, Results, Experiments) that cannot be matched to a registry entry trigger document rejection. Claims in other sections are replaced with visible placeholders. The writing agent can read the registry but cannot modify it. Citation verification. Every reference passes through a four-layer pipeline: DOI resolution via CrossRef, fuzzy title matching against OpenAlex, arXiv identifier lookup, and Semantic Scholar as a final fallback. An LLM-based relevance check then classifies each reference as Verified, Suspicious, or Hallucinated. References classified as Hallucinated are removed before any draft is finalized.

3.5

Human-in-the-Loop Collaboration

Full automation reduces output quality at critical junctures where domain judgment matters. Exhaustive step-by-step oversight eliminates the efficiency gains of automation. The useful region lies between these extremes: human expertise is most valuable at a small number of high-leverage decision points rather than distributed uniformly across the pipeline. AutoResearchClaw provides seven intervention modes that let researchers select their operating point along this spectrum. Intervention modes. Full-Auto runs the entire pipeline without human input. Gate-Only pauses at three fixed checkpoints: literature screening, experiment design, and final quality review. Thorough pauses at all phase boundaries, giving researchers visibility without requiring approval at every substep. CoPilot targets six high-leverage decision points, including hypothesis co-creation (Idea Workshop), experiment design review (Baseline Navigator), and collaborative paper drafting (Paper Co-Writer). Step-by-Step requires explicit approval at every stage. Two further regimes decompose CoPilot for the ablation in Section 4.4: Pre-Experiment retains intervention only at literature screening, hypothesis generation, and experiment design (early-pipeline), while Post-Experiment retains intervention only at result analysis, paper draft, and quality gate (late-pipeline). Our HITL ablation in Section 4.4 evaluates all seven modes empirically. SmartPause. Rather than relying on fixed checkpoints, SmartPause monitors the system’s estimated uncertainty at each stage. When uncertainty exceeds a learned threshold, the system pauses and presents the decision to the researcher. The threshold adapts based on historical approval patterns: stages where the researcher frequently overrides the system are paused more often, while stages with consistently high approval rates proceed autonomously.

3.6

Cross-Run Evolution

Existing autonomous research systems are stateless across runs: every run begins without knowledge of previous attempts, repeating failures that earlier runs already encountered. AutoResearchClaw maintains a persistent lesson store that converts past failures into future safeguards. At the end of each run, the system extracts structured lessons from repair attempts, Pivot/Refine decisions, HITL gate feedback, and verification results. Each lesson records a category, a severity score s(l) ∈ (0, 1], and a recommended mitigation. When a new run begins, relevant lessons are retrieved by category and ranked by a time-decayed weight:   ∆t w(l) = s(l) · exp − ln 2 · , (1) T1/2 5

where ∆t is the elapsed time since the lesson was recorded and T1/2 is a half-life hyperparameter controlling how quickly older lessons lose influence (default T1/2 = 30 days). Lessons are injected into prompts as naturallanguage overlays, requiring no model retraining and remaining applicable to any LLM backbone. This design means that recent failures strongly constrain subsequent runs, while lessons from completed, successful lines of work gradually fade from prominence.

4

Experiments

We evaluate AutoResearchClaw through three complementary studies. First, we benchmark against existing systems on ARC-Bench using an experiment-stage evaluation, because most baselines cannot reliably produce complete papers without human supervision (§4.2). Second, we conduct an end-to-end evaluation from idea to paper on 10 ARC-Bench topics across seven human-in-the-loop regimes, assessing full paper quality under varying levels of intervention (§4.4). Third, we run a component ablation that isolates the contribution of each mechanism (§4.5). We close with a case study illustrating how the mechanisms interact on a single topic (§4.6).

4.1

Experimental Setup

Benchmark. We introduce ARC-Bench, a 25-topic ML benchmark (ML01–ML25) spanning tabular ML, optimization, dimensionality reduction, NLP, AutoML, GP kernels, topic modeling, semi-supervised learning, dynamical systems, anomaly detection, feature selection, causal discovery, and learning-to-rank, together with a 20-topic scientific-domain extension covering 10 high-energy physics (P01–P10), 7 systems biology (B01–B07), and 3 statistics (S01–S03) tasks. Each topic specifies a research question, a target dataset (or reference figure/simulation, for science topics), and expected experimental deliverables (code, results, analysis writeup). ARC-Bench supports three evaluation modes. The experiment-stage mode evaluates systems at the experiment stage using a rubric-assisted strict judge, enabling fair comparison across systems with different end-to-end capabilities. The end-to-end mode evaluates the full pipeline from research idea to completed paper, assessing overall paper quality on a 1–10 scale with accept rate (≥ 5) as the primary metric. The end-to-end mode is used both for the HITL ablation on 10 ML topics (§4.4) and the scientific-domain coverage study (§4.3). The scientific-domain mode reuses the same code/exec/results/repro rubric on the 20 physics/biology/statistics topics, since most existing baselines cannot produce science-domain papers at all under fair-input conditions. Experiment-stage evaluation protocol. The strict judge grades each (framework × topic) cell along three dimensions weighted as CD:CE:RA = 25:25:50. Code Development (CD) assesses whether the implementation correctly instantiates the proposed method and baselines. Code Execution (CE) verifies that experiments run to completion and produce valid outputs. Result Analysis (RA) evaluates whether conclusions are grounded in actual measurements, hypotheses receive explicit verdicts, and limitations are honestly reported. RA receives double weight because it captures the scientific reasoning quality that distinguishes autonomous research from automated scripting. Two independent agent reviewers run the strict judge in parallel; per-leaf disagreements exceeding |∆| > 0.20 are re-adjudicated and final scores are averaged. Details and inter-rater agreement are in Appendix D.2. Baselines and implementation. We compare against AI Scientist v2 (Yamada et al., 2025) and AIDEML (Jiang et al., 2025), the two systems that provide end-to-end execution paths comparable to AutoResearchClaw (Table 1). Agent Laboratory (Schmidgall et al., 2025a) is excluded because it does not deliver end-to-end execution under fair-input conditions. All frameworks use the same LLM backbone (GPT-5.3-codex) and the same sandboxed execution environment with identical per-experiment time budgets. This controlled setup isolates the contribution of system design from backbone capability.

4.2

Main Results: Experiment-Stage Comparison

Table 2 reports mean scores across all 25 topics under the experiment-stage evaluation mode. AutoResearchClaw outperforms all baselines across all dimensions. AutoResearchClaw (CoPilot) achieves the highest overall strict score (0.648), outperforming AI Scientist v2 (0.419) by 54.7% and AIDEML (0.511) by 26.8%. Even in Full-Auto mode without human intervention, AutoResearchClaw (0.596) substantially exceeds both baselines, indicating that the gains are primarily driven by system design rather than human input.

6

Table 2 ARC-Bench experiment-stage results (25 topics, CD:CE:RA = 25:25:50). Framework AutoResearchClaw (CoPilot) AutoResearchClaw (Full-Auto) AIDE-ML AI Scientist v2

Code Dev

Code Exec

Result Analysis

Overall

0.968 0.938 0.958 0.712

0.578 0.562 0.415 0.442

0.523 0.442 0.336 0.261

0.648 0.596 0.511 0.419

Table 3 End-to-end HITL ablation across 10 topics and 7 intervention regimes. Paper quality scored 1–10; accept = score ≥ 5. Mode

Valid

Mean Q

Accept

Interventions

Full-Auto Gate-Only CoPilot Thorough Step-by-Step Pre-Experiment Post-Experiment

8/10 10/10 8/10 7/10 10/10 8/10 6/10

4.03 5.03 7.27 4.86 5.19 4.28 5.08

25.0% 50.0% 87.5% 42.9% 50.0% 37.5% 50.0%

0 3 6 8 23 3 3

The largest advantage is on Result Analysis. The performance gap is most pronounced on Result Analysis, where AutoResearchClaw (CoPilot) scores 0.523 against AI Scientist v2’s 0.261, a 100.4% relative improvement. This dimension evaluates whether conclusions are hypothesis-aligned, tables contain only verified numbers, and limitations are honestly reported. The advantage directly reflects multi-agent debate at the result analysis stage and the verified result registry: debate forces explicit per-hypothesis verdicts with critical scrutiny, and the registry ensures every reported number traces to an actual measurement. In contrast, AI Scientist v2’s single-agent analysis tends to oversell weak findings without cross-examination. Code Development is competitive; execution separates the systems. All systems score above 0.70 on Code Development, with AIDE-ML (0.958) nearly matching AutoResearchClaw (0.968). The differentiator is what happens after code is written. AIDE-ML’s execution success rate (0.415) is substantially lower, reflecting its lack of self-healing: experiments that encounter runtime errors are discarded rather than repaired. AutoResearchClaw’s self-healing executor raises execution success to 0.562 (Full-Auto) and 0.578 (CoPilot) by diagnosing failures and applying targeted fixes through the Pivot/Refine loop. Failure mode analysis. Among the 25 topics, AutoResearchClaw (Full-Auto) fails to produce valid results on 2 topics, both involving complex multi-file implementations with cascading dependencies. AI Scientist v2 fails on 6 topics, with failures concentrated in topics requiring iterative experiment refinement (dynamical systems, causal discovery) where single-attempt execution without recovery is insufficient. This pattern confirms that self-healing is most valuable on topics where the first implementation attempt is unlikely to succeed.

4.3

Cross-Domain Coverage: Physics, Biology, and Statistics

The ARC-Bench core (ML01–ML25) is intentionally ML-focused to enable a fair comparison against AIDE-ML and AI Scientist v2, both of which target ML pipelines. Autonomous research systems must, however, operate across scientific domains. We therefore extend ARC-Bench with 20 scientific-domain tasks that each require domain-specific software stacks: MadGraph5_aMC@NLO (Alwall et al., 2014) / Pythia8 / Delphes / FeynRules / MadAnalysis5 for high-energy physics; COBRApy (Ebrahim et al., 2013) / BiGG (King et al., 2016) genome-scale models / optlang / GLPK for systems biology; and double-machine-learning (Chernozhukov et al., 2018), bootstrap (Efron, 1979), and selective-inference machinery for statistics. AutoResearchClaw routes each topic through a sandboxed, domain-specialized agent during the experiment stage. The HEP agent is equipped with the FeynRules, MadGraph, and MadAnalysis skills drawn from Qiu et al. (2026), which introduced the ColliderAgent architecture we directly adopt here; the biology agent is equipped with GEM-builder, flux-balance analysis, and flux-analyzer skills; the statistics agent is equipped with Monte Carlo simulation and semiparametric inference skills. Each specialized agent runs inside a Claude Code subprocess with the requisite packages pre-installed, so the top-level orchestrator requires no domain-specific 7

Table 4 Scientific-domain coverage. Scores are computed with the same code:exec:results:repro rubric as Table 2. We evaluate biology on B01–B07, statistics on S01–S03, and HEP-ph on P01–P10 . A red cross indicates a code-execution failure caused by missing or unusable domain-specific software. Statistics tasks do not require specialized simulation stacks, so all systems are scored normally. Failed runs are counted as zero when computing the run-weighted overall mean over the 20 science-domain tasks. Framework

Biology

Statistics

HEP-ph

Overall

AutoResearchClaw (CoPilot) AIDE-ML AI Scientist v2

0.912 ✗ ✗

0.898 0.452 0.418

0.489 ✗ ✗

0.867 0.090 0.084

knowledge. This design is what enables AutoResearchClaw to reproduce experiments across heterogeneous scientific fields without per-domain engineering effort. Each task is graded with the same code / execution / results / repro rubric used in Table 2, with leaf criteria rewritten to reflect domain-appropriate targets (e.g., “loaded the correct iJO1366 model and set anaerobic medium bounds” for biology; “computed cross-section ratios within 3% of the published reference” for HEP). Per-task scores are aggregated to a column mean per domain. Table 4 reports the results. AIDE-ML and AI Scientist v2 fail to install the required HEP and biology stacks under fair-input conditions and produce no valid output on P01–P10 and B01–B07; on the statistics tasks both systems execute but their generic ML scaffolding yields incomplete implementations that miss the inferential targets (e.g., bias and coverage metrics not computed, no Neyman-orthogonal score, no estimand writeup). Sandboxed domain agents are necessary for cross-domain coverage. The biology column (mean 0.912 ranging from E. coli succinate knockout screens to M. tuberculosis essentiality and drug-target prioritization) is supported entirely by COBRApy-skilled sub-agents operating over BiGG genome-scale models; the orchestrator is never required to manipulate stoichiometric matrices directly. The statistics column (mean 0.898 on bootstrap confidence-interval coverage and DML/AIPW estimation) is supported by the Monte Carlo and semiparametricinference skill bundle; the third statistics run completed the pipeline but failed the requirements judge on missing metric artifacts and is therefore excluded from the column mean. The HEP column requires Lagrangian-to-UFO translation, MadGraph parton-level generatio, and quantitative reproduction of a published cross-section curve; AutoResearchClaw correctly reproduces the predicted shape and numerical cross-section values, but incurs scoring penalties for insufficient deliverable content and minor unsupported meta-claims. The qualitative conclusion of Table 4 is that both baselines score zero on physics and biology because their sandboxes do not include the required scientific software; AutoResearchClaw’s domain-skill installation step closes this gap.

4.4

End-to-End HITL Ablation

ARC-Bench’s experiment-stage mode enables fair cross-system comparison but does not evaluate the full research pipeline. To assess end-to-end quality from idea to completed paper, we run a HITL ablation on 10 ARC-Bench topics across seven intervention regimes, ranging from Full-Auto (zero interventions) to Step-by-Step (every stage). CoPilot targets six high-leverage decision points; Pre-Experiment and PostExperiment isolate early-stage and late-stage contributions respectively. Each mode receives the same payload at its covered stages. Setup details are in Appendix E. More intervention does not monotonically improve quality. Table 3 reports mode-level summaries. CoPilot achieves the highest mean paper-quality score (7.27) and accept rate (87.5%) with 19 targeted interventions. Step-by-Step requires 29 interventions but achieves only 5.19 and 50%. On matched topics, CoPilot beats Full-Auto by +3.21 and beats Step-by-Step by +2.16. The explanation is that Step-by-Step’s approve actions at non-critical stages add noise without information, while CoPilot concentrates expert judgment where it has the highest marginal impact. Pre-Experiment and Post-Experiment HITL address different failure modes. Splitting CoPilot reveals two complementary contributions. Pre-Experiment HITL (stages 5, 8, 9) fixes research design feasibility: on T02, it compresses a 240-cell factorial design to a feasible 60-cell layout and prescribes appropriate statistical tests. Pre-Experiment alone is widely valid (8/10) but rarely lifts quality (mean 4.28, 37.5% accept), because correct scoping alone does not fix downstream claim discipline. Post-Experiment HITL (stages 14, 17, 20)

8

Table 5 Component ablation in Full-Auto mode on the same 10 ARC-BENCH topics using a best-of-3 protocol over three reruns per configuration–topic pair. Completion counts topics for which at least one rerun completed a manuscript. Quality is the mean of the selected best scores over completed topics, and Accept counts selected completed outputs with score ≥ 5. Fabrication is determined by manual audit of the selected outputs. ‡ Score inflated by removing the verification gate. Configuration

Completion

Quality

Accept

Fabrication

10/10 10/10 6/10 9/10 10/10 4/10

5.62 4.25 4.83 5.14 5.48‡ 3.47

3/10 1/10 1/6 2/10 5/10‡ 0/4

✗ ✗ ✗ ✗ ✓ ✗

Full AutoResearchClaw w/o Debate w/o Self-Healing w/o Evolution w/o Verification w/o Debate & Healing

fixes claim faithfulness, ensuring conclusions match actual measurements and scope is honestly delimited. Post-Experiment produces some of the strongest individual papers but is valid on only 6/10 topics, because late-stage HITL cannot generate experimental evidence from nothing. CoPilot is best because it spans both halves: fixing feasibility early and faithfulness late. Gate-Only provides a cost-effective middle ground. Gate-Only (3 interventions) raises accept rate from 25% to 50% and is the only mode achieving 10/10 validity. For practitioners seeking minimal human involvement with meaningful quality improvement, Gate-Only represents an attractive operating point between Full-Auto and CoPilot.

4.5

Component Ablation

To isolate the contribution of each mechanism, we run a system-level ablation on the same 10 ARC-BENCH topics under Full-Auto mode. Each row in Table 5 removes one mechanism and keeps the others intact. Best-of-N protocol. Autonomous research agents are stochastic and path-dependent: a single unlucky branch in hypothesis generation, code repair, or drafting can dominate the final outcome. We therefore adopt a best-of-N protocol; Full-Auto numbers in this ablation reflect this setting and should be read against Table 5 rather than the single-run HITL results. Detailed ablations of design-space exploration parameters appear in Appendix F. Each mechanism addresses a distinct failure mode, even under a favorable rerun budget. Multiagent debate is the largest quality contributor (−1.37, p=0.003): without the Pragmatist filtering infeasible hypotheses and the Skeptic scrutinizing weak findings, both hypothesis quality and result analysis degrade. Self-healing is the largest completion contributor: removing it reduces completion from 10/10 to 6/10 even with three attempts per topic, because the first unrecovered runtime error terminates the run and surviving topics are disproportionately easier. Cross-run evolution provides a moderate reliability gain (−0.48 quality, −1 completion) by injecting lessons from prior failures, primarily avoiding known failure modes rather than raising the quality ceiling. Verification is the integrity backstop. Removing the verified registry raises apparent acceptance from 3/10 to 5/10, but manual inspection reveals that 3 of those 5 papers contain values absent from any measurement record. The verification gate correctly separates genuine from fabricated results; its cost to acceptance rate is the price of scientific integrity. The mechanisms interact super-additively. Combined removal of debate and self-healing drops completion to 4/10, mean quality to 3.47, and acceptance to zero. The interaction is intuitive: debate without self-healing produces ambitious hypotheses that crash on first failure, while self-healing without debate repairs experiments that test poorly formulated questions.

4.6

Case Study: Topic T10

Figure 2 compares Full-Auto and CoPilot on Topic T10, which studies cross-validation strategies for small-sample model selection. Both runs produce complete-looking manuscripts, but Full-Auto fails in a way that runtime

9

Figure 2 Case study comparing Full-Auto and CoPilot on Topic T10. Full-Auto suffers a silent semantic collapse: all cross-validation strategies report identical zero-bias outputs. CoPilot produces differentiated results, enabling a meaningful comparison across strategies.

metrics do not capture: its experiment collapses to identical outputs across conditions. A detailed artifact-level comparison is provided in Appendix G. The T10 case shows why execution success alone is not enough. Full-Auto completes a manuscript, but all eight cross-validation strategies collapse to identical zero-bias outputs. As a result, the paper cannot support a substantive comparison. CoPilot avoids this failure because the human guidance targets the experimental bottleneck directly. It asks the system to check whether cross-validation strategies produce different outcomes, whether leave-one-out cross-validation fits the time budget, and whether the manuscript’s claims stay within the logged results. The final CoPilot paper remains exploratory, but it reports nonzero contrasts across nine pipelines and states its limitations clearly. The Stage-20 quality gate therefore accepts it. Three observations follow. First, debate quality matters even when execution succeeds: the Contrarian’s concern about distinguishable ablations helps flag the collapsed design before it becomes the final experiment. Second, verification is necessary but not sufficient. Full-Auto passes the numeric gate because the zero values are real logged measurements, not fabricated numbers. However, the gate cannot tell whether those measurements answer the research question. Third, CoPilot improves quality not by adding more intervention, but by placing intervention at the right decision points. Guidance about experimental semantics produces evidence-grounded output, while late writing-stage guidance cannot recover missing experimental evidence. This pattern is consistent with the HITL results in Table 3: partial or mistimed intervention improves some failure modes, but does not match CoPilot’s combination of early experimental guidance and late claim checking. Broader failure and writing-quality audits are reported in Appendices H and I.

5

Conclusion

We presented AutoResearchClaw, a multi-agent autonomous research pipeline that unifies structured debate, self-healing execution, verifiable result reporting, cross-run evolution, and human-in-the-loop collaboration in a single self-reinforcing system. On ARC-Bench, AutoResearchClaw outperforms AI Scientist v2 by 54.7%, with the largest gains on result analysis where multi-agent debate and verified reporting produce hypothesis-aligned, grounded conclusions. An end-to-end HITL ablation across seven intervention regimes shows that targeted intervention at high-leverage decision points (CoPilot, 87.5% accept rate) consistently outperforms both full autonomy (25%) and exhaustive step-by-step oversight (50%), establishing that precise human-AI collaboration is a more effective paradigm than either extreme. Component ablation confirms 10

that the mechanisms are complementary: debate drives quality, self-healing drives completion, verification enforces integrity, and their combined removal is super-additive. We position AutoResearchClaw as a research amplifier that accelerates scientific exploration while keeping verifiability at the center, rather than a replacement for human scientific judgment. Detailed ethical issues are discussed in Appendix J.

References J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H.-S. Shao, T. Stelzer, P. Torrielli, and M. Zaro. The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations. Journal of High Energy Physics, 2014(7):79, 2014. doi: 10.1007/JHEP07(2014)079. Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Emergent autonomous scientific research capabilities of large language models. Nature, 624:570–578, 2023. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024. Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 (1):C1–C68, 2018. doi: 10.1111/ectj.12097. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. Proceedings of the International Conference on Machine Learning (ICML), 2024. Ali Ebrahim, Joshua A. Lerman, Bernhard O. Palsson, and Daniel R. Hyduke. COBRApy: COnstraints-Based reconstruction and analysis for Python. BMC Systems Biology, 7:74, 2013. doi: 10.1186/1752-0509-7-74. Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1–26, 1979. doi: 10.1214/aos/1176344552. Sasi Kiran Gaddipati et al. Aissistant: Human-ai collaborative review and perspective research workflows in data science. arXiv preprint arXiv:2509.12282, 2025. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864, 2025a. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, et al. Towards an AI co-scientist. arXiv preprint arXiv:2502.18864, 2025b. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. MetaGPT: Meta programming for a multi-agent collaborative framework. Proceedings of the International Conference on Learning Representations (ICLR), 2024. Peter Jansen, Marc-Alexandre Cote, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. DISCOVERYWORLD: A virtual environment for developing and evaluating automated scientific discovery agents. arXiv preprint arXiv:2406.06769, 2024. Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code. 2025. URL https://arxiv.org/abs/2502.13138. Zachary A. King, Justin Lu, Andreas Dräger, Philip Miller, Stephen Federowicz, Joshua A. Lerman, Ali Ebrahim, Bernhard O. Palsson, and Nathan E. Lewis. BiGG models: A platform for integrating, standardizing and sharing genome-scale models. Nucleic Acids Research, 44(D1):D515–D522, 2016. doi: 10.1093/nar/gkv1049. Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. Mlr-copilot: Autonomous machine learning research based on large language models agents. arXiv preprint arXiv:2408.14033, 2024. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.

11

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shailaja Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2023. Sriraam Natarajan, Saurabh Mathur, Sahil Sidheekh, et al. Human-in-the-loop or ai-in-the-loop? automate or collaborate? Proceedings of AAAI, 2025. Alexander Novikov, Ngan Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. ChatDev: Communicative agents for software development. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2024. Shi Qiu, Zeyu Cai, Jiashen Wei, Zeyu Li, Yixuan Yin, Qing-Hong Cao, Chang Liu, Ming xing Luo, Xing-Bo Yuan, and Hua Xing Zhu. An end-to-end architecture for collider physics and beyond, 2026. URL https://arxiv.org/abs/ 2603.14553. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625:468–475, 2024. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using LLM agents as research assistants. arXiv preprint arXiv:2501.04227, 2025a. Samuel Schmidgall et al. AgentRxiv: Towards collaborative autonomous research. arXiv preprint arXiv:2503.18102, 2025b. Chenyang Shao, Dehao Huang, Yu Li, Keyu Zhao, Weiquan Lin, Yining Zhang, Qingbin Zeng, Zhiyu Chen, Tianxing Li, Yifei Huang, et al. Omniscientist: Toward a co-evolving ecosystem of human and ai scientists. arXiv preprint arXiv:2511.16931, 2025a. Erzhuo Shao, Yifang Wang, Yifan Qian, et al. Sciscigpt: Advancing human-ai collaboration in the science of science. Nature Computational Science, 2025b. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. AI-Researcher: Automating scientific research through multi-agent collaboration. Advances in Neural Information Processing Systems, 38, 2025. Ziru Tian, Yanheng Ning, et al. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. Proceedings of the International Conference on Learning Representations (ICLR), 2025. Khanh-Tung Tran et al. Multi-agent collaboration mechanisms: A survey of LLMs. arXiv preprint arXiv:2501.06322, 2025. Xiaoman Wang et al. EvolveR: Self-evolving LLM agents through an experience-driven lifecycle. arXiv preprint arXiv:2510.16079, 2025. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. Proceedings of the Conference on Language Modeling (COLM), 2024. Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. SkillRL: Evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234, 2026. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025.

12

Appendix A

Full Stage Definitions

Table 6 provides the complete specification of all 23 pipeline stages. The 23-stage design reflects a tension between granularity and overhead. Early prototypes used a coarser 12-stage pipeline, but bundling too many responsibilities into a single LLM call (e.g. “search the literature, screen it, and extract knowledge” as one stage) led to poor intermediate quality that compounded downstream. Conversely, very fine-grained pipelines (we experimented with 30+) incurred excessive context-reconstruction overhead. The current 23-stage design was arrived at iteratively. Each stage defines a formal contract: a JSON schema specifying required input fields (with types and validation), expected output fields, acceptance criteria (e.g. “at least 2 hypotheses must be marked falsifiable”), and an error-code namespace (e.g. E-HYPO-*). Table 6 Complete 23-stage pipeline specification. #

Stage Name

Key Output

Special

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Topic_Init Problem_Decompose Search_Strategy Literature_Collect Literature_Screen Knowledge_Extract Synthesis Hypothesis_Gen Experiment_Design Code_Generation Resource_Planning Experiment_Run Iterative_Refine Result_Analysis Research_Decision Paper_Outline Paper_Draft Peer_Review Paper_Revision Quality_Gate Knowledge_Archive Export_Publish Citation_Verify

SMART goal, hardware profile Problem tree Search queries, sources.json candidates.jsonl Shortlisted papers Knowledge cards Gap analysis 2–4 hypotheses exp_plan.yaml + benchmarks Multi-file experiment code Time/resource estimates results.json per condition Improved code + metrics analysis.md + charts Proceed/Refine/Pivot Section outline Full paper draft Review feedback Revised paper quality_report.json Lessons extracted LATEX + sanitized BibTeX verification_report.json

Domain detect

API calls HITL Gate

Debate (K=3) HITL Gate Cascade† Docker sandbox Self-heal Debate (K=3) Decision Verified tables Multi-agent Length guard HITL Gate Noncritical Anti-fabrication 4-layer API

† Cascade: Beast Mode (external AI agent) → CodeAgent (multi-phase, blueprint, sequential, AST gates) → Legacy single-shot.

Each tier falls back to the next on failure.

B

Prompt Design

B.1

Prompt Architecture

The prompt system is organised into three nested layers. Primary layer. Twenty-three stage-specific prompts, each comprising a system message, a structured user template, an optional json_mode flag, and a max_tokens override. Section-level word-count targets are enforced post-generation: abstract 150–200, introduction 800–1000, related work 600–800, method 1000–1500, experiments 800–1200, results 600–800, discussion 400–600, conclusion 200–300. Reusable block layer. Shared text fragments injected into multiple stages to enforce consistent writing quality and experimental rigour: academic_style_guide, anti_hedging_rules, writing_structure, dataset_13

Algorithm 1 AutoResearchClaw: Autonomous Research Pipeline with Domain-Aware Routing Require: Research idea I, lesson store L, max pivots Np =2, max refines Nr =10 Ensure: Verified paper P, updated lesson store L′ 1: Detect hardware profile; detect research domain D via three-level cascade (forced override → keyword matching → LLM classification) 2: Select domain prompt bank B(D); instantiate PromptManager(D) and domain adapter A(D); inject relevant lessons from L 3: Scope topic from I using B(D).topic_init; retrieve & screen literature; synthesise knowledge cards 4: np ← 0 5: repeat [Debate] K=3 domain-specialised agents via A(D).debate_roles_hypothesis → hypotheses H (e.g. Innova6: tor/Pragmatist/Contrarian for ML; Theorist/Phenomenologist/Experimentalist for HEP-ph) 7: Design experiments from H with A(D)-injected condition terminology, paradigm constraints, and compute budget 8: Score complexity c; select generator via cascade (Beast → Agent → Legacy) 9: Validate code: AST checks, import verification, ablation-identity detection 10: nr ← 0 11: repeat 12: Execute in Docker sandbox (three-phase network isolation) with domain-specific image and pre-cached datasets 13: while experiment fails and repair budget remains do 14: Parse failure signature; generate targeted fix; re-execute 15: end while 16: [Debate] K=3 agents via A(D).debate_roles_analysis → result analysis 17: Decide d ∈ {Proceed, Refine, Pivot}; nr ← nr + 1 18: until d ̸= Refine or nr ≥ Nr 19: np ← np + 1 20: until d ̸= Pivot or np ≥ Np 21: Build verified registry R from experiment results 22: Draft paper with pre-built tables from R using A(D).preferred_template; review → revise 23: Verify: reject if unverified numbers appear in strict sections 24: Verify citations via 4-layer API pipeline (CrossRef, OpenAlex, arXiv, S2) 25: Extract lessons; update L → L′ with time-decayed weights 26: return P with verification reports

guidance, and hp_reporting. Sub-prompt layer. Specialised prompts for intra-stage operations: architecture_planning, generate_single_file, code_repair, hypothesis_synthesize, analysis_synthesize, and iterative_improve. Total prompt text across all three layers is approximately 46K tokens. Users can override any prompt via prompts.default.yaml; the PromptManager safely renders {word_chars} placeholders while leaving JSON schema syntax untouched. Domain-aware prompt selection. The PromptManager accepts a domain argument at instantiation and selects the corresponding stage bank. Two native banks exist: an ML bank (default) and a HEP-ph phenomenology bank. All other domains receive the ML bank augmented with a domain-adapter overlay. Both banks expose the same 23 stage keys and the same {placeholder} variables, enforced by a parity test suite, so all downstream pipeline logic is bank-agnostic. Domain detection runs a three-level cascade at pipeline start. Level 0 : a forced override via project.profile in config.yaml skips all detection and ensures cross-stage consistency. Level 1 : fast keyword matching against 350+ domain-specific rules ordered most-specific-first. Level 2 : LLM classification for topics not resolved by keywords, outputting one of 24 domain IDs. Level 3 : generic _generic.yaml fallback. Beyond the two native banks, AutoResearchClaw supports 20+ domains through a profile-and-adapter system in researchclaw/domains/. Each domain profile (a YAML file) specifies the experiment paradigm, condition terminology, standard baselines, typical file structure, core libraries, Docker image, metric types, statistical tests, output formats, and prompt guidance blocks. Each domain adapter injects domain-specific

14

content into stage prompts via {domain_context} and related placeholders. Table 7 lists domains with complete end-to-end configuration. Table 7 Domain support matrix. Bank: native 23-stage prompt bank. Profiles: YAML profiles with experimentparadigm configuration. Adapter: Python adapter with stage-level prompt injection. Template: preferred journal LATEX template. Domain

Bank

Profiles

Adapter

Template

Machine Learning High-Energy Physics Computational Biology Computational Chemistry Theoretical Physics Empirical Economics Mathematics Computational Neuroscience

ML HEP-ph Bio Chem Phy ML + adapter ML + adapter ML + adapter

8 1 3 2 1 1 2 1

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

NeurIPS/ICML/ICLR JHEP / PRD Bioinformatics JCTC / JCIM PRX / J. Comp. Phys. AER / JEEA Math. Comp. PLoS Comp. Bio.

Domain-differentiated debate roles. The most consequential domain-level difference is in multi-agent debate personas. For hypothesis generation (Stage 8), the ML bank uses Innovator (cross-domain analogies, high-risk hypotheses), Pragmatist (computational feasibility, incremental gains), and Contrarian (challenges assumptions, finds failure modes). The HEP-ph bank replaces these with Theorist (BSM Lagrangians, symmetries, UV completions), Phenomenologist (testable observable signatures, analytical cross-section estimates), and Experimentalist (detector acceptances, background shapes, overlooked systematics). For result analysis (Stage 14), ML uses Optimist / Skeptic / Methodologist; HEP-ph uses Model-Builder / Phenomenologist / Experimentalist, with all verdicts stated in natural units with explicit experimental bound citations.

B.2

Per-Stage Prompt Breakdown

Table 8 lists all 23 ML-bank stages with their system role, key prompt requirements, and output format. The full bank is approximately 28K tokens. Two stages with the highest prompt engineering load are detailed below the table. Stage 1 Deep Dive: Topic_Init

Stage 1 illustrates the prompt structure shared across all stages. The system message defines the agent role and novelty principles: “You are a rigorous research planner who identifies NOVEL, TIMELY research angles. You follow recent trends from top venues in the relevant domain and propose research that advances the frontier rather than repeating known results. NOVELTY PRINCIPLES: A good research angle addresses a GAP not yet covered by existing work. Avoid pure benchmark/comparison studies unless the methodology is novel. The research must be FEASIBLE with limited compute (single GPU, hours not days). Check: would a reviewer say ‘this is already well-known’? If so, find a sharper angle.”

The user template is parameterised by {topic}, {domains}, {project_name}, and {quality_threshold}, and requires six output sections: Topic, Novel Angle (specific unexplored aspect with trend validation—no fabricated paper titles), Scope, SMART Goal, Constraints, and Success Criteria. Stage 10 Deep Dive: Code_Generation

Stage 10 carries the highest prompt engineering load at approximately 1700 lines of specification. Four mandatory requirements are enforced at prompt level. Algorithm integrity. A method labelled “Bayesian Optimisation” must include a surrogate model, acquisition function, and model updates. A method labelled “PPO” must include a clipped surrogate objective, learned value baseline, and clip_eps in the loss. Dead parameters (declared but never consumed) are forbidden.

15

Table 8 ML prompt bank: per-stage system role, key user requirements, and output format. #

Stage

Phase A: Research Scoping 1 Topic_Init 2

Problem_Decompose

Phase B: Literature Discovery 3 Search_Strategy 4 Literature_Collect 5

Literature_Screen

6

Knowledge_Extract

Key requirements

Output

Novel angle, SMART goal, trend validation, benchmark specification ≥4 prioritised sub-questions, risks

Markdown

Merged search plan, source manifest ≥8 candidate papers with abstract, year, source Domain-match filter, recency preference, quality floor Structured knowledge cards: problem, method, data, metrics, findings

JSON JSON

Phase C: Knowledge Synthesis Cluster overview, gap list, prioritised 7 Synthesis opportunities 8 Hypothesis_Gen ≥2 falsifiable hypotheses; novelty argument, measurable prediction, failure condition, required baselines Phase D: Experiment Design 9 Experiment_Design

10

Code_Generation

11

Resource_Planning

YAML plan: objectives, datasets, baselines, proposed methods, ablations, metrics, risks, compute budget Multi-file Python; algorithm integrity; calibration loop; PyTorch detach rules; condition breadth-first ordering GPU/CPU time estimates, parallelism plan, fallback strategy

Phase E: Experiment Execution 12 Experiment_Run Execution stage; no LLM call Targeted fix based on failure signature; 13 Iterative_Refine repair budget tracking Phase F: Analysis & Decision 14 Result_Analysis

15

Research_Decision

Phase G: Paper Writing 16 Paper_Outline 17

Paper_Draft

18

Peer_Review

19

Paper_Revision

Phase H: Finalization 20 Quality_Gate 21 22

Knowledge_Archive Export_Publish

23

Citation_Verify

Markdown

JSON JSON

Markdown Markdown

YAML

Multi-file code

JSON

metrics.json Patched code

Per-hypothesis verdict; effect sizes; methodology audit; quality rating 1–10; K=3 debate consensus Proceed / Refine / Pivot with evidence-based justification

Markdown

Section structure, 3 title candidates, per-section content plan Full 8–10 page draft; verified-registry tables only; no fabricated numbers in strict sections ≥3 reviewer perspectives; soundness, novelty, presentation, significance Point-by-point response to reviews; section-length guard

Markdown

Quality report; noncritical warnings do not block Lesson extraction; time-decayed update LATEX + sanitised BibTeX; domaintemplate selection; anti-fabrication pass 4-layer API pipeline; reject unresolvable DOI in strict sections

JSON

16

Markdown

Markdown

Markdown Markdown

Markdown LATEX JSON

Calibration loop. A pilot run (3–5 seeds, 2 conditions) checks that the primary metric has nonzero variance across conditions. If all conditions score identically, the system prints WARNING: DEGENERATE_METRICS and applies difficulty adjustments before the full run proceeds. PyTorch detach rules. Any tensor from a previous forward pass used in the current loss must be .detach()’d. This prevents the “backward through the graph a second time” crash that causes near-total failure in naive RL implementations. Condition breadth-first ordering. One repetition per condition executes before any parameter sweep begins, so partial completions remain scientifically informative if execution is interrupted.

C

Sandbox Security Model

Each experiment executes inside a dedicated Docker container that is automatically removed after execution (--rm). The container runs as the host user’s UID:GID (not root). Resource limits: 8 GB memory, 2 GB shared memory, configurable wall-clock timeout (default 300–600 s). The Docker image pre-installs 80+ scientific Python packages including PyTorch (with torchvision, torchaudio, torchdiffeq), transformers, datasets, accelerate, gymnasium, scipy, sklearn, pandas, seaborn, networkx, timm, einops, albumentations, kornia. Frequently used datasets (CIFAR-10, CIFAR-100, FashionMNIST) are pre-cached as read-only mounts at /opt/datasets. Three-phase network model. The sandbox implements four network policies, with setup_only as the default. Phase 0 (Dependency Installation) and Phase 1 (Data Acquisition) enable network access; Phase 2 (Experiment Execution) disables network via iptables rules applied inside the container before the experiment script starts. Alternative policies are none, full, and pip_only. Code validation pipeline. (1) AST parsing for syntactic correctness; (2) AST security checks for forbidden function calls (os.system, os.popen, subprocess.run, shutil.rmtree) and banned builtins (eval, exec, compile, __import__); (3) module blacklist (subprocess, socket, http, urllib, requests, ftplib, smtplib, ctypes, signal); (4) import validation against an allowlist. Violations are classified as errors (block) or warnings (log). The evaluation harness (report_metric(), finalize()) is injected as a read-only Python module.

D

ARC-Bench Details

D.1

Benchmark Architecture

Topic specification. ARC-Bench consists of 25 CPU-executable ML research topics (T01–T25). Topics T01–T10 are shared with the HITL ablation, enabling end-to-end scenario from idea to paper. Each topic is a YAML entry in config/topics.yaml with five fields: id, topic, domains, metric_key, and metric_direction. Topic selection criteria: (1) CPU-executable in under 10 minutes on a single core using standard numpy/scipy/sklearn primitives; (2) involves a genuine scientific comparison with at least two distinct algorithmic approaches; (3) produces structured quantitative output that an LLM judge can evaluate without code execution. Topic list. Rubric structure. Each topic ships with a rubrics/T*.json defining a hierarchical tree of 8–11 leaf criteria across three categories: • Code Development (CD), weight 25: correctness and completeness of algorithm implementation. • Code Execution (CE), weight 25: whether the code ran successfully and produced machine-readable metric artefacts with multi-seed dispersion reporting.

17

Table 9 ARC-Bench topic list (T01–T25). ID

Topic

Algorithms / Methods

Primary Metric

Tabular ML T01 Dropout regularisation T02 Ensemble methods T04 Feature scaling on KNN T08 Class imbalance handling T09 RandomForest tuning T10 Cross-validation T14 Sparse linear models T15 Feature selection T18 Transfer learning T19 Semi-supervised learning T20 Active learning T22 Multi-label classification

MC-Dropout, standard Dropout, no Dropout Bagging, Boosting, Stacking StandardScaler, MinMax, Robust, None SMOTE, class weights, threshold tuning Grid search, random search, Bayes K-fold, stratified, leave-one-out Lasso, ElasticNet SelectKBest, RFE, noise injection Fine-tune, feature extract, from scratch Label propagation, self-training (10% labels) Uncertainty, margin, random sampling BR, CC, label powerset

ECE / Accuracy Accuracy Accuracy F1 (macro) CV score Accuracy MSE Accuracy Accuracy Accuracy Accuracy F1 (micro)

Optimisation & Search T03 Gradient-free optimisation T06 Adaptive LR schedules T13 GP kernel choice T21 Causal discovery

Nelder-Mead, Powell, CMA-ES StepLR, CosineAnnealing, ReduceOnPlateau RBF, Matérn, periodic (1-D / 5-D) PC, GES, NOTEARS

Regret Loss NLPD SHD

Dimensionality Reduction & Clustering T05 Dimensionality reduction PCA, t-SNE, UMAP T12 Clustering algorithms K-means, DBSCAN, GMM on synthetic shapes

Silhouette ARI

Text & Topic Models T07 Text feature extraction T17 Topic modelling

TF-IDF, Hashing, Count vectoriser LDA, NMF, LSA

Accuracy Coherence

Specialised Tasks T11 Anomaly detection T16 Time-series forecasting T23 Learning-to-rank T24 GP regression T25 Reservoir computing

IsolationForest, LOF, OCSVM ARIMA, exponential smoothing, MLP RankSVM, LambdaMART, listwise RBF, Matérn, ARD kernels ESN, MLP, GP on Lorenz-63

ROC-AUC RMSE NDCG@10 RMSE NRMSE

18

• Result Analysis (RA), weight 50: quality of scientific writeup—per-hypothesis verdicts supported by reported numbers, appropriate caveats, no fabricated values. Each leaf carries a weight in [0, 100] summing to 100 within its category and a score in [0, 1]. Aggregate scores are: P overall_strict = ℓ wℓ · sℓ / 100 (2) P results_only = ℓ∈CE∪RA wℓ · sℓ / 75 (3) Under results-only mode (default for framework comparison), CD leaves are excluded from aggregation. The complete rubric for T01 is shown in Box D.2. Judge system. Each (framework × topic) cell is graded against five artefact sources: (1) agent-produced code; (2) execution artefacts (CSVs, JSON metric files, stdout logs); (3) the agent-written README and claims file; (4) the topic rubric; (5) the topic manifest defining the research question, conditions, metrics, datasets, and hypotheses. The judge outputs per-leaf scores, rationale strings, and the two aggregates in Equations 2–3.

D.2

Strict Judge Protocol

The strict judge is designed to produce the same scoring distribution under three independent reviewer modes: a Claude Code subagent (Opus 4.7), a Codex CLI agent (GPT-5.4), and a human expert. Each reviewer operates from the same prompt and produces the same JSON output schema, enabling direct per-leaf cross-validation. Artefact sources and rubric structure follow Section D.1. Strict criteria. Four criteria apply uniformly to every leaf. (1) Implementation correctness: the algorithm is verified by reading code, not by label. Common failure patterns include MC-dropout that disables dropout at eval time, CMA-ES without covariance update, and NOTEARS implemented as plain Lasso. (2) Number grounding: every numerical claim in the writeup must trace to a captured artefact; fabricated numbers penalise the relevant leaf to 0.1–0.3. (3) Verdict-data consistency: a claimed supported hypothesis must be backed by measured evidence in the same direction; inverted verdicts score 0.1–0.3 regardless of prose quality. (4) Coverage: missing conditions, datasets, or seeds penalise execution leaves proportionally to manifest coverage. Timeout rule. If a run exceeded its wall-clock budget and the writing phase never executed, CE leaves are forced to 0.0 with no partial credit, CD retains rubric-correct credit, and RA leaves are capped at 0.1. Cross-validation. Two agent reviewers grade each cell independently. Per-leaf scores with |∆| > 0.20 are flagged and re-adjudicated; final scores are the mean of the two passes. Human expert scores follow the same protocol on a held-out subset. Mean per-leaf |∆| < 0.10 across the audited subset, with disagreements concentrated on partial-implementation cases where code reading is ambiguous. An example of the rubric is presented as follows: ARC-Bench Sample Rubric — T01: Dropout Regularization & Calibration Research question. Comparing dropout regularization strategies (standard, spatial, variational) for preventing overfitting in shallow MLPs on tabular classification benchmarks. Does dropout-variant choice affect calibration more than accuracy? Code Development (CD) — weight 25 t01-code-variants [10.0] Implement the main dropout variants—no-dropout baseline, standard element-wise dropout at one or more rates, and at least one stochastic-test-time / MC-style variant—as distinct code paths within a shared MLP architecture. t01-code-mlp [5.0] Shallow tabular MLP (small number of hidden layers, modest width) trained with a standard supervised classification objective on sklearn benchmarks. t01-code-datasets [5.0] Load multiple tabular sklearn datasets (e.g. breast_cancer, wine, digits) with a reasonable train/test split. t01-code-mc-pass [5.0] MC-dropout performs multiple stochastic forward passes at test time and averages

19

predicted probabilities; a single deterministic pass does not qualify. Code Execution (CE) — weight 25 t01-exec-metrics [16.67] Machine-readable artifact (results/metrics.json or stage-14/experiment_summary.json) with numeric accuracy and a calibration metric (ECE, NLL, or Brier score) covering all conditions on at least one dataset. t01-exec-seeds [8.33] Metrics aggregated over multiple random seeds per (condition, dataset) cell with dispersion reporting: std, stderr, CI, or min/max across seeds. A small-but-honest seed count with reported variance is preferable to a single deterministic run. Result Analysis (RA) — weight 50 t01-result-h1 [20.0] Compare calibration (ECE or analogous) between MC-style and standard dropout across evaluated datasets; convey whether MC calibration tends to be better (H1 supported / refuted / inconclusive). t01-result-h2 [10.0] Discuss whether accuracy differences among variants are small—i.e. calibration is the discriminative axis, not accuracy. Qualitative comparison grounded in reported numbers suffices; exact percentage-point thresholds are not required. t01-result-h3 [10.0] Compare no-dropout baseline vs. at least one dropout variant on calibration; convey whether baseline is clearly worse, comparable, or better. t01-result-writeup [10.0] README or writeup describes setup, presents key accuracy and calibration numbers, and conveys per-hypothesis outcomes (supported / refuted / inconclusive) with appropriate caveats on seed count, dataset scope, or calibration-metric choice. Judging note. Score on scientific substance and directional correctness of evidence, not on exact threshold satisfaction. Partial but well-motivated evidence deserves partial credit; rigidP wording or name mismatches should not penalise a substantively correct experiment. Aggregate: overall = ℓ wℓ sℓ /100; results_only = P w s /75. ℓ∈CE∪RA ℓ ℓ

E

HITL Ablation Details

Topic and mode design. 10 topics (T01–T10) span tabular ML, RL, MoE, NLP, physics-informed ML, and finance. The 7 modes vary the schedule of scripted expert interventions. The mapping between modes and stage-injection sets is shown in Table 10. Table 10 Mode-to-intervention mapping. CoPilot receives the most targeted interventions; Step-by-Step receives the most total interventions (mostly approve actions).

F

Mode

Receives Stages

Key Characteristic

Full-Auto Gate-Only CoPilot Thorough Step-by-Step Pre-Experiment Post-Experiment

none 5, 9, 20 5, 8, 9, 14, 17, 20 (+ smart pauses) Phase boundaries (8 stages) All 23 stages 5, 8, 9 14, 17, 20

No guidance injected Gate-only checkpoints Full intervention with auto-approve Broad but less targeted Every stage; mostly approve actions Early-pipeline only Late-pipeline only

Design-Space Exploration

Number of debate agents (K). We tested K ∈ {2, 3, 5} across 10 runs each. K=2 degenerates into a pro/con dynamic with −23% hypothesis diversity. K=5 raises tokens by +67% for only +8% diversity over K=3, since the additional agents largely echo the core three. K=3 is the diversity-per-token sweet spot for ML topics; specialised K=5 may help in domains requiring a dedicated domain expert (e.g. the HEP-ph bank already uses three distinct discipline perspectives). Evolution half-life (T1/2 ). We tested T1/2 ∈ {7, 15, 30, 60, ∞} days. T1/2 =7 expires useful lessons too fast; T1/2 =∞ accumulates contradictory advice past 15 runs. T1/2 =30 gave the best quality trajectory, persisting long enough to influence 3–5 subsequent runs while gradually fading. 20

Table 11 Case study on T10 (cross-validation strategies, small-sample model selection). Both runs complete a manuscript, but their evidence quality differs sharply. CoPilot reaches a score of 8.0 by targeting human input at the experimental bottleneck; Full-Auto scores 4.0 despite producing a paper.

G

Full-Auto (score: 4.0)

CoPilot (score: 8.0)

Debate

Hypotheses are generated and accepted without checking whether the planned CV strategies will produce distinguishable outcomes under the available compute budget.

Pragmatist flags that LOOCV may exceed time budget; Contrarian questions whether the ablation design can detect meaningful differences. Synthesizer incorporates both objections into a narrowed hypothesis set.

HITL

None.

Intervention at literature, hypothesis, design, writing, and quality stages. Human guidance explicitly asks the system to verify that CV strategies produce nonzero contrasts, handle LOOCV within budget, and avoid claiming metrics absent from the execution logs.

Execution

Plans eight CV strategies but every strategy reports identical zero estimation bias and zero variance. The ablation checker flags pairwise-identical conditions and a zero-variance warning; the run continues regardless.

Logs nine selection pipelines with nonzero contrasts across strategies. LOOCV, repeated stratified k-fold, and nested CV each produce distinguishable bias estimates. Ablation warnings are recorded as limitations rather than suppressed.

Verification

Artifact contains 51 metric keys with no empirical contrast between conditions. Registry is populated but scientifically uninformative. Claims pass the numeric gate but cannot support the paper’s research question.

Artifact contains 57 metric keys with differentiated values across nine conditions. Pre-built tables are injected from the verified registry. All numerical claims in strict sections trace to logged measurements.

Output

Paper is not fabricated in the strict sense but reports an all-zero result that cannot answer which CV strategy is preferable. Stage-20 score: 4.0.

Paper presents a calibrated exploratory result: comparisons are bounded, limitations are stated explicitly, and the Stage-20 gate accepts the manuscript because claims align with the logs. Stage-20 score: 8.0.

Case study Details

Here we present the detailed comparison between Full-Auto and CoPilot mode of AutoResearchClaw on Topic 10 in ARC-Bench, manifested in Table 11.

H

Failure Analysis

11 of 13 invalid canonical HITL runs fail at stage 17 (paper_draft). Stage 17 is the first hard anti-fabrication checkpoint and refuses to draft a paper when no usable metric exists upstream. The four recurring stage-17 failure subtypes are: (a) No real metrics: stages 10–14 do not produce a usable metric table; (b) Environment / dependency breakage: missing imblearn, LightGBM, etc.; (c) Dataset / resource failure: planned benchmark (e.g. FashionMNIST) cannot be loaded in sandbox; (d) Design / aggregation pathology: planned design too ambitious for the budget, invalid CV configuration (k must be in [2, 34]), only a tiny fraction of conditions complete. The stage-17 hard block is correct as a safety check but currently conflates heterogeneous causes; we propose graceful degradation that surfaces upstream cause in the draft header and limitations.

21

I

Writing-Quality Audit

We audited 20 canonical full-auto + step-by-step deliverables across T01–T10 for export defects: • abstract appears before \maketitle (20/20): consistent template misalignment. • Markdown-style \section{![Figure ...]} (17/20): image captions promoted to section headings. • Duplicated figure file (16/20): figure inserted twice. • “Learned Skills” / a-evolve content leaks into body (9/20). • Bracket-style pseudo-citations like [ray2021various; nanga2021review] (2/20). • Citation voids („) where keys were stripped without prose repair (small count). Local single-pass pdflatex compile rate across the 10 audited deliverables: 4/5 step-by-step pass, 3/5 full-auto pass. Note that local single-pass pdflatex differs from Overleaf’s latexmk-driven multi-pass with reference resolution; an Overleaf compile is not equivalent to a clean source. We treat compile-pass as necessary, not sufficient, for submission readiness. Citation count. Across the 5 audited topics, full-auto totals 94 citations and step-by-step totals 59. Per-paper minima can fall below NeurIPS norms (e.g. T03 step-by-step 2 citations, T05 full-auto 4, T09 step-by-step 7, T01 step-by-step 13). HITL improves citation discipline more reliably than breadth; we discuss mitigations (literature-retrieval rate-limit handling, related-work depth target enforcement) in the next section.

J

Ethical Considerations and Broader Impact

Autonomous research systems raise important questions about scientific integrity, academic labor, credit attribution, and the boundary between human and machine contributions to knowledge. We address these issues both through system design and through a conservative framing of AutoResearchClaw as a research amplifier rather than a replacement for human scientific judgment. Positive broader impact. AutoResearchClaw can accelerate early-stage scientific exploration by automating routine parts of the research cycle: literature scoping, experiment implementation, repair, result aggregation, drafting, and verification. This can help researchers test more hypotheses under limited time and compute budgets, expose negative or failed directions earlier, and preserve intermediate lessons that would otherwise be lost across attempts. The system may be especially useful for rapid prototyping, educational research workflows, benchmark construction, and preliminary feasibility studies. By combining execution logs, verified result registries, and explicit claim grounding, AutoResearchClaw can also encourage more transparent and reproducible research artifacts. Scientific integrity. The main risk of autonomous paper generation is that fabricated results or hallucinated citations could enter the scientific record. We treat this as a first-order design constraint. The verified result registry blocks ungrounded numerical claims in strict sections such as the Abstract, Experiments, and Results, while the citation verification pipeline removes references that cannot be resolved or validated before export. These safeguards do not guarantee that a scientific conclusion is correct, nor do they guarantee submission-ready formatting, but they reduce the risk that unsupported numbers or non-existent references are presented as evidence. Our ablation confirms the need for these safeguards: removing verification raises apparent acceptance, but manual inspection shows that several accepted papers then contain values absent from any measurement record. Impact on researchers and academic norms. We position AutoResearchClaw as a tool for accelerating exploration and preliminary investigation, not as an autonomous substitute for expert researchers. Our HITL ablation supports this framing: the best outputs come from targeted human input at critical decision points, not from full automation alone. We therefore recommend that such systems be used to augment researchers by handling routine execution and verification, while humans remain responsible for problem selection, interpretation, final claims, and submission decisions. We also encourage explicit disclosure when autonomous research tools are used, and we discourage their use for generating bulk low-quality submissions. Risks and safeguards. A system that lowers the cost of paper generation could contribute to submission flooding, superficial novelty claims, or over-reliance on automated judgments. AutoResearchClaw mitigates

22

these risks through sandboxed execution, network isolation, read-only evaluation harnesses, numeric claim verification, citation checks, and HITL gates. All generated code executes in isolated Docker containers with security checks. In our current implementation, each run costs approximately $3–15 in LLM usage, which makes large-scale misuse nonzero but still resource-constrained. The HITL experiments in this paper use scripted interventions rather than live human participants; future studies with live researchers would require appropriate IRB review.

23

Related documents

• AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration#214319
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery#144352
• Agentic Economies for Autonomous Scientific Discovery#1106261
• Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills#1011135
• A Persistent Fleet of AI Scientists Exhibits Cooperative and Autopoietic Behavior#1006961