Workflow Closure Is Not Scientific Closure in Auto-Research Systems
arXiv:2605.26200v1 [cs.SE] 25 May 2026
Shuai Wang1 , Xinyuan Tian1 , Pangpang Liu1 , Yize Zhao1 1 School of Public Health, Yale University, USA {shuai.wang.sw2572, yize.zhao}@yale.edu
Abstract This paper argues that workflow closure is not scientific closure in auto-research systems. Current systems can increasingly complete research-like loops internally, moving from idea generation to experiment execution, writing, and self-evaluation. That achievement is real, but it does not by itself give the resulting outputs scientific standing. We argue that trustworthy auto-research should not aim for autonomous self-sufficiency, but should aim for autonomous execution under nonautonomous epistemic control. Based on a survey of more than 100 recent papers and repositories in this rapidly emerging area, together with a structured audit of 21 representative systems, we diagnose a recurring and structurally connected failure pattern: objective collapse, in which single-proxy targets replace multi-objective scientific aims; validation collapse, in which internal self-evaluation replaces independent validation; and acceptance collapse, in which benchmark scores or publication-shaped artifacts replace mechanisms for domain-level critique, reuse, and integration. These collapses are not inherent limits of autonomy but correctable design choices. Accordingly, we outline potential remedies across objective signal, validation, and output pathway to spark community discussion. Keywords: Auto-research, LLM agents, AI scientist
1
Introduction
Auto-research has moved quickly from proposal to working systems. Karpathy’s autoresearch release [55] gave the community a spare, reusable template: propose a modification, run an experiment, inspect the outcome, and iterate inside a self-contained loop. Towards End-to-End Automation of AI Research [75], published in Nature, showed a more ambitious publication-shaped pipeline whose outputs passed workshop-level peer review at a major AI conference. Around these examples, auto-research has become a recognizable category of systems, repositories, and benchmarks [27, 2, 9, 45, 94, 117, 35, 29]. The important question is no longer whether such systems can automate parts of a research workflow. It is what kind of closure they achieve when they do. This work argues that workflow closure is not scientific closure in auto-research systems. Autoresearch should not be evaluated as scientifically trustworthy merely because it closes an end-to-end research workflow; it should be designed so that objective plurality, independent validation, and domain-level critique enter the loop as architectural requirements. The point is not that recent progress is illusory, or that any particular system must have overclaimed. The point is that the field now has artifacts that make workflow closure look uncomfortably close to scientific closure. The AI Scientist, for example, is described as moving from conception to publication and as having produced manuscripts that passed first-round workshop peer review [75]; repository-centered systems such as autoresearch make internally repeated propose–execute– evaluate loops easy to instantiate [55]. These are important achievements, but they also make the central inference newly tempting: if a system can generate ideas, run experiments, write a paper,
and evaluate its own output, then perhaps the research loop has been closed in the scientifically relevant sense. This inference is the target of this paper. A system achieves workflow closure when it can execute an end-to-end research pipeline, from ideation through experimentation to writing and evaluation, without human intervention within a single cycle. In such a system, the loop closes on itself: outputs feed back into subsequent decisions through the system’s own evaluators. By contrast, scientific closure requires that research outputs remain answerable to evaluators, constraints, and uptake processes outside the system that produced them. Here, the loop must close through the world: through competing objectives rather than a single proxy, through independent validation rather than self-confirmation, and through mechanisms by which relevant communities can evaluate, contest, reuse, and integrate findings. Workflow closure is therefore a real and important engineering achievement, but it does not by itself give the resulting outputs scientific standing. Analyzing the recent wave of LLM-workflow auto-research systems, we introduce the concept threelevel collapse for the recurring pattern that arises when these systems are designed around closurefor-autonomy: making the loop close internally, cheaply, and repeatedly. At the first level, objective collapse, objective signal design reduces the plurality of scientific aims to a single optimizable proxy, making the loop ratchetable on a measure that no longer reliably tracks the property it was meant to indicate. At the second level, validation collapse, validator design keeps validators inside the producer’s inductive boundary, such as self-review, same-family models, or benchmark-mediated signals, so that agreement among them cannot be assumed to constitute independence. At the third level, acceptance collapse, output pathway design terminates in scores or publication-shaped artifacts without standing pathways for the domain-level critique, reuse, and integration through which findings become usable knowledge. These are not three independent problems but three faces of the same design choice. The result is a drift toward local optimization: systems become increasingly good at improving what the loop can measure, validate, and present, while remaining insufficiently exposed to the external conditions required for scientific closure. However, these collapses are not inherent limits of autonomy but correctable design choices, and we propose potential remedies for each. Against this background, this position paper contributes: (i) a distinction between workflow closure and scientific closure as architectural properties of the auto-research loop; (ii) a three-level collapse framework that diagnoses where contemporary systems fall short, grounded in a survey of more than 100 papers and repositoriesA and a structured audit of 21 representative systems B; (iii) design principles for auto-research systems that achieve autonomous execution under non-autonomous epistemic control.
2
Workflow Closure vs. Scientific Closure
Workflow closure describes how a pipeline executes. A system achieves workflow closure when it can execute an end-to-end research pipeline: covering ideation, experimentation, writing, and evaluation, without human intervention inside a single cycle, with evaluation signals sourced from within the system rather than from external validators. The key property is that the loop closes on itself: outputs generated at one stage feed back into subsequent system decisions through the system’s own evaluators, scoring rules, or internal review mechanisms. Scientific closure describes how its outputs answer to the epistemic operations required for knowledge production. Three conditions distinguish it from workflow closure: First, plural-objectives, where the output is evaluated against multiple non-reducible objectives rather than collapsed into a single signal. Second, independent-validation, where the output is checked by validators that are not functionally identical to the producer. Third, domain-uptake, where the research community can contest, reuse, and integrate the output into ongoing practice. Here the loop closes through the world: through constraints, evaluators, and pathways outside the system that produced it. Achieving workflow closure does not imply scientific closure. Treating one as evidence of the other is a category error. Under closure-for-autonomy, each of the three conditions identified above is replaced by an internal substitute, producing what we call the three-level collapse: Objective Collapse into a single signal, Validation Collapse into self-checking, and Acceptance Collapse into terminal artifacts. Section 3 traces how each collapse arises, and how the three connect. 2
Figure 1: Workflow Closure vs. Scientific Closure in Auto-Research Systems
3
The Three-Level Collapse
Once systems are optimized for closure-for-autonomy, the three conditions for scientific closure introduced in Section 2 are replaced by internal substitutes. Objective plurality is reduced to a single internal signal; independent validation is replaced by evaluation within the loop’s own evaluative boundary; and domain-level uptake is replaced by termination in a score, report, or publication-shaped artifact. We call this pattern the three-level collapse: Objective Collapse, Validation Collapse, and Acceptance Collapse. These failures occur at three linked decision points in the loop: when the loop selects, when it validates, and when it terminates. 3.1
L1: Objective Collapse
L1—objective collapse occurs when progress under competing, non-reducible scientific objectives is reduced to a single internal signal. The relevant failure is not that a metric is used somewhere in the loop, but that retention, rejection, or ranking is governed by that metric alone. What Section 2 defined as objective plurality is thereby replaced by improvement under one comparable criterion. This collapse follows directly from closure-for-autonomy. A loop designed to run internally, repeatedly, and at low coordination cost favors objective signals that are easy to compute, compare, and feed back into the next cycle. Plural objectives resist this pressure because they introduce trade-offs, incomparabilities, and occasions for external judgment. A single signal, by contrast, makes the loop ratchetable: modifications can be retained when the score improves and reverted when it does not. This design has clear engineering value, but it weakens the requirement that outputs remain robust under the objective plurality that scientific closure demands. Objective collapse is a Goodhart-style failure [76, 56]: once a measure becomes the target of optimization, improvement on that measure no longer reliably tracks improvement in the broader property it was meant to indicate. Self-improving loops intensify the issue that the objective signal not only evaluates outputs but shapes the production of future ones, creating a stable pathway for over-optimization against the proxy. This is closely related to evidence that feedback loops with language models can induce in-context reward hacking [81]. In practice, L1 appears as monotonic improvement under the loop’s internal score while the relationship between that score and the broader research objective becomes increasingly opaque. 3.2
L2: Validation Collapse
L2—validation collapse occurs when independent validation is replaced by internal evaluation, such as self-review, same-family evaluators, sibling-model checking, or benchmark-mediated signals that remain within the loop’s own evaluative horizon. What Section 2 defined as independent validation is thereby replaced by success under internal validators. The relevant failure is not that internal evaluation occurs somewhere in the loop, but that validation is governed by internal evaluators alone. 3
Validation collapse is also encouraged by closure-for-autonomy. Independent validation is slow, costly, and difficult to standardize; it requires coordination with actors, distributions, or protocols that the loop does not itself control. An autonomous loop therefore has a strong incentive to internalize validation: a second model can score the first, an ensemble can simulate review, or a benchmark can serve as a standing gatekeeper. These choices preserve iteration speed and make the loop self-contained by removing the need for external validators. The technical issue is not simply bias but correlated error induced by shared inductive structure. When producer and validator are trained, prompted, parameterized, or selected from closely related evaluative regimes, agreement between them cannot be assumed to constitute independent validation. The operational distinction that matters is whether an external signal actually enters the loop, rather than appearing after the fact as annotation, which is a distinction we return to in Section 5. In practice, L2 appears as outputs that pass increasingly sophisticated review processes while remaining untested by validators outside the loop’s own evaluative family. 3.3
L3: Acceptance Collapse
L3—acceptance collapse occurs when domain-level evaluation and integration are replaced by terminal artifacts, such as a score, a report, or a publication-shaped object that the loop produces without any mechanism for the research community to contest, reuse, or integrate it. What Section 2 defined as outputs subject to community evaluation thereby becomes outputs treated as if community evaluation had already occurred. Acceptance collapse is likewise encouraged by closure-for-autonomy. Domain-level evaluation and integration are costly because they depend on research communities, review processes, and fieldspecific evaluation standards that the system does not control. A loop designed to close internally has an incentive to stop earlier, at the point where the artifact can be scored, displayed, or packaged as complete, without being exposed to domain-level critique. Acceptance collapse turns on a distinction between benchmark adequacy and domain standing. Benchmarks measure artifact quality under pre-specified tasks, but they do not instantiate the distributed process by which outputs become usable knowledge for a field. This process involves critique, reuse, and incorporation into ongoing practice, all of which are central to whether AI tools strengthen or weaken scientific understanding [77]. Acceptance collapse is therefore not an objection to benchmarks themselves, but to the use of benchmark success as a substitute for domain-level evaluation. In practice, L3 appears as a growing set of outputs that appear complete because they are score-bearing, legible, and artifact-rich, while lacking systematic route into external critique or reuse. 3.4
Why the Collapses Are Linked: Closure-for-Autonomy
The three collapses are dynamically linked, forming a feedback loop stabilized by closure-forautonomy. L1 enables L2 because a scalar objective can be checked internally, whereas plural objectives create pressure for external judgment. L2 enables L3 because, once internal validation is treated as sufficient, there is little architectural pressure to expose outputs to domain-level evaluation. L3 reinforces L1 because, once the endpoint is a score or artifact, the system is encouraged to optimize for the narrow criteria that make such endpoints look complete. The loop thus closes around what it can measure, validate, and terminate on its own. This is why incremental fixes do not break the pattern. Adding a second metric is insufficient if validation stays internal. Adding an external checker is insufficient if its output does not gate the loop. Adding a stronger benchmark is insufficient if outputs still have no pathway into external critique or reuse. Each fix targets one collapse while leaving the other two intact, so the loop continues to close on itself. The three-level collapse is therefore the structural cost of closure-for-autonomy.
4
Collapse Patterns in Contemporary Auto-Research
Section 3 introduced the three-level collapse as a diagnostic framework. This section does not attempt to turn the position paper into a comprehensive benchmark study; rather, it uses a survey-backed audit to examine whether the proposed failure mode is visible in selected contemporary systems. In Appendix A, we systematically survey more than 100 recent papers and open-source repositories 4
Table 1: Collapse audit of 21 auto-research systems in the full-coding audit pool. S = strong collapse; W = weak collapse (remediation attempted but not architecturally sufficient). System
L1 L2 L3 Notes
Karpathy autoresearch [55] uditgoenka/autoresearch [33] goal-md [14] autoresearch-anything [37] ADAS [30] AIDE [35] GEPA [7] HGM [19] Bilevel Autoresearch [89] ASI-Evolve [117] Omni-SimpleMem [71] Claudini [82] CORAL [87] SakanaAI/AI-Scientist [27, 75] AI-Scientist-v2 [28] AutoResearchClaw [2] Agent Laboratory [29, 92] CycleResearcher [109] AI-Researcher [9] ARIS [34] ClawTeam [10]
S S S S S S W S S W S S S S S S S S W S W
S S S S S S S S S S S W W W S W S S W W S
S S S S S S S S S S S S S W S S S S W S S
Canonical single-scalar loop Directly follows autoresearch loop Agent-constructed specialized metric Task-specific metric Meta-optimization on agent architecture Benchmark-driven and tree search Reflection adds minor L1 nuance Self-improvement on SWE-bench Loop-on-loop meta-optimization Three parallel objectives, scalar-aggregated Autoresearch on agent memory Held-out adversarial checks Evaluator separation, shared memory Real workshop review Extension of AI-Scientist pipeline Narrow external check (citation verification) Three-phase pipeline; AgentRxiv-based sharing Paired policy/reward LLMs, focused on writing Production development with external contact Cross-model review, skills-based Swarm, parallel GPU directions
on LLM-workflow auto-research systems. From this corpus, we identify a full-coding audit pool of 21 systems with sufficient loop structure and public information for L1/L2/L3 coding (Table 1). The scope is intentionally narrow: the audit includes LLM-workflow-based systems that implement an auto-research cycle, while excluding pre-LLM autonomous research systems, pure benchmarks, skill-only systems, and domain-specific deployment systems kept for contrast. Full methodology and per-system audit details are provided in Appendix B. Labels are assigned according to the role a design feature plays in the loop’s control structure. For L1, a system is coded strong when retention, ranking, or continuation is governed by a single scalar, and weak when it broadens objective evidence but still resolves progress through internal aggregation. For L2, a system is coded strong when validation comes from self-review, same-family models, benchmark feedback, or other evaluators inside the producer’s inductive boundary, and weak when it adds verifiers, cross-model review, citation checks, or adversarial probes that increase friction but do not gate the loop on independent validation. For L3, a system is coded strong when the terminal endpoint is a score, report, or publication-shaped output without a standing pathway by which the research community can evaluate, contest, or reuse it, and weak when external contact exists but is post-hoc or does not feed back into the loop. The label mitigated is reserved for designs that fully resolve the corresponding collapse, which no LLM-workflow auto-research system in the surveyed corpus achieves. The audit’s most informative finding is the asymmetry across the three collapse dimensions. Among the audited systems, L1-strong is 81.0% (17/21), L2-strong is 71.4% (15/21), and L3-strong is 90.5% (19/21). L3 is the least remediated layer: only 9.5% (2/21) of systems fall into the weak category, suggesting that domain-level evaluation and integration remain the weakest part of the current paradigm. L2 is the most remediated layer: 28.6% (6/21) are L2-weak, indicating that current systems most often try to move beyond fully internal closure through validation-side checks. L1 sits between them: 19.0% (4/21) are L1-weak, typically through mechanisms that broaden the information considered before selection, such as reflective critique, multi-objective aggregation, or parallel search. The broader corpus also contains contrast cases that clarify the boundary of the diagnosis. Four systems exhibit at least one mitigated label because an external operation is architecturally required before the loop can treat an output as valid. In the semi-autonomous formalization of the VML equilibrium, proof generation is coupled to Lean 4 verification and in-loop mathematician supervision, mitigating L1 and L2 because objective validity is fixed by formal proof correctness and validation 5
is not reducible to internal self-review [52]. In Latent-Y, antibody design campaigns are integrated with wet-lab execution, so L2 is mitigated by a reality-grounded validator rather than an internally accessible judge [100]. Robin extends this laboratory-grounded pattern to autonomous therapeutic discovery: although built on multi-agent literature search and data analysis that resemble closure-forautonomy systems on the surface, its architectural core is the in-vitro experimental loop in which candidate molecules selected by the agent are confirmed by laboratory assays before the loop treats them as validated [46]. CRISPR-GPT provides a fourth route: expert sign-off is architecturally required at critical junctures, making external human validation part of the loop rather than a post-hoc annotation [88]. These cases sit outside the main LLM-workflow closure-for-autonomy paradigm, but that is precisely why they are informative. They support a bounded claim: the three-level collapse is not an intrinsic cost of autonomous research as such, but a consequence of a specific contemporary design philosophy. The audit supports two bounded conclusions. First, among the audited systems, the three-level collapse is informatively asymmetric: L3 remains the least remediated dimension, L2 is where partial remediation is most often attempted, and L1 sits between them. Second, contrast cases outside the internally closed closure-for-autonomy paradigm show that at least one collapse dimension can be mitigated when external operations become part of the loop’s validity conditions. The audit does not show that the pattern is universal across all autonomous research, nor that the observed co-occurrence proves causation, nor that the audit pool is exhaustive. The weak labels in Table 1 mark departures from strong collapse within the same paradigm; Section 5 examines these departures together with related remediation attempts in the broader corpus.
5
How Current Systems Attempt to Remediate the Collapse
Multiple recent systems attempt partial remediation of the three-level collapse. In the closurefor-autonomy framework of Section 3, these attempts share two recurring limits. First, they are not systematized: added checks are usually ad hoc additions rather than pre-defined discriminative protocols. Second, they remain post-hoc rather than in-loop: external signals rarely become correction signals inside the system’s primary research loop. At the level of objective design, several systems broaden the internal evidence available to the loop before a decision is made. ASI-Evolve distributes search across neural architecture design, data curation, and reinforcement-learning algorithm discovery, demonstrating that auto-research can coordinate multiple research fronts within one system [117]. GEPA introduces reflective prompt evolution, using natural-language analysis of failed mutations to shape subsequent ones, which turns mutation search into a more interpretable process than pure scalar ratcheting [7]. ClawTeam expands exploration through swarm-style parallel GPU directions, increasing breadth relative to serial loops [10]. goal-md extends the paradigm further by having the agent construct its own fitness function from a natural-language goal before optimization begins [14]. These remedies enrich what the loop considers but not how it decides: a scalar aggregation still governs retention, ranking, or selection. They remain non-systematized, with each system broadening objective evidence in its own way, and partial rather than structural: the added evidence may enter the loop, but it does not become a non-reducible objective constraint or an external validity condition. At the level of validation, the engineering moves are even more substantial. Marco DeepResearch makes verification a first-class pipeline component, exposing the verifier as inspectable infrastructure rather than burying judgment inside the generator [122]. MiroThinker-H1 integrates adversarial probing into the mid-training stage, so verification pressure is applied before final outputs are frozen [101]. ARIS introduces cross-model review loops, increasing validator diversity beyond singlemodel self-review [34]. AutoResearchClaw adds citation verification and multi-batch coordination, bringing a narrow but genuine external check into the loop [2]. These attempts are non-systematized because each system defines validation differently and none establishes a general protocol for distinguishing substantive correctness from internally legible agreement. They also remain partial rather than structural: citation checks verify references rather than claims, cross-model review stays within the LLM family, and adversarial probes remain self-generated or system-internal. At the level of output pathways, remediation attempts are rarer and more externally meaningful. AI-Researcher provides a rare data point on production-facing deployment through novix.science, moving beyond benchmark-only evaluation [9]. SakanaAI/AI-Scientist submitted generated papers 6
to real workshop peer review, testing publication-shaped outputs against human expert judgment rather than only internal evaluators [27, 75]. Their common architectural limit is that the engagement remains non-systematized and post-hoc rather than in-loop. Production use is evidence of contact with the world, not yet a documented mechanism by which user feedback drives the loop’s evolution; a workshop submission is an event, not a standing pathway for contestation, reuse, and integration. The architectural lesson is that partial remediation on one dimension is readily reabsorbed by the closure structure of the other two. Breaking the collapse requires coordinated remediation across all three design choices: objective signal, validator, and output pathway. Among the audited systems, no system attempts that coordination, and Section 6 turns to its constructive implications.
6
Toward Scientific Closure: Design Implications for Auto-Research Systems
The diagnosis in Sections 3–5 has direct design implications. The relevant question is not which single architecture would solve the problem, but which architectural commitments would move auto-research from internal loop completion toward scientific closure. The position is not that autoresearch should be slowed or made to resemble traditional human research. It is that auto-research should preserve autonomous execution while giving up epistemic self-sufficiency. The design target therefore shifts from closure on itself toward closure through the world. Against objective collapse, objective signal design should treat plural objectives as architectural primitives rather than as inputs to be aggregated away at the moment of decision. Where appropriate, evaluation should operate over a Pareto frontier of non-reducible objectives, so that trade-offs remain visible to the loop and retention decisions preserve them rather than prematurely reduce them to a single notion of progress. Concretely, auto-research systems should maintain an objective ledger: a structured record of the aims a candidate output is supposed to satisfy, the evidence for each, the trade-offs among them, and the decision rule used when the loop proceeds despite unresolved conflict. For a machine-learning method paper, such a ledger might separate predictive performance, calibration, robustness, computational cost, interpretability, and reproducibility. For a biomedical discovery paper, it might separate mechanistic plausibility, assay feasibility, safety constraints, and clinical relevance. The point is not that every domain shares the same objective set, but that the loop represents objective plurality explicitly enough that competing criteria remain part of the control structure. Against validation collapse, validator design should make external validation both architecturally required and in-loop: a validator’s signal should enter the research loop as a condition of validity, not annotate outputs after the loop has already decided what counts as progress. Concretely, systems should maintain a validator provenance record: who or what validated each claim, the validator’s independence relation to the producer, the evidence inspected, and whether the result changed subsequent search. A same-family LLM reviewer may be useful for debugging, but it should not be counted as independent validation unless its signal is tied to an external constraint. Depending on the domain, the relevant validator could be a formal proof checker, a pre-locked holdout benchmark, a wet-lab assay, an external replication cohort, a human expert panel, or a protocol that forces claims through independently maintained tools and data. What matters is not the validator type, but that the validator lies outside the producer’s inductive boundary and that its signal materially alters whether outputs are retained, revised, or rejected. Against acceptance collapse, output pathway design should create architectural pathways through which outputs can enter domain-level evaluation, critique, reuse, and integration. The requirement is weaker than formal peer review or institutional approval: what matters is not whether a specific bureaucratic channel is traversed, but whether outputs become contestable and reusable outside the producing loop. Concretely, systems should emit a claim package rather than only a paper-shaped artifact: a set of atomic claims, their evidence traces, code and data provenance, failure cases, known limitations and sensitivities, and a route by which external users can reproduce, contest, or extend the claims. For domains with high downstream stakes, that package should also include stopping rules: conditions under which the system must defer, request external review, or decline to treat an output as ready for domain uptake. The design question is not how to assign a stronger terminal label to an artifact, but how to ensure that an artifact’s standing can be tested outside the loop. In this sense, acceptance is a process, not a score. 7
These three directions imply a single meta-principle: autonomous execution under non-autonomous epistemic control. Auto-research should move from loops that close on themselves to loops that close through external objectives, external validators, and external evaluation pathways. Autonomy as execution capability would be preserved; autonomy as epistemic self-sufficiency would be abandoned. Trustworthy auto-research then becomes a tractable architectural problem rather than an open-ended ideal: for each design choice, the question is whether the loop remains self-contained, or whether it closes through the world.
7
Related Work and Positioning
This section positions the paper against the two bodies of work most directly tied to the main argument: LLM-workflow auto-research systems, which establish the target phenomenon, and concurrent diagnostic positions, which diagnose adjacent failures at the level of behavior or epistemics rather than loop architecture. A more comprehensive survey of recent LLM-workflow auto-research is provided in Appendix A. LLM-workflow auto-research systems establish the target phenomenon of this paper. Recent surveys describe a coherent technical agenda in which proposal, execution, evaluation, and revision are composed into internally iterating research loops [108], with the broader categorical landscape situating autonomous scientific discovery alongside hybrid human-AI co-creation as identified phases of FM-driven scientific research [69]. A complementary lifecycle-stage view organizes the same field by epistemological phase rather than by loop architecture, mapping AI assistance across creation, writing, validation, and dissemination [58]. Concrete instantiations range from minimal repository templates [55, 33] to multi-stage publication-shaped pipelines [75, 92] and meta-level designs that optimize the loop itself [89]. These systems differ in domain, ambition, and degree of automation, but they share a control structure in which candidate outputs are generated, scored, retained, revised, and packaged through signals available inside the loop. The argument of this paper is therefore not about the quality or limitations of any single system. It is about what happens when this shared loop structure becomes the basis for deciding progress, validity, and standing. Concurrent positions are complementary, but they operate at a different level. Some locate failure in agent reasoning behavior, including weak evidence consideration, limited refutation-driven revision, and lack of convergent multi-test reasoning [90]. Others show empirically that nominally broad architecture search can reduce to narrow tuning within a single design region [63], or that agentgenerated papers that score competitively under manuscript-only review fall short under artifactaware review, with experimental rigor as the dominant failure mode [120]. Editorial and analytical commentary in Nature further argues that AI systems can empower scientific work but should not replace human researchers [36], and that AI tools can produce illusions of understanding when they bypass the distributed processes through which scientific knowledge is critiqued, integrated, and revised [77]. Together these works ask whether individual runs reason well, whether nominal search is genuinely broad, whether artifact quality survives artifact-aware scrutiny, and whether artifact production substitutes for knowledge production. Our diagnosis asks a different and compatible question: how the surrounding control structure decides that progress, validity, and standing have been achieved.
8
Limitations
This paper’s scope is LLM-workflow auto-research, and the diagnosis may not transfer unchanged to formal-verification, robotic-experiment, or other autonomous-research paradigms. The audit covers selected systems with sufficient loop structure and public information for full L1/L2/L3 coding, rather than the full population, so the reported rates should not be read as prevalence estimates across all uses of LLMs in science. The three-level collapse is the most common and structurally connected failure pattern we identify in this category, not an exhaustive account of all possible failures.
9
Alternative Views and Scope Boundaries
This section addresses five objections most directly tied to the paper’s central claim. 8
A first objection is that workflow closure is only an engineering milestone, not a claim of scientific standing. We agree that many system builders are appropriately cautious, and our argument does not depend on attributing overclaiming to any individual work. The issue is structural: once systems are benchmarked, advertised, and compared by their ability to move from idea generation to paper-shaped output, workflow completion can become the most visible proxy for research success. This paper calls on the community to resist that proxy before it becomes naturalized. A second objection is that benchmarks and internal validators are necessary because external validation does not scale. This is true, but it does not answer the concern. Benchmarks are indispensable for development, debugging, and comparison; internal validators are indispensable for rapid iteration. The problem arises when these instruments are treated as substitutes for the external operations they can only approximate. The constructive agenda in Section 6 therefore does not ask every iteration to wait for full peer review or wet-lab validation. It asks that systems represent the difference between internal progress signals and external validity conditions, and that the latter enter the loop at decision points where claims acquire standing. A third objection is that diverse model ensembles, adversarial agents, or specialized verifier models may provide independence without human or domain-level validation. Such methods can reduce correlated error and should be part of the design space. But independence is not a property of multiplicity alone: models trained on overlapping data, selected by the same benchmark culture, or rewarded under the same scoring rules can still share the producer’s blind spots. The relevant question is whether the validator introduces an external constraint that the producing loop cannot freely simulate, optimize against, or reinterpret as success. Artificial validators may eventually satisfy that condition, but their independence must be demonstrated rather than assumed. A fourth objection is that science has always relied on proxies, publication artifacts, and imperfect peer review, so auto-research is not categorically different. We agree that the problem is not unique to autoresearch, and much human-authored machine-learning work would also fail a demanding L3 criterion. What is distinctive is the speed and closure of the feedback loop. Human scientific communities contain partially independent frictions across laboratories, methods, incentives, instruments, and communities; auto-research systems can compress ideation, execution, validation, and writing into a single internally optimized loop. That compression is the source of their power, and also why external constraint must be designed architecturally. A fifth objection is that canonical systems such as Coscientist, FunSearch, AlphaEvolve, Google AI Co-Scientist, and OpenAI Deep Research falsify the universality claim. This objection clarifies rather than refutes the paper’s scope. The claim is not that every LLM-assisted scientific system exhibits the three-level collapse, but that collapse is characteristic of systems whose research loop closes primarily through internally available objectives, validators, and terminal artifacts. Coscientist, FunSearch, and AlphaEvolve differ because external, executable, or production-relevant constraints are constitutive parts of the architecture: Coscientist incorporates cloud-lab execution and chemical experimentation, FunSearch filters LLM-generated programs through executable evaluators, and AlphaEvolve depends on executable algorithmic evaluation and, in some settings, production-relevant computational constraints [41, 91, 79]. The same architectural pattern is visible in two recent systems published on Nature that further sharpen the scope line, one on the executable-evaluation axis and one on the laboratory-grounded axis. ERA couples LLM-driven code generation with tree search over executable quality metrics on public leaderboards across single-cell analysis, COVID-19 hospitalization forecasting, and additional scientific software tasks, so that retention is gated by external execution rather than by internal evaluators alone [40]. Robin proceeds further along the laboratory-grounded axis: it integrates multi-agent literature search and data analysis with an in-vitro experimental loop, autonomously identifying ripasudil and KL001 as therapeutic candidates for dry age-related macular degeneration whose efficacy is then confirmed in the laboratory [46]. In this design, lab assays are not post-hoc validation of a completed paper but constitutive validity conditions on each design cycle, in the same architectural sense as Coscientist. Google AI CoScientist and OpenAI Deep Research differ in a second way: the former is primarily a scientist-guided hypothesis-generation system whose outputs require downstream experimental validation, with its Nature 2026 deployment reporting in-vitro confirmation of drug repurposing candidates for acute myeloid leukemia [47], while the latter is a source-grounded research-synthesis product rather than an autonomous discovery loop [80]. These systems therefore support the paper’s scope line rather than undermine it.
9
10
Conclusion
Auto-research is increasingly at risk of being treated as if workflow closure were sufficient evidence of scientific closure, yet the inference from one to the other is structurally unsound. In this position paper, we identify the core risk as a three-level collapse: objective collapse, validation collapse, and acceptance collapse. We further derive design implications for objective signal design, validator design, and output pathway design, arguing that these collapses are correctable design choices rather than inherent limits of autonomy. The path forward is to preserve autonomous execution while giving up epistemic self-sufficiency: auto-research systems should be designed not to close on themselves, but to close through the world.
10
APPENDIX A
A Survey of LLM-Workflow Auto-Research
A.1
Scope and Methodology of the Survey
This appendix surveys the landscape of LLM-workflow auto-research as of early 2026. By LLMworkflow auto-research, we mean systems in which language-model agents plan, generate, evaluate, revise, or package research artifacts through a multi-stage workflow, executing more of the research process than question answering or single-step generation typically covers. The corpus surveyed here corresponds to the inventory described in the main text’s audit methodology, more than one hundred works spanning archival papers and open-source repositories, with the time window concentrated in 2025–2026 but including foundational earlier work such as Coscientist [41] and FunSearch [91]. Several categories are intentionally excluded: pure literature-retrieval systems labeled as “research” that do not iterate, generic coding agents not framed for research automation, traditional AutoML and hyperparameter optimization systems that predate the LLM-workflow paradigm, and general LLM-agent frameworks without a documented research-automation instantiation. Adjacent systems with constitutive external operations, such as formal verification, wet-lab integration, executable evaluation, are retained as boundary cases in Appendix A.4.3. The remainder of this appendix is organized along five lines. Appendix A.2 covers the core autoresearch loop ecosystem; Appendix A.3 covers the substrate capabilities, especially skill and memory systems, that auto-research loops increasingly compose over; Appendix A.4 covers domain-specific applications and boundary systems; and Appendix A.5 discusses concurrent diagnostic positions on auto-research as a category. The survey provides a timely and objective overview of this rapidly evolving field, while the position paper highlights perspectives and recommendations that deserve urgent attention. The two parts support each other. A.2
The Auto-Research Loop Ecosystem
This paper organizes the auto-research ecosystem into four subcategories that together cover the main design archetypes currently being explored. Foundational templates establish the minimal loop structure and seed a broad ecosystem of forks and adaptations. Publication-shaped end-to-end pipelines extend the loop across the full research workflow from ideation to manuscript drafting. Meta-level and self-improving systems treat the search procedure or the agent itself as the object being improved. Multi-agent collaborative systems distribute the loop across coordinated agents. The categories are not mutually exclusive, but together they capture the dominant designs in the current literature. A.2.1
Foundational Templates and Direct Descendants
The contemporary auto-research ecosystem traces a clear lineage to Karpathy’s autoresearch repository [55], which distills the auto-research loop into a minimal four-step pattern: modify code, verify against a benchmark, keep or discard based on the result, and repeat. The design is deliberately spare. A single LLM agent operates over training code on a single GPU, with no orchestration layer, no multi-agent coordination, and no external validators. This minimalism is what made the template generative: the entire loop fits in a few hundred lines, the modification–verification cycle is easy to reason about, and the closure-for-autonomy design pattern is exposed in its simplest form, where outputs at each step feeding back into the loop through internally computed scores. The template’s influence is visible both in repositories that fork it directly and in research that uses it as a controlled testbed. A first cluster of descendants ports the template across LLM platforms while preserving its core control structure. uditgoenka/autoresearch[33] adapts the loop to Claude Code; supratikpm/geminiautoresearch[32] targets the Gemini CLI; leo-lilinxiao/codex-autoresearch [15] brings the same pattern to Codex; and davebcn87/pi-autoresearch [3] together with drivelineresearch/autoresearch-claudecode[4] extend the loop to additional agent runtimes. A second cluster generalizes the template beyond its original ML-training setting. jmilinovich/goal-md [14] replaces the fixed benchmark with an agent11
constructed fitness function specified in a natural-language goal file, and zkarimi22/autoresearchanything [37] parameterizes the loop over arbitrary task definitions. Cross-platform packaging efforts including Entrpi/autoresearch-everywhere [6], mutable-state-inc/autoresearch-at-home [21], eimenhmdt/autoresearcher [5], greyhaven-ai/autocontext [8], and james-s-tayler/lazy-developer [12] further illustrate how rapidly the template propagated as a reusable substrate. He et al. [49] use the same template as the entry point for a methodological position statement on how the human role shifts from experimenter to research director, a perspective we return to in A.5. The template has also served as a testbed for controlled studies of what auto-research loops actually do. Ferreira et al. [45] compare classical hyperparameter-optimization algorithms against LLMbased search inside the auto-research framework, fixing the compute budget and varying only the optimizer. Within a fixed search space, classical methods such as CMA-ES and TPE consistently outperform LLM-based agents; however, when the LLM agent is allowed to edit training source code in an unconstrained space, the gap narrows substantially even with a self-hosted open-weight model. The authors further find that reliability dominates exploration breadth: methods that avoid out-of-memory failures outperform those with higher search diversity. Their experiments suggest that the auto-research loop benefits less from scaling the agent than from coupling it with a structured external optimizer. A complementary diagnostic study by Li [63], using the same template at 10,000-experiment scale, examines whether the loop’s nominal architecture search collapses into hyperparameter tuning within a narrow design region; we discuss its implications in A.5. A.2.2
Publication-Shaped End-to-End Pipelines
A second branch of the auto-research ecosystem aims for substantially broader scope than the foundational templates: rather than iterating over a single training script, these systems attempt to cover the full research pipeline from ideation through experimentation, writing, and self-review, producing outputs in publication-shaped forms. Lu et al.’s AI Scientist [75] is the most visible exemplar. Published in Nature, the system composes literature-grounded idea generation, experimental code synthesis, automated execution, and LaTeX paper drafting into a single end-to-end loop, with an internal LLM reviewer scoring the final manuscript. Several generated manuscripts were submitted to a workshop at a major machine-learning venue and received first-round peer-review acceptance scores. The system has set the architectural template that much of the subsequent publication-shaped category adapts or extends. Several systems extend the paradigm with collaborative or production-facing elements. Schmidgall and Moor’s AgentLaboratory [29, 92] structures the pipeline into three phases (literature review, experimentation, writing) and introduces AgentRxiv, a shared repository through which agents can access papers produced by other agent instances, introducing a form of inter-agent knowledge transfer. HKUDS AI-Researcher [9] takes a different path: it pairs the auto-research loop with a production-facing deployment at novix.science, exposing the system to external users beyond benchmark evaluation. SakanaAI/AI-Scientist-v2 [28] continues the original AI Scientist line with refinements to the experimental and writing modules. Adjacent systems including AutoResearchClaw [2, 72], which adds citation verification and multi-batch coordination, BloClaw [86], a multimodal scientific workspace, and the Sibyl auto-research system [31] further illustrate the architectural diversity within the paradigm. A further group situates the publication-shaped paradigm in distinct methodological or evaluative regimes. CycleResearcher [109] trains paired policy and reward LLMs through SimPO using historical ICLR review data, producing a system in which the writing process is optimized against learned reviewer preferences. Earlier systems such as MLR-Copilot [1] and ResearchAgent [13] established proto-versions of the pipeline with stronger human-in-the-loop elements at ideation and revision. Lighter-weight implementations including NanoResearch [23] and OpenAGS [22] demonstrate the template’s portability to smaller computational budgets, while AutoSOTA [66] narrows the scope to automated discovery of state-of-the-art configurations within specified model families. Beyond the publication-shaped pipelines themselves, several literature-handling components are frequently composed into auto-research loops at the literature-retrieval and ideation stages. LitLLM [16] provides an open framework for literature-aware LLM research workflows; LatteReview [26] formulates agent-driven literature review as a structured retrieval-and-synthesis task; and AwesomeLit [113] extends this line toward hypothesis generation, with agent-supported literature research feeding into 12
the formulation of new research questions. These components do not themselves close an autoresearch loop, but they supply the literature-grounding substrate that publication-shaped pipelines depend on. A.2.3
Meta-Level and Self-Improving Systems
A third branch of the auto-research ecosystem moves up a level: rather than executing research on a fixed task, these systems treat the search procedure itself or the agent that performs the search as the object being improved. ASI-Evolve [117] distributes autonomous search across three parallel research frontiers, including neural architecture design, training-data curation, and reinforcement-learning algorithm discovery, which is demonstrating that an auto-research system can coordinate substantive research directions across multiple sub-fields within a single coordinated framework, with cross-front signals aggregated into a system-level performance measure. Bilevel Autoresearch [89] takes the meta-move in a different direction: it instantiates an outer loop whose object of optimization is the inner research loop itself, with the outer loop proposing modifications to the inner loop’s search procedure based on how rapidly the inner loop improves on its task. The two systems together mark out the design space for meta-level auto-research: ASI-Evolve expands the breadth of what the loop searches, while Bilevel Autoresearch lifts the level of how the search is conducted. A complementary line targets self-improvement at fixed evaluative endpoints. HGM [19] formulates self-improvement against SWE-bench, with the agent revising its own codebase, tooling, and prompt structure across iterations as SWE-bench performance changes. The system illustrates the general pattern: a benchmark provides the optimization signal, and the agent’s own configuration becomes the modification surface. Claudini [82] applies the same logic in a substantively different domain, using an autoresearch-style loop to discover novel adversarial-attack algorithms against LLMs. Distinct from most systems in this category, Claudini supplements the internal search loop with a held-out adversarial evaluation that the loop cannot optimize against directly, demonstrating how the selfimproving template can be combined with a partial external check while remaining structurally loop-driven. GRAFT-ATHENA [102] extends self-improvement to evolutionary numerical-algorithm discovery, in which agentic teams iteratively refine numerical solvers against fixed quantitative performance criteria. Adjacent designs explore other surfaces on which the loop’s own machinery can be modified. ADAS [30] treats the multi-agent system architecture itself as the search space, with one agent proposing variants of agentic structure that are evaluated on downstream benchmark tasks. AIRA [83] pursues an analogous move at the neural-architecture level, with compose-and-design agents searching over foundation-model architectures beyond standard Transformers. GEPA [7] introduces reflective prompt evolution, using natural-language analysis of failed mutations to guide subsequent prompt modifications and yielding a more interpretable mutation process than pure scalar ratcheting. CausalEvolve [42] extends open-ended discovery with a causal scratchpad that traces how earlier modifications shape later candidates, and Xia et al. [112] apply the self-improvement template to the discovery of new LLM-RL algorithms. Open-source projects including autoevolve [20], self-improving-agent [25], and self-improving-coding-agent [18] further illustrate the breadth of self-modification patterns being explored in this part of the ecosystem. A.2.4
Multi-Agent Collaborative Research
A fourth branch of the auto-research ecosystem distributes the research process across multiple coordinated agents rather than executing it through a single agent operating sequentially. CORAL [87] formulates multi-agent research as an evolutionary process for open-ended discovery, with distinct agents filling separated roles in candidate generation and evaluation, and with task improvements emerging from the interaction between roles rather than from a single agent’s iteration. The architecture is notable for placing evaluator separation as an explicit design commitment, even though the evaluators themselves remain within the same model family as the producers. ClawTeam [10] approaches the multi-agent setting from an orthogonal direction. Rather than separating roles, it runs parallel agent instances across multiple GPU directions in a swarm-style configuration, broadening exploration by sampling more diverse research paths simultaneously before aggregating results through ranked selection. ARIS [34] moves in yet another direction along several axes at once: it draws reviewer agents from different foundation model families (Claude, Codex, and GPT-style backbones), so that producer and evaluator no longer share training distribution and 13
prompting regime; it organizes research capabilities as composable skill modules invoked on demand rather than as a monolithic pipeline; and it adopts a lightweight, loosely coupled architecture in which the main loop, reviewers, and skill modules communicate through decoupled interfaces and run asynchronously with modest hardware requirements. Shen et al. [94] contribute the most direct empirical examination of these design choices, comparing several multi-agent collaboration patterns on automated research tasks. Their results suggest that the marginal benefit of adding agents depends substantially on whether the collaboration pattern actually changes the information considered at decision points, rather than only the volume of generated candidates. Several adjacent systems explore narrower multi-agent configurations. ML-Agent [17] targets machine-learning research workflows with a multi-agent decomposition of the pipeline. hyperspaceai/agi [11] proposes a distributed multi-agent architecture for general research automation, and ABSTRAL [97] formulates the multi-agent system’s topology itself as the object of iterative refinement. A.2.5
Evaluation for Auto-Research Loops
A complementary line of work develops evaluation infrastructure that probes auto-research loops along the dimensions where they are most likely to fail. FML-bench [123] holds the ML research task fixed and compares agent search strategies, isolating which strategy choices drive performance and supplying a controlled testbed for search-dynamics analysis. SciIntegrity-Bench [119] scores AI scientist systems on faithful citation, honest result reporting, and reliable claim attribution, targeting the integrity of publication-shaped outputs at termination. MedProbeBench [73] extends evaluation to a demanding domain regime by assessing whether agents can synthesize and reconcile heterogeneous evidence sources at the level of expert clinical guideline development, operationalizing a domain-level integration standard. Together these benchmarks probe complementary properties of the auto-research loop: search dynamics during execution, artifact integrity at termination, and conformance to domain-level integration standards. A.3
Substrates and Capabilities
The systems in Appendix A.2 compose research loops out of agent capabilities and persistent state. Two adjacent research programs supply much of the underlying machinery: skill systems study how agents acquire, organize, and invoke reusable procedures, and memory systems study how agents accumulate and retrieve information across episodes. These programs are not themselves auto-research systems, which means they do not by themselves close a research-execution loop. However, they increasingly furnish the substrate on which auto-research loops operate, and a survey of the landscape would be incomplete without them. A.3.1
Skill Systems
Skill systems address how agents represent, acquire, and deploy reusable procedural knowledge. The recent survey [116] organizes the field along three axes: architecture (how skills are represented and composed), acquisition (how skills are learned, discovered, or distilled from experience), and security (how skill invocation can be constrained). The framing makes clear that the field is no longer about isolated tool use but about treating procedural capability itself as a structured object that agents can manipulate, share, and improve. Several systems illustrate the directions surveyed in that framing. Memento-skills [121] pushes the agent-designs-agent pattern to the skill layer, with agents generating and refining the skills of other agents in a meta-design loop. Seagent [99] formulates a self-evolving computer-use agent that acquires skills directly from its own interaction trajectories rather than from a fixed skill library, illustrating how experience-driven skill formation can operate without an externally curated taxonomy. A first thread of skill-systems work concerns acquisition and evolution. EvoSkill [39] discovers skills through evolutionary search across multi-agent configurations. XSkill [54] supports continual learning of skills across multimodal agent tasks, and AutoSkill [118] together with Trace2skill [78] target lifelong skill self-evolution and trajectory-to-skill distillation respectively. CycleQD [59] approaches skill acquisition through quality-diversity optimization, and SkillRL [111] evolves agents via recursive skill-augmented reinforcement learning. 14
A second thread concerns how acquired skills are organized and made retrievable at ecosystem scale. SkillFlow [61] proposes scalable skill retrieval as a system component, SkillNet [67] treats skill creation, evaluation, and connection as a unified workflow, the AI-Research-Skills project [24] assembles a curated skill library targeted at research workflows, and Li et al. [62] formulate orchestration and benchmarking of agent skills at ecosystem scale. Complementary work probes the limits of skill-based architectures: SWE-Skills-Bench [48] evaluates whether reusable skills actually help in real-world software engineering, SkillsBench [65] benchmarks how well agent skills transfer across diverse task settings, Li [64] examines when single-agent-withskills configurations can replace multi-agent systems and when they fail, and TARSE [103] studies test-time skill retrieval as an alternative to fixed skill libraries in reasoning agents. A.3.2
Memory Systems
Memory systems address how agents accumulate and retrieve information across episodes, supplying continuity that single-shot agent calls cannot provide. MuSEAgent [106] formulates a multimodal reasoning agent whose state persists across interactions through stateful experiences, with explicit mechanisms for encoding, retrieving, and updating episodic information as the agent operates over multimodal inputs. Omni-SimpleMem [71] takes an unusual inverse approach: rather than designing memory directly, it uses an autoresearch-style loop to discover memory architectures for lifelong multimodal agents, treating memory design itself as the object of autonomous experimentation. EvolveMem [74] pursues the same direction in a self-evolving variant, in which the memory architecture is iteratively rewritten by the agent itself during operation rather than fixed at design time. The three systems together illustrate the range of design points being explored, from memory as a built-in architectural component to memory as an emergent product of automated search. Memory Intelligence Agent [85] contributes a further framing of memory as a first-class agent capability rather than as a service of the underlying model. Memory considerations also appear, implicitly or explicitly, throughout the skill-systems work discussed in Appendix A.3.1: Seagent [99] acquires skills from interaction trajectories that function as a form of episodic memory, and Trace2skill [78] distills trajectory-local lessons into transferable skills, blurring the line between memory accumulation and skill formation. The boundary between these two systems is becoming blurred, as both are emerging as important components of higher-level agent capabilities that support auto-research. A.4
Domain Applications and Boundary Systems
This section turns from system designs to the domains in which auto-research is being deployed. Rather than re-categorizing systems already discussed in previous Appendices, we focus here on systems whose primary contribution lies in adapting auto-research to a particular scientific domain, where domain-specific data, evaluation standards, and stakes shape the design. The final subsection then turns outward to systems that fall outside the LLM-workflow auto-research paradigm but that are frequently raised as adjacent reference points. A.4.1
Biomedical and Medical Applications
Biomedical and medical applications constitute the largest concentration of domain-adapted autoresearch work in the current literature, reflecting both the volume of high-quality structured data and the particularly demanding evaluation standards of clinical and biological research. Wu et al. [110] present a publication-shaped pipeline specialized for medical research, with the loop adapted to work over clinical questions and biomedical evidence: literature retrieval is grounded in PubMed-scale corpora, hypothesis generation is conditioned on clinical task formulations, and output drafting follows medical-research conventions. The system illustrates the tightening that occurs when general auto-research scaffolding is applied to a domain where evaluators are well-defined and stakes are explicit. MedOpenClaw [93] pushes the design in a different direction. Rather than operating over curated datasets, the system reasons over uncurated full medical imaging studies—the kind of data clinicians actually encounter—and supplies an auditable reasoning trace through which its decisions can be inspected. The two design commitments together make MedOpenClaw a notable counterpoint to benchmark-tuned medical agents: input realism and reasoning transparency are treated as architectural requirements rather than as post-hoc explainability layers. 15
Several adjacent systems explore narrower clinical and biological problems. Fan et al. [44] extend medical imaging agents with experience-driven self-skill discovery, allowing the agent to accumulate domain-specific skills from execution trajectories rather than from a fixed skill library. Rhizome OS-1 [107] formulates a semi-autonomous operating system for small-molecule drug discovery, coordinating multiple computational and laboratory operations under a unified agent interface. ProtRLSearch [68] introduces a multi-round multimodal protein search agent trained via reinforcement learning, targeting structure-aware protein retrieval. MedMASLab [84] provides a unified orchestration framework for benchmarking multimodal medical multi-agent systems, supplying the evaluation infrastructure that the broader medical auto-research ecosystem increasingly relies on. A.4.2
Physics, Chemistry, Mathematics, and Other Sciences
Beyond biomedical research, auto-research has begun to take root in the physical sciences, robotics, and adjacent domains, although the overall volume of work remains smaller than in medicine. TRACE [70] formulates a multi-agent system for autonomous physical reasoning in seismology, with specialized agents handling waveform interpretation, source-mechanism inference, and seismic-event reasoning in coordination. The design illustrates how physical-science domains push auto-research toward representations of domain-specific physical knowledge that general scaffolding does not supply. Dr.Sai [50] approaches a different physics problem: real-world physics analysis at the BESIII collider, an experimental high-energy physics setting with mature analysis conventions and well-defined evaluation standards. The system demonstrates that agentic AI can interface with established experimental-physics workflows, although in such settings the human physicist remains tightly coupled to the analysis loop. Alexander et al. [38] extend the physical sciences direction toward theory induction, formulating autonomous discovery of particle physics theories from experimental data. The work sits at the intersection of symbolic search and LLM-assisted reasoning, illustrating one of the most ambitious targets for autonomous scientific discovery in the physical sciences. Xu et al. [114] extend autonomous discovery into cosmology, moving the agent from an assistant role toward hypothesis formation and analysis over cosmological data. STRIDE [98] pursues a related target in symbolic equation discovery, introducing a self-reflective agent framework that combines data-aware generation with mixed-fitting evaluation to improve the reliability of recovered equations. Nomad [53] pursues a more general autonomous exploration and discovery framework, designed to operate across domains rather than within a single scientific specialty. Somasekharan et al. [96] extend autonomous discovery into computational fluid dynamics, using physics-aware agents to explore open-ended CFD problems beyond predefined task templates. Xu et al. [115] approach the mathematical-sciences end of the spectrum with agentic MIP research, accelerating the generation of constraint handlers for mixedinteger programming through agent-driven search. A materials-science cluster pursues autonomous discovery through direct integration with experimental and computational characterization. Qumus [95] realizes an embodied AI experimentalist that integrates hypothesis generation, robotic experimentation, and analysis within a single minilaboratory for van der Waals heterostructures, while Lee et al. [60] coordinate multiple instruments in real time to autonomously discover phase-change memory materials in the Mn-Sb-Te ternary system. Cobelli et al. [43] target the upstream representation problem in materials discovery, using an autoresearch-style loop to design compositional descriptors that downstream models can exploit. Adjacent applications extend into robotics and the social sciences. Khandelwal and Gupta [57] formulate agent-driven autonomous reinforcement learning research for quadruped locomotion, with an agent iteratively proposing policy improvements that are evaluated in simulated and physical environments. LLM Agents as Social Scientists [105] extends the auto-research template into the social sciences, supplying a human-AI collaborative platform for social-science research automation. The two illustrate how the auto-research paradigm is being adapted both toward physically grounded experimentation and toward domains where the unit of investigation is social rather than natural. A.4.3
Boundary Systems Beyond Closure-for-Autonomy
A separate family of systems is frequently raised in connection with auto-research but lies outside the LLM-workflow paradigm surveyed in this appendix. What distinguishes these systems is that an external operation, such as formal verification, physical experimentation, expert sign-off, or executable evaluation, which is constitutive of their architecture rather than supplementary to it. 16
We catalog them here as a complement to the discussion of alternative views in Section 9, which addresses several of these systems in the context of the paper’s scope. A first cluster grounds the system in formal or physical execution. The semi-autonomous formalization of the Vlasov-Maxwell-Landau equilibrium [52] couples LLM-generated proof attempts to a Lean 4 verifier together with in-loop mathematician supervision, so that the system’s notion of a valid step is fixed by formal proof correctness. Coscientist [41] integrates LLM-driven chemical research with cloud-laboratory execution, with physical experimentation incorporated directly into the system’s operation. Latent-Y [100] extends this pattern to de-novo drug design, with antibody-design campaigns paired with wet-laboratory assays as integral validation steps in each design cycle. Robin demonstrates the same pattern in autonomous therapeutic discovery [46]: multi-agent literature search and data analysis are coupled with in-vitro assays as a constitutive validation step, and the system identifies ripasudil and KL001 as candidate treatments for dry age-related macular degeneration, with efficacy confirmed in the laboratory before the loop accepts the result. Although the multi-agent ideation and analysis layer resembles closure-for-autonomy systems on the surface, the lab-in-the-loop structure relocates the validity condition outside the producer’s inductive boundary. CRISPR-GPT [88] agentically automates gene-editing experiments while making expert validation an architectural step in the experiment-design pipeline. These four laboratory-grounded systems together demonstrate that auto-research-adjacent architectures in the life sciences typically incorporate physical or expert validation as constitutive components rather than as optional add-ons. A second cluster grounds the system in executable evaluation. FunSearch [91] filters LLM-generated mathematical and algorithmic programs through executable evaluators, ensuring that candidate outputs are vetted by formal program execution before being retained. AlphaEvolve [79] generalizes this approach into a coding agent for scientific and algorithmic discovery, with executable algorithmic evaluation serving as the validity condition. ERA [40] extends the same paradigm to scientific software generation, using tree search over executable quality metrics on public leaderboards and external baselines across single-cell analysis, COVID-19 hospitalization forecasting, and additional scientific software tasks. A further pair of systems sits adjacent to auto-research in a different way. Google’s AI Co-Scientist [47] is primarily a scientist-guided hypothesis-generation system whose outputs require downstream experimental validation, with its Nature 2026 deployment reporting in-vitro confirmation of drug repurposing candidates for acute myeloid leukemia, with the scientist positioned at the center of the system’s design. OpenAI’s Deep Research [80] is a source-grounded research-synthesis product rather than an autonomous discovery loop. The broader deep-research category, including its verification-centric and meta-evaluation lines, is surveyed and developed in [51, 122, 101, 104].
A.5
Concurrent Diagnostic Positions
Alongside the rapidly growing system literature surveyed in the preceding sections, a complementary body of work asks adjacent diagnostic questions about whether the outputs of these systems deserve the standing their surface form suggests. Five works are particularly relevant for situating the autoresearch landscape and warrant focused treatment. Ríos-García et al. [90] evaluate LLM-based scientific agents across eight domains and more than twenty-five thousand agent runs, combining performance decomposition with behavioral analysis of epistemological traces. Their central finding is that current systems can execute scientific workflows without reliably exhibiting the reasoning patterns that scientific practice typically requires: substantive evidence consideration, refutationdriven belief revision, and convergent multi-test reasoning often fail to appear in agent traces even when the workflow completes to publication-shaped form. The diagnosis locates the gap at the level of how individual runs handle evidence, contradiction, and convergence. Two related works use Karpathy’s autoresearch template as a controlled testbed for diagnostic study. Li [63] conducts a convergence analysis across ten thousand experiments and finds that what is nominally architecture search can reduce to hyperparameter tuning within a narrow design region, with the loop’s internal scoring driving the agent toward portions of the design space that the score most readily rewards. The result documents an empirical narrowing effect that follows from optimizing nominally broad search against a single internal signal. He et al. [49] take a different angle from the same starting point, framing the rise of auto-research as a methodological moment in which the human role shifts from experimenter to research director: as loops execute more of 17
the research workflow autonomously, the practitioner’s task increasingly becomes the specification, oversight, and stewardship of loops rather than the direct conduct of individual experiments. Zhang et al. [120] provide the most direct empirical test of whether artifact production aligns with scientific substance.Letting frontier coding agents carry out the full research loop across thirteen computer-science seeds, they produce 117 agent-generated papers. Under manuscript-only automated review the strongest system scores at the weighted-average level of ICLR 2025 submissions; under artifact-aware review, in which evaluators inspect the workspace alongside the manuscript,scores drop sharply, with manual auditing identifying experimental rigor as the dominant failure mode and none of the 117 papers meeting the acceptance bar of a top-tier venue. This empirical pattern aligns with the diagnostic framework of the present paper: the gap between manuscript-only and artifactaware review is what the L2 collapse predicts at the level of loop architecture, and the dominance of experimental rigor among failure modes is consistent with the L1 collapse acting upstream of validation. Messeri and Crockett [77] broaden the diagnostic frame beyond auto-research specifically, arguing that AI tools can produce illusions of understanding in scientific research when they bypass the distributed processes through which scientific knowledge is normally critiqued, integrated, and revised. An accompanying Nature editorial published alongside the recent Co-Scientist and Robin systems further argues that AI should empower scientific work but not replace human researchers [36], placing the question of autonomous research in an explicitly normative frame. Although their account is not specific to LLM-workflow auto-research, it identifies a gap between fluent artifact production and durable knowledge production that becomes increasingly visible as auto-research systems generate more publication-shaped outputs. A fifth complementary position takes the form of a contemporaneous life-cycle survey rather than a focused diagnosis. Kong et al. [58] organize AI-assisted research across creation, writing, validation, and dissemination, and identify a stage-dependent boundary between reliable assistance and unreliable autonomy, with human-governed collaboration framed as the most credible deployment mode. This stage-level reading and our loop-level reading are complementary rather than competing: where the former asks which research stages support autonomy, the latter asks how a loop’s internal control structure decides what counts as progress, validity, and standing. The five positions sketched here approach auto-research from distinct analytic angles, but none analyzes the field through the lens of scientific closure. Our work is complementary to these positions and fills this gap.
B
Audit Methodology and Per-System Coding
B.1
From Survey to Audit
Appendix A surveys the landscape of LLM-workflow auto-research at descriptive breadth, covering over one hundred works across systems, evaluation infrastructure, and adjacent diagnostic positions. This appendix narrows from that landscape to analytic depth: it documents the coding scheme used to assign the labels reported in Table 1 of Section 4, and provides the per-system evidence supporting each assignment. The 21 systems in the audit pool are a subset of the broader corpus, selected because their loop structure is publicly documented in sufficient detail to support coding against a fixed scheme. The purpose is not to recategorize the systems already described in Appendix A, but to make the basis for the audit’s quantitative claims inspectable and reproducible. Coding draws only on public artifacts, such as papers, repositories, documentation, prompts, and reported workflows. B.2
Coding Scheme
Each of the 21 systems is coded on three dimensions corresponding to the three-level collapse introduced in Section 3. Labels are assigned based on the role a design feature plays in the loop’s control structure. That is, whether and how it actually gates the loop’s selection, validation, or termination behavior. Each dimension admits three possible labels: • Strong: the dimension is not addressed; the auto-research system’s loop behavior is governed entirely by internal signals. 18
• Weak: the dimension is addressed by mechanisms that simulate external involvement from within the system’s inductive boundary. • Mitigated: the dimension is addressed by a genuinely external operation that lies outside the system’s inductive boundary and is integrated as a binding condition of loop validity. The separation between Weak and Mitigated is the central operational distinction in the audit: Weak captures systems whose external-looking checks are simulated from within the system’s inductive boundary, whereas Mitigated requires a genuinely external operation integrated into the loop as a binding condition of validity. The absence of Mitigated in Table 1 is a feature of the paradigm under study, not a gap in coverage. B.3
Per-System Audit Trace
This appendix records the per-system evidence supporting the labels reported in Section 4. For each of the 21 audited systems, Table 2 reports the objective signal, validator, and output pathway based on public documentation and literature synthesis, together with collapse codes for the three dimensions defined in Section 4 and Appendix B.2. The table is intended as an audit trace under those coding rules, not as a comprehensive system catalog. The 21 systems were selected because each has sufficient public documentation (papers, repositories, or both) to support coding without reliance on undocumented internal practices. No system in the audit pool receives a mitigated label on any dimension, under the rules in Appendix B.2. Table 2: Per-system audit. Each row records the method’s objective signal, validator, and output pathway, with collapse codes for the three dimensions (S = strong collapse; W = weak collapse). System
L1 L2 L3 Objective signal
Validator
Output pathway
Karpathy autoresearch[55] uditgoenkaautoresearch[33] goal-md[14]
S
S
S
Improved code solution
S
S
S
S
S
S
autoresearchanything[37] ADAS[30]
S
S
S
S
S
S
AIDE [35]
S
S
S
Internal inspection plus benchmark feedback Internal review and execution feedback Internal execution and metric feedback Internal execution and scoring Benchmark-mediated evaluation Internal execution feedback within the search regime
GEPA[7]
W
S
S
HGM[19]
S
S
S
Bilevel Autoresearch[89]
S
S
S
ASI-Evolve[117]
W
S
S
OmniSimpleMem[71]
S
S
S
Claudini[82]
S
W
S
CORAL[87]
S
W
S
Benchmark or experiment score Internal task metric or experiment result Agent-constructed fitness criterion Measurable task score Candidate architecture performance User-defined scalar metric(e.g., MLE-Bench, Kaggle leaderboard) Ranked task performance after reflection SWE-bench performance
Internal reflection and task feedback Internal execution and self-review Task score plus meta Internal benchmark and improvement rate execution feedback at both levels Aggregated system-level Experimental benchmark performance gain feedback and internal analyzer Benchmark improvement Internal experiment on memory tasks execution and benchmark feedback Attack success and transfer Held-out adversarial performance evaluation within benchmark regime Task improvement rates Evaluator separation and and evaluator scores task feedback inside system
19
Improved repository artifact Goal-conditioned code or task artifact Task-specific repository artifact Improved agent architecture Improved code solution
Optimized prompt, code, or text Improved agent or codebase Improved loop plus task artifact Discovered architectures, data, or algorithms Memory framework and design changes Released attack algorithms and evaluation code Improved mathematical, algorithmic, or systems solutions
System
L1 L2 L3 Objective signal
Validator
Output pathway
SakanaAI/AIScientist[27, 75]
S
W
W Automated reviewer score
Automated review plus workshop peer review
AI-Scientist-v2[28]
S
S
S
AutoResearchClaw[2] S
W
S
Agent Laboratory[29, 92]
S
S
S
Internal automated evaluation Citation verification plus internal review Internal review across same-family agents
Generated paper plus workshop-tested output (Submit to ICLR 2025 Workshop) Generated research artifact
CycleResearcher[109] S
S
S
AI-Researcher [9]
W
W
W Aggregated objectives across multiple tasks
ARIS[34]
S
W
S
Internal aggregated score
ClawTeam[10]
W
S
S
Aggregated ranked result after parallel search
B.4
Internal reviewer or evaluation score Paper-generation objective and internal selection Phase-internal completion plus paper-quality signals Averaged reviewer score used as SimPO reward
Publication-shaped output
Generated paper plus AgentRxiv-mediated access but not external correction Same-family reward model Watermarked LaTeX paper; submission to real peer trained on ICLR-2024 review data review prohibited Cross-model review and Novix.science code review production-facing deployment Cross-model review across Terminal research artifact Claude, Codex, and GPT-style systems Internal validation Terminal artifact
Coding Limitations
The coding is based on public artifacts and undocumented internal practices cannot alter labels. The Strong/Weak distinction is intentionally coarse and identifies structural patterns in loop control, not overall system quality, novelty, or usefulness. The audit identifies co-occurrence between closurefor-autonomy and the three collapse dimensions but does not establish that one dimension causally produces another.
20
References [1] du-nlp-lab/MLR-Copilot, Aug. 2025. [2] aiming-lab/AutoResearchClaw, Apr. 2026. [3] davebcn87/pi-autoresearch, Apr. 2026. [4] drivelineresearch/autoresearch-claude-code, Apr. 2026. [5] eimenhmdt/autoresearcher, Apr. 2026. [6] Entrpi/autoresearch-everywhere, Apr. 2026. [7] gepa-ai/gepa, Apr. 2026. [8] greyhaven-ai/autocontext, Apr. 2026. [9] HKUDS/AI-Researcher, Apr. 2026. [10] HKUDS/ClawTeam, Apr. 2026. [11] hyperspaceai/agi, Apr. 2026. [12] james-s-tayler/lazy-developer, Apr. 2026. [13] JinheonBaek/ResearchAgent, Mar. 2026. [14] jmilinovich/goal-md, Apr. 2026. [15] leo-lilinxiao/codex-autoresearch, Apr. 2026. [16] LitLLM/LitLLM, Apr. 2026. [17] MASWorks/ML-Agent, Mar. 2026. [18] MaximeRobeyns/self_improving_coding_agent, Apr. 2026. [19] metauto-ai/HGM, Apr. 2026. [20] MrTsepa/autoevolve, Mar. 2026. [21] mutable-state-inc/autoresearch-at-home, Apr. 2026. [22] openags/OpenAGS, Apr. 2026. [23] OpenRaiser/NanoResearch, Apr. 2026. [24] Orchestra-Research/AI-Research-SKILLs, Apr. 2026. [25] peterskoett/self-improving-agent, Apr. 2026. [26] PouriaRouzrokh/LatteReview, Apr. 2026. [27] SakanaAI/AI-Scientist, Apr. 2026. [28] SakanaAI/AI-Scientist-v2, Apr. 2026. [29] SamuelSchmidgall/AgentLaboratory, Apr. 2026. [30] ShengranHu/ADAS, Apr. 2026. [31] Sibyl-Research-Team/AutoResearch-SibylSystem, Apr. 2026. [32] supratikpm/gemini-autoresearch, Apr. 2026. [33] uditgoenka/autoresearch, Apr. 2026. [34] wanshuiyin/Auto-claude-code-research-in-sleep, Apr. 2026. 21
[35] WecoAI/aideml, Apr. 2026. [36] Why AI cannot do good science without humans. Nature, 653(8115):650–650, May 2026. [37] zkarimi22/autoresearch-anything, Apr. 2026. [38] S. Alexander, B. Bradley, L. Gouskos, and C. Niu. Autonomous Discovery of Particle Physics Theories from Experimental Data, Mar. 2026. [39] S. Alzubi, N. Provenzano, J. Bingham, W. Chen, and T. Vu. Evoskill: Automated skill discovery for multi-agent systems. arXiv preprint arXiv:2603.02766, 2026. [40] E. Aygün, A. Belyaeva, G. Comanici, M. Coram, H. Cui, J. Garrison, R. Johnston, A. Kast, C. Y. McLean, P. Norgaard, Z. Shamsi, D. Smalling, J. Thompson, S. Venugopalan, B. P. Williams, C. He, S. Martinson, M. Plomecka, L. Wei, Y. Zhou, Q.-Z. Zhu, M. Abraham, E. Brand, A. Bulanova, J. A. Cardille, C. Co, S. Ellsworth, G. Joseph, M. Kane, R. Krueger, J. Kartiwa, D. Liebling, J.-M. Lueckmann, P. Raccuglia, X. J. Wang, K. Chou, J. Manyika, Y. Matias, J. C. Platt, L. Dorfman, S. Mourad, and M. P. Brenner. An AI system to help scientists write expert-level empirical software. Nature, May 2026. [41] D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes. Autonomous chemical research with large language models. Nature, 624:570–578, 2023. [42] Y. Chen, C. Liu, Z. Chen, T. Liu, B. Han, and K. Zhang. CausalEvolve: Towards Open-Ended Discovery with Causal Scratchpad, Mar. 2026. [43] M. Cobelli and S. Sanvito. Agentic design of compositional descriptors via autoresearch for materials science applications. arXiv preprint arXiv:2605.14671, 2026. [44] L. Fan, P. Dai, Z. Deng, H. Wang, X. Gong, Y. Zheng, and Y. Ou. Evolving Medical Imaging Agents via Experience-driven Self-skill Discovery, Mar. 2026. arXiv:2603.05860 [cs]. [45] F. Ferreira, L. Wobbe, A. Krishnakumar, F. Hutter, and A. Zela. Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch, Mar. 2026. [46] A. E. Ghareeb, B. Chang, L. Mitchener, A. Yiu, C. J. Szostkiewicz, D. Shved, G. J. Gyimesi, J. M. Laurent, S. M. Wright, M. T. Razzak, A. D. White, S. C. Finnemann, M. M. Hinks, and S. G. Rodriques. A multi-agent system for automating scientific discovery. Nature, May 2026. [47] J. Gottweis, W.-H. Weng, A. Daryin, T. Tu, P. Sirkovic, A. Myaskovsky, G. Glowaty, F. Weissenberger, A. Orlandi, D. Popovici, A. Palepu, K. Rong, R. Tanno, K. Saab, F. Zhang, J. Blum, A. Carroll, K. Kulkarni, N. Tomašev, D. Zverinski, I. Rendulic, E. Vedadi, F. Hasler, L. Rimanic, M. Boia, I. Budiselic, B. Feinstein, M. Bellaiche, T. Sheffer, J. Freyberg, J. Ratcliff, O. Bertolli, K. Chou, A. Hassidim, B. Gokturk, A. Vahdat, Y. Guan, V. Dhillon, E. D. Vaishnav, B. Lee, T. R. D. Costa, J. R. Penadés, G. Peltz, Y. Matias, J. Manyika, D. Hassabis, Y. Xu, P. Kohli, A. Pawlosky, A. Karthikesalingam, and V. Natarajan. Accelerating scientific discovery with Co-Scientist. Nature, May 2026. [48] T. Han, Y. Zhang, W. Song, C. Fang, Z. Chen, Y. Sun, and L. Hu. Swe-skills-bench: Do agent skills actually help in real-world software engineering? arXiv preprint arXiv:2603.15401, 2026. [49] C. He, X. Zhou, D. Wang, H. Xu, W. Liu, and C. Miao. The AutoResearch Moment: From Experimenter to Research Director, Mar. 2026. [50] M. He, F. Jiang, J. Jiao, M. Li, K. Li, Y. Liao, B. Liu, T. Liu, F. Qi, Z. Shang, W. Song, Y. Sun, X. Wang, H. Wang, D. Xiong, C. Yuan, B. Zhang, Z. Zhang, and X. Zhu. Dr.Sai: An agentic AI for real-world physics analysis at BESIII, Apr. 2026. arXiv:2604.22541 [hep-ex] version: 1. [51] Y. Huang, Y. Chen, H. Zhang, K. Li, H. Zhou, M. Fang, L. Yang, X. Li, L. Shang, S. Xu, J. Hao, K. Shao, and J. Wang. Deep Research Agents: A Systematic Examination And Roadmap, Sept. 2025. arXiv:2506.18096 [cs]. 22
[52] V. Ilin. Semi-Autonomous Formalization of the Vlasov-Maxwell-Landau Equilibrium, Mar. 2026. [53] B. Jia, S. Kamboj, S. Katipomu, S. H. Han, N. Sengupta, and A. Jackson. Nomad: Autonomous Exploration and Discovery, Mar. 2026. [54] G. Jiang, Z. Su, X. Qu, and Y. R. Fung. Xskill: Continual learning from experience and skills in multimodal agents. arXiv preprint arXiv:2603.12056, 2026. [55] A. Karpathy. karpathy/autoresearch, Apr. 2026. [56] J. Karwowski, O. Hayman, X. Bai, K. Kiendlhofer, C. Griffin, and J. M. V. Skalse. Goodhart’s law in reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. [57] N. Khandelwal and S. S. Gupta. Agent-Driven Autonomous Reinforcement Learning Research: Iterative Policy Improvement for Quadruped Locomotion, Mar. 2026. [58] L. Kong, X. Sun, W. Chow, L. Li, K. Q. Lin, X. B. Zhang, S. Wang, R. Li, Q. Wu, W. Gao, Y. Wang, S. Xie, J. Liu, L. Qu, S. Li, L. X. Ng, B. R. Cottereau, Z. Liu, T.-S. Chua, and W. T. Ooi. AI for Auto-Research: Roadmap & user guide. May 2026. [59] S. Kuroki, T. Nakamura, T. Akiba, and Y. Tang. Agent skill acquisition for large language models via cycleqd. arXiv preprint arXiv:2410.14735, 2024. [60] C.-Y. Lee, H. Liang, R. Kim, A. McDannald, C. A. R. Ocampo, A. G. Kusne, and I. Takeuchi. Real-time multi-instrument autonomous discovery of novel phase-change memory materials. May 2026. [61] F. Li, P. Tagkopoulos, and I. Tagkopoulos. Skillflow: Scalable and efficient agent skill retrieval system. arXiv e-prints, pages arXiv–2504, 2025. [62] H. Li, C. Mu, J. Chen, S. Ren, Z. Cui, Y. Zhang, L. Bai, and S. Hu. Organizing, orchestrating, and benchmarking agent skills at ecosystem scale. arXiv preprint arXiv:2603.02176, 2026. [63] X. Li. Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 Experiments, Mar. 2026. [64] X. Li. When single-agent with skills replace multi-agent systems and when they fail. arXiv preprint arXiv:2601.04748, 2026. [65] X. Li, W. Chen, Y. Liu, S. Zheng, X. Chen, Y. He, Y. Li, B. You, H. Shen, J. Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026. [66] Y. Li, C. Shao, X. Liu, R. Zhao, P. Liu, H. Su, Z. Chen, Q. Yang, A. Xu, Y. Fang, et al. Autosota: An end-to-end automated research system for state-of-the-art ai model discovery. arXiv preprint arXiv:2604.05550, 2026. [67] Y. Liang, R. Zhong, H. Xu, C. Jiang, Y. Zhong, R. Fang, J.-C. Gu, S. Deng, Y. Yao, M. Wang, et al. Skillnet: Create, evaluate, and connect ai skills. arXiv preprint arXiv:2603.04448, 2026. [68] C. Liu, T. Li, M. Huang, X. Wei, P. Liu, Y. Shen, Y. Mao, and T. Cui. Protrlsearch: A multiround multimodal protein search agent with large language models trained via reinforcement learning. arXiv preprint arXiv:2603.01464, 2026. [69] F. Liu, J. Han, T. Lyu, W. Zhang, Z.-R. Yang, L. Dai, C. Liu, and H. Liu. Foundation models for scientific discovery: From paradigm enhancement to paradigm transition. Advances in Neural Information Processing Systems, 2025. [70] F. Liu, J. Xu, X. Cui, X. Wang, Z. Guo, J. Wang, S. M. Mousavi, X. Gu, H. Chen, B. Fei, L. Fang, F. Ling, Z. Li, and L. Bai. TRACE: A Multi-Agent System for Autonomous Physical Reasoning for Seismology, Mar. 2026. 23
[71] J. Liu, Z. Ling, S. Qiu, Y. Liu, S. Han, P. Xia, H. Tu, Z. Zheng, C. Xie, C. Fleming, et al. Omni-simplemem: Autoresearch-guided discovery of lifelong multimodal agent memory. arXiv e-prints, pages arXiv–2604, 2026. [72] J. Liu, S. Qiu, M. Li, B. Li, H. Ji, S. Han, X. Ye, P. Xia, Z. Dong, C. Zhang, L. Zhang, G. Chen, H. Tu, X. Yang, L. Feng, X. Zhao, H. Chen, J. Zhou, X. Wang, W. Zhang, H. Zhu, Y. Li, J. Mei, H. Fei, J. Zhang, L. Li, L. Zhang, Y. Zhou, S. Wang, C. Xiong, J. Zou, Z. Zheng, C. Xie, M. Ding, and H. Yao. AutoResearchClaw: Self-reinforcing autonomous research with human-AI collaboration. 2026. [73] J. Liu, J. Shen, S. Song, T. Li, X. Liu, R. Li, Z. Huang, J. Lin, J. Ning, C. Ji, S. Luo, W. Li, C. Ma, M. Hu, J. Xiong, J. Ye, B. Fu, N. Xu, Y. Chen, L. Jin, H. Chen, and J. He. MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline, Apr. 2026. arXiv:2604.18418 [cs] version: 1. [74] J. Liu, X. Ye, P. Xia, Z. Zheng, C. Xie, M. Ding, and H. Yao. Evolvemem: Self-evolving memory architecture via autoresearch for llm agents. arXiv preprint arXiv:2605.13941, 2026. [75] C. Lu, C. Lu, R. T. Lange, Y. Yamada, S. Hu, J. Foerster, D. Ha, and J. Clune. Towards end-to-end automation of AI research. Nature, 651(8107):914–919, Mar. 2026. [76] D. Manheim and S. Garrabrant. Categorizing variants of goodhart’s law. arXiv preprint arXiv:1803.04585, 2018. [77] L. Messeri and M. J. Crockett. Artificial intelligence and illusions of understanding in scientific research. Nature, 627:49–58, 2024. [78] J. Ni, Y. Liu, X. Liu, Y. Sun, M. Zhou, P. Cheng, D. Wang, X. Jiang, and G. Jiang. Trace2skill: Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158, 2026. [79] A. Novikov, M. Balog, M. P. Kumar, et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery, 2025. [80] OpenAI. Introducing deep research, 2025. [81] A. Pan, E. Jones, M. Jagadeesan, and J. Steinhardt. Feedback loops with language models drive in-context reward hacking. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39154–39200. PMLR, 2024. [82] A. Panfilov, P. Romov, I. Shilov, Y.-A. d. Montjoye, J. Geiping, and M. Andriushchenko. Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs, Mar. 2026. [83] A. Pepe, C.-Y. Lin, D. Magka, B. Acun, Y. N. Wu, A. Protopopov, C.-J. Wu, and Y. Bachrach. Agentic discovery of neural architectures: AIRA-Compose and AIRA-Design. May 2026. [84] Y. Qian, X. Hu, J. Yu, S. Xin, X. Chen, J. Zhang, P.-T. Jiang, J. Liu, and H. B. Li. Medmaslab: A unified orchestration framework for benchmarking multimodal medical multi-agent systems. arXiv preprint arXiv:2603.09909, 2026. [85] J. Qiao, W. Meng, Y. Cheng, Z. Lin, Z. Zhang, X. Tan, J. Gong, K. Shao, and Y. Xie. Memory intelligence agent. arXiv preprint arXiv:2604.04503, 2026. [86] Y. Qin, Y. Yan, J. Pang, and X. Zhang. BloClaw: An Omniscient, Multi-Modal Agentic Workspace for Next-Generation Scientific Discovery, Apr. 2026. [87] A. Qu, H. Zheng, Z. Zhou, Y. Yan, Y. Tang, S. Y. Ong, F. Hong, K. Zhou, C. Jiang, M. Kong, et al. Coral: Towards autonomous multi-agent evolution for open-ended discovery. arXiv preprint arXiv:2604.01658, 2026. [88] Y. Qu, K. Huang, M. Yin, K. Zhan, D. Liu, D. Yin, H. C. Cousins, W. A. Johnson, X. Wang, M. Shah, et al. Crispr-gpt for agentic automation of gene-editing experiments. Nature Biomedical Engineering, 10(2):245–258, 2026. 24
[89] Y. Qu and M. Lu. Bilevel Autoresearch: Meta-Autoresearching Itself, Mar. 2026. [90] M. Ríos-García, N. Alampara, C. Gupta, I. Mandal, S. Mannan, A. A. Aghajani, N. Krishnan, and K. M. Jablonka. Ai scientists produce results without reasoning scientifically. arXiv preprint arXiv:2604.18805, 2026. [91] B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi. Mathematical discoveries from program search with large language models. Nature, 625:468–475, 2024. [92] S. Schmidgall and M. Moor. AgentRxiv: Towards Collaborative Autonomous Research. arXiv preprint arXiv:2503.18102, Mar. 2025. Companion paper to Agent Laboratory repository. [93] W. Shen, Y. Hu, C. Liu, J. Wu, J. Zhu, C. Shen, M. Xu, Y. Jin, B. Wiestler, D. Rueckert, and J. Pan. MedOpenClaw: Auditable Medical Imaging Agents Reasoning over Uncurated Full Studies, Mar. 2026. [94] Y. Shen, Z. Yi, Z. Zhao, L. Sun, D. Li, C.-T. Lin, and Y. Shi. An Empirical Study of Multi-Agent Collaboration for Automated Research, Mar. 2026. [95] L. Shi, Z. J. Zheng, X. Juan, Y. Wang, M. Yin, M. Sengupta, K. Wolinski, Y. Jia, J. Shi, D. Saucedo, N. Saggi, H. Guan, K. Watanabe, T. Taniguchi, A. Yazdani, M. Wang, and S. Wu. Qumus: Realization of an embodied AI quantum material experimentalist. May 2026. [96] N. Somasekharan, R. Pathak, M. Dhanakoti, T. Zhang, L. Yue, A. Zhu, and S. Pan. Ai cfd scientist: Toward open-ended computational fluid dynamics discovery with physics-aware ai agents. arXiv preprint arXiv:2605.06607, 2026. [97] W. Song, J. Yue, and Z. Pang. Abstral: Automatic design of multi-agent systems through iterative refinement and topology optimization. arXiv preprint arXiv:2603.22791, 2026. [98] J. Su, S. Tu, B. Sun, and X. Liang. STRIDE: A self-reflective agent framework for reliable automatic equation discovery. May 2026. [99] Z. Sun, Z. Liu, Y. Zang, Y. Cao, X. Dong, T. Wu, D. Lin, and J. Wang. Seagent: Selfevolving computer use agent with autonomous learning from experience. arXiv preprint arXiv:2508.04700, 2025. [100] L. L. Team, S. M. Schmon, D. Pretorius, S. Mathis, R. Bartke-Croughan, A. Puvanendran, J. Vuckovic, H. Kenlay, M. Vlachynská, A. Bridgland, I. Grishin, S. Over, D. Li, B. Li, J. Crabbé, A. Hilmkil, A. W. R. Nelson, D. Yuan, A. Obika, and S. A. A. Kohl. Latent-Y: A Lab-Validated Autonomous Agent for De Novo Drug Design, Mar. 2026. [101] M. Team, S. Bai, L. Bing, L. Lei, R. Li, X. Li, X. Lin, E. Min, L. Su, B. Wang, L. Wang, L. Wang, S. Wang, X. Wang, Y. Zhang, Z. Zhang, G. Chen, L. Chen, Z. Cheng, Y. Deng, Z. Huang, D. Ng, J. Ni, Q. Ren, X. Tang, B. L. Wang, H. Wang, N. Wang, C. Wei, Q. Wu, J. Xia, Y. Xiao, H. Xu, X. Xu, C. Xue, Z. Yang, Z. Yang, F. Ye, H. Ye, J. Yu, C. Zhang, W. Zhang, H. Zhao, and P. Zhu. MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verification, Mar. 2026. [102] J. D. Toscano, Z. Chai, and G. E. Karniadakis. GRAFT-ATHENA: Self-improving agentic teams for autonomous discovery and evolutionary numerical algorithms. May 2026. [103] J. Wang, Z. Tao, H. Zeng, Z. Yang, H. Zamani, and H. Yu. Tarse: Test-time adaptation via retrieval of skills and experience for reasoning agents. arXiv preprint arXiv:2603.01241, 2026. [104] L. Wang, Y. He, P. Chen, A. Yehudai, Y. Liu, R. Ying, M. Shmueli-Scheuer, and A. Cohan. Time to REFLECT: Can we trust LLM judges for evidence-based research agents? 2026. [105] L. Wang, Y. Li, J. Wu, H. Gao, X. Bo, X. Chen, and J.-R. Wen. Llm agents as social scientists: A human-ai collaborative platform for social science automation. arXiv preprint arXiv:2604.01520, 2026. 25
[106] S. Wang, J. Jin, R. Fu, Z. Yan, X. Wang, M. Hu, E. Wang, X. Li, K. Zhang, L. Yao, W. Jiao, X. Cheng, Y. Lu, and Z. Ge. MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences, Mar. 2026. [107] Y. Wang, G. Sinenka, and X. Brace. Rhizome os-1: Rhizome’s semi-autonomous operating system for small molecule drug discovery. arXiv preprint arXiv:2604.07512, 2026. [108] J. Wei, Y. Yang, X. Zhang, Y. Chen, X. Zhuang, Z. Gao, D. Zhou, G. Wang, Z. Gao, J. Cao, Z. Qiu, M. Hu, C. Ma, S. Tang, J. He, C. Song, X. He, Q. Zhang, C. You, S. Zheng, N. Ding, W. Ouyang, N. Dong, Y. Cheng, S. Sun, L. Bai, and B. Zhou. From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery, Oct. 2025. arXiv:2508.14111 [cs]. [109] Y. Weng, M. Zhu, G. Bao, H. Zhang, J. Wang, Y. Zhang, and L. Yang. Cycleresearcher: Improving automated research via automated review. In The Thirteenth International Conference on Learning Representations, 2025. [110] H. Wu, B. Zheng, D. Song, Y. Jiang, J. Gao, L. Xing, L. Sun, and Y. Yuan. Towards a Medical AI Scientist, Mar. 2026. [111] P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, et al. Skillrl: Evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234, 2026. [112] S. Xia, Y. Zhang, A. Chen, S. Wu, S. Yuan, and Y. Xiao. From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents, Mar. 2026. [113] Z. Xie, Y. Guo, and K. Xu. AwesomeLit: Towards Hypothesis Generation with AgentSupported Literature Research, Mar. 2026. [114] L. Xu and T. Borrett. Beyond ai as assistants: Toward autonomous discovery in cosmology. arXiv preprint arXiv:2605.14791, 2026. [115] L. Xu, Y. Zhou, and S. Pokutta. Agentic mip research: Accelerated constraint handler generation. arXiv preprint arXiv:2605.09186, 2026. [116] R. Xu and Y. Yan. Agent skills for large language models: Architecture, acquisition, security, and the path forward. arXiv preprint arXiv:2602.12430, 2026. [117] W. Xu, T. Mi, Y. Liu, Y. Nan, Z. Zhou, L. Ye, L. Zhang, Y. Qiao, and P. Liu. ASI-Evolve: AI Accelerates AI, Mar. 2026. [118] Y. Yang, J. Li, Q. Pan, B. Zhan, Y. Cai, L. Du, J. Zhou, K. Chen, Q. Chen, X. Li, et al. Autoskill: Experience-driven lifelong learning via skill self-evolution. arXiv preprint arXiv:2603.01145, 2026. [119] Z. Yang, X. Liu, and X. Xu. Sciintegrity-bench: A benchmark for evaluating academic integrity in ai scientist systems. arXiv preprint arXiv:2605.10246, 2026. [120] Z. Zhang, N. Wang, S. Galhotra, and C. Cardie. How far are we from true auto-research? 2026. [121] H. Zhou, S. Guo, A. Liu, Z. Yu, Z. Gong, B. Zhao, Z. Chen, M. Zhang, Y. Chen, J. Li, et al. Memento-skills: Let agents design agents. arXiv preprint arXiv:2603.18743, 2026. [122] B. Zhu, Q. Jia, T. Lan, J. Ren, F. Gu, F. Jiang, L. Wang, Z. Xu, and W. Luo. Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design, Mar. 2026. [123] Q. Zou, H. H. Lam, W. Zhao, T. Chen, Y. Tang, S. Yu, Y. Zhu, S. Anumasa, Z. Zhang, T. Zhang, C. Liu, Z. Jiang, A. Goyal, and D. Liu. FML-bench: A controlled study of AI research agent strategies from the perspective of search dynamics. May 2026.
26