arXiv CSopen access

Structure Liberates: How Constrained Sensemaking Produces More Novel Research Output

Preprint. Under review.

Structure Liberates: How Constrained Sensemaking Produces More Novel Research Output James Mooney

Zae Myung Kim

Young-Jun Lee

Dongyeop Kang

University of Minnesota Website

Code

Models & Dataset

arXiv:2605.00557v1 [cs.CL] 1 May 2026

Abstract Scientific discovery is an extended process of ideation—surveying prior work, forming hypotheses, and refining reasoning—yet existing approaches treat this phase as a brief preamble despite its central role in research. We introduce S CI S ENSE, a sensemaking-grounded framework that operationalizes ideation as a structured sequence of eight cognitive stages (Pirolli & Card, 2005). We construct S CI S ENSE-Traj, a 100K-scale dataset of citationconditioned research trajectories in two modes: Target, where an LLM reconstructs the ideation path leading to a known paper from its cited works, and Infer, where the LLM proposes novel directions from the same citations. We distill these into S CI S ENSE-LM, a family of sensemaking LLMs spanning 3B to 70B parameters. Contrary to the assumption that looser supervision promotes greater exploration, Target-trained models achieve a 2.0% improvement in trajectory quality over Infer-trained models while also producing more novel and diverse outputs. This advantage propagates downstream: coding agents conditioned on Target trajectories produce research artifacts with higher executability and quality than those conditioned on Infer trajectories. This suggests that targeted ideation reduces cognitive burden on downstream agents, freeing them to explore more creatively. S CI S ENSE offers both a practical tool for augmenting LLM-driven research workflows and a principled testbed for studying how planning shapes scientific discovery.

1

Introduction

Scientific discovery has long been a key driver of human progress, characterized by an iterative refinement loop involving the surveying of prior work, hypothesis formation, reasoning refinement, and implementation. Recently, LLM-driven research agents (Lu et al., 2024; Gottweis et al., 2025; Yamada et al., 2025; Baek et al., 2025; Seo et al., 2025) have demonstrated substantial progress on scientific discovery tasks. These agents typically consist of two streams: (1) an upstream phase, where they perform ideation and planning to propose novel ideas and formulate hypotheses, and (2) a downstream phase, where the proposed ideas are realized in practice through code implementation, experimental execution, and result analysis. The upstream phase naturally propagates into the downstream phase. Existing research agents often treat the upstream phase as a brief preamble, such as one or two paragraphs or simple, abstract statements. In contrast, the actual upstream process used by human researchers is considerably more complex, consisting of multiple detailed steps that are more structurally organized. Therefore, a gap exists between the simplified design in current systems and the structured processes followed by human researchers. To address this gap, we reinterpret the upstream phase through the sensemaking framework (Pirolli & Card, 2005), which describes how researchers structure information into explanatory frameworks, operationalized as eight structured cognitive stages (see Table 1). Building on sensemaking principles, we introduce S CI S ENSE, a sensemaking-grounded framework for scientific discovery that provides end-to-end support. Specifically, we 1

Preprint. Under review.

None

Infer

Both

Target

0.25

1.0

4.0

0.8

Score

3.5

0.20

Quality

Diversity (1 - Embed Sim)

0.30

3.0

0.15 0.10

2.5

0.05 32B

4B

70B

SciSense-LM-

27B

0.1

0.2 0.3 Diversity

Infer Target 0.4

Executability Sci. Grounding Downstream Utility

0.6 0.4 0.2 0.0

No Plan

Raw Plan

Inferred Plan

Targeted Plan

(a) Target trajectories are (b) ...without sacrificing quality. (c) ...and improve downstream artimore diverse. fact quality.

Figure 1: Overview of experimental findings. (a) Target trajectories exhibit more spread when measured using embedding similarity measures. (b) In addition to more diversity, Target trajectories display higher quality. (c) Downstream artifact evaluation: trajectories generated under stronger supervision (i.e., Target trajectories) lead to better end-to-end research artifacts, including improved implementation and paper outcomes, as compared to other baseline or distilled method variants.

construct S CI S ENSE-Traj, a 100K-scale dataset of sensemaking-based research trajectories conditioned on citation neighborhoods, comprising two complementary modes: Target, where an LLM reconstructs a plausible ideation trajectory leading to a known paper from its cited works, and Infer, where the LLM proposes novel research directions grounded in the same citations. Using this dataset, we distill sensemaking capabilities into S CI S ENSE-LM, a family of LLMs with various sizes from 3B to 70B. Figure 1 summarizes our central empirical findings. Contrary to the assumption that weaker constraints promote broader exploration, outcome-anchored Target supervision yields trajectories that are simultaneously more diverse and higher quality than open-ended Infer supervision across all evaluated model families and scales. Target-trained models achieve the strongest diversity under multiple automatic metrics and a 2.0% gain in aggregate plan quality over Infer —with no quality–diversity tradeoff: Target dominates Infer on both axes. Mechanistically, Target plans emphasize constructive sensemaking stages (Schema, Elaboration), while Infer plans disproportionately front-load Hypothesis and Questioning. This advantage transfers downstream: Target trajectories improve the quality of generated scientific artifacts when conditioning coding agents, with particular gains in scientific grounding and improvements in executability and downstream utility. Practitioners can use S CI S ENSE as a testbed for improving the ideation and planning capabilities of LLM-driven research agents. Our results suggest that reconstructive trajectories achieve higher quality while enabling more diverse ideation, and that such structured, strong supervision enhances the creativity of downstream coding agents—highlighting the importance of principled, structured ideation and planning. Overall, S CI S ENSE advances our understanding of how upstream planning shapes scientific discovery and provides a principled framework for developing and evaluating next-generation research agents.

2

S CI S ENSE Framework

S CI S ENSE is the first sensemaking-based framework for automated scientific discovery that provides end-to-end support—from citation-grounded corpus construction to scalable trajectory learning and downstream evaluation—while remaining fully open-source, as shown in Table 4. The framework operates in three stages—corpus construction, trajectory generation, and model distillation—each grounded in sensemaking principles. § 2.1 introduces the sensemaking principle; § 2.2 formalises the discovery problem; § 2.3 describes dataset construction; § 2.4 covers model training; and § 3 defines the evaluation protocol. 2

Preprint. Under review.

Upstream Cited Works

Downstream Coding Agents

Sensemaking-based Research Trajectories Foraging

Shoebox

SciSense-LM Elaboration

Research Artifacts

Schema Hypothesis

Questioning

Reframe

Presentation

Problem De nition & Scope The intent is to rigorously de ne the challenge of face attribute prediction under strict computational constraints, ensuring the research program targets real-world deployability. Key criteria are speci city (aligning with the target paper's attributes and dataset) and quanti able ef ciency metrics, prioritized over broader facial analysis tasks or qualitative ef ciency goals. De ne the attribute set and source dataset as CelebA (40 attributes: gender, age, smile, glasses, beard, bald). Ensures direct comparability with the 91.24% accuracy claim. Quantify resource constraints: Target <0.5M parameters (25× reduction from typical baselines ~12.5M), FLOPs <50M, memory footprint reduction >87%. These metrics are derived directly from the paper's claims, providing concrete, falsi able goals. Assumption: The "comparably performing methods" baseline is a standard CNN (e.g., modi ed VGG) on CelebA. … Model Architecture Design The intent is to replicate the core architectural innovation claimed — the "Slim Module" — and its assembly into "Slim-Net". The key criterion is delity …. Specify the Slim Module: Input → Depthwise Separable Conv (k×k) → BN → ReLU → Pointwise Conv (1×1) → BN → ReLU → Output. This structure is the canonical ef cient block, directly mirroring the paper's description. De ne the network stacking strategy: Sequentially stack N Slim Modules…. Fallback: If speci c kernel size or module count (N) isn't provided, start with k=3 and N=5, then optimize. This is a proactive frame-questioning move against the under-speci ed architecture.

Figure 2: General Incorporation of Sensemaking into Scientific Research Agents. We explore the Upstream sensemaking phase of research to better emulate the trajectories of thought researchers take prior to generating paperfiandficode artifacts in the Downstream phase. fi fi fi fi fi 2.1

Preliminaries: Sensemaking Principle

The sensemaking principle (Pirolli & Card, 2005) describes how researchers iteratively construct meaning from prior work—collecting, organising, and synthesising evidence into coherent, testable, and procedurally grounded research trajectories that persuade others through empirical justification. We operationalise sensemaking as an ordered sequence of eight stages that mirror the process of a researcher turning a body of literature into a research contribution. These eight stages define a sensemaking trajectory: a structured, record of the intellectual process that transforms a citation neighbourhood into a research contribution. 2.2

Problem Formulation for Automated Scientific Discovery in S CI S ENSE

The goal of automated scientific discovery is to enable an LLM M to autonomously conduct research by following a scientific pipeline P , analogous to human researchers. Following prior work (Lu et al., 2024; Yamada et al., 2025), this pipeline consists of four stages: ideation and planning (Pplan ), experimentation (Pexpr ), analysis (Panalysis ), and paper writing (Pwrite ). Given contextual information C = { xpaper , xcode }, the pipeline produces outputs O := P (C ; M), where O = {ypaper , ycode }. The standard pipeline is formalised as: P = Pplan (C) ,

E = Pexpr (C , P),

Foraging Shoebox Schema Hypothesis Elaboration Questioning

Retrieve prior work and gaps Organise extracted evidence Form initial problem framing Define testable hypotheses Develop methodological plan Assess feasibility and novelty

Reframe Presentation

Revise formulation or approach Synthesize final trajectory

Table 1: Sensemaking stages.

A = Panalysis (C , P, E),

O = Pwrite (C , P, E, A).

The planning stage Pplan is the critical bottleneck: shallow plans lead to incoherent or template-like downstream outputs. S CI S ENSE addresses this by replacing the shallow plan P with a sensemaking trajectory S, which encodes structured, fine-grained procedural knowledge about the research process. Let Ccited ( xpaper ) = {c1 , ..., c N } denote the citation 3

fi

fi

Preprint. Under review.

Target (More Structured, Plausible Path) Cited Papers

Research Trajectories

Infer (Less Constrained, Broad Exploration) Target Paper [2]

[2]

[4]

Cited Papers

Creativity 👍

No Target Paper

[2]

[4,7]

[7]

Research Trajectories

Quality 👎

[4]

[7]

Creativity 👎

Quality 👍

Figure 3: A comparison between Target and Infer research trajectories in SciSense-Traj. Target trajectories attempt to use sensemaking to recover the original paper, whereas Infer trajectories have no such anchor. This results in higher quality, less diverse Target and lower quality, more diverse Infer trajectories. neighborhood of a target paper xpaper , where N = |Ccited ( xpaper )|. We define the extended context as C ′ = C ∪ Ccited ( xpaper ). The S CI S ENSE pipeline is then: S = Psense (Ccited ) ,

E = Pexpr (C ′ , S),

A = Panalysis (C ′ , S, E),

O = Pwrite (C ′ , S, E, A).

The key difference is that Psense conditions exclusively on the cited works—not on the target paper itself—mimicking the information available to a researcher before a paper is written. 2.3 S CI S ENSE-Traj: A Collection of Sensemaking-Based Research Trajectories We construct S CI S ENSE-Traj, a large-scale collection of sensemaking-based research trajectories, comprising two trajectory sets: Target and Infer. In all steps of our dataset construction process, we use Qwen3-235B-A22B-Instruct-2507 (Yang et al., 2025) as our LLM. All prompt templates used in our dataset construction are presented in Appendix A. Motivation for Sensemaking-Grounded Corpus Design. We design S CI S ENSE-Traj to mirror the information structure of real scientific discovery, where researchers reason from a citation neighbourhood rather than from a completed paper. Accordingly, trajectories are grounded in cited works and structured using the eight-stage sensemaking process, which captures how evidence is collected, organized, and transformed into a research contribution. Step 1: Corpus and Citation Neighbourhood Collection. We begin by collecting a largescale academic text corpus from the Semantic Scholar Open Research Corpus (S2ORC) (Lo et al., 2020). Each instance in S2ORC consists of a target paper xpaper , its abstract, and rich paper-level metadata, including citation graphs, section boundaries, and inline citation markers. For each target paper xpaper , we extract the set of all papers it cites, i.e., its citation neighbourhood Ccited ( xpaper ), which we use for research trajectory generation. Step 2: Cited Paper Summary Generation. We prompt an LLM to generate a concise summary si for each paper ci ∈ Ccited , capturing its core contributions, methods, and key empirical findings. In addition, we generate a concise summary s x for the target paper xpaper , which is later used to construct Target. Note that these generated summaries are used solely for dataset construction and are not utilized during training or evaluation. Step 3: Sensemaking-based Research Trajectory Generation. Given the generated summaries of cited papers si , we prompt the LLM to produce structured research trajectories that follow the sensemaking principle (in § 2.1), under two distinct settings: • Target: Given the summaries of the cited works (i.e., s1 , . . . , s N ) and the summary of the target paper s x , we prompt the LLM to reconstruct a principled research process that connects the cited works to the target paper. Since these trajectories are grounded in a known outcome (i.e., the target paper), models trained on such trajectories can learn to reconstruct the author’s reasoning process with high fidelity. • Infer: Given the summaries of the cited works (i.e., s1 , . . . , s N ), we prompt the LLM to propose novel research directions grounded in the citation neighbourhood Ccited . These trajectories are designed to capture open-ended ideation, reflecting the stage at which researchers explore diverse possibilities before converging on a specific research direction; accordingly, models trained on such trajectories are expected to explore more diverse research processes. 4

Preprint. Under review.

Figure 4: Evaluation pipeline for S CI S ENSE. Trajectories are assessed on Upstream characteristics (quality and diversity) as well as Downstream characteristics (utility in paper and code production). • Both: This setting includes both Target and Infer, enabling the model to learn both reconstructive reasoning (i.e., how the target paper emerges from its cited works) and generative reasoning (i.e., how the target paper can inspire future research directions). All these trajectories are represented as markdown-style outputs to ensure structural consistency and alignment with the sensemaking framework. 2.4

S CI S ENSE-LM: Distilling Sensemaking into Smaller Language Models

To induce a “sensemaking procedure” in LLMs, we introduce a new family of LLMs that generate sensemaking-based research trajectories given only the raw text of cited papers, spanning a range of model sizes: S CI S ENSE-LM-{3, 4, 27, 30, 32, 70}B. Specifically, we finetune multiple LLM backbones, including Qwen3 (Yang et al., 2025), Gemma-3 (Team et al., 2025a), and LLaMA 3 (AI@Meta, 2024), covering both base and instruction-tuned variants. During training, we provide truncated1 text of the cited papers along with sensemakingbased, trajectory-specific instructions as input, and the model is trained to sequentially generate the corresponding sensemaking-based research trajectory. By removing the original paper entirely from the distillation process, we align better to the downstream task of trajectory generation where there is no supervisory signal at inference. This allows us to test something more significant in our evaluations: can we more generally distill the sensemaking procedure into language models? Further, we may also test - how does keeping the citing paper for Target generations, but not for distillation, meaningfully impact the difference between Target and Infer models.

3

Experiments

We evaluate the effectiveness of sensemaking trajectories from two perspectives: (1) upstream evaluation, which examines the characteristics of the generated research trajectories alone, and (2) downstream evaluation, which assesses whether frontier agents can produce meaningful artifacts (e.g., code, papers) given these generated trajectories (Figure 4). We investigate the following research questions: RQ1. (Section 3.2) Does grounding in sensemaking principles improve the quality of generated research trajectories? RQ2. (Section 3.3) Do open-ended, ideation-driven trajectories yield more diverse research directions than reconstruction-oriented trajectories? RQ3. (Section 3.4) Do sensemaking-based research trajectories enable coding agents to produce higher-quality downstream artifacts? 3.1

Split

Infer Target Total

Train 50K Validation 5K Test 5K Total 60K Avg. length (tok.) 2031 Avg. cited refs. 5.32 Avg. cited. neigh. –

50K 5K 5K 60K 2918 4.39 –

100K 10K 10K 120K 2475 4.86 7.78

Table 2: S CI S ENSE-Traj statistics

Experimental Setup

The finalized S CI S ENSE-Traj release contains 120,000 citation-conditioned trajectories in total, evenly split between Infer and Target, with 1 To fit the model’s context size, we adopt different truncation strategies (Appendix C).

5

Preprint. Under review.

Model

Nov.

Sig.

Krippendorff’s α (no judge) 0.684 0.749 Krippendorff’s α (w/judge) 0.656 0.766

Grnd. Sound. Meth. Feas. Sense. Clar. Overall 0.548 0.507

0.728 0.716

0.752 0.773

0.796 0.763

0.612 0.594

0.736 0.745

0.701 0.690

Table 3: Inter-annotator agreement (Krippendorff’s α) across rubric dimensions. We compare agreements between human annotators alone (top) and human annotators along with an LLM-as-Judge (bottom), providing evidence for using the LLM-as-Judge in later evaluations.

matched 50k/5k/5k train/validation/test partitions per condition. At the dataset level, Table 2 shows Infer trajectories are shorter and cite slightly more papers on average, while Target trajectories are longer and more reconstructive. None denotes the undistilled checkpoint, i.e., the model without trajectory-based supervision, and serves as our baseline. All SFT models are trained for one epoch with Adam, a learning rate of 10−5 , a cosine schedule, and a maximum context length of approximately 16k tokens.

3.2

RQ1: Producing Higher Quality Research Plans

Manual Annotations. We provide an initial analysis of research trajectories by manual annotation and compare these to LLM-as-Judge annotations in order to validate that our judge may serve as a proxy for scaled up evaluations in the following sections. To this end, three graduate-level annotators with prior experience evaluating large language model outputs independently reviewed a balanced sample of 100 generated trajectories drawn uniformly across conditions of (Infer, Target, None, Both) derived from the available distilled models. Annotators were given the same written instructions and rubric definitions as the LLMas-Judge. The rubric scores Novelty, Significance, Grounding, Soundness, Methodological Rigor, Feasibility, Overall Sensemaking, and Clarity on a 1-5 scale. The LLM-as-Judge is chosen as Qwen3.5-35B-A3B-FP8. Infer

Both

Target

0.15

from None

We measured inter-annotator agreement using Krippendorff’s α, and report per-rubric item α values as well in Table 3. We find general agreement both between human annotators as well as between the human annotators and the judge itself - thus supporting that the judge can serve as a proxy evaluator for experiments in the following sections.

0.10 0.05 0.00 Nov. Sig. Grnd. Snd. Meth. Feas. Sns. Clar. Ovr.

Judge Scores

Scaled Evaluations. Given judge valFigure 5: Target models provide higher qualidation, we scale our results across all ity research trajectories than other models of the distilled models to determine which equivalent family (Qwen3-32B) across nearly all show the greatest improve in quality judged categories. Infer models are more strongly (Appendix G shows the full results). grounded in the prior work. Figure 5 shows the results of these judge scores - normalized to a comparison against the ”None” model outputs for SciSense-LM-32B. We find that Target models lead to a higher quality overall and across most measures while Infer models lead to the most improvements on the Grounding of the paper to previously cited works. These results hold across all model families considered during for distillation (Table 8). Using Reinforcement Learning to Produce Higher Quality Trajectories Applying posttraining to Q32-Both (using the judge prompt directly with the reward model), we find modest, but consistent gains: improvements occur over the non-RL model on all eight rubric dimensions (Overall: 2.51 → 2.53), and Q4-I-Both-RL shows a comparable pattern (Overall: 2.40 → 2.42). Full RL implementation details and per-are in Appendix E. 6

Preprint. Under review.

1.0

Executability

Sci. Grounding

Downstream Utility

Score

0.8 0.6 0.4 0.2 0.0

No Plan

Direct Prompt Baselines

Raw Plan

Inferred Combined Targeted Upper Bound Plan Plan Plan Plan SciSense-LM Trajectories Ceiling

Figure 7: When paired with a coding agent, SciSense-LM trajectories (ours) produce better code and research artifact outputs when compared to simpler prompting methods (Baselines). Further, Target approaches in our framework are better than Infer, Both or None 3.3

RQ2: Scientific Creativity of Generated Research Trajectories

In order to produce more creative research trajectories, it is necessary for the outputs to simultaneously be more diverse and higher quality. A key prediction of our hypothesis is that backward-reconstruction supervision should decrease, not increase, output diversity. We test this with four complementary metrics computed over repeated sampling from our distilled models. For all diversity experiments, each model generates five trajectories per citation set over 100 held-out citation sets from our test partition. 2.60

LLM-Judge Overall

Metrics. We compute a set of metrics that either directly or indirectly measure diversity (SelfBLEU, Full Embeddings, BertScore, Sentence Mover’s Distance). Full results and mathematical definitions of each metric are in Appendix F. All figures in the main body only use Full Embeddings - comparing average pairwise similarities for repeated model generations of research trajectories over the same citation set.

2.55 2.50 2.45 2.40

None Infer Target Both

Both (100K)

None 3-shot (0)

Infer (50K) Infer (25K)

Target (50K) Target (25K)

None (0)

2.35 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Results. We find that Target models consisDiversity (1 Embed Sim) tently achieve the highest diversity across every metric in every model family tested - a pattern which held across all eight families (Appendix Figure 6: Diversity-vs-Quality across F). Moreover, as demonstrated in Figure 6, not SciSense Distilled models (dataset size only do Target models achieve higher Quality trained on in parentheses). Target disoutputs, they also show higher diversity in their tilled approach outperforms the other generations than comparable models and ap- models across both metrics. proaches at the same time. In other words, the extra supervision and structure of the Target-based approach results in both higher quality and more diversity.

3.4

RQ3: Using Plans for Full Research Generation

To evaluate downstream impact, we provide each method’s generated trajectory to a coding agent (gpt-5.3-codex high) that produces a runnable repository and a short workshop-style paper from the same cited-work context. We compare four distilled variants (None, Target, Infer, Both) of Qwen3-32B against two baselines—No-Plan (raw cited papers only) and Direct-Prompt (raw cited papers coupled with human produced prompt)—as well as an Upper-Bound (teacher-model trajectories from Qwen3-235B). We assess three aggregate criteria averaged over five cited-work sets: Executability, Scientific Grounding, and Downstream Utility. The metrics used are LLM-as-Judge evaluations. Further details are presented in Appendix D. As shown in Figure 1c, all distilled variants outperform both baselines across 7

Preprint. Under review.

every dimension, with the largest gains in scientific grounding and consistent improvements in executability and utility. These results indicate that distilled trajectory conditioning creates higher-quality paper artifacts and repositories than either unstructured inputs or zero-shot planning - and further demonstrates that the Target approach is again the most reasonable for producing the highest quality research output, as compared to the Infer approach. 3.5

Further Analysis of Sensemaking-based Research Trajectories

Sensemaking Stage Distributions. We examTarget ine trajectories by determining how much time ging ora Infer F they spend in each of the stages from Table 1. ox b e Figure 8 presents the per-stage annotation freSho quencies across the eight stages of the sensemakema Sch is ing principles, averaged over 1K generated rees oth search trajectories from each of S CI S ENSE-LMHyp ion t 32B-Target and S CI S ENSE-LM-32B-Infer. Overall, ora Elab ing Target plans concentrate effort in the constructive n stio middle stages: Schema (+4.2% over Infer) and Que ing Foraging (+3.7%), consistent with the backwardam Refr tion reconstruction objective, which rewards building a and filling coherent representational structures. resent P Elaboration also favours Target (+1.7%). Infer 0% 5% 10% 15% 20% 25% Annotation frequency (%) plans allocate more to Hypothesis (+4.4%) and Presentation (+2.1%), reflecting the open-ended ideation setting where the model must generate Figure 8: Per-stage annotation frequenand interrogate claims without a known target. cies across the eight sensemaking stages for Target and Infer trajectories. Case Study. As shown in Figure 9, we compare Target and Infer trajectories generated from the same face-attribute citation neighborhood, revealing distinct patterns: the Target trajectory is more schema-first and artifact-grounded, whereas the Infer trajectory begins with a high-level hypothesis and explores a broader, more speculative design space. These findings indicate that the two supervision regimes lead to different reasoning patterns: Target emphasizes early grounding in concrete datasets, constraints, and implementation details, whereas Infer fosters hypothesis-driven exploration across a broader, more variable design space before settling on a concrete formulation. Infer (forward / open ideation)

Target (reconstructive / paper-aligned) §01 — Problem Definition & Scope Foraging Rigorously define face attribute prediction under strict computational constraints, targeting real-world deployability. Key criteria: quantifiable efficiency metrics, prioritised over broader facial-analysis tasks.

§01

—

Hypothesised Breakthrough A unified multi-task deep learning framework for facial attribute recognition in unconstrained environments, addressing attribute heterogeneity and label imbalance. Hypothesis

Schema #01 Novelty: mixed objective optimisation with adaptive loss re-weighting for joint learning without compromising rare-label performance.

Shoebox #01 Define attribute set and source dataset as CelebA (40 attributes: gender, age, smile, glasses, beard, bald). Ensures comparability with the 91.24% accuracy claim.

Elaboration Architecture builds on a deep CNN backbone — Inception-style or ResNeXt — with shared and task-specific feature pathways. Foraging Analyse facial attribute types by semantic scope (local vs. global) and distribution (balanced vs. imbalanced) to define task groups and design a modular network.

Quantify constraints: <0.5M params (25× reduction), FLOPs <50M, memory >87% reduction. Concrete, falsifiable goals from the paper’s claims. Schema

Schema #03 Slim Module: Input → DW Conv (k×k) → BN → ReLU → PW Conv (1×1) → BN → ReLU → Output. Mirrors the paper’s description.

Questioning #04 Open Question: How to optimally partition the network for maximal cross-task transfer without negative interference?

Knowledge distillation: train ResNet50 teacher on CelebA, then train Slim-Net student with cross-entropy + KL-divergence loss. Transfers knowledge from powerful but inefficient model. Elaboration

Schema CNN backbone with Inception or ResNeXt blocks. Mixed objective loss with domain-adaptive re-weighting.

Figure 9: Comparison of the Target and Infer trajectories for the same citation. 8

Preprint. Under review.

Target

Infer

§01 — Evaluation & Extension Elaboration Evaluate word alignment quality against IBM Model 5 using standard alignment metrics. This preserves the paper-aligned validation target.

§01 — Evaluation & Validation Elaboration Run ablation studies that isolate nonisomorphic mappings from isomorphic baselines. This makes the contribution of structural flexibility directly testable.

Questioning #01 Also measure BLEU and human evaluation. Alignment quality alone is not enough to establish translation quality.

Shoebox #01 Evaluation depends on human annotators and parallel corpora with aligned references. These are the critical external resources for credible validation. Schema Treat local distortion as the central explanatory variable, rather than folding all gains into a single end-to-end score. This turns evaluation into a falsifiable account of what actually helps.

Ablate the contribution of reordering, insertion, and lexical translation operations. This checks whether each stochastic operation is actually necessary. Reframe Move beyond the original setup by integrating a modern neural language model for target-string scoring. The extension directly addresses fluency limitations. Elaboration

Questioning #02 Open question: BLEU may not track human judgment. Metric mismatch can hide real translation quality failures. Reframe If metric mismatch appears, switch to minimum error rate training so optimisation is pulled toward the evaluation target. This reframes the pipeline around the observed bottleneck. Presentation Once the system stabilises, automatic and human evaluation can run in parallel. The trajectory ends in a deployable validation workflow.

Questioning #02 Test low-resource, structurally distant language pairs. Generalisability remains uncertain without this stress test. Reframe Add target-side parse structure to form a bidirectional tree-transduction system. This recasts the model as a stronger syntactic generator.

Target

Infer §01 — Problem Statement & Motivation Hypothesis Prioritise a lexicon-free, end-to-end CTC-based neural ASR framework. The conjectured breakthrough is that transcription can be simplified without hand-built lexical machinery. Schema Prefer this over lexicon-based pipelines because it removes brittle components and improves adaptability. The architecture is organised around robustness rather than pipeline modularity.

§01 — Deployment, Ethics & Validation Hypothesis Extend the original documentation framework into software documentation, treating the method as a broader NLG system. This turns a narrow application into a generalisable hypothesis. Elaboration Validate generated descriptions with developers to ensure technical correctness. The trajectory ties usefulness to expert-facing evaluation. Questioning #01 Developer review is necessary because software documentation has domain-specific conventions and code examples. Validation must respect the target domain’s specialised norms.

Questioning #01 Trade-off: higher initial training complexity is accepted in exchange for robustness and lower maintenance. This is the key open cost of the proposed direction.

Reframe Broaden the agenda to bias, transparency, and scalability, moving beyond the original paper’s immediate scope. This reframes the project around deployment realities. Foraging Justify these additions through modern NLP ethics and known computational constraints. The shift is motivated by both social and systems-level evidence.

Elaboration Make OOV handling a primary objective so novel words can be transcribed without a fixed lexicon. This converts a known failure mode into a concrete design requirement. Foraging Use prior evidence on dynamic-domain ASR to justify eliminating the lexicon altogether. The proposal is grounded in earlier observations that lexicon-free systems adapt more readily.

Questioning #02 Unchecked bias or opacity can undermine trust at deployment. Practical adoption depends on more than raw generation quality.

Figure 10: Four sensemaking-annotated excerpts derived from unpaired research trajectories.

4

Related Work

We make a systemic comparison of S CI S ENSE to prior framework in Table 4. Scientific Research Agents Web-enabled “deep research” agents—Tongyi DeepResearch (Team et al., 2025b), DR Tulu (Shao et al., 2025), and TTD-DR (Han et al., 2025)—excel at end-to-end report generation but operate as black boxes, making it unclear which part 9

Preprint. Under review.

Framework

Sensemaking Cit.-Aware Scale. Data Down. Eval. Open Src.

AutoScience (Autoscience Institute, 2025) AI Scientist (Lu et al., 2024) ResearchAgent (Baek et al., 2025) SciAgent (Ma et al., 2024)

✗ ✗ ▲ ▲

✗ ▲ ▲ ✗

▲ ✗ ✗ ✗

✓ ✓ ✓ ✓

✗ ✓ ✓ ✓

S CI S ENSE (Ours)

✓

✓

✓

✓

✓

Table 4: Comparison of existing scientific research frameworks against S CI S ENSE. ✓= full support; ▲= partial support; ✗ = not supported. of the research loop improves and why. Towards autonomous scientific innovation, AIR ESEARCHER (Tang et al., 2025) and Google’s AI CO - SCIENTIST (Gottweis et al., 2025) target hypothesis generation and literature-to-writing pipelines, while Cheng et al. (2025) argue that closed-loop optimization requires strong verifiers. Our work is complementary: rather than automating the full pipeline, we specifically operationalize the ideation stage through structured, citation-grounded trajectories, providing a controlled supervision signal these systems lack. Sensemaking and Research Evaluation Existing benchmarks measure research abilities in realistic settings—L IVE R ESEARCH B ENCH (Wang et al., 2025) targets coverage and citation quality, D EEP S CHOLAR -B ENCH (Patel et al., 2025) emphasizes nugget coverage and attribution, and AS TA B ENCH (Bragg et al., 2025) spans literature understanding through end-to-end workflows—but typically entangle multiple skills, making it hard to isolate procedural planning. Our framework factors sensemaking into a single controllable primitive: given only cited works, generate a trajectory that either reconstructs how citations led to the target paper or proposes grounded forward directions, enabling clean comparisons across supervision regimes. Work on MLE-bench trajectories (Chan et al., 2025) finds that broader initial exploration correlates with stronger downstream outcomes—a finding we both confirm and complicate, showing that structured reconstruction produces more diverse outputs than open-ended ideation.

5

Conclusion

We introduced citation-conditioned research trajectories as a controllable primitive for studying scientific ideation in LLMs, isolating the sensemaking stage that precedes implementation and enabling direct comparison between open-ended exploration and structured reconstruction. Our central finding is counterintuitive: models trained on constrained, targeted-reconstruction trajectories produce outputs that are more diverse and more novel than those trained explicitly for open-ended exploration—an advantage that propagates downstream into stronger execution success, tighter plan–artifact alignment, and higher judged quality. We attribute this to the structure of the training signal itself. Each Target example is anchored to a distinct paper whose contribution lies at some displacement from its citation neighbourhood, causing models to internalise a distribution over diverse attractor points and generate trajectories that move toward distinctive contributions. Infer training, by contrast, anchors every example to a single teacher model’s generative prior, amplifying structural regularities through distillation rather than capturing the heterogeneity of real research. Looking forward, we hope this work encourages the community to treat ideation—not just execution—as a first-class target for evaluation and improvement, and serves as a step toward agentic systems capable of producing genuinely novel and impactful research.

Ethics Statement This work uses publicly available metadata and abstracts from the Semantic Scholar Open Research Corpus (S2ORC) (Lo et al., 2020), which is released under the Semantic Scholar Dataset License Agreement. No private, proprietary, or personally identifiable data was 10

Preprint. Under review.

collected or used. All trajectory generation was performed using publicly available LLMs, and no human subjects were involved beyond the voluntary participation of graduatestudent co-authors in the annotation study described in § 3.2. Generating 120K sensemaking trajectories with a 235B-parameter teacher model, followed by fine-tuning eight model families and running RL post-training, incurs non-trivial computational cost and associated energy consumption. We mitigate this by training each SFT model for only one epoch and restricting RL post-training to the two strongest checkpoints. We release S CI S ENSE-Traj, S CI S ENSE-LMweights, and all prompt templates to support reproducibility and downstream research. While sensemaking-based trajectories are designed to improve the quality of AI-assisted scientific ideation, we acknowledge that such tools could be misused to generate plausible-sounding but unverified research proposals at scale. We encourage users to treat generated trajectories as starting points for human-guided research rather than as finished scientific contributions.

Reproducibility Statement We take several steps to ensure reproducibility. All prompt templates used for trajectory generation (Target, Infer) and evaluation are provided in Appendix A and in the supplementary code repository. The teacher model used for dataset construction is Qwen3-235B-A22B-Instruct-2507 (Yang et al., 2025), and the LLM-as-judge is Qwen3.5-35B-A3B-FP8. Full model configurations, including all backbone variants, hyperparameters (learning rate 10−5 , Adam optimizer, cosine schedule, one epoch, ∼16K token context length). RL post-training details, including the GRPO configuration, reward model setup, and infrastructure layout, are provided in Appendix E. The S CI S ENSE-Traj dataset (120K trajectories with matched train/validation/test splits), S CI S ENSE-LMmodel weights, and evaluation code are available at the anonymous repository.2

Acknowledgments This material is based upon work supported by the National Science Foundation (NSF) and the National Artificial Intelligence Research Resource (NAIRR) under Grant No. NAIRR240308. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF or NAIRR.

References AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/ main/MODEL CARD.md. Autoscience Institute. Carl: Automated research synthesis, 2025. URL https://www. autoscience.ai/carl. Accessed: 2026-03-29. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6709–6738, 2025. Jonathan Bragg, Mike D’Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi, Sergey Feldman, Dany Haddad, Jena D. Hwang, Peter Jansen, Varsha Kishore, Bodhisattwa Prasad Majumder, Aakanksha Naik, Sigal Rahamimov, Kyle Richardson, Amanpreet Singh, Harshit Surana, Aryeh Tiktinsky, Rosni Vasu, Guy Wiener, Chloe Anastasiades, Stefan Candra, Jason Dunkelberger, Dan Emery, Rob Evans, Malachi Hamada, Regan Huff, Rodney Kinney, Matt Latzke, Jaron Lochner, Ruben Lozano-Aguilera, Cecile Nguyen, 2 https://anonymous.4open.science/r/sciphi-prod-F6B6

11

Preprint. Under review.

Smita Rao, Amber Tanaka, Brooke Vlahos, Peter Clark, Doug Downey, Yoav Goldberg, Ashish Sabharwal, and Daniel S. Weld. Astabench: Rigorous benchmarking of ai agents with a scientific research suite, 2025. URL https://arxiv.org/abs/2510.21652. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2025. URL https://arxiv.org/abs/2410.07095. Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya Agrawal, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica. Barbarians at the gate: How ai is upending systems research, 2025. URL https://arxiv.org/abs/2510.06189. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R D Costa, José R Penadés, Gary Peltz, Yunhan Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Towards an ai co-scientist, 2025. URL https://arxiv.org/abs/2502. 18864. Rujun Han, Yanfei Chen, Zoey CuiZhu, Lesly Miculicich, Guan Sun, Yuanjun Bi, Weiming Wen, Hui Wan, Chunfeng Wen, Solène Maı̂tre, George Lee, Vishy Tirumalashetty, Emily Xue, Zizhao Zhang, Salem Haykal, Burak Gokturk, Tomas Pfister, and Chen-Yu Lee. Deep researcher with test-time diffusion, 2025. URL https://arxiv.org/abs/2507.16075. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–4983, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.447. URL https://aclanthology.org/2020. acl-main.447/. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024. Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, Aixin Sun, Hany Awadalla, and Weizhu Chen. Sciagent: Tool-augmented language models for scientific reasoning, 2024. URL https://arxiv.org/abs/2402.11451. Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, and Carlos Guestrin. Deepscholar-bench: A live benchmark and automated evaluation for generative research synthesis, 2025. URL https://arxiv.org/abs/2508.20033. Peter Pirolli and Stuart Card. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, volume 5, pp. 2–4. McLean, VA, USA, 2005. Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning. arXiv preprint arXiv:2504.17192, 2025. Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, and Pang Wei Koh. Dr tulu: Reinforcement learning with evolving rubrics for deep research, 2025. URL https://arxiv.org/abs/2511. 19399. 12

Preprint. Under review.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. Ai-researcher: Autonomous scientific innovation, 2025. URL https://arxiv.org/abs/2505.18705. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025a. URL https://arxiv.org/abs/2503.19786. Tongyi DeepResearch Team, Baixuan Li, Bo Zhang, Dingchu Zhang, Fei Huang, Guangyu Li, Guoxin Chen, Huifeng Yin, Jialong Wu, Jingren Zhou, Kuan Li, Liangcai Su, Litu Ou, Liwen Zhang, Pengjun Xie, Rui Ye, Wenbiao Yin, Xinmiao Yu, Xinyu Wang, Xixi Wu, Xuanzhong Chen, Yida Zhao, Zhen Zhang, Zhengwei Tao, Zhongwang Zhang, Zile Qiao, Chenxi Wang, Donglei Yu, Gang Fu, Haiyang Shen, Jiayin Yang, Jun Lin, Junkai Zhang, Kui Zeng, Li Yang, Hailong Yin, Maojia Song, Ming Yan, Minpeng Liao, Peng Xia, Qian Xiao, Rui Min, Ruixue Ding, Runnan Fang, Shaowei Chen, Shen Huang, Shihang Wang, Shihao Cai, Weizhou Shen, Xiaobin Wang, Xin Guan, Xinyu Geng, Yingcheng Shi, Yuning Wu, Zhuo Chen, Zijian Li, and Yong Jiang. Tongyi deepresearch technical report, 2025b. URL https://arxiv.org/abs/2510.24701. 13

Preprint. Under review.

Jiayu Wang, Yifei Ming, Riya Dulepet, Qinglin Chen, Austin Xu, Zixuan Ke, Frederic Sala, Aws Albarghouthi, Caiming Xiong, and Shafiq Joty. Liveresearchbench: A live benchmark for user-centric deep research in the wild, 2025. URL https://arxiv.org/abs/2510.14240. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL https://arxiv.org/abs/2503.14476.

14

Preprint. Under review.

A

Citation Neighborhood Pipeline Details

We illustrate the full multi-stage pipeline: 1) extracting citation neighborhoods from S2ORC; 2) summarizing individual papers; 3) generating target and inferred research trajectories; 4) enforcing structured outputs via prompting constraints. A.1

YAML prompt template libraries

We package our prompt text as YAML template libraries to keep the pipeline reproducible and easy to audit: prompt wording is versioned separately from orchestration code, and each stage calls a named template with explicit inputs. Listing 1 (analysis prompts.yaml) contains templates for structured paper- and citation-network analysis (e.g., role-based paper summaries, cluster profiling, and gap finding). Listing 2 (insight prompts.yaml) contains templates for cross-cluster synthesis, prompting the model to generate “crosspollination” research ideas by connecting complementary clusters and transferring methods across areas. Listing 1: YAML prompt set for paper- and citation-network analysis (e.g., cluster profiling, gap finding). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

paper_analysis: foundational: template: | You are reviewing a foundational paper in its field. Title: {title} Abstract: {abstract} Known citations: {citing_papers} Citation count: {citation_count} Provide a concise expert summary covering: - Why this work shaped the field - Which assumptions or design choices enabled its impact - What elements no longer hold bridge: template: | Analyze this bridge paper that connects disparate subfields. Title: {title} Abstract: {abstract} Influenced papers: {citing_papers} Describe: - The conceptual translation it performs - How it reframes terminology or methods - What tensions remain unresolved frontier: template: | Summarize this frontier paper. Title: {title} Abstract: {abstract} Early citations: {citing_papers} Explain: - The emerging problem it addresses - How it builds on recent momentum - Where validation is still thin method: template: | Examine this methodological paper. Title: {title} Abstract: {abstract} Downstream users: {citing_papers} Detail: - The core methodological advance

15

Preprint. Under review.

45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109

- Typical contexts where it excels - Failure modes or assumptions to test supporting: template: | Provide a brief expert summary of the following paper. Title: {title} Abstract: {abstract} Focus on: - Central claim - Evidence cited - Key dependencies paper_contributions: foundational: template: | Distill the lasting contributions of this foundational work. Title: {title} Built upon: {building_papers} Builds on: {backbone_papers} Highlight: - Breakthrough idea - Enduring techniques - Concepts others adapted bridge: template: | Identify the synthesis that this bridge paper enabled. Title: {title} Builds on: {backbone_papers} Follow-on work: {building_papers} Capture: - Cross-domain translation - Tooling or frameworks introduced - How it changed the direction of citing papers frontier: template: | Extract the contributions of this frontier paper. Title: {title} Builds on: {backbone_papers} Describe: - Novel problem framing - Prototype methods or datasets - Hypotheses pending validation method: template: | Summarize the methodological toolkit from this paper. Title: {title} Builds on: {backbone_papers} Downstream use: {building_papers} Include: - Core algorithm or protocol - Required inputs and outputs - Extensions that citing papers added supporting: template: | Note the supporting contributions of this paper. Title: {title} Builds on: {backbone_papers} Address: - Specific niche it serves

16

Preprint. Under review.

110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174

- How it reinforces accepted results - Any incremental advances paper_gaps: foundational: template: | Examine blind spots of this foundational paper. Title: {title} Abstract: {abstract} Identify: - Assumptions that deserve re-testing - Datasets or contexts it skipped - Technical bottlenecks that still persist bridge: template: | Surface the limitations of this bridge paper. Title: {title} Abstract: {abstract} Cover: - Concepts that were mistranslated - Areas the bridge failed to integrate - Experiments or evaluations still missing frontier: template: | Flag the uncertainties within this frontier paper. Title: {title} Abstract: {abstract} Discuss: - Uncontrolled variables - Dependencies on unstable baselines - Steps needed for broad adoption method: template: | Analyze the weak spots of this methodological work. Title: {title} Abstract: {abstract} Outline: - Scalability constraints - Sensitivity to hyperparameters or data - Generalization risks supporting: template: | Note gaps or follow-up questions raised. Title: {title} Abstract: {abstract} Include: - Underexplored use cases - Missing comparisons - Simplifying assumptions that may fail network_insights: cluster_profile: template: | Assess research cluster {cluster_id}. Representative papers: {paper_titles} Key exemplars: {representative_papers} Timeline: {timeline} Provide: - Dominant questions

17

Preprint. Under review.

175 176 177 178 179 180 181 182 183 184 185 186

- Shared methodologies - Internal disagreements - Signals of maturity or stagnation gap_analysis: template: | Detect research gaps for cluster {cluster_id}. Core papers: {paper_titles} Report: - Methods not yet cross-applied - Contradictory findings to reconcile - High-value datasets or benchmarks still missing

Listing 2: YAML prompt set for cross-cluster synthesis and generating “cross-pollination” research ideas. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

A.2

cross_pollination: novel_connections: template: | You have a citation network with clusters grouped by role: {major_clusters} Temporal activity profile: {temporal_flow} Produce novel research directions by: - Linking complementary clusters - Transferring mature methods to rising areas - Challenging dominant assumptions Return bullets that can inspire concrete projects.

Prompt Structures

We use controlled prompt templates to elicit consistent, comparable outputs from the teacher model across ideation modes. Listing 3 shows the inferred-planning prompt: given only citedpaper synopses (with the target text withheld), the model must hypothesize a plausible breakthrough and produce a staged research program, grounding each inference with inline paper-ID citations. Listing 4 shows the complementary targeted-reconstruction prompt: given a prescribed section outline and target synopsis, the model generates an executable trajectory with concrete checkpoints paired with meta-level rationales, enabling a faithful reconstruction-style trajectory that remains auditable and systematically comparable to the inferred setting. Listing 3: Inferred prompt: generate a hypothesized breakthrough + research program from cited synopses only. 1

2 3 4 5 6 7 8 9

You are an AI research strategist tasked with inferring the most probable and highimpact research plan based solely on the cited works provided (the target paper text is unavailable). Produce a two-part Markdown document that hypothesizes what the target contribution could be and charts a credible path to deliver it. Output format: 1. A section titled ``Hypothesized Breakthrough'' (rendered as `## Hypothesized Breakthrough`) that summarizes: - The likely research objective and novelty inferred from the cited corpus. - The expected system / method architecture. - The anticipated evaluation setting and success criteria. Include inline references to the relevant cited papers using [[paper_id]] to justify each inference. 2. A section titled ``High-Impact Research Program'' (rendered as `## High-Impact Research Program`) with subsections for Idea Formulation, Modeling,

18

Preprint. Under review.

10 11 12 13 14

15 16 17 18 19 20 21 22

Experimentation, Validation, and Analysis (render each as `###` headings). For each subsection: - Articulate specific actions, checkpoints, and decision gates that synthesize insights from the cited works. - Include meta-level rationales that explain why the step is essential or risky, citing the supporting papers with [[paper_id]]. - Highlight any contingency plans or open questions implied by the cited works. Citation rule: every time you lean on a cited paper for evidence, design inspirations, cautions, or evaluation criteria, cite it inline using [[paper_id ]]. Example snippet (do not reuse verbatim): - Hypothesized Objective: Unify latent dense retrieval with multi-hop reasoning to handle knowledge gaps in open QA [[173990818]], [[52822214]]. Cited paper synopses: {cited_synopses} Produce the requested Markdown now.

Listing 4: Targeted-reconstruction prompt: turn the target outline/synopsis into an executable trajectory with checkpoints + meta rationales. 1

2 3 4 5 6 7

8

9

10 11 12 13

14 15 16

17 18 19 20 21

You are an AI research strategist tasked with turning the target paper into a reproducible, high-agency research program. Follow the section outline supplied below and write a single Markdown document that blends execution guidance with meta-level reasoning. Section outline (use these headings verbatim, in order, expanding each with the requested depth): {structure_plan} For every section and subsection: - Begin with a short paragraph that states the intent of that stage and the key decision criteria, and briefly explain why these criteria were prioritized instead of obvious alternatives. Cite the motivating papers using [[paper_id]]. - Follow with 1-3 bullet points detailing concrete actions or checkpoints (the implementation blueprint), each paired with a ``Meta rationale''' sentence that exposes the authors' or your inferred reasoning (e.g., trade-offs, precedent, or risk mitigation) and cites the relevant papers. - Where a decision depends directly on the target paper, describe the rationale in prose (e.g., ``the canonical pipeline prioritizes retrieval transparency'') and contrast it with at least one cited work using [[paper_id]]. - Surface assumptions, dependencies, and fallback options for the step; note any conditions that would cause you to revisit the decision. Global requirements: - Interpret each top-level bullet from the outline as an H2 heading (`## {{title}}`) and each nested bullet as an H3 heading (`### {{title}}`). Do not reproduce the bullet list; use Markdown headings only. - Keep the tone analytic and directive; make it easy for a research team to follow without losing the ``why.'' - Cite external papers using [[paper_id]]; do not emit [[target]] or other placeholders for the target work-reference it descriptively instead. - Highlight potential parallelization or sequencing constraints when relevant (e.g., checkpoints that can run concurrently versus those gated by data availability). - Never mention the target paper's title, authors, venue, or publication year; refer to it only implicitly. Context for reference: - Target synopsis (for reference only; do not restate metadata): {target_synopsis} - Cited paper synopses: {cited_synopses}

19

Preprint. Under review.

22 23

Produce only the Markdown document that follows the prescribed section structure.

B

Citation Statistics

A central requirement of citation-conditioned research is that models must reason over prior work in a disciplined and appropriate manner. We therefore evaluate citation behavior along two axes: structural similarity to the teacher trajectory and semantic appropriateness of citation usage. Citation Similarity Metrics We first measure how closely the student trajectory matches the citation structure of the teacher trajectory. For each generated trajectory, we parse cited paper identifiers and compute: (i) the L1 absolute difference in total citation count between P(T ) and P(S) ; (ii) the L1 difference in the number of unique cited papers; (iii) the Jaccard distance between the two citation sets: JaccardDistance(C (T ) , C (S) ) = 1 −

|C ( T ) ∩ C (S) | . |C ( T ) ∪ C (S) |

These metrics quantify whether student models attend to a similar body of evidence as the teacher when constructing research trajectories. Section to contain larger evaluation results across many different models (trained and untrained) Checkpoint Lenchar (Inf) Lenchar (Tgt) Cites (Inf) Cites (Tgt) Uniq (Inf) Uniq (Tgt) Rep2 (Inf) Rep2 (Tgt) Rep16 (Inf) Rep16 (Tgt) Q32-Target Q32-Infer Q32-Both

13919.258 11016.898 11303.008

14027.972 11760.246 12319.485

29.086 36.180 37.023

28.082 35.187 32.037

5.906 5.977 5.914

5.515 5.970 5.522

0.240 0.188 0.190

0.247 0.202 0.234

0.039 0.007 0.012

0.041 0.012 0.027

Q4-Target Q4-I-Infer Q4-I-Both

16723.875 12327.438 14708.986

15216.052 16914.440 16173.500

36.961 52.703 51.672

34.022 53.284 48.052

9.641 5.977 6.031

9.082 6.082 5.828

0.350 0.242 0.281

0.348 0.307 0.314

0.132 0.060 0.075

0.107 0.098 0.082

Q303-Target Q303-I-Infer Q303-I-Both Q303-Infer Q303-Both Q303-None

16464.223 12413.791 15064.660 15419.668 15562.858 15285.785

15427.216 16871.356 16095.015 15658.179 15575.903 15252.127

42.962 55.633 54.287 44.094 43.344 37.438

39.649 55.485 50.694 43.037 41.918 35.649

8.156 6.000 6.164 7.352 7.180 6.875

7.761 5.985 5.940 7.343 7.045 6.985

0.323 0.250 0.286 0.292 0.291 0.271

0.312 0.309 0.318 0.297 0.300 0.286

0.097 0.059 0.074 0.071 0.073 0.056

0.080 0.100 0.083 0.068 0.070 0.060

Q4-Infer Q4-Both Q4-None

16608.129 16721.926 16419.836

15974.329 15922.202 15606.097

42.180 41.625 37.305

39.724 40.157 35.590

7.531 7.398 7.055

7.321 7.269 7.052

0.306 0.304 0.288

0.293 0.296 0.283

0.074 0.074 0.062

0.066 0.069 0.060

Table 5: Generation statistics for inferred vs. target ideations. We report average generation length (chars), citation usage (total/unique), and repetition proportions (2-gram and 16gram). Cell color indicates per-column magnitude (darker = larger). Table 5 summarizes how our different SFT formatting/generation regimes trade off length, citation density, and repetition. Across checkpoints, the I-Infer setting tends to allocate a larger fraction of the budget to citations (higher # citations and comparable or slightly higher unique citations) while remaining relatively concise in total characters. In contrast, Target generations are typically longer in raw character count but do not consistently increase citation coverage, suggesting that additional non-citation context dominates the extra budget. The repetition metrics (Rep2 and Rep16 ) generally increase with longer outputs, indicating that expanding the character budget can add redundant text rather than new evidence.

C

SFT Ablations

We ablate two SFT formatting knobs: the amount of prepended context (character budget) and the maximum number of citations retained. Figure 11 shows that expanding the character budget helps early but quickly plateaus when citation capacity is already high. In contrast, Figure 12 shows that, under a fixed 5k character budget, increasing the citation cap continues to reduce validation loss. Together, these trends justify allocating limited context to citations and pruning non-citation characters. 20

Preprint. Under review.

Figure 11: SFT ablation: character budget (max citations=100). Increasing the prepended character budget reduces validation loss initially, then saturates, indicating diminishing returns from additional non-citation context.

D

Downstream Research Generation Details

We describe the procedure used to generate outputs and evaluation results in Section 3.4. D.1

Downstream Research Artifact Generation

Filtering and Selection of Research Plans. We began from a pool of automatically generated research plans, where each plan belonged to one of four supervision conditions: N ONE, I NFER, TARGET, and B OTH. For each underlying seed instance, we retained only cases for which all four conditions produced corresponding plans, enabling matched qualitative comparison across supervision regimes. We then manually filtered these matched sets using four criteria. First, the proposed research trajectory had to fall within computer science or otherwise be executable entirely through computation, as our evaluation setting does not support physical experimentation or human-subject actuation. Second, the plan had to appear feasible under modest compute constraints, excluding projects that would likely require prohibitively large-scale training or data collection. Third, the core artifacts required by the plan, including datasets, pretrained models, and software dependencies, had to be publicly available. These artifacts were downloaded in advance into a shared local repository so that downstream execution would not depend on open-ended discovery. Fourth, we required that all retained plan sets preserve correspondence across the previous three conditions for the same seed context. The resulting benchmark is therefore intentionally small and manually curated, prioritizing feasibility and controlled comparison over scale. This yielded a total of 5 unique citation sets, with 4 generated trajectories from our distill models from the S CI S ENSE-LM-32B class. Agentic Execution Environment. For each retained instance, we instantiated an autonomous coding workflow in which a coding agent was provided with: (i) an optional single research trajectory, (ii) the cited papers referenced by that trajectory, (iii) local system instructions describing the available infrastructure (e.g., DGX/Slurm usage, filesystem 21

Preprint. Under review.

Figure 12: SFT ablation: citation cap (character budget = 5k). Under a fixed character budget, allowing more citations consistently improves validation loss, motivating our choice to maximize citation retention while pruning surrounding text. layout, environment setup conventions, and artifact locations), and (iv) a specification of the desired outputs, namely a runnable repository, a workshop-style research paper, and brief reproduction instructions. To reduce uncontrolled variance, the agent’s external web-search capabilities were disabled. Instead, it operated only over the supplied plan, cited materials, and local artifacts. Before implementation, the agent was prompted to translate the high-level plan into an executable task list, including hypotheses, required resources, implementation milestones, and evaluation steps. It was then asked to iteratively construct the repository, validate the environment, run a minimal end-to-end experiment, and finally produce a paper summarizing the motivation, method, experiments, limitations, and reproduction details. For paper writing, we used gpt-5.4 xhigh, requesting output as a Markdown file to simplify downstream compilation and inspection. Operational Use of the Research Agent. In practice, each run followed a staged protocol. The agent first summarized the provided plan and cited works into a concise execution brief, identifying the central claim, concrete implementation tasks, expected datasets or models, and likely risks. It next generated a repository scaffold containing environment files, scripts, and an initial README with explicit execution steps. The agent was then instructed to prioritize a “minimum viable research artifact”: a small but functioning pipeline that instantiated the core idea of the plan, rather than attempting premature optimization or overly ambitious reproduction. After obtaining a runnable baseline, the agent was allowed to refine experiments, improve documentation, and draft the accompanying paper. This staged procedure was designed to increase the chance that any benefit of the research plan would appear not only in final prose quality, but also in task decomposition, implementation coherence, and the ability to reach a credible experimental endpoint. Evaluation Scope. Our goal is primarily qualitative: to assess whether higher-quality research plans lead to more coherent autonomous execution and more credible final artifacts. We therefore compare outputs at the level of repository completeness, executability, faithfulness to the proposed plan, experimental plausibility, and paper quality, rather than treating the benchmark as a large-scale statistical evaluation. Because all four plans in a 22

Preprint. Under review.

matched set originate from the same seed context, observed differences can be interpreted as evidence about how the structure and specificity of planning affect downstream agentic research behavior under controlled constraints. D.2

Downstream Artifact Evaluation Details

Artifact bundle. Each artifact bundle contains: (i) the input plan or prompt, (ii) the citedwork context, (iii) the generated repository, (iv) experiment outputs, and (v) the generated paper. Evaluation is performed over the final artifact bundle rather than over plans in isolation. To avoid self-preference bias—since the coding agent and paper-writing model both belong to the GPT family—all three criteria below are scored by an LLM judge from a different model family, claude-sonnet-4-6 (Anthropic), operating over the full bundle. Each artifact is scored ten times per rubric with randomized presentation order, and the mean is taken. All sub-scores are produced on a 1–5 Likert scale and linearly rescaled to [0, 1] before aggregation. A. Executability. The executability judge receives the repository contents, any logged execution traces, and the reproduction instructions. It scores four dimensions, each on a 1–5 scale: 1. Environment setup: Are dependencies specified, and could the environment be reproduced from the provided files (e.g., requirements.txt, Dockerfiles)? 2. Data preparation: Is the data-loading or preprocessing pipeline present, and does it reference accessible artifacts? 3. Experiment execution: Do the core experiment scripts run to completion (as evidenced by logs or saved outputs), and do they implement the methodology described in the plan? 4. Paper generation: Is a compiled or compilable paper produced, and does it contain sections covering motivation, method, results, and limitations? The executability score is the mean of the four rescaled dimension scores: E XEC =

1 4 ŝi . 4 i∑ =1

B. Scientific Grounding. The grounding judge receives the plan, the cited papers, the repository, and the generated paper. It scores two complementary aspects: Plan coverage (P LAN C OV): the degree to which the plan’s stated requirements—benchmarks, architectural choices, training procedures, ablations, and evaluation metrics—are realized in the code and paper. The judge assigns a single 1–5 score reflecting the fraction of plan elements that are meaningfully addressed. Citation coverage (C ITE C OV): the degree to which the cited works are (a) correctly referenced and (b) substantively used to motivate design choices, baselines, or experimental comparisons. The judge assigns a single 1–5 score capturing both resolution (are the cited works identifiable in the text?) and usage (do they inform the methodology or discussion beyond superficial mention?). The grounding score is: G ROUND =

1 \ 1 \ P LAN C OV + C ITE C OV, 2 2

each ∈ [0, 1].

C. Downstream Utility. Downstream utility combines benchmark performance with holistic quality judgments over the repository and paper. Experimental quality (B ENCH N ORM): Because different plans within the same cited-work set may target different benchmarks or evaluation strategies, we do not require a direct 23

Preprint. Under review.

Family Qwen3

Gemma-3 Llama 3

Model variant Qwen3-4B Qwen3-4B-Instruct-2507 Qwen3-32B Qwen3-30B-A3B-Instruct-2507 Gemma-3-4b-it Gemma-3-27b-it Llama3.2-3B-Instruct Llama3.1-70B-Instruct

Params/Active 4B/4B 4B/4B 32B/32B 30B/3B 4B/4B 27B/27B 3B/3B 3B/3B

Training split Target / Infer / Both Target / Infer / Both Target / Infer / Both Target / Infer / Both Target / Infer / Both Target / Infer / Both Target / Infer / Both Target / Infer / Both

Table 6: Model configurations used for S CI S ENSE-LMfine-tuning. In all cases we train for one epoch, we use the Adam Optimizer, an initial learning rate of 0.00001 with a cosine LR schedule, the global batch size is varied based on the parallelism required for the models (all models below except for the Qwen3-4B* models below use a batch size of 1). There is no weight decay, and the maximum context length is 16K tokens. All models are trained with full Supervised Fine Tuning. numeric comparison across artifacts. Instead, the judge is shown the artifact’s reported experimental results and, when available, the upper-bound artifact’s results for the same set. It assigns a 1–5 score reflecting the rigor, plausibility, and informativeness of the reported experiments—considering whether appropriate benchmarks were chosen, whether the evaluation methodology is sound, and whether the results credibly support the artifact’s claims. When artifacts do share a common benchmark, the judge additionally considers relative performance against the upper bound. Repository quality (R EPO J UDGE): The judge scores technical coherence, reproducibility, adequacy of experiments, and faithfulness to the plan on a 1–5 scale. Paper quality (PAPER J UDGE): The judge scores scientific clarity, evidence-grounded claims, citation use, and overall workshop-paper quality on a 1–5 scale. The utility score is:

\ \ \ U TILITY = 0.50 B ENCH N ORM + 0.25 R EPO J UDGE + 0.25 PAPER J UDGE,

each ∈ [0, 1].

LLM-as-Judge protocol. All sub-scores across the three criteria are produced by claude-sonnet-4-6 (Anthropic), chosen to be from a different model family than the generation pipeline to mitigate self-evaluation bias. Each artifact bundle is evaluated three times per rubric under independent randomized ordering of artifacts within a cited-work set; the reported score is the mean across the ten runs. Judges are provided with detailed rubric descriptions and anchor examples for each point on the 1–5 scale to reduce variance. To mitigate positional bias, the order in which competing artifacts appear in the prompt is shuffled across runs. Overall score.

The final score is: O VERALL = 0.35 E XEC + 0.30 G ROUND + 0.35 U TILITY.

We weight executability and utility slightly more heavily than grounding because our central claim is that planning enables better downstream scientific artifacts, not only more structured intermediate plans.

E

Reinforcement Learning Post-Trained S CI S ENSE-LM

While SFT teaches models to imitate teacher-generated trajectories, it does not directly optimise for the rubric criteria that define trajectory quality. We therefore apply a reinforcement learning (RL) post-training stage to a subset of our best SFT checkpoints, using an LLM-as-judge reward signal aligned with the evaluation rubric described in § G. We use Group Relative Policy Optimization (GRPO; Shao et al., 2024) with the DAPO loss variant (Yu et al., 2025). For each training prompt, the policy model generates G =8 candidate 24

Preprint. Under review.

trajectories (temperature 0.7, top-p 0.95, max 4 096 tokens). Each candidate is then scored by a separate reward model—Qwen3.5-27B-FP8 (Yang et al., 2025)—that acts as an expert scientific reviewer. The reward model evaluates every candidate against the same 8-item rubric used for final evaluation (Novelty, Significance, Grounding, Soundness, Methodology, Feasibility, Sensemaking, Clarity; each scored 1–5). To keep scoring reliable within the reward model’s context budget, rubric items are evaluated in three multi-turn groups of 3+3+2 items; each group’s scores are parsed from structured XML tags (<score N>), and the final reward is the mean across all eight items. Rewards are normalised at the group level (scale rewards=group), and the KL penalty coefficient is set to β=0.001. Training uses a learning rate of 5×10−6 with a cosine schedule (5% warmup), gradient checkpointing, and bf16 mixed precision. Four vLLM reward-model servers (each using tensor parallelism across 8 A100-80GB GPUs) serve the judge model; reward requests are load-balanced across servers via round-robin and executed asynchronously (asyncio.gather over all G candidates in parallel). Training runs on 4 additional nodes (32 GPUs total) using DDP via HuggingFace Accelerate. We apply RL post-training to the Both SFT checkpoints for Qwen3-32B and Qwen3-4B-Instruct, as these showed the strongest combined quality– diversity profiles after SFT.

F

Diversity Comparisons

We present a set of comparisons relying on different measures of diversity in text outputs. Table 7 demonstrates this overperformance across different model families and sizes. The sections below describe the approaches in detail. For all below experiments, we present each model with a fixed citation set, then have it generate five separate research plans. This process is repeated for 100 total citation sets from our test data set. Embedding-based diversity. To measure semantic similarity among repeated samples for the same prompt, we embedded each full generation using the long-context model jinaai/jina-embeddings-v3. When necessary, outputs were chunked to satisfy the model context limit and chunk embeddings were mean-pooled to obtain a single vector per generation. Within each bundle of five sampled generations, embeddings were ℓ2 -normalized and all pairwise cosine similarities were computed. Bundle-level similarity was defined as the mean over the ten unordered pairs, Simemb ( B) =

1 ∑ cos(x, y). (52) x,y∈ B x <y

Similarity for model type k was computed by averaging over bundles Gk , and diversity was defined as  1 1  (infer) (k) (overall) (target) Demb = 1 − Simemb ( B), Demb = Demb + Demb . ∑ | Gk | B∈G 2 k

Self-BLEU diversity. As a lexical complement to the embedding analysis, we computed self-BLEU over the same five-sample bundles. For each bundle B, each sample was treated in turn as the hypothesis and the remaining four samples as references. Sentence-level BLEU-4 with uniform n-gram weights was computed and averaged across the five hypothesis choices to obtain a bundle-level similarity score SimBLEU ( B). Diversity was defined analogously by averaging across bundles in category k,  1 1  (infer) (k) (overall) (target) DBLEU = 1 − SimBLEU ( B), DBLEU = DBLEU + DBLEU . ∑ | Gk | B∈G 2 k

Chunked BERTScore diversity. Texts were segmented into sentence-based chunks under a fixed token budget. For a document pair ( x, y), BERTScore F1 was computed for all chunk pairs, yielding a similarity matrix S ∈ Rm×n . Let wi and v j denote chunk-length weights. 25

Preprint. Under review.

Family

Cond. Self-BLEU ↓ Embedding-Based ↓ BERTScore ↑ Sentence Movers ↑

Q303-I

None Infer Both Target

0.2046 0.2301 0.1822 0.1365

0.912 0.846 0.813 0.741

0.3538 0.3692 0.3801 0.3961

0.2666 0.2659 0.2878 0.2929

Q32

None Infer Both Target

0.2440 0.2921 0.2233 0.1563

0.932 0.872 0.842 0.778

0.3480 0.3419 0.3659 0.3907

0.2450 0.2317 0.2661 0.3027

Q4

None Infer Both Target

0.2415 0.2821 0.2202 0.1640

0.938 0.884 0.854 0.796

0.3520 0.3504 0.3745 0.3992

0.2600 0.2549 0.2856 0.3250

Q4-I

None Infer Both Target

0.1830 0.2339 0.2101 0.1496

0.926 0.862 0.828 0.762

0.3582 0.3552 0.3758 0.4049

0.2665 0.2616 0.2838 0.3150

L3.2-3B-I

None Infer Both Target

0.2380 0.2760 0.2170 0.1610

0.939 0.886 0.857 0.799

0.3500 0.3470 0.3720 0.4020

0.2620 0.2550 0.2870 0.3290

None Infer L3.1-70B-I Both Target

0.1980 0.2230 0.1760 0.132

0.912 0.838 0.804 0.736

0.3400 0.3350 0.3600 0.388

0.2380 0.2250 0.2600 0.300

G3-4B-I

None Infer Both Target

0.236 0.271 0.215 0.158

0.937 0.882 0.853 0.794

0.351 0.348 0.373 0.401

0.261 0.253 0.286 0.327

G3-27B-I

None Infer Both Target

0.205 0.236 0.184 0.139

0.918 0.848 0.816 0.748

0.344 0.339 0.363 0.392

0.242 0.229 0.264 0.304

Table 7: Diversity metrics grouped by model family and supervision condition. For each family, bold indicates the condition with the greatest diversity in that column. Columns marked with ↑ assign greater diversity to larger values, while columns marked with ↓ assign greater diversity to smaller values. Directional similarity was defined as ∑m wi max j Sij s ( x → y ) = i =1 m , ∑ i =1 wi

s(y → x ) =

∑nj=1 v j maxi Sij ∑nj=1 v j

and symmetric document similarity as SimBS ( x, y) =

1 s( x → y) + s(y → x ) . 2

For a bundle B, bundle-level similarity was defined as SimBS ( B) =

1

∑ SimBS (x, y).

(52) x,y∈ B x <y

26

,

Preprint. Under review.

Diversity was then computed as (k)

DBS = 1 −

1 SimBS ( B), | Gk | B∑ ∈G

(overall)

DBS

=

k

 1  (infer) (target) DBS + DBS . 2

Sentence Mover’s Similarity diversity. Texts were chunked identically and each chunk was embedded using a sentence embedding model. For a document pair ( x, y), cosine distance between chunk embeddings defined a transport cost matrix C. Let a and b be normalized chunk-length weight vectors. Optimal transport cost was defined as Cost( x, y) =

min ⟨ T, C ⟩,

SimSMS ( x, y) = 1 − Cost( x, y),

T ∈Π( a,b)

where Π( a, b) denotes the set of transport plans with marginals a and b. Bundle-level similarity was computed as SimSMS ( B) =

1

∑ SimSMS (x, y).

(52) x,y∈ B x <y

Diversity was computed analogously, (k)

DSMS = 1 −

1 SimSMS ( B), | Gk | B∑ ∈G

(overall)

DSMS

k

G

=

 1  (infer) (target) DSMS + DSMS . 2

LLM-as-Judge

Table 8 shows the results of LLM-as-Judge by rubric metric across different model families. The LLM-as-Judge Prompt template is as follows: You must evaluate the proposal rigorously, fairly, and transparently, using the detailed rubric below. Your evaluation should reflect expert-level scientific judgment and careful sensemaking, not superficial pattern matching. :contentReference[oaicite:0]index=0 INPUTS. (1) Reference Papers (already published): Each paper is identified by a normalized ID such as [[R1]], [[R2]], [[R3]], etc. You may be given titles, abstracts, or excerpts. (2) Proposal to Evaluate: A Markdown document describing a novel scientific idea and proposed approach. The proposal is REQUIRED to use structured Markdown sections, normalized in-text citations (e.g., [[R1]]), and include a References section mapping citation IDs to full bibliographic entries. YOUR RESPONSIBILITIES. (A) Read and internalize the full rubric before reviewing any reference papers or the proposal. (B) Review the provided reference papers and briefly think about their key claims or contributions using only the information explicitly provided. (C) Review the generated research proposal carefully, evaluating it only relative to the provided reference papers and the rubric criteria. (D) Evaluate the proposal one rubric item at a time, in rubric order (Item 1–Item 8), following these rules: focus on exactly one rubric item per turn; think carefully and rigorously before assigning a score; assign a score from 1 to 5 based strictly on the rubric definitions; output the numeric score surrounded by <score> and </score> tags (e.g., <score>4</score>); the score must appear clearly and unambiguously in each turn. (E) Judge novelty only relative to the provided reference papers and what is explicitly stated in the proposal; if broader novelty is unclear or cannot be determined, state this explicitly. (F) Verify citation integrity: every in-text citation must appear in the References section, and references must not be fabricated, duplicated, or internally inconsistent. (G) Base all judgments strictly on evidence from the proposal text; quote or reference specific sections when helpful; do not infer unstated intentions or assume missing details. PROPOSAL STRUCTURE (FLEXIBLE GUIDELINE). The proposal is expected to be written in structured Markdown and to cover all major intellectual components needed for evaluation. However, the exact sectioning and ordering may vary depending on the nature of the work (e.g., theoretical, empirical, systems,

27

Preprint. Under review.

Model

Nov. Sig. Grnd. Sound. Meth. Feas. Sense. Clar. Overall

Q32 Family Q32-None Q32-Infer Q32-Both Q32-Target

1.66 1.70 1.72 1.79

2.67 2.71 2.73 2.80

1.22 1.38 1.31 1.29

2.98 3.03 3.06 3.08

3.02 3.08 3.13 3.14

2.74 2.79 2.83 2.85

2.01 2.05 2.09 2.11

3.08 3.12 3.19 3.20

2.42 2.48 2.51 2.53

Q303 Family Q303-None Q303-Infer Q303-Both Q303-Target

1.67 1.72 1.77 1.81

2.67 2.72 2.78 2.82

1.28 1.46 1.39 1.34

2.80 2.87 2.92 2.90

2.91 2.97 3.04 3.01

2.65 2.71 2.77 2.78

1.94 2.00 2.05 2.06

2.97 3.03 3.08 3.07

2.36 2.44 2.48 2.47

Q4-I Family Q4-I-None Q4-I-Infer Q4-I-Both Q4-I-Target

1.71 1.76 1.78 1.82

2.70 2.75 2.77 2.81

1.39 1.56 1.48 1.44

2.64 2.71 2.73 2.72

2.82 2.88 2.90 2.89

2.60 2.67 2.68 2.69

1.86 1.91 1.93 1.94

2.88 2.93 2.96 2.95

2.33 2.40 2.40 2.41

Q4 Family Q4-None Q4-Infer Q4-Both Q4-Target

1.60 1.64 1.68 1.70

2.56 2.60 2.66 2.68

1.18 1.34 1.26 1.23

2.60 2.66 2.71 2.72

2.74 2.80 2.85 2.86

2.52 2.57 2.62 2.61

1.79 1.84 1.89 1.88

2.77 2.82 2.88 2.87

2.22 2.28 2.32 2.31

L3.1-70B-I Family L3.1-70B-I-None 1.68 L3.1-70B-I-Infer 1.73 L3.1-70B-I-Both 1.78 L3.1-70B-I-Target 1.82

2.69 2.74 2.79 2.84

1.30 1.47 1.40 1.36

2.88 2.95 3.00 2.98

2.99 3.05 3.10 3.08

2.71 2.77 2.82 2.83

1.98 2.03 2.08 2.09

3.01 3.08 3.14 3.13

2.41 2.48 2.51 2.50

L3.2-3B-I Family L3.2-3B-I-None L3.2-3B-I-Infer L3.2-3B-I-Both L3.2-3B-I-Target

1.61 1.66 1.69 1.72

2.57 2.62 2.67 2.70

1.21 1.37 1.29 1.26

2.63 2.69 2.74 2.75

2.77 2.83 2.88 2.89

2.54 2.59 2.64 2.63

1.81 1.86 1.91 1.90

2.80 2.85 2.91 2.90

2.24 2.31 2.34 2.34

G3-27B-I Family G3-27B-I-None G3-27B-I-Infer G3-27B-I-Both G3-27B-I-Target

1.67 1.72 1.76 1.81

2.68 2.73 2.77 2.83

1.27 1.45 1.38 1.34

2.90 2.97 3.01 2.99

3.00 3.06 3.11 3.09

2.72 2.78 2.82 2.83

1.99 2.04 2.08 2.09

3.03 3.09 3.15 3.14

2.41 2.48 2.51 2.50

G3-4B-I Family G3-4B-I-None G3-4B-I-Infer G3-4B-I-Both G3-4B-I-Target

1.62 1.67 1.70 1.74

2.58 2.64 2.69 2.72

1.23 1.40 1.31 1.28

2.65 2.71 2.76 2.77

2.79 2.85 2.90 2.91

2.56 2.61 2.66 2.65

1.82 1.87 1.92 1.91

2.82 2.87 2.93 2.92

2.26 2.33 2.36 2.36

Table 8: Rubric evaluation scores with overall mean. Bold indicates column-wise maxima within each family.

interdisciplinary). Reviewers should evaluate content coverage, not rigid adherence to headings. A typical proposal may include: Title; one-sentence pitch; background and gap (with citations [[R#]]); core hypothesis or key insight; proposed approach or methodology; evaluation plan (metrics, baselines, datasets, controls); expected outcomes and contributions; risks, limitations, and alternative hypotheses; ethics and responsible research (if applicable); and references (normalized). Variations are acceptable if all necessary elements for rigorous evaluation are present, the structure is coherent and navigable, citations are normalized and correctly mapped, and the scientific argument can be clearly followed. If critical components are missing or unclear, this should negatively affect relevant rubric scores (especially Clarity, Methodological Rigor, and Sensemaking). RUBRIC (Score each criterion from 1–5). Shared scale: 5 = Excellent (clear, rigorous, compelling; minimal gaps); 4 = Strong (solid and persuasive with minor gaps); 3 = Adequate (reasonable but underspecified or uneven); 2 = Weak (substantial flaws or missing elements); 1 = Poor (incorrect, incoherent, or violates requirements). 1) Novelty & Differentiation. Evaluate whether the proposal introduces a genuinely new idea relative to [[R#]]. Consider whether it explicitly compares against prior

28

Preprint. Under review.

work, whether novelty is conceptual, methodological, or contextual, whether it is non-obvious, and whether it clearly articulates what is new. 2) Significance & Potential Contribution. Assess importance and potential impact if successful. Consider whether a clear gap is identified, whether success would advance the field, whether contributions are well-scoped, and whether beneficiaries are identified. 3) Grounding in Prior Work & Citation Integrity. Evaluate engagement with prior literature and correctness of citations. Check that references are accurate, properly formatted, and complete. 4) Conceptual Soundness & Plausibility. Assess internal coherence and plausibility of the core idea. Consider assumptions, logical consistency, distinction between speculation and fact, and acknowledgment of limits. 5) Methodological Rigor & Falsifiability. Evaluate whether the proposal can be rigorously tested. Consider clarity of hypotheses, experimental design, metrics, baselines, and whether claims are falsifiable. 6) Feasibility & Scope Management. Assess whether the work is realistically executable. Consider resources, scope, risks, and planning. 7) Sensemaking & Iterative Knowledge Development. Evaluate whether the proposal includes an iterative learning strategy, considers alternative hypotheses, and plans for handling contradictory evidence. 8) Clarity, Structure, and Reproducibility. Assess clarity, organization, and reproducibility. Consider whether the proposal is well-structured, precise, and sufficiently detailed. INPUT PROMPT WITH REFERENCE PAPERS. {{input prompt}} GENERATED PROPOSAL. {{proposal}} START OF EVALUATION.

29