AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery

Preprint

AI-D RIVEN R EPRODUCIBILITY AND A NALYTICAL ROBUSTNESS IN S CIENTIFIC D ISCOVERY Shicheng Liu1†, Krista Opsahl-Ong2 , Andrew Drozdov2 , Julian Killingback4,2 Arnav Singhvi2 , Ronak Pradeep2 , Michael Bendersky2 James C. Dickerson3∗, Monica S. Lam1∗ 1

Department of Computer Science, Stanford University Databricks AI Research 3 Division of Oncology, Stanford University School of Medicine 4 University of Massachusetts Amherst

arXiv:2610.23845v1 [cs.AI] 16 Oct 2026

2

A BSTRACT Reproducible and analytically robust research findings are essential to scientific progress, but evaluating these properties typically requires costly manual effort. We investigate whether LLM agents can help automate reproduction and robustness analysis, using a medical case study grounded in the Surveillance, Epidemiology, and End Results (SEER) database, a widely used U.S. cancer registry underlying tens of thousands of published studies. First, we introduce SEERRepro, a benchmark of 20 published SEER-based papers together with the data and resources required to reproduce their findings. Second, we create an AI agent that successfully reproduces 77.4% of 2,190 atomic claims, while providing executable baselines for further analysis. Third, we develop an agentic analysis pipeline to test the robustness of the medical findings by identifying analytical choices underlying each finding, proposing defensible alternatives, and rerunning the analysis to determine whether those choices affect the conclusion. Across published papers in SEER-Repro, our pipeline identifies potentially consequential analytical choices in 9.5% of fully reproduced findings and flags them for expert review. These results show how LLM agents can support scientific review by reproducing findings, systematically testing their robustness, and directing expert attention toward potentially consequential analytical choices.

1

I NTRODUCTION

Reproducibility and analytical robustness are the bedrock on which scientific discovery rests. Reproducibility ensures that research findings can be independently verified, while analytical robustness asks whether conclusions remain stable under reasonable alternative choices. Traditionally, meta-analyses and systematic reviews have addressed these challenges through laborious, manual efforts to reproduce experiments or reanalyze data using alternate specifications (Delios et al., 2022; Brodeur et al., 2026). However, the immense cost and expertise required for such metascientific investigations have limited their scalability. Scientific claims also depend on the analytical decisions made by their authors (Simonsohn et al., 2020; Bertran et al., 2026). Recent “many-analyst” studies in psychology and the social sciences show that independent teams addressing the same hypothesis and dataset can reach conflicting conclusions (Silberzahn et al., 2018; Breznau et al., 2022; Botvinik-Nezer et al., 2020). Large language models (LLMs) offer a way to explore this analytical diversity at scale. Importantly, these AI-driven results exhibit notable steerability: shifting the design of the analyst persona or changing the underlying LLM meaningfully alters the distribution of analytic outcomes, even when methodological rigor is maintained throughout (Bertran et al., 2026; Miao et al., 2026). †

Work done at Databricks AI Research.

* Joint senior authors. ‡ Correspondence to shicheng@cs.stanford.edu, lam@cs.stanford.edu.

1

jcdicker@stanford.edu,

and

Preprint

SEER-Repro environment Published paper

Data + manuals

AI-assisted robustness assessment 3

1

2

Extract claims

Reproduce findings

Test robustness

Identify findings and extract atomic claims

Re-run analyses and check published claims

Test defensible choices; compare conclusions

Claim 1 Claim 2 Claim 3

Reported

Evidence for expert review

Reproduced

3 1 2 Numerical sensitivity

Baseline

Alternative

9 7 5 6 8 4 Secondary change Major effects on or qualification main conclusions

Figure 1: AI-assisted robustness assessment pipeline to support expert review.

Does this sensitivity to analytical choices documented in psychological and social sciences extend to medicine, where findings profoundly inform clinical decisions and patient care? Addressing this question is critical, as the validity of medical research underpins clinical decisions that affect countless lives. In this paper, we investigate the potential of AI-driven analysts to enhance both reproducibility and analytical robustness in scientific discovery, with a particular focus on medical epidemiology using the Surveillance, Epidemiology, and End Results (SEER) database. Maintained by the National Cancer Institute since 1973, the program integrates population-based cancer-registry records on patient demographics, tumor characteristics, treatment, and survival. In its 2024 usage metrics, NCI reports that 17 core registries cover roughly 48% of the U.S. population and collect approximately 950,000 incident cases per year (National Cancer Institute, 2026d; 2025). From 1975 through 2024, approximately 25,000 publications used SEER data in their primary analysis, while more than 126,000 publications referenced SEER data (National Cancer Institute, 2025). This scale provides a large body of realistic, high-impact questions, while differences in data and analytical definitions, such as registry coverage and cohort selection, make faithful reconstruction challenging. We argue that, before LLMs can be trusted to produce new scientific results, they must first demonstrate the ability to accurately reproduce existing scientific studies and assess their analytical robustness. Motivated by this prerequisite, this paper addresses two key research questions: First, can we create an AI agent to reproduce published medical epidemiological studies? And if so, what is the reproducibility rate of such papers? We construct SEER-Repro, the first benchmark consisting of 20 original recent or highly cited papers published on the SEER data. We develop an AI agent to reproduce the results of each paper by reading the paper and database documentation, writing and executing the code to access and process the SEER data accordingly. Across 2,190 atomic claims, our pipeline reproduced 77.4%, while 3.5% were close and 19.2% were not reproduced. Error analysis identifies unavailable data as the main barrier to reproduction, despite extensive efforts to recover all required data. Second, can we create an AI agent to evaluate the analytical robustness of findings in SEER-Repro? And if so, how robust are those findings to reasonable alternative analytical choices? We formulate the task of analytical robustness evaluation as follows: Given a research report and access to the underlying data, the task is to investigate if there are well-justified analytical choices that can change the conclusions of the report. We refer to such choices as consequential. Evaluating analytical robustness is challenging because gold-standard judgments do not exist. We therefore investigate three complementary approaches: (1) asking a coding agent to independently analyze the data and comparing its conclusions with those of the published paper; (2) systematically perturbing individual analytical choices through a structured agentic pipeline; and (3) asking coding agents to assess analytical robustness directly. To rate how strongly alternative analyses could change published conclusions, we adapted GRADE (Guyatt et al., 2008), an established framework for evaluating clinical evidence that rates the importance of outcomes on a nine-point scale, with scores of 7-9 indicating outcomes critical to decision-making. 2

Preprint

Our results show that major effects (scores 7–9) are flagged in 9.5% of fully reproduced findings (55 of 579). Expert review of 10 flagged cases found that 9 did not establish a consequential change under a defensible alternative, while 1 required further review. Thus, in contrast to the sensitivity reported for the social science datasets studied by Bertran et al. (2026) and Miao et al. (2026), our findings suggest that the medical papers in SEER-Repro exhibit a high degree of robustness to the analytical alternatives tested—an encouraging sign for the reliability of published medical research. Overall, our results show how LLM agents can strengthen the scientific process by reproducing findings, systematically testing their robustness, and directing expert attention toward potentially consequential analytical choices.

2

R ELATED W ORK

2.1

R EPRODUCTION AND ROBUSTNESS TO A NALYTICAL C HOICE IN S CIENTIFIC R ESEARCH

Multiverse and specification-curve analyses examine results across defensible analytical specifications, while many-analyst studies demonstrate that independent teams can reach different conclusions from the same data and question (Steegen et al., 2016; Simonsohn et al., 2020; Silberzahn et al., 2018). Related efforts test published findings through reproduction and reanalysis: Delios et al. (2022) extend archival analyses to new observations, and Brodeur et al. (2026) reproduce analyses and conduct robustness checks for 110 economics and political science articles. These studies motivate assessing both whether a finding can be reproduced and whether it survives alternative analytical choices. In this work, we investigate whether an LLM-assisted pipeline can help with this important scientific problem. Recent work automates robustness exploration with AI agents. Bertran et al. (2026) generate analyses with multiple AI analysts and screen their methodological validity with an AI analysis agent. Miao et al. (2026) sample agent-generated specifications and estimate how often defensible paths produce claims at least as extreme as the reported finding. Such analytical variation can reflect subjective judgments about how to operationalize a research question. We investigate how this problem manifests in medical epidemiology, where choices about cancer cohorts, outcome definitions, and statistical analyses can alter findings relevant to clinical practice and public health. 2.2

DATA -D RIVEN D EEP R ESEARCH B ENCHMARKS AND E VALUATION

Deep-research benchmarks such as DeepResearchBench, ReportBench, and DeepScholar-Bench assess report quality, evidence coverage, and citation grounding (Du et al., 2025; Li et al., 2025; Patel et al., 2025). Data-oriented benchmarks evaluate insight generation and executable scientific workflows, including InsightBench, DiscoveryBench, and ScienceAgentBench (Sahu et al., 2025; Majumder et al., 2025; Chen et al., 2025). BLADE additionally evaluates conceptual variables, transformations, and models across multiple valid analyses (Gu et al., 2024). DataSTORM, Hunt, and DAgent extend this setting to database investigation and report generation (Liu et al., 2026a;b; Xu et al., 2025). Our evaluation asks whether a report’s conclusions survive defensible changes to the analytical choices that produced them, using executed alternatives linked to individual findings. This report-level assessment of analytical robustness is not covered by these prior benchmarks and evaluations.

3

M ETHODOLOGY

3.1

AUTOMATED P IPELINE FOR R EPRODUCING P UBLISHED F INDINGS

In this section, we provide an overview of our proposed agent for reproducing published scientific findings. Our process starts with the paper p that we are interested in reproducing: Extract findings and atomic claims: First, we extract the empirical findings Fp = {f1 , . . . , fm }. These findings represent the core contributions of the paper and underpin the conclusion it draws. We exclude sections of the paper pertaining to related work, background context, or methodology. i Each finding fi is then decomposed into a set of atomic claims Ai = {aij }nj=1 . One atomic claim involves one specific numerical or qualitative target while one finding could group multiple atomic 3

Preprint

Algorithm 1 Perturbation Agent Input: Paper p, reproduced findings Fprep , reproduction evidence Ep , LLM-generated reports Rp from Sec 3.2.1 Output: A list of outcomes yk (s) ∈ AnalysisOutcome and an associated 1–9 score for yk (s) = DirectChange 1: Cp ← S ELECTA NALYTICAL C HOICES(p, Rp , Fprep , Ep ) 2: Kp ← P ROPOSE A LTERNATIVES(p, Rp , Cp , Ep ) 3: for each choice–alternative comparison (k, s) ∈ Kp do 4: ek (s) ← E XECUTE A NALYSES(k, s). 5: yk (s) ← A SSESS C ONSEQUENCES(k, s, ek (s)). 6: If yk (s) = DirectChange, assign a 1–9 consequentiality score. 7: end for 8: return comparison outcomes and consequentiality scores.

claims together. For instance, “Among women with adenoid cystic carcinoma of the breast, the mean age at diagnosis was 63 years, the median was 62 years, and the range was 33–97 years” is a finding, while its individual targets such as “Among women with adenoid cystic carcinoma of the breast, the mean age at diagnosis was 63 years” (and similarly for “median age: 62 years” and “age range: 33–97 years”) each become a separate atomic claim. Reproduce findings: We then ask a coding agent to reproduce each finding fi using the original paper p, the data used by the paper, and any additional documentation for the data. The agent identifies the relevant data, writes analysis code, and executes it to produce evidence ei for whether the finding can be validated or not. The evidence ei includes the output of any analytical code and its results. Grade individual claims: Finally, for each atomic claim and evidence pair (aij , ei ), a claim verification agent checks the method, extracts the reproduced result, and assesses the quantitative or qualitative agreement with the known target values in aij to produce a final verdict vij ∈ {R EPRODUCED, C LOSE, N OT R EPRODUCED}. For numerical targets, we consider a claim reproduced if the relative error is at most 5%, while errors greater than 5% and at most 10% are counted as close. Qualitative claims are deemed either reproduced or not reproduced. The full protocol for grading is presented in Appendix F. 3.2

A NALYTICAL ROBUSTNESS A NALYSIS

The reproduction artifacts, including the exact code derived, then serve as a foundation for analytic robustness evaluation. Our overall approach uses 3 complementary methods illustrated in Figure 2. 3.2.1

I NDEPENDENT R EPORT G ENERATION

Our first method asks whether an independent analysis of the same research question would reach a conclusion that contradicts the original paper (Figure 2 (A)). For each paper p, a coding agent receives a research question specifying the topic and scope derived from p, together with access to relevant data and additional documentation about the data, if available, and is tasked with independently answering the research question. The agent chooses its analytical approach, writes and executes code, and produces a research report r. We then judge whether the conclusions in r contradict those in p. Appendix C documents the report-generation setup and the prompts for all 20 research questions. 3.2.2

P ERTURBATION AGENT

Our second method uses the Perturbation Agent illustrated in Figure 2 (B) and Algorithm 1. Appendix D details the implementation of each component, including prompts. Eligible findings. For a published paper p, we start from the findings reproduced according to Section 3.1: Fprep = {fi ∈ Fp : vij is reproduced for every aij ∈ Ai }. 4

(1)

Preprint

A Independent Report Generation

? Published Paper Research question

>

Derived research question

Analyze data; write a report

Compare with published findings

B Perturbation Agent A finding Analysis

1–9

Published findings Select choices

Review alternatives

Run alternative analyses

Assess consequences

SEER-Repro

C Direct Coding Agent Analysis Data + manuals Reproduction evidence

>

Analysis

>

1–9

Paper + reproduction evidence

Agent selects and tests alternatives

Analysis report + execution evidence

Figure 2: AI-driven analytical robustness with 3 complementary methods discussed in Sec 3.2 Only fully reproduced findings enter the robustness analysis: their reconstructed results have passed the reproduction criteria in Section 3.1, and their reproduction code provides the baseline for testing alternative analytical choices. S ELECTA NALYTICAL C HOICES: First, the pipeline identifies analytical choices made in p that contribute to the known outcome for the findings Fprep , such as cohort definitions, endpoints, or statistical models. The set Rp of independent reports generated for p is used as an additional reference. These choices are then ordered by scientific importance and feasibility to select up to 20 choices which form the final set of choices Cp . P ROPOSE A LTERNATIVES: Next, the pipeline establishes how the paper p implements each selected choice in Cp , using the reproduction evidence Ep = {ei }m i=1 , where ei contains the evidence for finding fi . It then proposes defensible alternatives. The resulting set Kp contains comparisons (k, s), where each k ∈ Cp is a selected analytical choice and s is a proposed alternative setting. E XECUTE A NALYSES: For each comparison (k, s), we run the baseline unchanged and validate it against the available evidence. We then apply the alternative choice while holding the remaining analysis fixed. A SSESS C ONSEQUENCES: Finally, the pipeline assigns each comparison an outcome yk (s) from the enumerated type AnalysisOutcome = {DirectChange, Qualification, StableUnderTest, Inconclusive, Unavailable, NotTested}.

(2)

DirectChange denotes a change to a stated finding; Qualification narrows its scope or evidentiary strength without directly changing its conclusion; and StableUnderTest means the completed comparison does not meet the criterion for a consequential change. Inconclusive means the evidence cannot support a verdict, Unavailable indicates missing required inputs or baseline evidence, and NotTested records comparisons excluded before execution. 5

Preprint

Table 1: Atomic reproduction outcomes for the 20-paper SEER-Repro benchmark using GPT-5.6 Sol. Appendix Table 8 gives exact outcome counts and percentages; Appendix G lists full paper titles; Appendix Table 9 separates quantitative and qualitative claims. Paper (shorter name)

Journal (year)

Cit. Findings

Atomic claims Reproduced

Close

Not reproduced

Bone sarcoma survival by tumor site (Hu Cancers (2021) 37 et al., 2021) Sex disparities in cancer incidence (Cook Cancer Epidemiol. 466 et al., 2009) Biomarkers Prev. (2009) Inflammatory breast cancer incidence and J. Natl. Cancer Inst. 702 survival (Hance et al., 2005) (2005) JAMA Otolaryngol. Head 1,628 Thyroid cancer trends (Davies & Welch, Neck Surg. (2014) 2014) Age at oropharyngeal cancer Cancers (2020) 29 diagnosis (Cline et al., 2020) Adenoid cystic carcinoma of the Breast Cancer Res. 188 breast (Ghabach et al., 2010) (2010) Adult diffuse glioma incidence (Li et al., Cancer Med. (2018) 72 2018) Childhood cancer incidence and J. Natl. Cancer Inst. 438 mortality (Linet et al., 1999) (1999) Head and neck mucosal melanoma J. Skin Cancer (2012) 83 incidence (Marcus et al., 2012) Esophageal adenocarcinoma incidence and Cancer (2013) 510 mortality (Hur et al., 2013) Second cancers after young-onset Cancer Med. (2018) 43 colorectal cancer (He et al., 2018) Lung cancer incidence by sex, race, and PLOS ONE (2015) 368 histology (Meza et al., 2015) Age disparities in colorectal cancer JAMA Surg. (2015) 1,102 incidence (Bailey et al., 2015) Fatal prostate cancer incidence by Eur. Urol. (2017) 111 race (Kelly et al., 2017) Intracranial glioma incidence and Front. Oncol. (2021) 116 mortality (Lin et al., 2021) Long-term mortality in young cancer Cancers (2021) 23 survivors (Moke et al., 2021) Breast cancer incidence by age and tumor Breast Cancer Res. 352 features (Jemal et al., 2007) (2007) Esophageal versus gastric cardia Gut (2002) 266 adenocarcinoma (El-Serag et al., 2002) Metastatic cancer incidence and Commun. Med. (2023) 68 survival (Hudock et al., 2023) Cervical cancer incidence by race and Cancer (2004) 448 histology (Wang et al., 2004)

56

188

187 (99.5%)

36

151

140 (92.7%)

40

131

119 (90.8%)

33

91

80 (87.9%)

75

154

129 (83.8%)

51

128

103 (80.5%)

60

111

89 (80.2%)

52

144

113 (78.5%)

20

54

42 (77.8%)

33

52

40 (76.9%)

59

124

95 (76.6%)

36

111

80 (72.1%)

27

82

59 (72.0%)

32

63

45 (71.4%)

67

94

66 (70.2%)

62

178

123 (69.1%)

31

67

39 (58.2%)

34

51

29 (56.9%)

51

96

52 (54.2%)

46

120

64 (53.3%)

Total

901

2,190 1,694 (77.4%) 76 (3.5%) 420 (19.2%)

The outcome label only describes what happens to the tested finding. For DirectChange outcomes, we additionally grade the effect on the source’s conclusions from 1 to 9, using a scale adapted from GRADE’s outcome-importance framework (Guyatt et al., 2008). Scores 1–3 indicate quantitative sensitivity without substantive interpretive change, 4–6 indicate qualifications or changes to secondary findings, and 7–9 indicate major qualification, loss of support, or reversal of a principal conclusion.

3.2.3

D IRECT C ODING AGENT A NALYSIS

Our third method asks a coding agent to identify and test consequential analytical choices directly (Figure 2 (C)). For each paper, the agent receives the paper p, the available reproduction evidence Ep , and access to the data and data documentation when available. We then instruct the agent to select scientifically defensible alternatives addressing the same research question, validate the baseline analysis, and, where feasible, execute comparisons that vary one choice while holding the remaining analysis fixed (with Prompt 30). It produces a report describing the choices tested, the supporting evidence, and their effects on the source’s conclusions. After the analysis, we grade all output choices using the same 1–9 scale as in Section 3.2.2. 6

Preprint

Table 2: Classification of all 420 non-reproduced atomic claims from 20 papers in SEER-Repro. Category

Meaning

Data issues

Unavailable data (212 cases) or a data-related explanation such as mismatches attributed to revised records and population denominators (39 cases). The available paper and documentation do not uniquely specify the target analysis. Method substitutions involving approximations of unavailable software such as NCI Joinpoint. Departures from the reported method or comparison (25), incorrect cohort (13), statistical inference (3), denominator or standardization (6), or an omitted analysis (1). Suspected errors in our claim extraction or grading.

251 59.8

20

4.8

The evidence does not identify a sufficiently specific cause of non-reproduction.

26

6.2

Insufficiently specified analysis Unavailable-software approximations Agent-side errors

Suspected evaluation errors Unresolved cause

Claims

4

E XPERIMENTS

4.1

SEER-R EPRO C ONSTRUCTION C HALLENGES

%

47 11.2 28

6.7

48 11.4

We apply our pipeline to published SEER findings. Reconstructing these studies presents two challenges: Historical data availability. Detroit was excluded from the April 2022 release, changing SEER 9, 13, and 18 to SEER 8, 12, and 17 (National Cancer Institute, 2026b). We focus on SEER 9 studies because a retained supplement covers 1975–2018, including Detroit (National Cancer Institute, 2026f). Current releases cannot recover the exact historical SEER 13 and 18 cohorts. Appendix A details remaining limitations. Fragmented data and documentation. Epidemiological studies often rely on external population data and relevant manuals. We curate these resources manually through an iterative process with expert input. Missing inputs often become apparent only as individual analyses are reconstructed, and identifying what is missing can itself require tracing the analysis step by step. For example, reproducing an incidence rate requires identifying an appropriate population denominator, locating its source, and checking that its geographic coverage, calendar years, and demographic definitions match the cancer records. Selecting among candidate sources also requires assessing their suitability and consulting the relevant manuals. This process therefore requires manual curation and expert judgment, rather than simply retrieving a known list of files. We curated SEER 9 case table, four county-population tables, 34 SEER*Explorer tables, and associated documentation on coding, staging, registry mappings, and population methods (Appendix B). These 39 tables contain a combined 713 columns and 198,255,448 rows (Table 6). The accompanying 40 documentation resources include 11 PDF manuals and reference documents totaling 816 pages, 28 web pages, and one variable-dictionary spreadsheet (Table 5). These data tables and documentation resources form the execution environment for SEER-Repro.

4.2

SEER-R EPRO B ENCHMARK AND R EPRODUCTION O UTCOME

We collect the ten most cited eligible papers and ten more recent papers in SEER-Repro. Using GPT5.6 Sol xhigh via Codex, our reproduction agent reproduces 1,694 of 2,190 atomic claims (77.4%); 76 are close (3.5%) and 420 are not reproduced (19.2%; Table 1). We classify the 420 failures using Prompt 6 and review the diagnoses against saved evidence. Data issues account for 251 (59.8%), insufficiently specified analyses for 47 (11.2%), unavailable-software approximations for 28 (6.7%), and agent-side errors for 48 (11.4%; Table 2). 7

Preprint

Table 3: Model-assessed consequential links (scores 7–9): finding counts by paper and analyst. Counts include individual links and members of consequential joint links, deduplicated per analyst. Eligible findings

Paper

Findings with consequential links Perturbation Agent

Direct coding agent

Sol Opus Kimi GLM Codex

Found by

≥2 Claude ≥ 1 Code analyst analysts

Bone sarcoma survival by tumor site Sex disparities in cancer incidence Inflammatory breast cancer incidence and survival Thyroid cancer trends Age at oropharyngeal cancer diagnosis Adenoid cystic carcinoma of the breast Adult diffuse glioma incidence Childhood cancer incidence and mortality Head and neck mucosal melanoma incidence Esophageal adenocarcinoma incidence and mortality Second cancers after young-onset colorectal cancer Lung cancer incidence by sex, race, and histology Age disparities in colorectal cancer incidence Fatal prostate cancer incidence by race Intracranial glioma incidence and mortality Long-term mortality in young cancer survivors Breast cancer incidence by age and tumor features Esophageal versus gastric cardia adenocarcinoma Metastatic cancer incidence and survival Cervical cancer incidence by race and histology

55 31 31 24 53 37 40 28 12 23 44 13 10 20 47 40 15 15 22 19

2 – – – – – – – – – – 1 – – – 2 – – – –

– – 3 – 3 – – – – – – – – – – 4 2 – 1 –

2 – – – – – – – – – – – – 5 1 6 – – 2 1

– – – – – – – 1 1 2 – – 1 – – – – – – –

– – 2 2 – 4 2 – 2 – 1 – – 3 – 2 – 5 1 2

– – 3 – – – – – 1 – 2 – – 3 – – – – 3 –

4 – 4 2 3 4 2 1 2 2 2 1 1 6 1 6 2 5 4 3

– – 3 – – – – – 2 – 1 – – 3 – 4 – – 1 –

Total

579

5

13

17

5

26

12

55

14

The benchmark packages 39 data tables and 40 documentation resources (Appendix B), together with the 2,190 atomic claims over 20 papers. Each claim aij ∈ Ai links to its paper p, parent finding fi , verdict vij , and reproduction evidence ei , including code and available execution outputs. 4.3

ROBUSTNESS A NALYSIS OF SEER-R EPRO

4.3.1

I NDEPENDENT R EPORT G ENERATION

For each paper, we independently ask Opus 4.8 max via Claude Code and GPT-5.6 Sol xhigh via Codex to generate a report. GPT-5.6 Sol then compares each of the 40 reports with its corresponding paper for contradictions on their shared research question (Prompt 1). The judge identifies no contradictions in the central conclusions of the 39 comparable reports (19 Sol and 20 Opus). One Sol report analyzes a nonoverlapping study period and is not comparable. It marked 3 Opus reports as containing contradictions in supporting statements, but these do not overturn their central conclusions upon closer scrutiny. These results suggest that independent analyses generally reach central conclusions non-contradictory with the published papers. 4.3.2

P ERTURBATION AGENT AND D IRECT C ODING AGENT A NALYSIS

We run the Perturbation Agent with GPT-5.6 Sol (OpenAI, 2026), Opus 4.8 (Anthropic, 2026), Kimi K3 (Kimi Team, 2026), and GLM 5.3 (Z.ai, 2026), together with direct coding agent analyses through GPT-5.6 Sol xhigh via Codex and Opus 4.8 max via Claude Code. These correspond to methods B and C in Figure 2. GPT-5.6 Sol then assigned scores of 7–9 to 54 results (Prompt 31). These scores indicate a model-assessed major qualification, loss of support, or reversal of a principal conclusion and help identify shortlisted candidates for expert examination. We use GPT-5.6 Sol to link these high-score results back to the original findings in each paper (Prompt 32). A model-assessed consequential link requires reported evidence of a reversal, loss of support, major qualification, or material change to a quantity central to one specific finding or a conclusion that the paper draws jointly from multiple findings. In the latter case, we link the analysis to each supporting finding. Table 3 shows that 55 of the 579 fully reproduced findings (9.5%) have such a link from at least one analyst, and 14 (2.4%) from at least two analysts. 8

Preprint

Table 4: Three expert-reviewed cases, one per assessment category. Alternative analysis and result

Expert assessment

Bone sarcoma survival by tumor site. Sol, Perturbation Agent; score 9. Adjusted cancer-specific mortality did not significantly improve in the 2000s or 2010s relative to the 1990s.

Replace the SEER “Bones and Joints” recode with direct primary-site codes C40.0–C41.9. The implementation added 3,974 records; both period comparisons then showed significant improvements.

Likely less defensible alternative. The added records change the disease population, and closer examination of the database contents and manuals suggests that the alternative is less defensible than the original choice.

Adenoid cystic carcinoma of the breast. Codex, direct analysis; score 8. Nonsignificant standardized incidence ratios (SIRs) were interpreted as an absence of associated cancers.

Interpret the estimates as no statistically detected excess, while acknowledging that the confidence intervals still permit increased risk. Failure to reject the null does not establish absence of excess risk.

Limited qualification. The distinction is reasonable, but the paper already qualifies its conclusions with caveats. The reviewer judged this a clarification of interpretation rather than a conclusion-changing result.

Head and neck mucosal melanoma incidence. Claude Code, direct analysis; score 8. Increasing incidence was attributed primarily to nasal-cavity disease.

Directly compare nasal and non-nasal trends rather than infer a difference from their separate significance tests. The analyst reported positive trends in both groups and a nonsignificant interaction (p = 0.46).

Flagged for further review. The comparison may identify an overstatement, but noisy data and model-fit concerns complicate interpretation. Closer manuscript review and statistical consultation are needed; a nonsignificant interaction does not establish equal trends.

Discussion: As mentioned in Section 3.2, we restrict this analysis to fully reproduced findings. This provides executable baselines whose results passed the reproduction criteria in Section 3.1. The analysts can therefore compare an alternative against a reconstructed result supported by reproduction evidence. We further consider two questions: Do different analysts complement each other? Results in Table 3 show that most of the affected findings—41 of 55 (74.5%)—are identified by only one analyst. This suggests that different analysts identify largely complementary sets of potentially consequential analytical choices.

Number of graded results

Paper and original conclusion

Sol (Perturbation) 7–9: 4/27 (14.8%) Kimi (Perturbation) 7–9: 10/104 (9.6%) Codex 7–9: 18/93 (19.4%)

50 40 30 20 10 0

Opus (Perturbation) 7–9: 9/109 (8.3%) GLM (Perturbation) 7–9: 5/67 (7.5%) Claude Code 7–9: 8/46 (17.4%) High consequence (scores 7–9)

1

2

3 4 5 6 7 Consequentiality score

8

9

Figure 3: GPT-5.6 Sol consequence scores for analyses of published papers; shading marks scores 7–9.

Distribution of consequence scores. Among results with a 1–9 grade (Figure 3), scores 7–9 account for 7.5–14.8% of Perturbation Agent comparisons, 19.4% of Codex results, and 17.4% of Claude Code results. Thus, most graded results do not reach the high-consequence range. 4.3.3

E XPERT A NALYSIS

One coauthor with medical expertise and familiarity with SEER data manually examined 10 cases scored 7–9 in depth (five each from Perturbation Agent and Direct Coding Agent Analysis). Among the 5 cases from the Perturbation Agent, 2 suggested consequential changes, but closer expert scrutiny indicated that the alternatives were likely less defensible than the original analytical choices; the remaining 3 offered only limited qualifications to the papers’ conclusions. Among the 5 cases from direct coding agent analysis, 4 likewise offered only limited qualifications. The remaining 1 case identified a potentially overstated comparison between nasal and non-nasal incidence trends, but determining its consequence requires closer manuscript review and statistical consultation. Thus, 9 cases did not establish a consequential change under a defensible alternative, while 1 remains flagged for further review. Table 4 illustrates 3 cases with one from each category: a likely less defensible alternative, a limited qualification to the paper’s conclusions, and a potentially consequential issue requiring further review. Overall, our pipeline narrows the material requiring expert attention by autonomously reconstructing published findings, executing alternative analyses, and prioritizing potentially consequential results. Across 579 fully reproduced findings, it identifies 55 with model-assessed consequential links, providing a focused shortlist and supporting evidence for expert review. Experts can therefore concentrate on whether the flagged alternatives are scientifically defensible and will materially change the conclusions, rather than reconstructing and testing each analysis from scratch manually. 9

Preprint

5

C ONCLUSION

We introduced SEER-Repro, a benchmark of 20 published SEER studies, and an AI-assisted pipeline for evaluating reproducibility and robustness to analytical choices. Our reproduction agent recovered 77.4% of atomic claims. Automated robustness assessments then prioritized 55 of 579 fully reproduced findings (9.5%) for expert review, providing a focused shortlist with supporting analytical evidence. Expert examination of 10 flagged cases clarified why 9 did not establish consequential changes under defensible alternatives and identified 1 potentially consequential issue requiring further investigation. These results illustrate how automated analysis can focus expert attention on questions that warrant closer scientific scrutiny. SEER provides an early, high-stakes testbed for this approach. Longer term, such checks could become part of scientific infrastructure: published findings could be routinely reassessed as data, methods, and analytical alternatives evolve, with potentially consequential sensitivities prioritized for expert review.

10

Preprint

AI U SE S TATEMENT Uses of generative AI. In this work, we used generative AI tools to help develop theoretical models or conceptual frameworks; propose or refine hypotheses; design or provide feedback on research methodology or experiments; implement methods; clean and reformat datasets; support qualitative and thematic data analysis; interpret results; create or modify scientific figures or images; create or edit software code; draft parts of the paper; create artifacts; summarize or analyze existing literature; discover research topics or identify gaps; brainstorm; source and search for information; improve the paper’s readability; identify relevant literature; and format references. Activities without generative AI. We did not use generative AI tools to suggest experimental parameters, suggest a structure for the paper, or propose a title or keywords. Activities not applicable to this work. Generating synthetic datasets, formulating mathematical claims, providing critical ingredients for mathematical proofs, assisting with proof writing, translating, formulating survey or interview questions, and transcribing research recordings are not applicable to this work. Additional disclosure and author responsibility. We have reviewed all AI-assisted work. For example, the final writing has been double-checked by authors. We have reviewed all relevant literature manually found by LLMs. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. E THICS S TATEMENT This work aims to support scientific reliability by evaluating the reproducibility and analytical robustness of published cancer epidemiology findings. We acknowledge that errors in our pipeline, including but not limited to data selection, LLM-generated code, interpretation, and grading, may lead to incorrect assessments. Failure to reproduce a finding may also reflect unavailable historical data (as documented in Section 4.1). Sensitivity to an alternative analysis does not invalidate the original study and requires careful expert judgment to assess its significance (as discussed in Section 4). Our goal is to help experts identify findings that warrant closer examination, making rigorous scientific review more accessible while keeping expert judgment central to the process. R EPRODUCIBILITY S TATEMENT We document the SEER data releases and registry coverage in Appendix A, and the database resources, manuals, source links, and file formats in Appendix B. Table 1 identifies the exact 20 papers evaluated in SEER-Repro. Appendix C provides the research prompts and report-generation settings, while Appendix D describes the analysis implementation and Appendix H provides the LLM prompts. Section 4 identifies the tested analysis models (GPT-5.6 Sol, Opus 4.8, GLM 5.3, and Kimi K3) and the coding-agent configurations used in the comparison experiments. ACKNOWLEDGMENT We thank Ivan Zhou, Qing (Veronica) Lyu, Xinyi (Cindy) Wang, Muhammad Shahir Abdurrahman, and Xing Chen at Databricks; and Harshit Joshi, Zikai (Cyrus) Zhou, Yanzhe Zhang, and Yuchen Jiang at Stanford for discussions that shaped the project. Shicheng Liu’s academic research outside Databricks was supported in part by the Verdant Foundation, the Hasso Plattner Institute, Itaú Unibanco, BMO Financial Group, and the Stanford HumanCentered Artificial Intelligence (HAI) Institute. We acknowledge the National Artificial Intelligence Research Resource (NAIRR) Pilot and Microsoft Azure for contributing to the results in this work.

11

Preprint

R EFERENCES Anthropic. Introducing Claude Opus 4.8, May 2026. URL https://www.anthropic.com/ news/claude-opus-4-8. Christina E. Bailey, Chung-Yuan Hu, Y. Nancy You, Brian K. Bednarski, Miguel A. RodriguezBigas, John M. Skibber, Scott B. Cantor, and George J. Chang. Increasing Disparities in the Age-Related Incidences of Colon and Rectal Cancers in the United States, 1975-2010. JAMA Surgery, 150(1):17–22, 2015. doi: 10.1001/jamasurg.2014.1756. Martin Bertran, Riccardo Fogliato, and Zhiwei Steven Wu. Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse. Proceedings of the National Academy of Sciences, 123(29):e2606495123, 2026. doi: 10.1073/pnas.2606495123. Rotem Botvinik-Nezer, Felix Holzmeister, Colin F. Camerer, et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582:84–88, 2020. doi: 10.1038/ s41586-020-2314-9. Nate Breznau, Eike Mark Rinke, Alexander Wuttke, et al. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proceedings of the National Academy of Sciences, 119(44):e2203150119, 2022. doi: 10.1073/pnas.2203150119. Abel Brodeur, Derek Mikola, Nikolai Cook, et al. Reproducibility and robustness of economics and political science research. Nature, 652:151–156, 2026. doi: 10.1038/s41586-026-10251-x. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. In International Conference on Learning Representations, 2025. Brittany J. Cline, Matthew C. Simpson, Matthew Gropler, Aleksandr R. Bukatko, Eric Adjei Boakye, Kahee A. Mohammed, and Nosayaba Osazuwa-Peters. Change in Age at Diagnosis of Oropharyngeal Cancer in the United States, 1975–2016. Cancers, 2020. doi: 10.3390/cancers12113191. Michael B. Cook, Sanford M. Dawsey, Neal D. Freedman, Peter D. Inskip, Sara M. Wichner, Sabah M. Quraishi, Susan S. Devesa, and Katherine A. McGlynn. Sex Disparities in Cancer Incidence by Period and Age. Cancer Epidemiology, Biomarkers & Prevention, 18(4):1174–1182, 2009. doi: 10.1158/1055-9965.epi-08-1118. Louise Davies and H. Gilbert Welch. Current Thyroid Cancer Trends in the United States. JAMA Otolaryngology–Head & Neck Surgery, 140(4):317–322, 2014. doi: 10.1001/jamaoto.2014.1. Andrew Delios et al. Examining the Generalizability of Research Findings from Archival Data. Proceedings of the National Academy of Sciences, 119(30):e2120377119, 2022. doi: 10.1073/ pnas.2120377119. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. arXiv preprint arXiv:2506.11763, 2025. URL https://arxiv.org/abs/2506.11763. H B El-Serag, A C Mason, N Petersen, and C R Key. Epidemiological differences between adenocarcinoma of the oesophagus and adenocarcinoma of the gastric cardia in the USA. Gut, 50(3): 368–372, 2002. doi: 10.1136/gut.50.3.368. Bassam Ghabach, William F Anderson, Rochelle E Curtis, Mark M Huycke, Jackie A Lavigne, and Graça M Dores. Adenoid cystic carcinoma of the breast in the United States (1977 to 2006): a population-based cohort study. Breast Cancer Research, 2010. doi: 10.1186/bcr2613. Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, and Tim Althoff. BLADE: Benchmarking Language Model Agents for Data-Driven Science. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. 12

Preprint

Gordon H. Guyatt, Andrew D. Oxman, Regina Kunz, Gunn E. Vist, Yngve Falck-Ytter, Holger J. Schünemann, and GRADE Working Group. What is “quality of evidence” and why is it important to clinicians? BMJ, 336(7651):995–998, 2008. doi: 10.1136/bmj.39490.551019.BE. URL https://pmc.ncbi.nlm.nih.gov/articles/PMC2364804/. Kenneth W. Hance, William F. Anderson, Susan S. Devesa, Heather A. Young, and Paul H. Levine. Trends in Inflammatory Breast Carcinoma Incidence and Survival: The Surveillance, Epidemiology, and End Results Program at the National Cancer Institute. JNCI: Journal of the National Cancer Institute, 97(13):966–975, 2005. doi: 10.1093/jnci/dji172. Xingkang He, Wenrui Wu, Yu’e Ding, Yue Li, Jianmin Si, and Leimin Sun. Excessive risk of second primary cancers in young-onset colorectal cancer survivors. Cancer Medicine, 2018. doi: 10.1002/cam4.1437. Xianglin Hu, Kai Deng, Hui Ye, Zhengwang Sun, Wending Huang, Yangbai Sun, and Wangjun Yan. Trends in Tumor Site-Specific Survival of Bone Sarcomas from 1980 to 2018: A Surveillance, Epidemiology and End Results-Based Study. Cancers, 2021. doi: 10.3390/cancers13215381. Nicholas L. Hudock, Kyle Mani, Chachrit Khunsriraksakul, Vonn Walter, Larissa Nekhlyudov, Ming Wang, Eric J. Lehrer, Maria R. Hudock, Dajiang J. Liu, Daniel E. Spratt, and Nicholas G. Zaorsky. Future trends in incidence and long-term survival of metastatic cancer in the United States. Communications Medicine, 3(1), 2023. doi: 10.1038/s43856-023-00304-x. Chin Hur, Melecia Miller, Chung Yin Kong, Emily C. Dowling, Kevin J. Nattinger, Michelle Dunn, and Eric J. Feuer. Trends in esophageal adenocarcinoma incidence and mortality. Cancer, 119 (6):1149–1158, 2013. doi: 10.1002/cncr.27834. Ahmedin Jemal, Elizabeth Ward, and Michael J Thun. Recent trends in breast cancer incidence rates by age and tumor characteristics among U.S. women. Breast Cancer Research, 9(3), 2007. doi: 10.1186/bcr1672. Scott P. Kelly, Philip S. Rosenberg, William F. Anderson, Gabriella Andreotti, Naji Younes, Sean D. Cleary, and Michael B. Cook. Trends in the Incidence of Fatal Prostate Cancer in the United States by Race. European Urology, 71(2):195–201, 2017. doi: 10.1016/j.eururo.2016.05.011. Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. URL https://arxiv.org/abs/2607.24653. Kai Li, Dan Lu, Yazhou Guo, Changwei Wang, Xiao Liu, Yu Liu, and Dezhong Liu. Trends and patterns of incidence of diffuse glioma in adults in the United States, 1973-2014. Cancer Medicine, 2018. doi: 10.1002/cam4.1757. Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, and Kai Jia. ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks. arXiv preprint arXiv:2508.15804, 2025. Dongdong Lin, Ming Wang, Yan Chen, Jie Gong, Liang Chen, Xiaoyong Shi, Fujun Lan, Zhongliang Chen, Tao Xiong, Hu Sun, and Shu Wan. Trends in Intracranial Glioma Incidence and Mortality in the United States, 1975-2018. Frontiers in Oncology, 2021. doi: 10.3389/fonc.2021.748061. M. S. Linet, L. A. G. Ries, M. A. Smith, R. E. Tarone, and S. S. Devesa. Cancer Surveillance Series: Recent Trends in Childhood Cancer Incidence and Mortality in the United States. JNCI Journal of the National Cancer Institute, 91(12):1051–1058, 1999. doi: 10.1093/jnci/91.12.1051. Shicheng Liu, Yucheng Jiang, Sajid Farook, Camila Nicollier Sanchez, David Fernando Castro Pena, and Monica S. Lam. DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytelling. Conference on Language Modeling, 2026a. Wei Liu, Peijie Yu, Michele Orini, Yali Du, and Yulan He. Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models. International Conference on Machine Learning, 2026b. 13

Preprint

Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBench: Towards Data-Driven Discovery with Large Language Models. In International Conference on Learning Representations, 2025. David M. Marcus, Rebecca P. Marcus, Roshan S. Prabhu, Taofeek K. Owonikoko, David H. Lawson, Jeffrey Switchenko, and Jonathan J. Beitler. Rising Incidence of Mucosal Melanoma of the Head and Neck in the United States. Journal of Skin Cancer, 2012. doi: 10.1155/2012/231693. Rafael Meza, Clare Meernik, Jihyoun Jeon, and Michele L. Cote. Lung Cancer Incidence Trends by Gender, Race and Histology in the United States, 1973–2010. PLOS ONE, 2015. doi: 10.1371/ journal.pone.0121323. Jiacheng Miao, Jonathan K. Pritchard, and James Zou. The Agentic Garden of Forking Paths. arXiv preprint arXiv:2607.01507, 2026. Diana J. Moke, Ziwei Song, Lihua Liu, Ann S. Hamilton, Dennis Deapen, and David R. Freyer. A Population-Based Analysis of 30-Year Mortality among Five-Year Survivors of Adolescent and Young Adult Cancer: The Roles of Primary Cancer, Subsequent Malignancy, and Other Health Conditions. Cancers, 2021. doi: 10.3390/cancers13163956. National Cancer Institute. SEER Program Annual Usage Metrics, 2024. https://seer. cancer.gov/about/factsheets/metrics.pdf, 2025. Accessed September 15, 2026. National Cancer Institute. Behavior Variable Changes. https://seer.cancer.gov/ data-software/documentation/seerstat/nov2020/behavior-change. html, 2026a. Accessed September 23, 2026. National Cancer Institute. SEER Data Change History. https://seer.cancer.gov/data/ data-changes.html, 2026b. Accessed September 23, 2026. National Cancer Institute. U.S. County Population Data, 1969–2024. https://seer.cancer. gov/popdata/, 2026c. Accessed September 23, 2026. National Cancer Institute. Surveillance, Epidemiology, and End Results Program. https:// seer.cancer.gov/, 2026d. National Cancer Institute. Registry Groupings in SEER Data and Statistics. https://seer. cancer.gov/registries/terms.html, 2026e. Accessed September 23, 2026. National Cancer Institute. SEER*Stat Database Details: Supplemental SEER November 2020 Submission. https://seer.cancer.gov/data-software/documentation/ seerstat/supplemental2020/, 2026f. Accessed September 23, 2026. OpenAI. GPT-5.6 Sol. Model documentation, 2026. URL https://developers.openai. com/api/docs/models/gpt-5.6-sol. Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, and Carlos Guestrin. DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis. arXiv preprint arXiv:2508.20033, 2025. URL https://arxiv.org/ abs/2508.20033. Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, Nicolas Chapados, Christopher Pal, Sai Rajeswar Mudumba, and Issam Hadj Laradji. InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation. International Conference on Learning Representations, 2025. Raphael Silberzahn, Eric L. Uhlmann, Daniel P. Martin, et al. Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science, 1(3):337–356, 2018. doi: 10.1177/2515245917747646. Uri Simonsohn, Joseph P. Simmons, and Leif D. Nelson. Specification Curve Analysis. Nature Human Behaviour, 4:1208–1214, 2020. doi: 10.1038/s41562-020-0912-z. 14

Preprint

Sara Steegen, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science, 11(5):702–712, 2016. doi: 10.1177/1745691616658637. Sophia S. Wang, Mark E. Sherman, Allan Hildesheim, James V. Lacey, and Susan Devesa. Cervical adenocarcinoma and squamous cell carcinoma incidence trends among white women and black women in the United States for 1976–2000. Cancer, 100(5):1035–1044, 2004. doi: 10.1002/cncr. 20064. Wenyi Xu, Yuren Mao, Xiaolu Zhang, Chao Zhang, Xuemei Dong, Mengfei Zhang, and Yunjun Gao. DAgent: A Relational Database-Driven Data Analysis Report Generation Agent. arXiv preprint arXiv:2503.13269, 2025. Z.ai. GLM-5.3. Model documentation, 2026. URL https://docs.z.ai/guides/llm/ glm-5.3.

15

Preprint

A PPENDIX A SEER Data Evolution and Study Scope

17

B Supplementary SEER Tables and Manuals

18

C Details on LLM-Generated Research Reports

21

D Perturbation Agent: Implementation Details

22

E Reproduction Outcome Breakdowns in SEER-Repro

23

F Reproduction Grading Details

23

F.1

Judge Inputs and Evidence Assessment . . . . . . . . . . . . . . . . . . . . . . .

23

F.2

Numerical Extraction and Comparison . . . . . . . . . . . . . . . . . . . . . . . .

25

F.3

Grading Qualitative Claims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

G Full Titles of Benchmark Papers

27

H Prompts Used

28

16

Preprint

A

SEER DATA E VOLUTION AND S TUDY S COPE

Registry Overview. SEER 9, SEER 13, and SEER 18 identify different geographic cohorts, with standard diagnosis-year coverage beginning in 1975, 1992, and 2000, respectively. Figure 4 summarizes their expansion and subsequent changes. With the November 2021 submission, released in April 2022, Detroit was excluded from the new databases, yielding SEER 8, 12, and 17. Massachusetts was subsequently excluded from the November 2024 submission, changing SEER 22 to SEER 21 (National Cancer Institute, 2026b;e).

Historical registry expansion

SEER 9

SEER 13

SEER 18

1975 onward

1992 onward

2000 onward

Connecticut Iowa New Mexico Utah Hawaii Detroit San Francisco –Oakland Seattle –Puget Sound Atlanta

SEER 9 + four registries Los Angeles San Jose –Monterey Rural Georgia Alaska Native Tumor Registry

SEER 13 + five registries Greater California Kentucky Louisiana New Jersey Greater Georgia

April 2022 release: Detroit excluded from new groupings SEER 9

SEER 13

− Detroit

SEER 8

− Detroit

SEER 12

SEER 18 − Detroit

SEER 17

Retained separately: supplemental SEER 9, 1975–2018 Includes Detroit; November 2020 submission with updated population denominators

Figure 4: Evolution of the SEER registry groupings relevant to our study. Historical SEER 13 and 18 expand the SEER 9 cohort; their successor groupings exclude Detroit. The separately retained SEER 9 supplement covers 1975–2018. Registry memberships and retained coverage follow NCI documentation (National Cancer Institute, 2026e;f). Why we focus on SEER 9? The supplemental November 2020 database preserves SEER 9 records, including Detroit, through 2018. We use this retained cohort and select papers whose analyses do not require additional registries. It is in general difficult to reconstruct historical SEER 13 and SEER 18 datasets using the newer, Detroit-excluding releases (SEER 12 and SEER 17), since each case does not record the exact associated registry. Release and variable differences remain. Retaining the registry grouping does not guarantee identical historical inputs. For example, the November 2020 submission changed the default malignant-behavior definition and excluded approximately 10,000 previously included, no-longerreportable borderline cases, predominantly ovarian tumors (National Cancer Institute, 2026a). Our local failure diagnostics also identify unavailable radiation-treatment fields, preventing reconstruction of some treatment proportions even when registry coverage matches. We retain these limitations when interpreting claim-level reproduction outcomes. For instance, these cases would count towards “Data Issues” in Table 2.

17

Preprint

B

S UPPLEMENTARY SEER TABLES AND M ANUALS

The curated SEER-Repro environment contains the SEER 9 case table, 4 population denominator tables, 34 SEER*Explorer aggregate tables, and 40 documentation files or reference pages. Population denominators. The four population tables provide county-level annual estimates by age, sex, and race, with an expanded race/Hispanic-origin scheme available from 1990. Each series is supplied both in 20 age groups and in single-year ages through 89 with a 90+ category (Table 6). These are all-state files: rates for a SEER cohort require restricting them to the corresponding registry counties (National Cancer Institute, 2026c). SEER*Explorer aggregates. We include 34 workbooks from the official November 2024 SEER*Explorer archive as queryable tables. Table 6 lists each table with its column count, row count, and description. They supplement case-level computation with aggregated statistics. Manuals and reference definitions. Table 5 lists the 40 documentation resources supplied in SEER-Repro: 28 HTML pages, 11 PDFs, and one XLSX spreadsheet. The collection covers SEER*Stat dictionaries, population denominators and standardization, coding and staging, variable recodes, and registry coverage. It includes both recent manuals and a historical-manual index. Table 5: Documentation resources provided in SEER-Repro. Source URLs are those recorded for the local copies; formats were verified from the files. Resource SEER*Stat resources (6) SEER*Stat November 2025 submission overview Variable dictionary spreadsheet for SEER*Stat variables

Source URL

Format

https://seer.cancer.gov/data-software/ documentation/seerstat/nov2025/

HTML

https://seer.cancer.gov/data-software/ documentation/seerstat/nov2025/ seerstat-variable-dictionary-nov2025.xlsx

XLSX

Variable dictionary PDF for SEER*Stat https://seer.cancer.gov/data-software/ variables documentation/seerstat/nov2025/

PDF

seerstat-variable-dictionary-nov2025.pdf

Text data file description for SEER Research and Research Plus

https://seer.cancer.gov/data-software/ documentation/seerstat/nov2025/TextData. FileDescription-nov2025.pdf

PDF

SEER product comparison and availability notes SEER data documentation hub

https: //seer.cancer.gov/data/product-comparison.html

HTML

https: //seer.cancer.gov/data-software/documentation/

HTML

https://seer.cancer.gov/popdata/ https://seer.cancer.gov/popdata/download.html

HTML HTML

https://seer.cancer.gov/popdata/popdic.html

HTML

https://seer.cancer.gov/popdata/singleages.html

HTML

https://seer.cancer.gov/popdata/modifications.html

HTML

https://seer.cancer.gov/popdata/hurricane_adj.html

HTML

https://seer.cancer.gov/popdata/W%26P_ICen_Tech_ Documentation.pdf

PDF

https: //seer.cancer.gov/data-software/uspopulations.html

HTML

https://seer.cancer.gov/stdpopulations/

HTML

Population resources (10) U.S. county population data overview Population data download page and file layout links Population data dictionary for fixed-width fields Single-year-age population estimate methodology NCI modifications to county population estimates Katrina/Rita population adjustment notes Woods & Poole technical documentation linked by SEER U.S. populations hub Standard population data hub

Continued on next page

18

Preprint

Table 5 (continued) Resource

Source URL

Format

Use of the 2000 U.S. standard population Coding and staging (10) SEER coding and staging manual page Historical coding manuals index

https://seer.cancer.gov/stdpopulations/ 2000stdpop-use.html

HTML

https://seer.cancer.gov/tools/codingmanuals/

HTML HTML

https://seer.cancer.gov/tools/codingmanuals/ historical.html

2025 SEER Program Coding and Staging Manual 2025 county codes

https://seer.cancer.gov/manuals/2025/SPCSM_2025_ MainDoc.pdf

PDF

https://seer.cancer.gov/manuals/2025/SPSCM_2025_ Appendix_A.pdf

PDF

2025 country and state codes

https://seer.cancer.gov/manuals/2025/SPCSM_2025_ Appendix_B.pdf

PDF

2025 race and nationality descriptions

https://seer.cancer.gov/manuals/2025/SPCSM_2025_ Appendix_D.pdf

PDF

2025 reportable and non-reportable examples 2025 summary of changes

https://seer.cancer.gov/manuals/2025/SPCSM_2025_ Appendix_E.pdf

PDF

https://seer.cancer.gov/manuals/2025/SPCSM_2025_ Changelog.pdf

PDF

2026 SEER Program Coding and Staging Manual 2026 county codes

https://seer.cancer.gov/manuals/2026/SPCSM_2026_ MainDoc.pdf

PDF

https://seer.cancer.gov/manuals/2026/SPSCM_2026_ Appendix_A.pdf

PDF

https://seer.cancer.gov/analysis/

HTML HTML HTML HTML HTML HTML

Variable and recode definitions (10) Variable and recode definitions hub Site recode documentation Incidence site recode variables Cause of death recode Cause-specific death classification Race and Hispanic ethnicity definitions Cancer stage variable overview Summary Stage 2018 documentation Months survived based on complete dates Tumor size over time recode Registry coverage (4) SEER registries overview List of SEER registries Registry groupings used in SEER data and statistics Population characteristics of SEER registry areas

https://seer.cancer.gov/siterecode/ https://seer.cancer.gov/analysis/incidence.html https://seer.cancer.gov/codrecode/ https://seer.cancer.gov/causespecific/ https://seer.cancer.gov/seerstat/variables/seer/ race_ethnicity/

https://seer.cancer.gov/survivaltime/

HTML HTML HTML

https://seer.cancer.gov/tumorsize/

HTML

https://seer.cancer.gov/registries/

HTML HTML HTML

https://seer.cancer.gov/analysis/stage.html https://seer.cancer.gov/tools/ssm/

https://seer.cancer.gov/registries/list.html https://seer.cancer.gov/registries/terms.html https: //seer.cancer.gov/registries/characteristics.html

HTML

Table 6: The 39 curated tables, with schema column counts and loaded row counts. SEER*Explorer tables use the November 2024 archive; RUCC denotes rural–urban continuum codes. Table

Cols.

SEER 9 case records

250

Population 1969–2024, 20 age groups Population 1990–2024, 20 age groups Population 1969–2024, single-year ages

Rows Description

5,075,266 Case-level cancer-registry export from the SEER 9 Program. 13 17,485,723 County denominators; White/Black/Other. 13 22,178,259 County denominators; expanded race/Hispanic origin. 13 69,655,933 County denominators; White/Black/Other. Continued on next page

19

Preprint

Table 6 (continued) Table Population 1990–2024, single-year ages Recent trends annual SEER incidence rates Recent trends SEER incidence trend estimates 5-year age adjusted incidence rates Long-term trends annual SEER incidence rates Long-term trends SEER incidence trend estimates Incidence rates by age at diagnosis Stage distribution Median age at diagnosis Recent trends annual SEER incidence rates by RUCC Recent trends SEER incidence trend estimates by RUCC 5-year age adjusted incidence rates by RUCC Recent trends annual U.S. mortality rates Recent trends U.S. mortality trend estimates 5-year age adjusted mortality rates Long-term trends annual U.S. mortality rates Long-term trends U.S. mortality trend estimates Mortality rates by age at death Median age at death Recent trends annual U.S. mortality rates by RUCC Recent trends U.S. mortality trend estimates by RUCC 5-year age adjusted mortality rates by RUCC Recent trends annual SEER survival rates Recent trends SEER survival trend estimates 5-year relative survival Long-term trends annual SEER survival rates Long-term trends SEER survival trend estimates Conditional survival Survival by survival time Complete prevalence Limited duration prevalence Risk of diagnosis Risk of death Preliminary estimates annual SEER incidence rates Preliminary estimates SEER incidence trend estimates

Cols.

Rows Description

13 78,667,287 County denominators; expanded race/Hispanic origin. 13 1,648,320 Annual incidence rates. 17 104,360 Incidence trend estimates. 11 12

67,360 Five-year age-adjusted incidence rates. 488,160 Annual incidence rates.

16

38,815 Incidence trend estimates.

12

11

59,360 Incidence rates stratified by age at diagnosis. 51,440 Stage distribution. 1,592 Median age at diagnosis estimates. 20,700 Annual incidence rates by rural–urban category. 3,832 Incidence trend estimates by rural–urban category. 900 Five-year age-adjusted incidence rates by rural–urban category. 218,880 Annual U.S. mortality rates. 30,836 U.S. mortality trend estimates. 9,120 Five-year age-adjusted mortality rates. 167,580 Annual U.S. mortality rates.

15

14,939 U.S. mortality trend estimates.

10

14 15

18,240 Mortality rates stratified by age at death. 912 Median age at death estimates. 12,096 Annual U.S. mortality rates by rural–urban category. 1,957 U.S. mortality trend estimates by rural–urban category. 612 Five-year age-adjusted mortality rates by rural–urban category. 631,440 Annual SEER survival rates. 39,890 Survival trend estimates.

12 13

66,640 Five-year relative survival estimates. 48,504 Annual SEER survival rates.

14

2,184 Survival trend estimates.

8 6 13 17 11 11 15 10

6 13 17 11

12 12 7 9 9 9 13 17

20

260,160 Conditional survival estimates. 715,440 Survival estimates by survival-time interval. 752 Prevalence. 103,920 Prevalence. 179,520 Risk-of-diagnosis estimates. 179,520 Risk-of-death estimates. 4,200 Preliminary annual incidence rates. 799 Preliminary incidence trend estimates.

Preprint

C

D ETAILS ON LLM-G ENERATED R ESEARCH R EPORTS

For each of the 20 cases in SEER-Repro, we supply the same neutral research prompt to two coding agents: Claude Code with Opus at the maximum effort setting, and Codex with GPT-5.6 Sol at the xhigh effort setting. Both agents have access to all databases and manuals included in SEER-Repro. The prompts specify the research topic and scope without supplying the published findings. Table 7 lists the exact neutral prompt used for each case. Table 7: Neutral research prompts supplied to both report-generation agents for the 20 benchmark cases, in the same paper order as Table 1. Paper (shorter name)

Neutral research prompt

Bone sarcoma survival by tumor site

Analyze tumor-site-specific survival trends for bone sarcomas in SEER 9 from 1980 to 2018, especially variation by histology, site, stage, age, and calendar period.

Sex disparities in cancer incidence

Analyze sex disparities in cancer incidence across SEER 9 from 1975 to 2004, especially male-to-female incidence rate ratios by cancer site, histology, period, and age.

Inflammatory breast cancer incidence and survival

Analyze inflammatory breast carcinoma incidence and survival trends in SEER 9 from 1988 to 2000, especially differences from locally advanced and non-T4 breast cancer by race and subtype.

Thyroid cancer trends

Analyze thyroid cancer incidence and mortality trends in the United States from 1975 to 2009, especially by tumor size, stage, histology, age, sex, and race.

Age at oropharyngeal cancer diagnosis

Analyze changes in age at diagnosis of oropharyngeal cancer in SEER 9 from 1975 to 2016, especially by sex, race, and human-papillomavirus-related tumor site.

Adenoid cystic carcinoma of the breast

Analyze breast adenoid cystic carcinoma in SEER 9 from 1977 to 2006, especially incidence, clinicopathologic characteristics, subsequent primary cancers, and relative survival.

Adult diffuse glioma incidence Analyze incidence trends in adult diffuse gliomas in SEER 9 from 1973 to 2014, especially differences between glioblastoma and nonglioblastoma and their age, period, and birth-cohort effects. Childhood cancer incidence and mortality

Analyze childhood cancer incidence and mortality trends in the United States from 1975 to 1995, especially by International Classification of Childhood Cancer diagnostic group.

Head and neck mucosal melanoma incidence

Analyze incidence and survival trends for mucosal melanoma of the head and neck in SEER 9 from 1987 to 2009, especially by anatomic site, age, sex, race, and stage.

Esophageal adenocarcinoma incidence and mortality

Analyze temporal trends in esophageal adenocarcinoma incidence and incidence-based mortality in the United States from 1975 to 2009, especially by cancer stage, sex, and race.

Second cancers after young-onset colorectal cancer

Analyze the risk of second primary cancers among young-onset colorectal cancer survivors in SEER 9 from 1973 to 2013, including standardized incidence ratios and absolute excess risks.

Lung cancer incidence by sex, race, and histology

Analyze lung cancer incidence trends in SEER 9 from 1973 to 2010, especially differences by sex, race, histology, and stage distribution.

Age disparities in colorectal cancer incidence

Analyze age-related disparities in colon and rectal cancer incidence trends in the United States from 1975 to 2010, especially differences between young adults and older patients by stage.

Fatal prostate cancer incidence Analyze trends in the incidence of fatal prostate cancer in the United States by race from 1975 to 2002, especially differences by race and age and changes following the introduction of PSA screening. Continued on next page

21

Preprint

Table 7 continued Paper (shorter name)

Neutral research prompt

Intracranial glioma incidence and mortality

Analyze intracranial glioma incidence, incidence-based mortality, and relative survival trends in SEER 9 from 1975 to 2018, especially by histology, age, sex, race, and tumor characteristics.

Long-term mortality in young cancer survivors

Analyze 30-year mortality among five-year survivors of adolescent and young adult cancer diagnosed in SEER 9 from 1975 to 2012, distinguishing primary cancer, subsequent malignancy, and other causes of death.

Breast cancer incidence by age Analyze recent trends in female breast cancer incidence in the United States, and tumor features especially by age group, tumor size, stage, and estrogen/progesterone receptor status. Esophageal versus gastric cardia adenocarcinoma

Analyze epidemiologic differences between esophageal adenocarcinoma and gastric cardia adenocarcinoma in the nine SEER population-based registries from 1977 to 1996, especially temporal, ethnic, age, and birth-cohort patterns.

Metastatic cancer incidence and survival

Analyze incidence trends and long-term survival of metastatic cancer in the United States from 1988 to 2018, especially trends in metastasis to specific sites such as liver, lung, bone, and brain.

Cervical cancer incidence by race and histology

Analyze cervical adenocarcinoma and squamous cell carcinoma incidence trends among white and black women in the United States from 1976 to 2000, especially by histology, age, race, and disease stage.

D

P ERTURBATION AGENT: I MPLEMENTATION D ETAILS

We implement the components in Section 3.2.2 with LLM calls and coding agents. This appendix describes their inputs, prompts, configuration, and the validation controls applied at each stage. Section 4 reports the models evaluated. Our experiments include a published paper and Sol- and Opus-generated reports (Appendix C) for each research question. The semantic-call wrapper can request a corrected complete JSON object after schema or parsing failures (Prompt 29). Select Analytical Choices (Step 1). For each published paper p, we identify candidate analytical choices relevant to its fully reproduced findings Fprep . An LLM receives the paper, eligible findings, and reproduction evidence Ep , and extracts choices such as cohort definitions, endpoints, subgroup definitions, statistical models, and interpretation rules (Prompt 7). For each generated report r ∈ Rp , a separate call extracts choices from the report and its available execution log, linking them to the paper’s eligible findings where applicable (Prompt 8). We verify finding identifiers and supporting quotations, attempt to repair invalid candidates (Prompt 9), and remove evidence or candidates that remain ungrounded. The validated candidates from all sources are then pooled: an LLM merges descriptions of the same scientific decision and ranks the resulting choices by scientific importance, uncertainty, feasibility, and redundancy, selecting at most 20 choices before any alternative analysis is run (Prompt 10). Review and repair calls resolve problems with the proposed selection (Prompts 11–15), yielding the finalized choice set Cp used to propose alternatives. Propose Alternatives. (Step 2 of the Perturbation Agent in Figure 2) For each selected analytical choice, an LLM examines how each paper or report implements that choice, using the source text and available execution evidence (Prompt 16) and documents the exact setting. Using these original settings and findings, another LLM proposes a defensible alternative to test across sources, adding a second only when it represents a materially different approach (Prompt 18). A separate LLM reviews whether each proposed alternative is scientifically justified and identifies any conditions required for its use before execution (Prompt 20). Choices without a defensible alternative are not tested. Stage-specific repair calls correct invalid baseline mappings, alternatives, or alternative reviews using the associated validation errors (Prompts 17, 19, and 21). Deterministic checks enforce valid identifiers and complete coverage of the required records. 22

Preprint

Execute Analyses. (Step 3 of the Perturbation Agent in Figure 2) For each choice–alternative comparison (k, s) ∈ Kp , the planning LLM receives the baseline settings, the reviewed alternative, linked findings, feasibility conditions, and execution evidence; it decides whether the comparison should be executed and draws up a concrete plan for its implementation (Prompt 22). Invalid plans are repaired against the validation errors, with deterministic checks of identifiers and comparison coverage (Prompt 23). Executable comparisons are then fed to coding agents for implementation (Prompt 24). The executor first runs the baseline unchanged and checks it against the frozen validation criteria. It then copies the baseline, changes only the specified choice and its listed dependent calculations, and saves the baseline output, alternative output, validation evidence, and exact code diff. These saved artifacts form the comparison evidence ek (s) returned by the Execute Analyses stage in Algorithm 1. The pipeline prepares the SEER-Repro environment for execution. The launch prompt (Prompt 28) directs the coding agent to follow the execution instructions (Prompt 24). The agent recovers the baseline implementation from preserved code, structured execution traces, or legacy transcripts, in that order of preference; it constructs a fresh implementation only when no prior implementation evidence survives. It then validates the unchanged baseline, executes the specified alternatives, and saves the results and supporting evidence. If a Codex session ends successfully without the required results file, a continuation prompt asks it to finish using the same workspace (Prompt 26). If the file remains missing, a final prompt stops further analysis and requires it to record existing results, marking unfinished comparisons as unavailable (Prompt 27). Assess Consequences. (Step 4 of the Perturbation Agent in Figure 2) A separate assessment LLM receives each comparison, its eligibility evidence, frozen comparison criteria, and the saved execution evidence ek (s) (Prompt 25). This implements the Assess Consequences stage in Algorithm 1, assigning an outcome yk (s) using the AnalysisOutcome type defined in Equation 2. For DirectChange outcomes, a further LLM call assigns a scientific-consequence score from 1 to 9 based on the report’s conclusions and the baseline-to-alternative evidence (Prompt 31), adapted from GRADE’s nine-point outcome-importance framework (Guyatt et al., 2008). The score bands are interpreted as described in Section 3.2.2.

E

R EPRODUCTION O UTCOME B REAKDOWNS IN SEER-R EPRO

Table 8 provides the counts and percentages for all three reproduction outcomes for each paper, alongside its numbers of findings and atomic claims. Table 9 reports the 20-paper breakdown using cumulative 1%, 5%, and 10% tolerances for quantitative claims and binary outcomes for qualitative claims. Of 1,695 quantitative claims, 1,299 are reproduced (76.6%), 76 are close, and 320 are not reproduced. Of 495 qualitative claims, 395 are reproduced (79.8%) and 100 are not reproduced. Together these yield the 1,694 reproduced, 76 close, and 420 not-reproduced claims reported in the main text.

F

R EPRODUCTION G RADING D ETAILS

Findings are extracted (Prompt 2), decomposed into atomic targets (Prompt 3), and attempted to be reproduced (Prompt 4). The reproduction grader separates evidence assessment from numerical comparison. An LLM extracts the result reported for each atomic claim and assesses its method and answer provenance (Prompt 5); deterministic code then applies the numerical grading policy. The reproduction agent’s own statement that a finding was reproduced does not determine the final verdict. F.1

J UDGE I NPUTS AND E VIDENCE A SSESSMENT

Inputs and unit of assessment. One judge call receives the reproduction report for an original finding together with its atomic targets. Each target includes its identifier, claim text, reference answer, numerical or semantic evaluation mode, verification target, source quotation, and required analytical assumptions. The judge is instructed to assess every atomic target independently: reproducing one component of a finding does not establish that its other components were reproduced. The grading 23

Preprint

Table 8: Tabular breakdown of the reproduction outcomes shown in Table 1, in the same paper order. Each outcome cell gives the number of atomic claims and the percentage of that paper's atomic claims. Findings counts refer to distinct parent findings represented by these claims. Paper (shorter name) Bone sarcoma survival by tumor site Sex disparities in cancer incidence Inflammatory breast cancer incidence and survival Thyroid cancer trends Age at oropharyngeal cancer diagnosis Adenoid cystic carcinoma of the breast Adult diffuse glioma incidence Childhood cancer incidence and mortality Head and neck mucosal melanoma incidence Esophageal adenocarcinoma incidence and mortality Second cancers after young-onset colorectal cancer Lung cancer incidence by sex, race, and histology Age disparities in colorectal cancer incidence Fatal prostate cancer incidence by race Intracranial glioma incidence and mortality Long-term mortality in young cancer survivors Breast cancer incidence by age and tumor features Esophageal versus gastric cardia adenocarcinoma Metastatic cancer incidence and survival Cervical cancer incidence by race and histology Total

Atomic Findings claims

Reproduced

Close

Not reproduced

56 36 40

188 151 131

187 (99.5%) 140 (92.7%) 119 (90.8%)

0 (0.0%) 1 (0.7%) 2 (1.5%)

1 (0.5%) 10 (6.6%) 10 (7.6%)

33 75 51 60 52 20 33

91 154 128 111 144 54 52

80 (87.9%) 129 (83.8%) 103 (80.5%) 89 (80.2%) 113 (78.5%) 42 (77.8%) 40 (76.9%)

1 (1.1%) 9 (5.8%) 2 (1.6%) 4 (3.6%) 6 (4.2%) 1 (1.9%) 3 (5.8%)

10 (11.0%) 16 (10.4%) 23 (18.0%) 18 (16.2%) 25 (17.4%) 11 (20.4%) 9 (17.3%)

59

124

95 (76.6%) 14 (11.3%)

15 (12.1%)

36

111

80 (72.1%)

7 (6.3%)

24 (21.6%)

27 32 67 62 31

82 63 94 178 67

59 (72.0%) 45 (71.4%) 66 (70.2%) 123 (69.1%) 39 (58.2%)

7 (8.5%) 2 (3.2%) 2 (2.1%) 5 (2.8%) 3 (4.5%)

16 (19.5%) 16 (25.4%) 26 (27.7%) 50 (28.1%) 25 (37.3%)

34

51

29 (56.9%)

2 (3.9%)

20 (39.2%)

51 46

96 120

52 (54.2%) 64 (53.3%)

4 (4.2%) 1 (0.8%)

40 (41.7%) 55 (45.8%)

901

2,190 1,694 (77.4%) 76 (3.5%) 420 (19.2%)

Table 9: Atomic reproduction outcomes by claim type for the 20-paper SEER-Repro benchmark. Quantitative claims Paper

Cit.

Bone sarcoma survival by tumor site 37 Sex disparities in cancer incidence 466 Inflammatory breast cancer incidence and 702 survival Thyroid cancer trends 1,628 Age at oropharyngeal cancer diagnosis 29 Adenoid cystic carcinoma of the breast 188 Adult diffuse glioma incidence 72 Childhood cancer incidence and mortality 438 Head and neck mucosal melanoma incidence 83 Esophageal adenocarcinoma incidence and 510 mortality Second cancers after young-onset colorectal 43 cancer Lung cancer incidence by sex, race, and 368 histology Age disparities in colorectal cancer incidence 1,102 Fatal prostate cancer incidence by race 111 Intracranial glioma incidence and mortality 116 Long-term mortality in young cancer survivors 23 Breast cancer incidence by age and tumor 352 features Esophageal versus gastric cardia 266 adenocarcinoma Metastatic cancer incidence and survival 68 Cervical cancer incidence by race and histology 448 Total

Qualitative claims

1% tolerance

5% tolerance

10% tolerance

All

Reprod.

Not reprod.

119 (88.1%) 87 (75.0%) 78 (74.3%)

135 (100.0%) 114 (98.3%) 98 (93.3%)

135 (100.0%) 115 (99.1%) 100 (95.2%)

135 (100.0%) 116 (100.0%) 105 (100.0%)

52 (98.1%) 26 (74.3%) 21 (80.8%)

1 (1.9%) 9 (25.7%) 5 (19.2%)

52 (61.9%) 101 (73.2%) 75 (64.7%) 56 (65.9%) 56 (52.3%) 28 (56.0%) 15 (40.5%)

74 (88.1%) 117 (84.8%) 92 (79.3%) 67 (78.8%) 84 (78.5%) 38 (76.0%) 25 (67.6%)

75 (89.3%) 126 (91.3%) 94 (81.0%) 71 (83.5%) 90 (84.1%) 39 (78.0%) 28 (75.7%)

84 (100.0%) 138 (100.0%) 116 (100.0%) 85 (100.0%) 107 (100.0%) 50 (100.0%) 37 (100.0%)

6 (85.7%) 12 (75.0%) 11 (91.7%) 22 (84.6%) 29 (78.4%) 4 (100.0%) 15 (100.0%)

1 (14.3%) 4 (25.0%) 1 (8.3%) 4 (15.4%) 8 (21.6%) 0 (0.0%) 0 (0.0%)

31 (32.3%)

70 (72.9%)

84 (87.5%)

96 (100.0%)

25 (89.3%)

3 (10.7%)

51 (52.0%)

69 (70.4%)

76 (77.6%)

98 (100.0%)

11 (84.6%)

2 (15.4%)

20 (39.2%) 28 (60.9%) 53 (58.2%) 65 (47.1%) 19 (37.3%)

29 (56.9%) 31 (67.4%) 63 (69.2%) 98 (71.0%) 26 (51.0%)

36 (70.6%) 33 (71.7%) 65 (71.4%) 103 (74.6%) 29 (56.9%)

51 (100.0%) 46 (100.0%) 91 (100.0%) 138 (100.0%) 51 (100.0%)

30 (96.8%) 14 (82.4%) 3 (100.0%) 25 (62.5%) 13 (81.2%)

1 (3.2%) 3 (17.6%) 0 (0.0%) 15 (37.5%) 3 (18.8%)

7 (25.0%)

10 (35.7%)

12 (42.9%)

28 (100.0%)

19 (82.6%)

4 (17.4%)

10 (15.6%) 25 (42.4%)

26 (40.6%) 33 (55.9%)

30 (46.9%) 34 (57.6%)

64 (100.0%) 59 (100.0%)

26 (81.2%) 31 (50.8%)

6 (18.8%) 30 (49.2%)

976 (57.6%)

1,299 (76.6%)

1,375 (81.1%)

1,695 (100.0%)

395 (79.8%)

100 (20.2%)

24

Preprint

call receives the report and structured targets, rather than independently inspecting the complete code and execution trace or rerunning the analysis. Result availability. For each target, the judge records whether the requested result is reported, not reported, or unclear. A missing reproduction report is assigned a not-reproduced verdict directly. When a report exists but omits the requested result or provides ambiguous evidence, the target likewise receives no reproduction credit. Method validity. The judge assesses whether the reported computation uses a valid method for the specific target. This includes compatibility with required cohort bounds, disease and variable coding, outcome measures, grouping, denominators, standardization conventions, temporal data definitions, and statistical-model settings. A conflicting analytical assumption makes the method invalid; insufficient methodological evidence makes it unclear. The report need not repeat every required assumption verbatim, but it must establish a compatible computation. Both invalid and unclear methods lead to a not-reproduced verdict, even when the extracted numerical value matches the reference. Answer provenance. The judge also flags evidence that the result was obtained by recalling, copying, or searching for the published answer instead of computing it from the supplied data. Such results receive no reproduction credit. Consulting documentation, variable definitions, or coding guidance is permitted when the agent subsequently performs the computation. An exact numerical match alone is not evidence of copying. F.2

N UMERICAL E XTRACTION AND C OMPARISON

Only claims passing the evidence checks in Section F.1 proceed to numerical grading. The process has three steps. 1. Extract the computed result. The LLM extracts each required value, its unit, and its relation (=, <, ≤, >, or ≥), using unrounded values when available. It follows the target’s requested representation—for example, distinguishing the magnitude of a decline from a signed change. It does not calculate numerical error or assign the numerical verdict. 2. Compare each required component. For an ordinary point estimate with reference value t ̸= 0 and computed value x, deterministic code calculates ε=

|x − t| . |t|

The component is Reproduced if ε ≤ 5%, Close if 5% < ε ≤ 10%, and Not reproduced otherwise. Counts also use this rule unless the target specifies another tolerance. The following cases modify this default comparison: Case

Comparison rule

Rounding allowance

If the target specifies its reported decimal precision, accept differences up to half a unit in the last reported decimal place—for example, 0.05 for one decimal place.

Explicit tolerance

Use the target’s exact-equality or absolute-error thresholds instead of the default relative thresholds, subject to the rounding allowance.

Zero reference

Under the default relative rule, require x = 0, unless the rounding allowance applies.

Inequality

Check whether the computed value satisfies the target inequality. If only a bound is reported, it must imply the target bound. A bound cannot establish a required point value. These comparisons have no Close category.

Interval

For an acceptable range, check whether the computed scalar lies within its boundaries, respecting strict or inclusive limits. For a reported interval, compare its two endpoints separately. 25

Preprint

3. Combine the component verdicts. Required components are matched in their declared order; optional uncertainty components are excluded. The least favorable component verdict determines the atomic claim’s verdict. Missing or unparseable required values yield “Not reproduced”. F.3

G RADING Q UALITATIVE C LAIMS

For qualitative or categorical claims, the judge determines whether the reproduced result supports the same conclusion as the published claim. A claim is marked reproduced only when the conclusions are equivalent and the evidence passes the availability, method-validity, and answer-provenance checks described above. Approximately equivalent, conflicting, or unclear conclusions are marked not reproduced. Unlike numerical claims, qualitative claims have no close category. Implementation checks and saved records. The implementation verifies that the judge returns the requested claim identifiers and that extracted numerical results have the required structure. Saved records include the extracted result, assessment fields, rationale, final verdict, and grading-policy version. Numerical thresholds can therefore be reapplied to saved extractions without another judge call. Table 9 reports outcomes by claim type. Scope of the grades. These grades assess whether the reproduction report provides sufficient evidence for the original claim. They do not independently verify the complete analysis code or establish that undocumented analytical choices match those of the original authors.

26

Preprint

G

F ULL T ITLES OF B ENCHMARK PAPERS

Table 10 maps the shorter names used in Table 1 to the full titles of all 20 papers in SEER-Repro, in the same order.

Table 10: Full titles corresponding to the shorter paper names in Table 1. Shorter name

Full paper title

Bone sarcoma survival by tu- Trends in Tumor Site-Specific Survival of Bone Sarcomas from 1980 to 2018: mor site A Surveillance, Epidemiology and End Results-Based Study Sex disparities in cancer inci- Sex Disparities in Cancer Incidence by Period and Age dence Inflammatory breast cancer in- Trends in Inflammatory Breast Carcinoma Incidence and Survival: The cidence and survival Surveillance, Epidemiology, and End Results Program at the National Cancer Institute Thyroid cancer trends

Current Thyroid Cancer Trends in the United States

Age at oropharyngeal cancer Change in Age at Diagnosis of Oropharyngeal Cancer in the United States, diagnosis 1975–2016 Adenoid cystic carcinoma of Adenoid cystic carcinoma of the breast in the United States (1977 to 2006): a the breast population-based cohort study Adult diffuse glioma incidence Trends and patterns of incidence of diffuse glioma in adults in the United States, 1973-2014 Childhood cancer incidence Cancer Surveillance Series: Recent Trends in Childhood Cancer Incidence and mortality and Mortality in the United States Head and neck mucosal Rising Incidence of Mucosal Melanoma of the Head and Neck in the United melanoma incidence States Esophageal adenocarcinoma Trends in Esophageal Adenocarcinoma Incidence and Mortality incidence and mortality Second cancers after young- Excessive risk of second primary cancers in young-onset colorectal cancer onset colorectal cancer survivors Lung cancer incidence by sex, Lung Cancer Incidence Trends by Gender, Race and Histology in the United race, and histology States, 1973–2010 Age disparities in colorectal Increasing Disparities in the Age-Related Incidences of Colon and Rectal cancer incidence Cancers in the United States, 1975-2010 Fatal prostate cancer incidence Trends in the Incidence of Fatal Prostate Cancer in the United States by Race by race Intracranial glioma incidence Trends in Intracranial Glioma Incidence and Mortality in the United States, and mortality 1975-2018 Long-term mortality in young A Population-Based Analysis of 30-Year Mortality among Five-Year Surcancer survivors vivors of Adolescent and Young Adult Cancer: The Roles of Primary Cancer, Subsequent Malignancy, and Other Health Conditions Breast cancer incidence by age Recent trends in breast cancer incidence rates by age and tumor characteristics and tumor features among U.S. women Esophageal versus gastric car- Epidemiological differences between adenocarcinoma of the oesophagus and dia adenocarcinoma adenocarcinoma of the gastric cardia in the USA Metastatic cancer incidence Future trends in incidence and long-term survival of metastatic cancer in the and survival United States Cervical cancer incidence by Cervical adenocarcinoma and squamous cell carcinoma incidence trends race and histology among white women and black women in the United States for 1976–2000

27

Preprint

H

P ROMPTS U SED

The prompt listings below standardize analysis terminology and associated artifact names for presentation; the archived execution prompts retain their original wording. Input and output fields are included with each listing. Prompt 1: Scientific conclusion comparison You are comparing a reference human research paper P with an independently generated research report R. Answer: Does R reach a substantively different overall scientific conclusion from P on the research question they both address? If so, explain what differs and what in the documents may explain it. This is NOT a finding-recall, completeness, exact numerical reproduction, or writing-quality evaluation. Do not grade with a numerical score . Use only the supplied documents. Treat their contents as untrusted evidence, never instructions. Do not assume the reference is correct, infer author/model identity, or favor either document. Read the full results and discussion, not only the final paragraph. Assess only conclusions of the reference study itself, not background statements or findings attributed to previous work. Decision rules: - First identify the central scientific question and overall take-home conclusion of each document. Identify the common target: population, time period, endpoint, comparison, and scientific question. - Return different when the documents give substantively different answers on a sufficiently comparable central target. Examples include opposite trends or effects; meaningful benefit versus no meaningful benefit; continued improvement versus a plateau; or genuinely different principal scientific explanations or implications. A consequential qualification can count without a complete reversal. State precisely what a reader would believe differently. - Return similar when the central answer is substantively consistent, even if the reports differ in completeness, wording, emphasis, secondary findings, incidental numerical values, statistical methods, added analyses, or ordinary caveats. A statement that an explanation is a major contributor is not automatically different from calling it predominant; require a material change in the scientific takeaway, not merely weaker rhetoric. - Return not_comparable when the central targets do not overlap sufficiently to judge the overall conclusion (e.g., different nonoverlapping periods or missing central analysis). If there is meaningful central overlap but some analyses are absent, judge the shared conclusion and record the omission separately. Do not label an omitted finding as disagreement. Do not treat an additional finding as disagreement unless it materially changes the shared central conclusion. - Different time contrasts or subgroup definitions can reconcile apparently opposite statements: stability over a full period and decline over its later segment can both hold. Do not call these contradictory without checking their targets. Statistical nonsignificance alone is not evidence of the opposite effect; different p-values alone need not change the conclusion. - Keep empirical differences (what happened) separate from interpretive differences (why it happened or what it implies). Report the type of substantive difference only if the overall verdict is different. - Explain WHY cautiously. Text can establish that the analyses used different cohorts, endpoints, time windows, definitions, models, adjustments, or interpretive assumptions; it usually cannot establish that such a choice caused a conclusion difference. Separate a documented analytical difference from a plausible causal explanation. Do not speculate about coding errors, model bias, missing variables, or mechanisms absent from the documents . Use insufficient_information when the reason cannot be established. This exercise does not validate execution or establish which conclusion is true. Return only JSON with this exact structure: { "verdict": "similar" | "different" | "not_comparable", "human_takeaway": "Brief central conclusion of the current reference study.", "report_takeaway": "Brief central conclusion of the report.", "shared_question_and_scope": "What can actually be compared, including material scope differences.", "difference_type": "none" | "empirical" | "interpretive" | "both" | "not_comparable", "substantive_differences": [ { "difference": "A material difference in the shared central conclusion, not merely an omitted or secondary finding.", "type": "empirical" | "interpretive", "human_quote": "Short contiguous verbatim reference passage.", "report_quote": "Short contiguous verbatim report passage.", "why_material": "What a reader would conclude differently and why this is not just scope , emphasis, or omission." } ], "explanation": [

28

Preprint

{ "factor": "study_period" | "population_or_cohort" | "endpoint_or_definition" | " statistical_method_or_adjustment" | "interpretation_or_causal_attribution" | " other_documented_factor" | "insufficient_information", "observation": "What the documents establish about a possible explanation for the different conclusions or inability to compare.", "evidential_status": "documented_difference_causal_role_unproven" | " plausible_explanation" | "insufficient_information", "human_quote": "Short contiguous verbatim passage, or null if absent.", "report_quote": "Short contiguous verbatim passage, or null if absent.", "causal_limit": "What cannot be established about why the conclusions differ from these texts alone." } ], "coverage_or_emphasis_notes": "Omissions, scope shifts, secondary differences, or caveats that should not themselves be counted as substantive disagreement.", "overall_evidence": { "human_quote": "Short contiguous verbatim passage supporting the human takeaway.", "report_quote": "Short contiguous verbatim passage supporting the report takeaway." }, "rationale": "Two to four sentences justifying the overall decision." } For similar: difference_type=none, substantive_differences=[], explanation=[] (put nonsubstantive distinctions in coverage_or_emphasis_notes). For different: difference_type=empirical, interpretive, or both consistently with listed differences; provide 1--3 substantive_differences and 1--3 explanation entries. If reasons are unknown, use insufficient_information rather than invent them. For not_comparable: difference_type=not_comparable, substantive_differences=[], and 1--3 explanation entries about scope or missing central analysis. Every quotation must be an exact contiguous source substring after whitespace normalization , preferably 10--35 words and at most 70 words. Retain Markdown formatting within quoted text. Do not insert ellipses or join separated passages. Only explanation quotations may be null. For documented differences or plausible explanations, provide evidence from both documents; for insufficient_information, null is allowed. Do not fill fields with unsupported causal claims. Input fields: reference_paper: full paper text analysis_report: full report text Output fields: { verdict: "similar" | "different" | "not_comparable", human_takeaway: string, report_takeaway: string, shared_question_and_scope: string, difference_type: "none" | "empirical" | "interpretive" | "both" | "not_comparable", substantive_differences: [{ difference: string, type: "empirical" | "interpretive", human_quote: string, report_quote: string, why_material: string }], explanation: [{ factor: "study_period" | "population_or_cohort" | "endpoint_or_definition" | " statistical_method_or_adjustment" | "interpretation_or_causal_attribution" | " other_documented_factor" | "insufficient_information", observation: string, evidential_status: "documented_difference_causal_role_unproven" | " plausible_explanation" | "insufficient_information", human_quote: string | null, report_quote: string | null, causal_limit: string }], coverage_or_emphasis_notes: string, overall_evidence: {human_quote: string, report_quote: string}, rationale: string }

Prompt 2: Finding extraction You are a meticulous oncology-epidemiology research assistant. You are given the full text of a published study that analyzed the SEER cancer registry. Your job is to extract a bank of ATOMIC VERIFIABLE CLAIMS: the study’s concrete, data-derived findings and reproductionrelevant study context, each stated so it could be independently checked from the raw data or used to reproduce a data-derived result.

29

Preprint

Rules for an atomic verifiable claim: - It states ONE verifiable claim only. Split compound findings into separate claims. - It may be numeric or non-numeric, but it must be empirically checkable from the study data or necessary for reproducing such a check. - If the paper reports concrete numbers for the claim, include them in "numbers" with units and context. - If the claim is qualitative but data-derived, such as a distribution shape, ordering, rank, threshold crossing, curve pattern, subgroup comparison, table pattern, or figure pattern, keep "numbers" as an empty list and describe what should be checked in " verification_target". - Also extract operational study context when it is necessary to reproduce findings: cohort definitions, SEER variable/code definitions, inclusion/exclusion criteria, grouping rules, endpoint definitions, statistical methods, plotting methods, adjustment methods, and table /figure construction rules. - Preserve enough detail that another agent could decide what data subset, variables, grouping, transformation, model, plot, or comparison is needed. - Capture the verbatim sentence(s), table row, table note, figure caption, or methods text that support the claim. - Extract findings and reproducibility context from results, tables, figures, figure captions, table notes, and methods sections. - Skip background, motivation, citations of other studies, unsupported interpretation/ speculation, and purely clinical discussion not tied to a reproducible data operation. Return a JSON object with a single key "claims" whose value is a LIST of objects. Each object MUST use exactly these field names, and no others: - "claim": string -- the one-sentence claim, written to be checkable. - "evidence_type": one of "numeric", "categorical", "distributional", "trend_shape", " ranking", "table_fact", "figure_pattern", "method_context", "other". - "numbers": a LIST of objects, each with "value" (string), "unit" (string), and "context" (string). Use an empty list if the claim is non-numeric. - "verification_target": string -- what an analysis agent should compute, plot, compare, inspect, or apply to verify this claim or use it during reproduction. - "source_quote": string -- the verbatim supporting sentence(s), table row, table note, figure caption, or methods text from the paper. Use these exact keys. Do not rename fields. Return only the JSON object, no prose or markdown fences. Paper title: {{title}} Full text: {{full_text}}

Input fields: title: paper title full_text: full paper text Output fields: { claims: [{ claim: string, evidence_type: "numeric" | "categorical" | "distributional" | "trend_shape" | "ranking" | "table_fact" | "figure_pattern" | "method_context" | "other", numbers: [{value: string, unit: string, context: string}], verification_target: string, source_quote: string }] }

Prompt 3: Atomic claim decomposition You are splitting ONE grouped SEER reproduction claim into independently testable scientific findings. For numeric results, "atomic" means one substantive reported scalar. Related numbers retain shared context but become separate atomic claims so each can be reproduced and graded independently. Splitting rules: - Preserve every independently testable scientific result in the source claim. - The curated source_claim.numbers array is the complete numeric target inventory. Every numeric child must correspond to an entry in that array. Never create a target from a number that appears only in source_quote or the full paper text. - Split every substantive point value into its own atomic claim, including each baseline/follow-up value, each subgroup value, numerators, denominators,

30

Preprint

derived percentages, absolute differences, ratios, and other reported results. - Example: "1,524 of 3,223 cases, or 47%" becomes three atomic claims: 1,524, 3,223, and 47%. None is merely optional support. - Example: an increase from 4.9 to 14.3 becomes one claim for 4.9 and one claim for 14.3. Preserve the year and baseline/follow-up role in each claim. - Keep a confidence interval with its point estimate in one atomic claim. Never turn confidence-interval endpoints into standalone claims. - Treat any parenthetical interval immediately following and enclosing a point estimate as optional uncertainty, even when source_claim.numbers calls it a "range" rather than a confidence interval. Example: ‘9.4 (8.9-9.9)‘ is one atomic claim for 9.4 with optional lower=8.9 and upper=9.9, not a second required range claim. - Keep a meaningful range together as one atomic claim. Never turn range endpoints into standalone claims. - Treat cohort definitions and analysis settings as context, not independent findings. Study years, age eligibility, registry counts, stage codes, and similar design numbers belong in required_analysis_assumptions unless the grouped claim explicitly presents that item as the substantive result. - Split genuinely independent outcomes, subgroups, tumor types, or conclusions when each can stand alone scientifically. - Create a qualitative direction or comparison claim only when that conclusion is independently testable and not redundant with the numeric child claims. - Do not invent facts, precision, cohorts, units, or conclusions. - Keep enough cohort, outcome, subgroup, geography, and time context for each atomic claim to stand alone. - Use the full paper to resolve context needed to make the exact result independently reproducible, but include only cohort and analysis details that apply to that result; do not automatically propagate broad paper-level restrictions into every atomic claim. - A numeric atomic answer normally contains exactly one required component and exactly one substantive number in answer.value. Do not include contextual design numbers or other substantive results in answer.value. - A genuinely range-valued result is the sole exception: keep its lower and upper endpoints together as two required components with roles "range_lower" and "range_upper". Set range_semantics="reported_interval" when the paper reports both endpoints as the result and both must be reproduced. Set range_semantics="acceptable_interval" when the range is a criterion and any scalar inside it satisfies the result. Do not use this exception for before/after or group pairs. Otherwise set range_semantics=null. - Example: "the IRR declined to 4-5 at the oldest ages" is an acceptable_interval. A computed oldest-age IRR of 4.814 satisfies it; the agent need not manufacture or reproduce endpoints 4 and 5. - Every required substantive component uses role "primary" except the two endpoints of a genuinely range-valued result. - Preserve the numeric relation for every component. Use relation "eq" for a point estimate or an endpoint that the agent must reproduce, "lt" for "less than" or "below", "lte" for "at most" or "not exceeding", "gt" for "more than" or "above", and "gte" for "at least" or "no less than". Never discard an inequality or turn a lower/upper bound into an equality. - Relation and comparator are separate: relation describes which values satisfy the finding, while comparator describes exact versus tolerance-based numeric matching for equality targets. - Use comparator "relative_numeric" for approximate estimates, percentages, rates, ratios, and confidence limits. Use "exact_numeric" for exact counts or genuinely exact result parameters. - Every required numeric component must include tolerance. Use {"kind":"relative","reproduced":0.05,"close":0.10} for estimates, rates, percentages, ratios, APCs, p-values, and counts. Use an absolute tolerance appropriate to the paper’s precision for years, ages, timing, and targets at or near zero; for example, an approximate turning year reported to the nearest year normally uses {"kind":"absolute","reproduced":1,"close":2}. Use {"kind":"exact","reproduced":0,"close":0} only when exact identity is scientifically required. - Set reported_decimal_places to the number of decimal places visibly reported for an equality target, so an unrounded reproduction that rounds to the paper’s displayed value can match. Set it to null for inequalities and values whose displayed precision is not meaningful. Do not use this field to make vague words such as "about" artificially permissive. - Use one canonical representation for changes and keep claim, label, value, and component consistent. "Decreased by 5%" and "magnitude of the decline" require the positive magnitude 5. "Increased by 5%" requires positive 5. "APC was -5%" or "signed APC" requires signed -5. Never pair change-magnitude wording with a negative target. Preserve threshold relations on the magnitude: "the magnitude of the decline was less than 1%" stores positive 1 with relation="lt". - Keep confidence or uncertainty bounds with their point estimate in answer.uncertainty, not as separate atomic claims and not in components. uncertainty contains kind, level, lower, and upper. Use kind "interquartile_range" for an IQR. Each bound uses the same fields as a component but must have required=false. A correct point estimate can therefore reproduce the finding even when its uncertainty is not reported.

31

Preprint

- Use answer_type "continuous_numeric" when all required result values are percentages, rates, APCs, p-values, ratios, estimates, confidence limits, or other approximate numeric values. - Use answer_type "exact_numeric" only when the substantive result is a fixed numeric parameter or meaningful exact range. - Use answer_type "count" when all required result values are exact counts. - Use answer_type "categorical" for exact named categories or labels. - Use answer_type "qualitative" for directions, rankings, comparisons, and patterns requiring semantic grading. - Copy or closely preserve the source quote that supports each atomic claim. - Populate required_analysis_assumptions with the answer-safe protocol needed to compute this exact result. This setup is shown to the QA agent separately from the question, so it may be more specific than the source sentence. - In required_analysis_assumptions.cohort, preserve every applicable registry, diagnosis-year, sex, race, stage, histology, behavior, primary-sequence, reporting-source, and eligibility restriction. Always state applicable age bounds explicitly and preserve inclusive semantics, for example "Include women age >=40 years" or "Restrict to ages 45-84 years, inclusive". - In required_analysis_assumptions.coding, record the paper’s exact variable, recode, code system and edition, code list, precedence rule, missing-value handling, and exclusion logic when stated. - In required_analysis_assumptions.denominator_and_standardization, record the denominator population, rate scale, standard population, age bins, weight handling, and whether weights are renormalized after an age restriction. - In required_analysis_assumptions.temporal_data, record the data release or vintage, follow-up cutoff, calendar grouping, and era-specific field mapping. - In required_analysis_assumptions.statistical_model, record aggregation, smoothing, regression, joinpoint, survival, forecasting, significance, and model-selection settings needed for the target when the paper states them. - Do not invent a protocol detail. If a necessary choice is not specified in the paper, add a concise statement in the relevant category beginning with "Not specified by the source:" so the ambiguity remains traceable. - Neither required_analysis_assumptions nor agent_methodology_context may reveal the target answer, point estimate, comparison conclusion, or trend. Answer-bearing text belongs in evaluator_evidence. Return only JSON with exactly one top-level key, "atomic_claims". Each item must contain exactly: - "claim" - "evidence_type" - "answer": {"value", "unit", "label", "answer_type", "components", "uncertainty", " range_semantics"} - "verification_target" - "source_quote" - "required_analysis_assumptions": {"cohort", "coding", "denominator_and_standardization", "temporal_data", "statistical_model"} - "agent_methodology_context" - "evaluator_evidence" Each required numeric component must contain exactly "value", "unit", "label", "role", "relation", "comparator", "required", "reported_decimal_places", and "tolerance". tolerance contains exactly "kind", "reproduced", and "close". Optional uncertainty components use the same fields but may set tolerance=null. Relation must be one of "eq", "lt", "lte", "gt", or "gte". For categorical or qualitative answers, set components=[], uncertainty=null, and range_semantics=null. Example estimate with confidence interval: {"value":"2.9 (95% CI 2.7-3.1)","unit":"relative rate","label":"relative rate","answer_type ":"continuous_numeric","components":[{"value":"2.9","unit":"relative rate","label":"point estimate","role":"primary","relation":"eq","comparator":"relative_numeric","required":true ,"reported_decimal_places":1,"tolerance":{"kind":"relative","reproduced":0.05,"close ":0.10}}],"uncertainty":{"kind":"confidence_interval","level":"95%","lower":{"value ":"2.7","unit":"relative rate","label":"lower confidence limit","role":"range_lower"," relation":"eq","comparator":"relative_numeric","required":false,"reported_decimal_places ":1,"tolerance":null},"upper":{"value":"3.1","unit":"relative rate","label":"upper confidence limit","role":"range_upper","relation":"eq","comparator":"relative_numeric"," required":false,"reported_decimal_places":1,"tolerance":null}},"range_semantics":null} Example split for a before/after finding (two items in atomic_claims): {"value":"4.9","unit":"per 100,000","label":"1975 baseline incidence","answer_type":" continuous_numeric","components":[{"value":"4.9","unit":"per 100,000","label":"1975 baseline incidence","role":"primary","relation":"eq","comparator":"relative_numeric"," required":true,"reported_decimal_places":1,"tolerance":{"kind":"relative","reproduced ":0.05,"close":0.10}}],"uncertainty":null,"range_semantics":null} {"value":"14.3","unit":"per 100,000","label":"2009 follow-up incidence","answer_type":" continuous_numeric","components":[{"value":"14.3","unit":"per 100,000","label":"2009 follow -up incidence","role":"primary","relation":"eq","comparator":"relative_numeric","required": true,"reported_decimal_places":1,"tolerance":{"kind":"relative","reproduced":0.05,"close ":0.10}}],"uncertainty":null,"range_semantics":null}

32

Preprint

Example lower-bound finding, "the IRR reached at least 0.7": {"value":"at least 0.7","unit":"IRR","label":"lower-bound IRR","answer_type":" continuous_numeric","components":[{"value":"0.7","unit":"IRR","label":"lower-bound IRR"," role":"primary","relation":"gte","comparator":"relative_numeric","required":true," reported_decimal_places":null,"tolerance":{"kind":"relative","reproduced":0.05,"close ":0.10}}],"uncertainty":null,"range_semantics":null} Example upper-bound finding, "the percentage did not exceed 9%": {"value":"not exceeding 9","unit":"%","label":"upper-bound percentage","answer_type":" continuous_numeric","components":[{"value":"9","unit":"%","label":"upper-bound percentage ","role":"primary","relation":"lte","comparator":"relative_numeric","required":true," reported_decimal_places":null,"tolerance":{"kind":"relative","reproduced":0.05,"close ":0.10}}],"uncertainty":null,"range_semantics":null} Worked example: Source finding: "Overall, the age-adjusted CRC incidence rates significantly decreased between 1975 and 2010 (APC -0.92; 95% CI -1.14 to -0.70, P<0.001)." Better atomic claim (with this paper’s context): "Among histologically confirmed colon or rectal cancers diagnosed in SEER 9 from 1975 through 2010, excluding nursing-home, autopsy, and death-certificate reports, the age-adjusted incidence rate decreased by 0.92% per year (95% CI -1.14% to -0.70%; two-sided P<0.001)." Because this claim asks how much incidence decreased, use positive 0.92 as the sole required answer component and label it "magnitude of annual incidence decline". Store -1.14 and -0.70 as optional confidence bounds because those are signed APC limits, and do not create a separate p-value claim. If the claim were instead worded "the signed APC was -0.92%", preserve -0.92 as the target. Paper metadata: {{paper_metadata}} Grouped source claim: {{source_claim}} Full paper text: {{paper_context}}

Input fields: paper_metadata: {paper_id, pmid, pmcid, title, seer_registries, starting_question, source_insight_id} source_claim: one grouped finding, including its numeric inventory paper_context: full paper text Output fields: { atomic_claims: [{ claim: string, evidence_type: string, answer: { value: string, unit: string, label: string, answer_type: "continuous_numeric" | "exact_numeric" | "count" | "categorical" | " qualitative", components: [{ value: string, unit: string, label: string, role: "primary" | "comparison" | "supporting" | "range_lower" | "range_upper", relation: "eq" | "lt" | "lte" | "gt" | "gte", comparator: "relative_numeric" | "exact_numeric", required: boolean, reported_decimal_places: integer | null, tolerance: { kind: "relative" | "absolute" | "exact", reproduced: number, close: number } | null }], uncertainty: { kind: "confidence_interval" | "uncertainty_interval" | "interquartile_range" | " range", level: string, lower: { value: string, unit: string, label: string, role: "primary" | "comparison" | "supporting" | "range_lower" | "range_upper", relation: "eq" | "lt" | "lte" | "gt" | "gte",

33

Preprint

comparator: "relative_numeric" | "exact_numeric", required: boolean, reported_decimal_places: integer | null, tolerance: { kind: "relative" | "absolute" | "exact", reproduced: number, close: number } | null }, upper: { value: string, unit: string, label: string, role: "primary" | "comparison" | "supporting" | "range_lower" | "range_upper", relation: "eq" | "lt" | "lte" | "gt" | "gte", comparator: "relative_numeric" | "exact_numeric", required: boolean, reported_decimal_places: integer | null, tolerance: { kind: "relative" | "absolute" | "exact", reproduced: number, close: number } | null } } | null, range_semantics: "reported_interval" | "acceptable_interval" | null }, verification_target: string, source_quote: string, required_analysis_assumptions: { cohort: [string], coding: [string], denominator_and_standardization: [string], temporal_data: [string], statistical_model: [string] }, agent_methodology_context: [string], evaluator_evidence: [string] }] }

Prompt 4: Reproduction execution # Task: reproduce one published finding from the SEER 9 data You are given direct query access to the SEER 9 cancer-registry table on a SQL warehouse. Your job is to **independently reproduce the finding below** by analyzing the data yourself (SQL to pull it, then any computation you like), then report whether your computed result or observed pattern matches it. ## Temporal constraint (important) The finding comes from a study published in **{{publication_year}}**. To make this a fair reproduction of what that study could have done, restrict yourself to data and resources that existed **on or before {{publication_year}}**: - Do not use methods, software versions, or published results that postdate {{publication_year}}. The only data-release exceptions are the APPROVED staged population denominators and SEER*Explorer aggregate tables described below. Use those staged resources regardless of their packaging vintage, restrict their rows to the study’s years and cohort, and do not go looking for an older vintage. The SEER 9 case data itself is appropriate to use as provided. ## Do NOT look up the source (independence requirement) This reproduction may use the provided source paper text for methodological context, but the numbers must still be computed independently from the SEER 9 data. Therefore: - You may read ‘paper_context.txt‘ to understand cohort definitions, exclusions, coding choices, statistical methods, tables/figure captions, and other context needed to reproduce the target claim. - Do not search for, open, or fetch the paper, its abstract, or secondary pages from the web -- e.g. PubMed, PubMed Central (ncbi.nlm.nih.gov / pmc), journal sites, ResearchGate, or any page quoting the published numbers. Use the staged ‘paper_context.txt‘ copy only. - Use the STAGED population tables for denominators (see "Population denominators" below) -- do NOT fetch population data from the web. The only acceptable web/reference use is generic, non-paper methodology documentation (e.g. how age-adjustment / APC works). - The target claim below is given ONLY so you can state whether you matched or verified it. Treat any target numbers as a checksum -- do NOT copy results out

34

Preprint

of ‘paper_context.txt‘ or reverse-engineer your cohort/method to land on them. Use the paper for methods/context, compute or inspect the result yourself, then compare. ## The finding to reproduce {{finding_json}} ## Paper context The full source paper text is available in ‘paper_context.txt‘. Use it as context for the study’s cohort, variable definitions, exclusions, and statistical methods. The final reproduced number must come from your own data analysis. ## Local SEER manuals and documentation Official NCI/SEER documentation is available under ‘manuals/‘. Start with ‘manuals/README.md‘, which maps each local file to its source URL and summarizes the contents. Useful paths include: - ‘manuals/seerstat_nov2025/‘: SEER*Stat variable dictionary and Research / Research Plus data description. - ‘manuals/population/‘: population denominator dictionary, modifications, single-year-age methods, hurricane-adjustment notes, and standard-population references. - ‘manuals/coding_manuals/‘: SEER Program Coding and Staging Manuals plus county, state, race, and reportability appendices. - ‘manuals/variable_recodes/‘: site recodes, stage variables, cause-of-death recodes, cause-specific death classification, race/ethnicity, months survived, and tumor-size recodes. - ‘manuals/registry/‘: SEER registry lists, groupings, and population characteristics. Prefer these local manuals over web search for SEER documentation. You can use ‘rg‘ and ‘find‘ to locate HTML/README content; for PDFs, use ‘pdftotext‘, e.g. ‘pdftotext manuals/seerstat_nov2025/TextData.FileDescription-nov2025.pdf - | rg -i "summary stage"‘. For the XLSX variable dictionary, use Python/pandas/openpyxl if the PDF or HTML copy is not enough. ## Official SEER*Explorer aggregate tables The warehouse also contains 34 staged tables from the official November 2024 SEER*Explorer archive. Their fully qualified names and columns are listed in ‘schema.txt‘; all names begin with ‘{{catalog}}.{{schema}}.seer_explorer_‘. These tables provide published aggregate incidence, mortality, survival, prevalence, lifetime-risk, rural/urban, and preliminary-estimate data. Read ‘manuals/seer_explorer_footnotes.md‘ before using an Explorer table. It records data sources, age-adjustment and trend methodology, coding caveats, and tab-specific qualifications. These are approved staged data resources even though the archive packaging is a current release. Restrict queries to the study’s years, population, site, sex, race/ethnicity, age, and stage definitions, and state clearly when an Explorer aggregate rather than the case-level table supports the result. ## How to get the data Read the SEER 9 table with the provided tool: python seer_sql.py "SELECT ... FROM {{case_table}} WHERE ..." - See ‘schema.txt‘ for all column names and notes on the all-STRING typing and missing-value sentinels. - The table is ‘{{case_table}}‘. ## Population denominators -- use ONLY the staged tables The SEER-9 population denominator tables in the database (described in ‘schema.txt‘, ‘{{catalog}}.{{schema}}.seer_pop_*‘) are the APPROVED, authoritative denominators for this task. They cover **1969-2024** with one row per year x county x age x sex x race, so restrict to the years you need with a ‘WHERE year BETWEEN ...‘ clause. - Do NOT fetch external population files (SEER popdata, Census, NBER, etc.) over the web or in your own code -- use the staged tables as-is. - These tables are a current-release vintage. The small population-estimate revision vs. the vintage the original study used is well within tolerance and is NOT worth pursuing -- do not hunt for an older-vintage file. - Restrict the population to the SEER-9 counties (per the FIPS map in ‘schema.txt‘) so the denominator matches the case cohort. ## How to compute -- work however you like You are NOT limited to SQL. Pull whatever data you need with ‘seer_sql.py‘, then analyze it however is most appropriate: - numpy, scipy, pandas, and statsmodels are PRE-INSTALLED -- just ‘import‘ them (no pip needed, and network installs will fail). Use them to save intermediate CSVs and fit regressions (age-adjustment, trend/APC models). - There is no NCI Joinpoint binary here; implement joinpoint-style APC as a

35

Preprint

segmented / piecewise log-linear regression with statsmodels (with its standard errors / CIs) rather than a hand-rolled fit. - Create as many working files in this directory as you want; you have a full scratch workspace here. - You can also choose to just stick with SQL. ## Ground rules - **Read-only on the database only.** The SEER table is shared and must not be modified: no INSERT/UPDATE/DELETE, DROP, CREATE/ALTER TABLE, or new tables/views -- issue **SELECT queries only** against it. This is the one hard restriction. - Everything else (local files, scripts, packages) is fair game. ## What to deliver Write your final report to a file named exactly ‘report.md‘ in this directory. It must contain: 1. How you computed it: the SQL you pulled, plus any Python/analysis you ran. Just put all code under code blocks inside the markdown file. 2. The numbers, pattern, figure/table property, or context item you computed, inspected, or applied. 3. A clear verdict: does your computation reproduce the finding (and how closely)? 4. If it does not match, your best explanation why (cohort definition, coding, age-adjustment standard, SEER 9 vs other registry coverage, etc.). Only ‘report.md‘ is collected after the run; scratch files you create elsewhere are discarded. Input fields: publication_year: source publication year finding_json: one grouped finding catalog, schema, table: database identifiers Files: insight.json, paper_context.txt, schema.txt, manuals/, seer_sql.py Output fields: report.md: string (Markdown report) Contents: computation code; computed or inspected results; reproduction verdict; explanation of discrepancies.

Prompt 5: Atomic reproduction grading You are grading whether a data-analysis agent reproduced EACH atomic result derived from one grouped SEER finding. The report was originally produced for the grouped finding. Inspect every atomic target independently. A value or conclusion for one target must not affect any other target. Your job is evidence extraction and method assessment, not numeric grading: - answer_status is "reported" when the report gives the target result, "not_reported" when it does not, and "unclear" when the evidence is ambiguous. - method_status is "valid" when the reported result was produced with a sound method for this target, "invalid" for a materially wrong cohort, measure, denominator, grouping, or calculation, and "unclear" when the report lacks enough methodological detail. - Treat required_analysis_assumptions as part of the target protocol when assessing method_status. The agent report does not need to repeat every assumption verbatim, but its computation must be compatible with applicable cohort bounds, coding rules, denominator and standardization conventions, temporal data definitions, and statistical-model settings. A conflicting assumption is method_status "invalid"; insufficient evidence is "unclear". - if_cheating is true only when the report indicates that the agent obtained the target result by recalling it, copying it from the source paper, or searching online for the reported answer instead of computing it from the provided data. Looking up documentation, variable definitions, or coding guidance is allowed when the agent then performs the computation itself. Do not infer cheating merely because a computed value matches the target. When the report does not establish a credible computation, use method_status "unclear" or "invalid" as appropriate even if if_cheating is false. - For numeric targets, return one object per corresponding required result in computed_components. Each object has value, relation, and unit. Use the report’s unrounded computed value whenever it is available, in target order and target units. Most targets require one number. A reported interval requires two endpoints; an acceptable interval requires the one computed scalar being tested. Never duplicate one scalar to manufacture two endpoints. Also put a compact legacy rendering of these same values in computed_value. Ignore optional confidence bounds. Set semantic_verdict to "not_applicable". Do not calculate relative error, apply a tolerance, or assign a final verdict. - Normalize change direction to the representation requested by the atomic

36

Preprint

target. If the target asks for the magnitude of a decline or says "decreased by", convert a report’s signed APC of -1.09 to the positive magnitude 1.09. If the target asks for a signed APC, preserve -1.09. Do not otherwise change signs. - For a lower- or upper-bound target, extract the report’s actual computed value, not the target boundary. Example: for target ">5", return "5.51" when the report computed 5.51. For target "P<0.0001", return "2.94e-26" when that is the computed P value. If the report supplies only a bound, preserve its relation in computed_components and operator in computed_value, such as "<0.0001". Never replace a reported value with the reference threshold. - Preserve the unit and scale actually computed. For example, distinguish a percentage from a proportion and a rate per 100,000 from a raw count. Use an empty unit only when the report truly does not state or establish one. - For qualitative or categorical targets, set semantic_verdict to "equivalent", "approximately_equivalent", "conflicting", or "unclear" by comparing the report’s conclusion with the target. Use "not_applicable" only for numeric targets. Return ONLY one JSON object with key "grades". "grades" must contain exactly one object for every supplied atomic_claim_id, with no missing or extra IDs. Each grade object must contain exactly these keys: atomic_claim_id, answer_status, computed_components, computed_value, method_status, if_cheating, semantic_verdict, key_caveat, and rationale. computed_components must be [] for non-numeric targets. Each numeric computed component must have exactly value, relation, and unit, where relation is eq, lt, lte, gt, or gte. if_cheating must be a JSON boolean. Use only the enum values specified above. ## ATOMIC targets and grading policies {{targets}} ## Agent REPORT {{report}}

Input fields: targets: [{atomic_claim_id, claim, answer, evaluation_mode, verification_target, source_quote, required_analysis_assumptions}] report: reproduction report Output fields: { grades: [{ atomic_claim_id: string, answer_status: "reported" | "not_reported" | "unclear", computed_components: [{ value: string, relation: "eq" | "lt" | "lte" | "gt" | "gte", unit: string }], computed_value: string, method_status: "valid" | "invalid" | "unclear", if_cheating: boolean, semantic_verdict: "equivalent" | "approximately_equivalent" | "conflicting" | "unclear" | "not_applicable", key_caveat: string, rationale: string }] }

Prompt 6: Reproduction failure classification You are analyzing failed atomic targets from a scientific reproduction agent that queried staged NCI SEER data. Classify why each target was not reproduced. Use the reproduction report and grader records as evidence. The grader record contains the target, extracted computed value, and prior rationale, but you must distinguish the underlying cause rather than restating the numeric mismatch. Assign exactly one primary_issue per atomic target: - data_unavailable: required records, variables, geography, endpoint, years, or historical schema are absent from the staged data. - target_under_specified: the paper’s exact operational definition, model, test, or context cannot be uniquely inferred from the target and available documentation. - agent_omission: the target appears computable, but the agent did not attempt it or failed to report the result. - cohort_or_coding_error: the agent used the wrong population filter, disease/histology definition, stage, treatment, or tumor-size coding.

37

Preprint

- denominator_or_standardization_error: the agent used the wrong denominator, rate scale, age standard, weighting, or change formula. - statistical_method_error: the agent used an unsuitable test, regression, joinpoint/time segmentation, aggregation, or other statistical procedure. - survival_or_endpoint_error: the agent used the wrong mortality/survival endpoint, censoring, event definition, or follow-up. - valid_alternative_estimate: the agent performed a defensible computation, but a datavintage, extract, small-sample, or reasonable operationalization difference produced a materially different estimate. - qualitative_result_conflict: the computed direction, ranking, or pattern directly conflicts with the target and no narrower cause is supported. - grading_artifact: the substantive result should likely count, but atomicization, parsing, rounding, inequality, or tolerance policy caused rejection. - contamination_or_lookup: the answer was copied, recalled, or obtained online instead of being computed from staged data. - insufficient_evidence: evidence is inadequate to choose another category. issue_codes are narrower contributing diagnoses. Put the code that best supports primary_issue first. Multiple codes are allowed only when independently supported. Use only these exact issue_codes strings: missing_records, missing_variable, missing_geographic_granularity, missing_external_endpoint, temporal_coverage_or_schema, paper_method_not_identifiable, claim_context_incomplete, not_attempted, result_not_reported, wrong_cohort_filter, wrong_disease_or_histology_coding, wrong_stage_treatment_or_size_coding, wrong_denominator, wrong_age_standardization, wrong_rate_or_change_calculation, wrong_statistical_test, wrong_model_or_time_segmentation, small_sample_instability, wrong_survival_endpoint, wrong_censoring_or_followup, data_vintage_or_extract_difference, defensible_operationalization_difference, qualitative_direction_or_ranking_conflict, numeric_tolerance_or_rounding, atomic_target_or_reference_issue, copied_or_recalled_answer, tool_or_query_failure, cause_unclear. Use only these exact responsibility strings: benchmark_data, benchmark_specification, agent , evaluator, scientific_variation, unclear. Use only these exact computability strings: computable_from_staged_data, partially_computable, not_computable_from_staged_data, unclear. Use only the strings high, medium, or low for confidence. The evidence field must be one string, not a list. Important distinctions: - Missing data is not an agent error when the report correctly analyzes and documents the absence. - A value outside tolerance is not automatically an agent error. Use valid_alternative_estimate when the method is defensible and comparable. - Use grading_artifact for values that agree after the benchmark’s intended rounding/ inequality semantics or for a defective atomic reference. - Qualitative grading is intentionally binary. A qualitative grade of close_to_reproduced or approximately_equivalent is therefore an effective failure, not automatically a grading artifact. Diagnose the missing, weakened, or conflicting semantic content. Use grading_artifact only if the report actually establishes the complete target and the judge or reference rejected it incorrectly. - Use target_under_specified only when a hidden paper-specific choice prevents unique reproduction, not merely because the agent chose poorly. - Use agent_omission when data and method appear available but no result was attempted or reported. - Documentation or web lookup is allowed when it supports a real computation. contamination_or_lookup requires that the reported result itself was not computed. - Keep diagnosis and evidence specific to this case. Do not invent unavailable facts. Return JSON with one classifications item for every input atomic_claim_id and no others. Do not create any enum values beyond those listed above. Each item must contain: atomic_claim_id, primary_issue, issue_codes, responsibility, computability, diagnosis, evidence, recommended_action, confidence. FAILED TARGETS: {{failure_rows_json}} REPRODUCTION REPORT: {{report}} Input fields: failure_rows_json: [{atomic_claim_id, claim, target_value, computed_value, rationale, grading_method, judge_verdict, numeric_verdict, if_cheating}] report: reproduction report Output fields: { classifications: [{ atomic_claim_id: string,

38

Preprint

primary_issue: "data_unavailable" | "target_under_specified" | "agent_omission" | " cohort_or_coding_error" | "denominator_or_standardization_error" | " statistical_method_error" | "survival_or_endpoint_error" | "valid_alternative_estimate" | " qualitative_result_conflict" | "grading_artifact" | "contamination_or_lookup" | " insufficient_evidence", issue_codes: ["missing_records" | "missing_variable" | "missing_geographic_granularity" | "missing_external_endpoint" | "temporal_coverage_or_schema" | " paper_method_not_identifiable" | "claim_context_incomplete" | "not_attempted" | " result_not_reported" | "wrong_cohort_filter" | "wrong_disease_or_histology_coding" | " wrong_stage_treatment_or_size_coding" | "wrong_denominator" | "wrong_age_standardization" | "wrong_rate_or_change_calculation" | "wrong_statistical_test" | " wrong_model_or_time_segmentation" | "small_sample_instability" | "wrong_survival_endpoint" | "wrong_censoring_or_followup" | "data_vintage_or_extract_difference" | " defensible_operationalization_difference" | "qualitative_direction_or_ranking_conflict" | " numeric_tolerance_or_rounding" | "atomic_target_or_reference_issue" | " copied_or_recalled_answer" | "tool_or_query_failure" | "cause_unclear"], responsibility: "benchmark_data" | "benchmark_specification" | "agent" | "evaluator" | "scientific_variation" | "unclear", computability: "computable_from_staged_data" | "partially_computable" | " not_computable_from_staged_data" | "unclear", diagnosis: string, evidence: string, recommended_action: string, confidence: "high" | "medium" | "low" }] }

Prompt 7: Scientific frame and eligible claims You construct an outcome-blind scientific frame for one SEER research question. The claim_reproduction_index contains only original paper insights whose required atomic claims were all strictly reproduced. Extract headline and supporting claims only from this supplied index; do not extract additional result claims directly from the paper. Copy each selected entry’s source_insight_id exactly into the corresponding output claim. Then enumerate scientifically material analytical assumptions from the original paper plus ontology gaps suggested by the filtered reproduction_index. The reproduction_index contains execution evidence only for the same fully reproduced insights eligible for this analysis. Do not use, infer, or predict sensitivity results. Do not manufacture candidates to hit a quota. Candidate IDs must be H01, H02, ... in importance-neutral document order. Each candidate is one underlying decision, not a collection of loosely related settings. Include assumptions that are necessary but unspecified. Link every candidate to at least one extracted full claim. Cover every ontology family as covered, unresolved, or genuinely not applicable. Evidence quotes must be exact contiguous substrings from original_paper or serialized reproduction_index. The claim_reproduction_index establishes claim identity and eligibility but is not an evidence-quote source. Use reproduction_index only for what its metadata and claim text actually establish; it is reconstruction evidence, not author execution. Return only the JSON object required by the schema. Input fields: paper_id: integer research_question: string original_paper: string claim_reproduction_index: fully reproduced original findings and atomic verdicts reproduction_index: filtered reproduction metadata and claim text assumption_families: array of the ten family names panel_ceiling: 20 outcome_blindness: string Output fields: { paper_id: integer, research_question: string, claims: [{ claim_id: string, source_insight_id: string, claim: string, importance: "headline" | "supporting", question_scope: string, evidence: [{source: "report" | "query_log" | "reproduction_index", quote: string, location: string}] }], candidates: [{ candidate_id: string,

39

Preprint

family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", name: string, decision: string, original_value: string, observability: "explicit" | "execution_observed" | "implicit_unspecified" | "unknown", linked_claim_ids: [string], applicability: string, consequence_mechanism: string, evidence: [{source: "report" | "query_log" | "reproduction_index", quote: string, location: string}] }], ontology_coverage: [{ family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", status: "covered" | "unresolved" | "not_applicable", rationale: string }] }

Prompt 8: Candidate choices in generated reports Extract scientifically material analytical assumptions from one generated report and its own original query log. Use the supplied canonical paper claims only for linkage; do not force the report to address every claim. Do not use or predict sensitivity outcomes. Candidate IDs must use candidate_prefix followed by sequential two-digit numbers. One candidate must represent one underlying decision. Distinguish explicit report statements, execution-observed settings, necessary but unspecified choices, and genuinely unknown decisions. Do not treat absent analysis as an assumption. Evidence quotes must be exact contiguous substrings of report or query_log. Return only the JSON object required by the schema. Input fields: artifact: source artifact identity and metadata canonical_claims: claims from paper_frame report: string query_log: string observability_limitations: string assumption_families: array of the ten family names candidate_prefix: "S" for the Sol report; "O" for the Opus report outcome_blindness: string Output fields: { artifact_id: string, source: "codex" | "opus", candidates: [{ candidate_id: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", name: string, decision: string, original_value: string, observability: "explicit" | "execution_observed" | "implicit_unspecified" | "unknown", linked_claim_ids: [string], applicability: string, consequence_mechanism: string, evidence: [{source: "report" | "query_log" | "reproduction_index", quote: string, location: string}] }], coverage_gaps: [string] }

Prompt 9: Candidate extraction repair Repair one generated-report candidate set using the supplied validation error. Preserve the artifact identity, candidate order, candidate IDs, scientific decisions, and all already valid evidence. Change only evidence fields needed to make every quote an exact contiguous substring of the named report or query log. Do not import wording or decisions from another source. Return only the complete repaired JSON object required by the schema. Input fields: artifact: source artifact identity and metadata canonical_claims: claims from paper_frame

40

Preprint

report: string query_log: string observability_limitations: string assumption_families: array of the ten family names candidate_prefix: "S" for the Sol report; "O" for the Opus report outcome_blindness: string draft_candidates: previous candidate output validation_error: string Output fields: { artifact_id: string, source: "codex" | "opus", candidates: [{ candidate_id: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", name: string, decision: string, original_value: string, observability: "explicit" | "execution_observed" | "implicit_unspecified" | "unknown", linked_claim_ids: [string], applicability: string, consequence_mechanism: string, evidence: [{source: "report" | "query_log" | "reproduction_index", quote: string, location: string}] }], coverage_gaps: [string] }

Prompt 10: Shared analytical choice selection Create one shared canonical assumption panel for the paper from the candidate pool. Every canonical item must be a neutral scientific decision axis, not an assertion that one source -specific value is the common baseline. Source-specific values, implementations, and disagreements are resolved later in source mapping. Merge candidates that express the same underlying scientific decision across Human, Sol, and Opus evidence even when their chosen values differ. Do not merge merely correlated decisions and do not split incidental implementation details unless changing them independently defines a different meaningful sensitivity question. A single source candidate may imperfectly mention more than one decision. Do not broaden a canonical item merely to absorb all of that candidate’s wording. Attach the candidate to its primary decision axis, preserve nearby decisions as separate canonical items when scientifically warranted, and rely on the later source-mapping stage to inspect the full source evidence. Fixed, data-selected, and differently bounded values may share one canonical axis when they are alternative settings of the same scientific choice. Rank prospectively using only the supplied criteria. Candidate frequency, report verbosity, source identity, and source omissions are not importance evidence. No empirical outcomes are available. Select at most panel_ceiling assumptions and return fewer when remaining candidates are redundant or low value. Preserve meaningful family diversity near the selection boundary. Canonical names and definitions must state what decision is being made and its scope without selecting one source’s value. Canonical IDs must be P<paper_id>-C01, P<paper_id>C02, ... in selected rank order. candidate_refs use <pool key>__<candidate_id>, for example human_paper__H01 or sol_p22_open__S03. Each candidate reference may appear in at most one selected assumption. Score totals must equal the sum of component scores. Return only the JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame candidate_pool: object mapping source artifact to its candidate array ranking_weights: object of string-valued score ranges: headline_link="0-4", consequence_potential="0-4", defensible_choices="0-3", justification_uncertainty="0-3", feasibility="0-2", redundancy_penalty="0 to -3" panel_ceiling: 20 selection_rules: array outcome_blindness: string Output fields: { paper_id: integer, assumptions: [{ canonical_id: string,

41

Preprint

family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", name: string, definition: string, linked_claim_ids: [string], applicability_rule: string, candidate_refs: [string], merge_rationale: string, distinct_from: [string], score: { headline_link: integer, consequence_potential: integer, defensible_choices: integer, justification_uncertainty: integer, feasibility: integer, redundancy_penalty: integer, total: integer }, selection_rationale: string }], excluded: [{ name: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", candidate_refs: [string], reason: string }], diversity_rationale: string, outcome_blindness_statement: string }

Prompt 11: Analytical choice review Review the proposed shared canonical panel without using empirical outcomes. Check that it captures the highest-priority nonredundant decisions, uses neutral decision axes, merges equivalent scientific choices, preserves distinctions between different sensitivity questions, follows the score ranges, avoids source-frequency bias, and has reasonable family diversity. Do not reject a canonical item merely because sources chose conflicting values or implementations; those differences belong in source mapping. Fixed, data-selected, and differently bounded values can be valid members of one axis when switching among them is one coherent sensitivity question. Conversely, reject an item when its components can be varied independently to answer meaningfully different scientific questions. A non-atomic source candidate may be attached to its primary axis without forcing every decision mentioned by that candidate into the same canonical item. Review every canonical assumption in order. Approve only if no scientifically material repair remains; optional wording refinements or lower-ranked alternatives are not required repairs. Return only the JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame candidate_pool: object mapping source artifact to its candidate array panel: canonical_panel output panel_ceiling: 20 Output fields: { paper_id: integer, approved: boolean, assumption_reviews: [{canonical_id: string, approved: boolean, issue: string}], missing_high_priority_assumptions: [string], redundancy_issues: [string], diversity_issues: [string], outcome_leakage_issues: [string], required_repairs: [string] }

Prompt 12: Analytical choice repair Repair the draft canonical panel exactly once using the supplied review. The repaired panel must remain outcome-blind and use only the original claims and candidate pool. Every

42

Preprint

repaired item must be a neutral decision axis; source-specific values and disagreements belong in later source mappings. Do not overmerge distinct decisions to absorb every part of a non-atomic source candidate, and do not split alternative values that answer the same coherent sensitivity question. Fix required omissions, redundancy, merging, scoring, or diversity issues without manufacturing low-value assumptions. Reassign sequential canonical IDs in final selected rank order and keep at most panel_ceiling assumptions. Return only the JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame candidate_pool: object mapping source artifact to its candidate array ranking_weights: object of string-valued score ranges: headline_link="0-4", consequence_potential="0-4", defensible_choices="0-3", justification_uncertainty="0-3", feasibility="0-2", redundancy_penalty="0 to -3" panel_ceiling: 20 selection_rules: array outcome_blindness: string draft_panel: proposed panel review: panel review, or binding_validation_error and binding_repair_requirements Output fields: { paper_id: integer, assumptions: [{ canonical_id: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", name: string, definition: string, linked_claim_ids: [string], applicability_rule: string, candidate_refs: [string], merge_rationale: string, distinct_from: [string], score: { headline_link: integer, consequence_potential: integer, defensible_choices: integer, justification_uncertainty: integer, feasibility: integer, redundancy_penalty: integer, total: integer }, selection_rationale: string }], excluded: [{ name: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", candidate_refs: [string], reason: string }], diversity_rationale: string, outcome_blindness_statement: string }

Prompt 13: Final analytical choice repair Perform the final bounded repair of the shared canonical panel using the supplied review. The repaired panel must remain outcome-blind and use only the original claims and candidate pool. Every item must be a neutral scientific decision axis; source-specific values and disagreements belong in later source mappings. Resolve every scientifically material issue in the review. Do not overmerge independently variable decisions, but do not split fixed, data-selected, or differently bounded values when they answer the same coherent sensitivity question. A non-atomic source candidate may be attached to its primary axis without forcing all decisions mentioned by that candidate into one item. Rerank prospectively, displace lower-priority items when necessary, reassign sequential canonical IDs, and keep at most panel_ceiling assumptions. Return only the JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame candidate_pool: object mapping source artifact to its candidate array

43

Preprint

ranking_weights: object of string-valued score ranges: headline_link="0-4", consequence_potential="0-4", defensible_choices="0-3", justification_uncertainty="0-3", feasibility="0-2", redundancy_penalty="0 to -3" panel_ceiling: 20 selection_rules: array outcome_blindness: string draft_panel: proposed panel review: panel review Output fields: { paper_id: integer, assumptions: [{ canonical_id: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", name: string, definition: string, linked_claim_ids: [string], applicability_rule: string, candidate_refs: [string], merge_rationale: string, distinct_from: [string], score: { headline_link: integer, consequence_potential: integer, defensible_choices: integer, justification_uncertainty: integer, feasibility: integer, redundancy_penalty: integer, total: integer }, selection_rationale: string }], excluded: [{ name: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", candidate_refs: [string], reason: string }], diversity_rationale: string, outcome_blindness_statement: string }

Prompt 14: Final analytical choice review Perform the final outcome-blind specification review of the shared canonical panel. Check that the panel captures the highest-priority nonredundant decisions, uses neutral decision axes, preserves distinctions between independently variable scientific questions, follows the score ranges, avoids source-frequency bias, and has reasonable family diversity. Do not reject an item merely because sources chose conflicting values or implementations; those differences belong in source mapping. Fixed, data-selected, and differently bounded values can share one axis when switching among them is one coherent sensitivity question. A non-atomic source candidate may be attached to its primary axis without forcing every decision it mentions into the same item. Review every canonical assumption in order. Approve only if no scientifically material repair remains. Optional wording refinements and lower-ranked candidate substitutions are not required repairs. Return only the JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame candidate_pool: object mapping source artifact to its candidate array panel: canonical_panel output panel_ceiling: 20 Output fields: { paper_id: integer, approved: boolean, assumption_reviews: [{canonical_id: string, approved: boolean, issue: string}], missing_high_priority_assumptions: [string], redundancy_issues: [string], diversity_issues: [string],

44

Preprint

outcome_leakage_issues: [string], required_repairs: [string] }

Prompt 15: Analytical choice resolution Produce the final outcome-blind canonical panel by resolving the supplied review as a binding checklist, not as optional advice. This is a fresh resolution stage after prior bounded repairs failed. For every entry in ‘review.required_repairs‘, ‘review.missing_high_priority_assumptions‘, ‘ review.redundancy_issues‘, ‘review.diversity_issues‘, and every rejected ‘ assumption_reviews‘ item, make a concrete corresponding change in the returned panel. Do not return an unchanged panel. If a repair says to add or replace an axis, create that separate neutral scientific decision axis and attach the specified candidate evidence; do not hide it inside another independently variable decision. If the panel is already at 20 items, remove or replace the lowest-priority selected item needed to satisfy the repair. If a repair moves or splits candidate support, update candidate_refs, merge_rationale, distinct_from, exclusions, ranking, scores, and diversity_rationale consistently. Before returning, internally analyze the result as if each required repair were a unit test : 1. Every required missing axis is present as its own canonical assumption when requested. 2. Every requested replacement actually removes the rejected item or recasts it exactly as directed. 3. Candidate references support the primary axis to which they are attached. 4. Independently variable population, measurement, time, endpoint, model, inference, subgroup, and interpretation decisions are not overmerged. 5. The panel contains no more than ‘panel_ceiling‘ assumptions, IDs are sequential, scores and selection rationales are prospective, and the diversity rationale describes the returned panel rather than an earlier draft. 6. The panel remains outcome-blind and uses only the supplied claims and candidate pool. Return only the canonical-panel JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame candidate_pool: object mapping source artifact to its candidate array ranking_weights: object of string-valued score ranges: headline_link="0-4", consequence_potential="0-4", defensible_choices="0-3", justification_uncertainty="0-3", feasibility="0-2", redundancy_penalty="0 to -3" panel_ceiling: 20 selection_rules: array outcome_blindness: string draft_panel: proposed panel review: panel review binding_repair_checklist: object Output fields: { paper_id: integer, assumptions: [{ canonical_id: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", name: string, definition: string, linked_claim_ids: [string], applicability_rule: string, candidate_refs: [string], merge_rationale: string, distinct_from: [string], score: { headline_link: integer, consequence_potential: integer, defensible_choices: integer, justification_uncertainty: integer, feasibility: integer, redundancy_penalty: integer, total: integer }, selection_rationale: string }], excluded: [{ name: string,

45

Preprint

family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", candidate_refs: [string], reason: string }], diversity_rationale: string, outcome_blindness_statement: string }

Prompt 16: Source-specific baseline mapping Map one source artifact to every assumption in the frozen canonical panel, in panel order. Mapping state and baseline provenance are separate judgments. An explicit narrative value is not automatically a complete executable baseline. A missing denominator, definition, or window that the analysis necessarily uses is implicit_unspecified, not not_applicable. Use not_applicable only when the source makes no analysis or conclusion requiring the decision. Every not_applicable mapping must use baseline_state unavailable, empirical_feasibility not_applicable, omission false, source_value not_applicable, and an empty evidence list. For Human evidence, reproduction_index supports conditional reconstructability only; it is not author execution. For generated reports, query_log is that artifact’s own execution evidence. Assess justification independently of observability. Evidence quotes must be exact contiguous substrings of report, query_log, or serialized reproduction_index. Evidence may be empty only for a genuinely not-applicable or unsupported implicit mapping, with the limitation explained. Return only the JSON object required by the schema. Input fields: artifact: source artifact identity and metadata research_question: string claims: claims from paper_frame canonical_panel: canonical_panel output report: string query_log: string reproduction_index: filtered reproduction metadata and claim text observability_limitations: string mapping_definitions: object baseline_states: array Output fields: { paper_id: integer, artifact_id: string, source: "human" | "codex" | "opus", mappings: [{ canonical_id: string, mapping: "explicit" | "execution_observed" | "implicit_unspecified" | "not_applicable", baseline_state: "author_traceable" | "execution_traceable" | "report_specified" | " conditionally_reconstructable" | "unavailable", source_value: string, evidence: [{source: "report" | "query_log" | "reproduction_index", quote: string, location: string}], justification: "justifiable" | "conditional" | "not_justifiable" | "unresolved", justification_rationale: string, empirical_feasibility: "feasible" | "uncertain" | "unavailable" | "not_applicable", feasibility_rationale: string, omission: boolean }], overall_limitations: [string] }

Prompt 17: Source mapping repair Repair one source-to-panel mapping using the supplied validation error. Preserve the artifact identity, canonical panel order, evidence-grounded scientific judgments, and all already valid rows. Change only fields needed to satisfy the mapping contract or exactquote validation. Required invariants: - Every canonical assumption appears exactly once and in panel order. - If mapping is not_applicable, baseline_state must be unavailable, empirical_feasibility must be not_applicable, and omission must be false. - If mapping is implicit_unspecified, omission must be true. - Evidence quotes must be exact contiguous substrings of the named report, query_log, or serialized reproduction_index.

46

Preprint

- Human reproduction evidence supports at most conditional reconstructability; it is not author execution. Return only the complete repaired JSON object required by the schema. Input fields: artifact: source artifact identity and metadata research_question: string claims: claims from paper_frame canonical_panel: canonical_panel output report: string query_log: string reproduction_index: filtered reproduction metadata and claim text observability_limitations: string mapping_definitions: object baseline_states: array draft_mapping: previous mapping output validation_error: string Output fields: { paper_id: integer, artifact_id: string, source: "human" | "codex" | "opus", mappings: [{ canonical_id: string, mapping: "explicit" | "execution_observed" | "implicit_unspecified" | "not_applicable", baseline_state: "author_traceable" | "execution_traceable" | "report_specified" | " conditionally_reconstructable" | "unavailable", source_value: string, evidence: [{source: "report" | "query_log" | "reproduction_index", quote: string, location: string}], justification: "justifiable" | "conditional" | "not_justifiable" | "unresolved", justification_rationale: string, empirical_feasibility: "feasible" | "uncertain" | "unavailable" | "not_applicable", feasibility_rationale: string, omission: boolean }], overall_limitations: [string] }

Prompt 18: Shared alternative generation Generate shared outcome-blind alternatives for the frozen canonical panel. Alternatives attach to the canonical scientific decision, not to individual sources. Generate one primary alternative by default and a second only when it represents a materially different defensible direction. It is acceptable to generate none. Each alternative must be scientifically defensible, sufficiently specified, linked to claims, and either preserve the same question or define a bounded claim-support test. Freeze dependent updates, feasibility requirements, executor family, and observable consequence criteria before execution. Do not use or predict sensitivity outcomes. IDs must be <canonical_id>-ALT1 and optionally ALT2 in order. Return one assessment for every canonical assumption in panel order and only the JSON required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame canonical_panel: canonical_panel output source_mappings: object mapping artifact ID to source_mapping output seer9_fields: array of available database column names alternative_ceiling: 2 rules: array Output fields: { paper_id: integer, assessments: [{ canonical_id: string, original_judgment: { label: "justifiable" | "conditional" | "not_justifiable" | "unresolved", rationale: string, conditions: [string] }, alternatives: [{ alternative_id: string, value: string,

47

Preprint

definition: string, scope_relation: "same_question" | "claim_support", scope_rationale: string, dependent_updates: [string], feasibility_requirements: [string], executor_family: "population" | "measurement" | "time" | "missingness" | "denominator " | "endpoint" | "model" | "inference" | "subgroup" | "interpretation", linked_claim_ids: [string], consequence_criteria: [string], judgment: { label: "justifiable" | "conditional" | "not_justifiable" | "unresolved", rationale: string, conditions: [string] } }], no_alternative_reason: string }] }

Prompt 19: Alternative generation repair Repair the supplied shared-alternatives draft so it covers every canonical panel assumption exactly once and in the same order as canonical_panel. Preserve valid existing scientific alternatives. Add the omitted assessments, using zero alternatives with a clear no_alternative_reason when no bounded same-question alternative is defensible, or at most alternative_ceiling alternatives when they are defensible. Alternatives remain shared scientific definitions rather than source-specific inventions. Do not use or predict empirical outcomes. Return only the complete corrected JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame canonical_panel: canonical_panel output source_mappings: object mapping artifact ID to source_mapping output seer9_fields: array of available database column names alternative_ceiling: 2 rules: array draft_alternatives: previous alternative output validation_error: string Output fields: { paper_id: integer, assessments: [{ canonical_id: string, original_judgment: { label: "justifiable" | "conditional" | "not_justifiable" | "unresolved", rationale: string, conditions: [string] }, alternatives: [{ alternative_id: string, value: string, definition: string, scope_relation: "same_question" | "claim_support", scope_rationale: string, dependent_updates: [string], feasibility_requirements: [string], executor_family: "population" | "measurement" | "time" | "missingness" | "denominator " | "endpoint" | "model" | "inference" | "subgroup" | "interpretation", linked_claim_ids: [string], consequence_criteria: [string], judgment: { label: "justifiable" | "conditional" | "not_justifiable" | "unresolved", rationale: string, conditions: [string] } }], no_alternative_reason: string }] }

Prompt 20: Prospective alternative review

48

Preprint

Review every proposed shared alternative independently of empirical outcomes. Determine whether its scientific justification, scope relation, specification, and in-principle SEER9 feasibility support admission. A claim-support alternative may qualify interpretation but cannot directly refute a different estimand. Do not invent replacement alternatives. Return one review per proposed alternative, using exact IDs, and only the JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame canonical_panel: canonical_panel output source_mappings: object mapping artifact ID to source_mapping output seer9_fields: array of available database column names alternative_ceiling: 2 rules: array alternatives: shared_alternatives output Output fields: { paper_id: integer, reviews: [{ canonical_id: string, alternative_id: string, admitted: boolean, same_question_valid: boolean, specification_valid: boolean, feasible_in_principle: boolean, judgment: { label: "justifiable" | "conditional" | "not_justifiable" | "unresolved", rationale: string, conditions: [string] }, rationale: string }], panel_level_issues: [string] }

Prompt 21: Alternative review repair Repair the proposed-alternative review using the supplied validation error. Preserve all scientifically valid admission decisions and rationales from the draft. Return exactly one review for every proposed alternative, in proposed order, using exact alternative IDs. Remove duplicates, invented alternatives, and placeholder rows for assumptions with no proposed alternative. Do not change the proposed alternatives or use empirical outcomes. Return only the complete JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame canonical_panel: canonical_panel output source_mappings: object mapping artifact ID to source_mapping output seer9_fields: array of available database column names alternative_ceiling: 2 rules: array alternatives: shared_alternatives output draft_review: previous review output validation_error: string Output fields: { paper_id: integer, reviews: [{ canonical_id: string, alternative_id: string, admitted: boolean, same_question_valid: boolean, specification_valid: boolean, feasible_in_principle: boolean, judgment: { label: "justifiable" | "conditional" | "not_justifiable" | "unresolved", rationale: string, conditions: [string] }, rationale: string }],

49

Preprint

panel_level_issues: [string] }

Prompt 22: Execution planning Create the cheap eligibility matrix and grouped execution plan for one frozen paper panel. Decide every source-artifact × proposed-alternative pair. Respect source mapping, baseline provenance, alternative admission, evidence availability, and applicability. Untested, unavailable, or implicit choices are never stable. Group executable comparisons into at most job_ceiling paper-family jobs. Prefer deterministic parameterized execution and shared extracts. A job can serve multiple sources and alternatives when definitions and data dependencies overlap. Use fallback_agent only when an admitted important comparison cannot be represented by a deterministic family. Nonexecutable comparisons must have an empty job_id. Comparison IDs must be <artifact_id>__< alternative_id>. Job IDs must be P<paper_id>-J01, P<paper_id>-J02, ... . Return only the JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame canonical_panel: canonical_panel output source_mappings: object mapping artifact ID to source_mapping output alternatives: shared_alternatives output alternative_review: alternative_review output seer9_fields: array of available database column names reproduction_index: filtered reproduction metadata and claim text job_ceiling: 6 executor_policy: string Output fields: { paper_id: integer, comparisons: [{ comparison_id: string, canonical_id: string, alternative_id: string, artifact_id: string, source: "human" | "codex" | "opus", decision: "execute" | "document_only" | "unavailable" | "not_applicable" | " scientifically_rejected", reason: string, baseline_state: "author_traceable" | "execution_traceable" | "report_specified" | " conditionally_reconstructable" | "unavailable", job_id: string }], jobs: [{ job_id: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", canonical_ids: [string], alternative_ids: [string], artifact_ids: [string], executor_mode: "deterministic" | "fallback_agent", executor_family: "population" | "measurement" | "time" | "missingness" | "denominator" | "endpoint" | "model" | "inference" | "subgroup" | "interpretation", shared_extract: string, required_fields: [string], validation_checks: [string] }], document_only_findings: [string], unavailable_findings: [string], planning_limitations: [string] }

Prompt 23: Execution plan repair Repair the supplied draft execution plan so it exactly matches the frozen execution-plan schema and constraints. Preserve the scientific alternatives, source-specific decisions, comparison coverage, and validation intent. Decide every source-artifact × proposedalternative pair exactly once. Use no more than job_ceiling jobs by consolidating compatible jobs that can share extracts or execution families; do not drop executable comparisons merely to meet the cap. Keep non-executable comparisons jobless. Use comparison

50

Preprint

IDs <artifact_id>__<alternative_id> and sequential job IDs P<paper_id>-J01, P<paper_id>J02, ... . Return only the complete corrected JSON object required by the schema. Input fields: paper_id: integer research_question: string claims: claims from paper_frame canonical_panel: canonical_panel output source_mappings: object mapping artifact ID to source_mapping output alternatives: shared_alternatives output alternative_review: alternative_review output seer9_fields: array of available database column names reproduction_index: filtered reproduction metadata and claim text job_ceiling: 6 executor_policy: string draft_execution_plan: previous plan validation_error: string Output fields: { paper_id: integer, comparisons: [{ comparison_id: string, canonical_id: string, alternative_id: string, artifact_id: string, source: "human" | "codex" | "opus", decision: "execute" | "document_only" | "unavailable" | "not_applicable" | " scientifically_rejected", reason: string, baseline_state: "author_traceable" | "execution_traceable" | "report_specified" | " conditionally_reconstructable" | "unavailable", job_id: string }], jobs: [{ job_id: string, family: "population" | "measurement" | "time" | "missingness" | "denominator" | " endpoint" | "model" | "inference" | "subgroup" | "interpretation", canonical_ids: [string], alternative_ids: [string], artifact_ids: [string], executor_mode: "deterministic" | "fallback_agent", executor_family: "population" | "measurement" | "time" | "missingness" | "denominator" | "endpoint" | "model" | "inference" | "subgroup" | "interpretation", shared_extract: string, required_fields: [string], validation_checks: [string] }], document_only_findings: [string], unavailable_findings: [string], planning_limitations: [string] }

Prompt 24: Analysis execution # Task: execute one frozen matched assumption analysis You are the scientific execution component of a frozen SEER-9 assumption-analysis pipeline. You have received exactly one blinded source artifact. Execute every comparison whose decision is ‘execute‘ in ‘execution_plan.json‘. The canonical assumptions, alternatives, linked claims, source mapping, and consequence criteria are frozen. You may not add, remove , merge, reinterpret, or tune them after seeing results. Do not infer or speculate about who produced the source artifact or whether it came from a publication or a model. ## Inputs - ‘run_manifest.json‘: exact jobs and comparison IDs that must be covered. - ‘execution_plan.json‘: grouped jobs, required fields, validation checks, and all sourcealternative decisions. - ‘paper_frame.json‘: research question and frozen source claims. - ‘canonical_panel_selected.json‘: frozen canonical assumptions. - ‘canonical_alternatives.json‘: frozen alternatives and consequence criteria. - ‘alternative_admission.json‘: prospective admission judgments. - ‘source_mapping.json‘: the frozen baseline mapping for the single supplied source artifact. - ‘source_evidence.json‘: the source report and its available execution evidence.

51

Preprint

- ‘supporting_evidence_manifest.json‘ and ‘supporting_evidence/‘: additional reconstruction or triangulation evidence when available. This evidence is not proof that the source’s baseline specification was executed exactly. - ‘code_capsule_manifest.json‘ and ‘code_capsules/‘: baseline implementation evidence for the single supplied artifact. ‘captured_workspace‘ is preserved executable work, ‘ structured_trace_recovery‘ is identity-neutral structured tool-call evidence from the original execution, ‘transcript_recovery‘ is a legacy implementation recovered from transcript/query evidence, and ‘fresh_reconstruction‘ means no prior implementation survived. - ‘schema.txt‘, ‘manuals/‘, ‘seer_sql.py‘, and ‘operator_helpers.py‘. - ‘result_schema.json‘: required output schema. ## Local SEER manuals and documentation Official NCI/SEER documentation is staged in the execution environment under ‘manuals/‘. Start with ‘manuals/README.md‘, which maps each file to its source URL and summarizes its contents. Useful paths include: - ‘manuals/seerstat_nov2025/‘: SEER*Stat variable dictionary and Research / Research Plus data description. - ‘manuals/population/‘: population denominator dictionary, modifications, single-year-age methods, hurricane-adjustment notes, and standard-population references. - ‘manuals/coding_manuals/‘: SEER Program Coding and Staging Manuals plus county, state, race, and reportability appendices. - ‘manuals/variable_recodes/‘: site recodes, stage variables, cause-of-death recodes, cause -specific death classification, race/ethnicity, months survived, and tumor-size recodes. - ‘manuals/registry/‘: SEER registry lists, groupings, and population characteristics. Prefer these staged official manuals over unstated conventions or memory. Use ‘rg‘, ‘find‘, and ‘pdftotext‘ to locate the exact definitions needed for each baseline and alternative, and cite the relevant manual paths in saved validation evidence. ## Official SEER*Explorer aggregate tables The warehouse also contains 34 staged tables from the official November 2024 SEER*Explorer archive. Their names and columns are listed in ‘schema.txt‘; all names begin with ‘seer. seer.seer_explorer_‘. These tables provide published aggregate incidence, national mortality, survival, prevalence, lifetime-risk, rural/urban, and preliminary-estimate data. In particular, use the U.S. mortality tables when the research question requires calendaryear national vital-statistics mortality rather than mortality conditional on a SEER diagnosis. Read ‘manuals/seer_explorer_footnotes.md‘ before using an Explorer table. It records data sources, age-adjustment and trend methodology, coding caveats, and tab-specific qualifications. These are approved staged data resources even though the archive packaging is a current release. Restrict queries to the study’s years, population, site, sex, race/ ethnicity, age, and stage definitions, and state clearly when an Explorer aggregate rather than the case-level table supports the result. Do not copy a published target result from source literature; independently select and analyze the applicable staged rows. No prior sensitivity outcome, adjudication, or verdict is available or permitted. Do not search the web, inspect files outside ‘/work‘, or instantiate another agent. ## Execution rules 1. Use ‘seer.seer.seer9‘ as the case backbone. Auxiliary population tables are allowed only as denominators. 2. Use only ‘seer_sql.py‘ for SQL. It is read-only, caches complete results, and records ‘ query_log.jsonl‘. Retrieve aggregate sufficient statistics whenever possible. 3. Implement all new or modified analysis under ‘analysis/‘. Save executable Python/R/SQL code, intermediate checks, compact result tables, validation receipts, and unified diffs. Do not rely on narrative assertions alone. 4. Treat the supplied artifact as the only baseline in this workspace. Select one of its code capsules without attempting to identify or compare it with another source. Prefer a successful ‘captured_workspace‘ capsule, then ‘structured_trace_recovery‘. Use legacy ‘ transcript_recovery‘ only when neither is available for that estimand, and use ‘ fresh_reconstruction‘ only when no implementation evidence survives. 5. Materialize one frozen baseline implementation per ‘source_artifact::job_id‘ and reuse it across every related comparison in that job. Do not independently reconstruct the same source/estimand for each alternative. 6. Before changing any assumption, run the selected baseline implementation unchanged. Apply every job-level validation check, save the validation evidence under ‘analysis/‘, and record ‘passed‘, ‘failed‘, or ‘unavailable‘. A reproduction report or a matching published number is supporting evidence, not automatic baseline certification. A comparison cannot be ‘complete‘ unless unchanged baseline validation passes. 7. Create the alternative by copying the validated baseline implementation and changing only the frozen canonical decision and its listed dependent updates. Keep all other code, filters, definitions, windows, estimators, and thresholds fixed. Save an exact unified diff from baseline to alternative. 8. Do not tune code, windows, definitions, model choices, or thresholds to reproduce a reported number or to create a consequential result.

52

Preprint

9. If baseline evidence, a required field, same-question comparability, or computation is unavailable, return ‘inconclusive‘ or ‘unavailable‘. Never call an unexecuted comparison stable. 10. Execution outputs contain measurements only. Do not assign ‘direct_claim_change‘, ‘ interpretive_qualification‘, or ‘stable_under_test‘; a separate adjudication stage applies those labels. 11. Interpretation-family jobs must use a frozen deterministic rubric over the supplied claims and empirical outputs. Save the rubric and its application under ‘analysis/‘. ## Required output Write ‘execution_bundle.json‘ conforming exactly to ‘result_schema.json‘. - Include every job in ‘run_manifest.json‘ exactly once and in order. - Include every executable comparison assigned to that job exactly once and no others. - For each comparison, provide ‘status‘, structured ‘baseline‘, structured ‘alternative‘, structured ‘metrics‘, ‘evidence_files‘, and ‘limitations‘. - For each comparison, provide ‘code_lineage‘ with the capsule origin and ID, the exact ‘ artifact_id::job_id‘ reuse key, SHA256 records for baseline files, baseline-validation evidence, alternative files, and the exact unified baseline-to-alternative diff. - Completed comparisons require nonempty baseline, alternative, and metrics objects. - Completed comparisons also require a nonempty baseline implementation, passed baseline validation with evidence, a nonempty alternative implementation, and a nonempty unified diff. - Inconclusive or unavailable comparisons require a specific limitation and saved diagnostic evidence. - Every evidence path and implementation path must begin with ‘analysis/‘, exist, and be nonempty. Before finishing, validate the JSON locally against ‘result_schema.json‘ and verify exact ID coverage against ‘run_manifest.json‘. Input fields: run_manifest.json: job and comparison IDs execution_plan.json: grouped jobs and comparison decisions paper_frame.json: research question and claims canonical_panel_selected.json: canonical assumptions canonical_alternatives.json: alternatives and consequence criteria alternative_admission.json: admission judgments source_mapping.json: baseline mapping source_evidence.json: report and execution evidence supporting_evidence_manifest.json, supporting_evidence/: additional evidence when available code_capsule_manifest.json, code_capsules/: baseline implementation evidence schema.txt, manuals/, seer_sql.py, operator_helpers.py result_schema.json: output schema Output fields: execution_bundle.json: { paper_id: integer, executor: string, implementation_files: [string], jobs: [{ job_id: string, family: string, status: "complete" | "partial" | "inconclusive", comparisons: [{ comparison_id: string, status: "complete" | "inconclusive" | "unavailable", baseline: object, alternative: object, metrics: object, code_lineage: { origin: "captured_workspace" | "structured_trace_recovery" | "transcript_recovery" | "fresh_reconstruction", capsule_id: string, reused_implementation_key: string, baseline_files: [{path: string, sha256: string}], baseline_validation: { status: "passed" | "failed" | "unavailable", evidence_files: [{path: string, sha256: string}] }, alternative_files: [{path: string, sha256: string}], baseline_to_alternative_diff: {path: string, sha256: string} | null }, evidence_files: [string], limitations: [string] }], limitations: [string] }],

53

Preprint

limitations: [string] }

Prompt 25: Consequence adjudication Adjudicate the completed matched source--alternative comparisons using only the supplied frozen claims, canonical assumptions, admitted alternatives, consequence criteria, and deterministic family results. Do not invent analyses, alternatives, thresholds, claims, or evidence. Use exactly one outcome per comparison: - ‘direct_claim_change‘: the frozen criterion shows a source claim changes truth value, direction, significance, or declared categorical conclusion. - ‘interpretive_qualification‘: the result materially narrows scope, robustness, or evidentiary strength without directly changing the claim. - ‘stable_under_test‘: the completed admitted comparison does not meet the frozen consequence criterion. - ‘inconclusive‘: execution or evidence cannot support a verdict. - ‘unavailable‘: execution established that required data, fields, baseline settings, or source evidence are unavailable. Apply the alternative’s listed consequence criteria explicitly. A numerical difference alone is not automatically consequential. A ‘claim_support‘ alternative ordinarily yields an interpretive qualification rather than a direct claim change unless the frozen criterion explicitly determines the source claim’s truth value. A runtime result whose status is ‘ unavailable‘ must remain ‘unavailable‘; a runtime result whose status is ‘inconclusive‘ must remain ‘inconclusive‘. Never convert missing, failed, unavailable, or inconclusive evidence into stability. Cite only supplied evidence-file paths. Return every supplied comparison exactly once and only the JSON object required by the schema. Input fields: paper_id: integer batch: {index: integer, count: integer} claims: claims from paper_frame canonical_assumptions: relevant canonical_panel.assumptions alternatives: relevant shared alternative records comparisons: executable comparison records from execution_plan family_results: saved execution results and review evidence outcome_rule: string Output fields: { paper_id: integer, reviews: [{ comparison_id: string, outcome: "direct_claim_change" | "interpretive_qualification" | "stable_under_test" | " inconclusive" | "unavailable", affected_claim_ids: [string], criterion_application: [string], rationale: string, evidence_files: [string], confidence: "high" | "medium" | "low" }], limitations: [string] }

Prompt 26: Execution continuation The previous turn stopped before producing /work/execution_bundle.json. Continue in the same workspace and reuse all existing files under /work/analysis. Do not narrate another plan and do not restart completed analysis. Finish every remaining frozen comparison, write /work/execution_bundle.json, validate it against /work/result_schema.json and the exact IDs in /work/execution_plan.json, and do not finish until the required bundle exists with exact comparison coverage. Input fields: run_manifest.json: job and comparison IDs execution_plan.json: grouped jobs and comparison decisions paper_frame.json: research question and claims canonical_panel_selected.json: canonical assumptions canonical_alternatives.json: alternatives and consequence criteria alternative_admission.json: admission judgments source_mapping.json: baseline mapping source_evidence.json: report and execution evidence

54

Preprint

supporting_evidence_manifest.json, supporting_evidence/: additional evidence when available code_capsule_manifest.json, code_capsules/: baseline implementation evidence schema.txt, manuals/, seer_sql.py, operator_helpers.py result_schema.json: output schema analysis/: existing code, results, and evidence Output fields: execution_bundle.json: { paper_id: integer, executor: string, implementation_files: [string], jobs: [{ job_id: string, family: string, status: "complete" | "partial" | "inconclusive", comparisons: [{ comparison_id: string, status: "complete" | "inconclusive" | "unavailable", baseline: object, alternative: object, metrics: object, code_lineage: { origin: "captured_workspace" | "structured_trace_recovery" | "transcript_recovery" | "fresh_reconstruction", capsule_id: string, reused_implementation_key: string, baseline_files: [{path: string, sha256: string}], baseline_validation: { status: "passed" | "failed" | "unavailable", evidence_files: [{path: string, sha256: string}] }, alternative_files: [{path: string, sha256: string}], baseline_to_alternative_diff: {path: string, sha256: string} | null }, evidence_files: [string], limitations: [string] }], limitations: [string] }], limitations: [string] }

Prompt 27: Execution finalization STOP ANALYSIS AND FINALIZE NOW. Do not run more SQL, inspect more manuals or source evidence, or write narrative reports. Reuse the analysis already present under /work/ analysis. Your first tool call must create /work/execution_bundle.json covering every exact comparison ID in /work/execution_plan.json. For any comparison whose evidence cannot be completed from existing files, record it as unavailable with a precise limitation rather than doing more research. Then validate the bundle against /work/result_schema.json and the frozen plan, repair only structural validation defects, and do not finish until /work/ execution_bundle.json exists and validates. Input fields: run_manifest.json: job and comparison IDs execution_plan.json: grouped jobs and comparison decisions paper_frame.json: research question and claims canonical_panel_selected.json: canonical assumptions canonical_alternatives.json: alternatives and consequence criteria alternative_admission.json: admission judgments source_mapping.json: baseline mapping source_evidence.json: report and execution evidence supporting_evidence_manifest.json, supporting_evidence/: additional evidence when available code_capsule_manifest.json, code_capsules/: baseline implementation evidence schema.txt, manuals/, seer_sql.py, operator_helpers.py result_schema.json: output schema analysis/: existing code, results, and evidence Output fields: execution_bundle.json: { paper_id: integer, executor: string, implementation_files: [string], jobs: [{

55

Preprint

job_id: string, family: string, status: "complete" | "partial" | "inconclusive", comparisons: [{ comparison_id: string, status: "complete" | "inconclusive" | "unavailable", baseline: object, alternative: object, metrics: object, code_lineage: { origin: "captured_workspace" | "structured_trace_recovery" | "transcript_recovery" | "fresh_reconstruction", capsule_id: string, reused_implementation_key: string, baseline_files: [{path: string, sha256: string}], baseline_validation: { status: "passed" | "failed" | "unavailable", evidence_files: [{path: string, sha256: string}] }, alternative_files: [{path: string, sha256: string}], baseline_to_alternative_diff: {path: string, sha256: string} | null }, evidence_files: [string], limitations: [string] }], limitations: [string] }], limitations: [string] }

Prompt 28: Execution task Read AGENTS.md and execute the complete frozen analysis for the single supplied source artifact. Save execution_bundle.json and all analysis evidence exactly as instructed. Do not infer the source identity, search the web, or launch other agents. Input fields: run_manifest.json: job and comparison IDs execution_plan.json: grouped jobs and comparison decisions paper_frame.json: research question and claims canonical_panel_selected.json: canonical assumptions canonical_alternatives.json: alternatives and consequence criteria alternative_admission.json: admission judgments source_mapping.json: baseline mapping source_evidence.json: report and execution evidence supporting_evidence_manifest.json, supporting_evidence/: additional evidence when available code_capsule_manifest.json, code_capsules/: baseline implementation evidence schema.txt, manuals/, seer_sql.py, operator_helpers.py result_schema.json: output schema Output fields: execution_bundle.json: { paper_id: integer, executor: string, implementation_files: [string], jobs: [{ job_id: string, family: string, status: "complete" | "partial" | "inconclusive", comparisons: [{ comparison_id: string, status: "complete" | "inconclusive" | "unavailable", baseline: object, alternative: object, metrics: object, code_lineage: { origin: "captured_workspace" | "structured_trace_recovery" | "transcript_recovery" | "fresh_reconstruction", capsule_id: string, reused_implementation_key: string, baseline_files: [{path: string, sha256: string}], baseline_validation: { status: "passed" | "failed" | "unavailable", evidence_files: [{path: string, sha256: string}] }, alternative_files: [{path: string, sha256: string}],

56

Preprint

baseline_to_alternative_diff: {path: string, sha256: string} | null }, evidence_files: [string], limitations: [string] }], limitations: [string] }], limitations: [string] }

Prompt 29: Structured output repair Return a corrected complete JSON object matching the supplied schema. Preserve supported scientific content, fix the validation error, and do not add commentary. Validation error: {{validation_error}} Input fields: schema: JSON Schema defining output fields, nested structure, and types inputs: task input values Previous assistant response validation_error: parsing or schema error Output fields: object (field names, nested structure, and value types defined by the supplied schema)

Prompt 30: One-shot analytical-choice analysis # Consequential-assumption analysis Analyze the completed research report in ‘report.md‘ for consequential analytical assumptions. A consequential analytical assumption is a choice in population, measurement, time window, denominator, endpoint, model, inference, subgrouping, missing-data handling, or interpretation for which a plausible and scientifically justifiable alternative can materially change a substantive reported finding or conclusion. Use all available evidence: - the research question in ‘question.txt‘; - the completed report in ‘report.md‘; - the original report-generation SQL history in ‘original_query_log.jsonl‘, when present; - the surviving original-agent output files under ‘original_agent_evidence/‘, when present; - for published Human papers, the selected GPT-5.6 Sol / SEER-9 reproduction reports, SQL logs, transcripts, and provenance under ‘reproduction_evidence/‘; - the source-specific baseline implementation manifest in ‘code_capsule_manifest.json‘ and preserved or recovered implementation evidence under ‘code_capsules/‘; - the SEER schema in ‘schema.txt‘; - local official documentation under ‘manuals/‘; - live read-only SEER 9 access through ‘seer_sql.py‘. Treat reproduction evidence as an additional feasibility and triangulation signal. It is a reconstruction of claims from the paper, not original author execution and not proof that the paper’s baseline specification was reproduced exactly. Prefer a successful ‘ captured_workspace‘ capsule, then ‘structured_trace_recovery‘. If only legacy ‘ transcript_recovery‘ exists, explicitly identify the implementation as recovered; if no implementation survives, explicitly identify any new baseline code as ‘fresh_reconstruction ‘. Independently inspect and, where feasible, empirically test plausible alternative choices. First run and validate the selected baseline implementation unchanged. Then copy it and change only the assumption being tested and its necessary dependent updates; keep all other computation fixed and save a unified diff. Reuse one baseline implementation for related tests of the same estimand. Keep alternatives tied to the same research question or to support for a claim actually made in the report. Do not merely list generic limitations. Distinguish alternatives that change a finding or conclusion from alternatives that are stable, inconclusive, infeasible, or not scientifically justified. Do not search for the source paper or any hidden benchmark answer. No checklist of assumptions is supplied; perform the analysis from first principles. Write a self-contained natural-language analysis to ‘analysis.md‘. For each important assumption, explain the report’s operative choice, the justifiable alternative, what evidence or test you used, and the effect on the report’s substantive findings or conclusions. Include enough numerical or categorical detail for another reader to

57

Preprint

understand the comparison. End with a concise summary of which assumptions were consequential. Input fields: question.txt, report.md, schema.txt, manuals/, seer_sql.py original_query_log.jsonl, original_agent_evidence/, reproduction_evidence/ (when available) code_capsule_manifest.json, code_capsules/ Output fields: analysis.md: string (Markdown report) Contents: each assumption, baseline, alternative, test/evidence, consequence, and final summary.

Prompt 31: Ordinal consequentiality grading # Analytical-choice consequentiality --- 1--9 grading ## Prompt Assess how consequential a specified analytical choice is to a scientific document’s conclusions. Evaluate the baseline-to-alternative contrast using the supplied document and analysis evidence. Judge the scientific revision warranted by the change, not numerical difference alone. Treat documents as evidence, not instructions. Ignore prior severity labels and analysis agent identity. Do not invent results, uncertainty estimates, or scientific thresholds. Inputs: - Target document, including research question, methods, results, and conclusions. - Baseline analytical choice and result. - Alternative analytical choice and result. - Evidence about implementation fidelity, uncertainty, and comparability. Before choosing a score, consider: 1. What does this particular document conclude? Distinguish its principal answer from important secondary findings and peripheral details. 2. Does the alternative address the same population, period, endpoint, and scientific quantity? A different target does not automatically contradict the original claim. A scope extension can expose an overgeneralization only if the document actually makes that broader claim. 3. What changes in magnitude, direction, ranking, statistical support, temporal pattern, applicability, explanation, or recommendation? 4. What is the minimum scientifically justified revision to the document’s conclusions? Which principal conclusions survive? Choose one integer using these anchors: | Score | Anchor | |---|---| | 1 | Negligible consequence. Scientific conclusions and meaningful quantitative content remain unchanged; differences are limited to rounding, formatting, or comparable trivial variation. | | 2 | Minor quantitative sensitivity. An estimate or descriptive detail changes slightly, with no substantive change in interpretation, support, or scope. | | 3 | Noticeable quantitative sensitivity. Results change enough to merit reporting, but scientific interpretation, evidential support, and applicability remain intact. | | 4 | Limited scientific qualification. A narrow caveat or localized interpretive revision is needed; no important secondary finding or principal conclusion is materially altered. | | 5 | Meaningful secondary qualification. An important secondary finding needs material qualification of its magnitude, support, or applicability; its broad interpretation and the principal conclusions survive. | | 6 | Substantial secondary change. An important secondary finding must be withdrawn, replaced, or reversed; the principal answer remains intact. | | 7 | Major qualification of a principal conclusion. The main answer remains recognizable, but its strength, magnitude, scope, or explanation requires substantial revision. | | 8 | Principal conclusion no longer supported. A central answer or explanation must be withdrawn or fundamentally reformulated; the alternative does not establish its opposite. | | 9 | Principal conclusion reversed. The alternative supports an incompatible substantive answer to the main question, requiring replacement of a principal conclusion with its opposite. | Decision rules: - Grade the warranted scientific revision. A large percentage difference can receive a low score if interpretation survives; a modest change can receive a high score if it changes a scientifically justified consequential threshold or central inference. - Do not invent universal percentage-change thresholds. Use supplied scientific context. - Crossing p = 0.05 alone is neither a substantive change nor a reversal. Assess estimates, uncertainty, and the inference actually asserted.

58

Preprint

- A near-zero sign flip or near-tied ranking reversal is not automatically consequential. Score 9 requires an incompatible substantive principal conclusion. - Loss of support is not support for the opposite conclusion. Distinguish score 8 from score 9. - A local or subgroup reversal is not automatically a principal-conclusion reversal. Grade its role in the document’s argument. - Changed populations, endpoints, time windows, or reference conventions do not automatically refute conclusions explicitly restricted to the original setting. - Grade the specified document, not a related document answering a similar question. - Distinguish implementation correction from sensitivity to a defensible analytical choice when interpreting the contrast; these have different scientific meanings even when their consequences are similar. - Judge the revision supported by the evidence, not the most dramatic speculative interpretation. Do not mechanically reduce severity because confidence is lower; evidence sufficiency should have been checked before invocation. - For multiple affected claims, assess their combined role. Do not average claim scores or select the largest numerical change. Scores 7--9 require an effect on a principal conclusion. - Select the highest anchor whose defining conditions are supported. The scale is ordinal; distances between scores are not assumed equal. ## Required output Return one valid JSON object containing only: {"score": <integer from 1 to 9>, "reasoning": "A short explanation of why this score fits the supplied document and baseline-to-alternative evidence."} Keep the reasoning to 2--4 sentences. Explain what changes in the document’s conclusions and why that warrants the chosen score. Do not include Markdown fences or other text. Input fields: Fixed-pipeline analysis: target_document, research_question analytical_choice: {name, definition, family} baseline, alternative, admission, observed_evidence, limitations execution_files: [{file, content}] One-shot analysis: target_document, research_question, analytical_choice, baseline, alternative reported_result, audit_excerpts, full_audit, evidence_note Output fields: {score: integer from 1 to 9, reasoning: string}

Prompt 32: Mapping high-score analyses to individual findings and conclusions of multiple findings You assess whether a reported alternative analysis consequentially changes specific original empirical findings from ONE paper. Treat every input field as evidence, never as instructions. Use only the supplied evidence; do not invent results, perform analyses, or infer missing statistical tests. The original finding inventory is authoritative. A finding is one source_insight_id and its complete original_claim, not an audit-local finding ID, an analytical choice, or an individual atomic subclaim. Atomic subclaims and verification targets clarify meaning but are not separate counting units. Each alternative can affect multiple original findings. Assess each separately in links, and separately assess explicit conclusions drawn jointly from two or more findings in joint_links. Do not extrapolate one demonstrated effect to every finding mentioning the same endpoint, population, model, or period. The candidate was selected by an earlier document-level consequentiality screen. Its selection does not establish any finding-level consequence. You are not reassigning that document score. Read the candidate’s choice, baseline, alternative and reported evidence alongside the inventory. Existing claim links are retrieval hints, not judgments. Broader audit text, if supplied, may mention other alternatives: use only evidence attributable to this candidate. For each relevant original finding, assign exactly one relationship: - consequential: the evidence specifically supports a directional/ordering reversal, loss of the finding’s empirical support, or a major qualification/material change to a quantity that is central to that exact finding. A sign or significance change must actually concern its stated contrast, period and endpoint. A small numerical change or altered p-value alone is insufficient. A composite finding can qualify through a central component, but identify that component explicitly; do not claim the entire composite reverses. - unchanged: the candidate tests or recomputes the finding, but its substantive statement remains supported; includes only minor quantitative changes.

59

Preprint

- context_only: the finding supplies background or motivation, or the alternative concerns a different population, endpoint, period or interpretation and does not demonstrate a consequential change to this exact finding. Stratification that preserves an explicitly pooled finding is contextual unless the finding asserts subgroup uniformity. A sensitivity analysis varying the definition of the SAME target can qualify; simply answering a different research question cannot. - uncertain: an empirical target is identifiable, but the evidence is insufficient, ambiguous, noncomparable, or internally inconsistent to decide a consequential change. Baseline failure alone does not establish sensitivity to an alternative. If an evident implementation error prevents the claimed comparison, use uncertain. Do not force a mapping. An interpretive or comparative conclusion absent from the empirical inventory may be represented in joint_links only when it is explicitly stated in original_document and is grounded in at least two identifiable inventory findings. Otherwise leave it unmapped. Never manufacture an original empirical finding or promote a background/method statement into one. Omit unrelated findings. For consequential links, quote the specific observed evidence exactly and explain how it changes the original statement, not just why it is topically related. Report concerns about validity or comparability separately; these assessments are of reported evidence, not independent code validation or verified scientific reversals. Do not reject a reported empirical change solely because the alternative’s scientific defensibility has not been independently established. Joint-conclusion assessment: - original_document is the original source paper, not the alternative analyst’s interpretation of that paper. Use an exact quotation from original_document to establish that the joint conclusion was actually asserted. A candidate’s claim about what the paper implies is insufficient on its own. - A joint conclusion asserts a comparison, ordering, explanation, attribution, or synthesis supported by two or more inventory findings. It is more than a list of separate results. Individual results can remain unchanged while the inference drawn from them requires a major qualification. - Select the smallest set of findings directly supporting the challenged inference. Explain the role of every member. Do not add findings merely because they concern the same disease , endpoint, cohort, or time period. Do not add an overall finding that is only background to the tested contrast. - Apply the same consequential/unchanged/context_only/uncertain labels to the joint conclusion itself. consequential requires reported evidence of a reversal, loss of support, or major qualification of that specific conclusion. The earlier document score is not evidence. Use uncertain when the supplied evidence does not establish the claimed effect. - Identify what the alternative actually tests. Separate a descriptive ranking or contribution to an aggregate from statistical evidence of differing trends and from a causal explanation. Non-significance does not prove no effect, equality, or equivalence, and a non-significant difference does not by itself reverse a descriptive ordering or establish which subgroup contributed most to an aggregate change. - Do not upgrade individual links because their findings support a challenged joint conclusion. Preserve their individual labels. Membership in a consequential joint link means the finding supports a challenged interpretation; it does not imply that the finding itself changed. - Report each distinct joint conclusion once. Omit unrelated joint conclusions. Report validity and comparability concerns separately, as for individual links. Return only a JSON object with exactly these keys: { "record_id": "input record_id", "links": [{ "source_insight_id": "exact inventory ID", "relationship": "consequential|unchanged|context_only|uncertain", "change_type": "reversal|loss_of_support|major_qualification|material_magnitude_change| none", "evidence_quote": "exact contiguous candidate-evidence excerpt, or empty if none", "reasoning": "finding-specific explanation", "validity_concern": "specific concern, or none identified in supplied evidence" }], "joint_links": [{ "source_insight_ids": ["exact inventory ID", "another exact inventory ID"], "original_conclusion": "faithful statement of the original joint conclusion", "source_quote": "exact contiguous excerpt from original_document", "relationship": "consequential|unchanged|context_only|uncertain", "change_type": "reversal|loss_of_support|major_qualification|material_magnitude_change| none", "evidence_quote": "exact contiguous candidate-evidence excerpt, or empty if none", "reasoning": "explain how the alternative affects the joint inference", "member_roles": [{"source_insight_id": "exact member ID", "role": "how this finding supports the joint inference"}], "validity_concern": "specific concern, or none identified in supplied evidence" }], "unmapped_reason": "explain why no individual or joint link maps, or empty if links are present" }

60

Preprint

Each finding ID may occur only once in links. Each joint link must have at least two distinct valid inventory IDs, with exactly one nonempty member_roles entry for each ID and a nonempty source_quote present verbatim in original_document. Do not duplicate the same joint conclusion. consequential requires a nonempty exact candidate-evidence quote and a change_type other than none; all other relationships require change_type none. Use empty arrays where appropriate. Do not output Markdown. Input fields: record_id: string paper_id: integer original_document: string (full original source paper) original_findings: [{source_insight_id, original_claim, verification_target, atomic_claims: [{claim}]}] candidate: available research_question, analytical_choice, baseline, alternative, observed_evidence, reported_result, audit_excerpts, evidence_note; claim_link_hints: [{ source_insight_id, claim}] Output fields: { record_id: string, links: [{ source_insight_id: string, relationship: "consequential" | "unchanged" | "context_only" | "uncertain", change_type: "reversal" | "loss_of_support" | "major_qualification" | " material_magnitude_change" | "none", evidence_quote: string, reasoning: string, validity_concern: string }], joint_links: [{ source_insight_ids: [string, string, ...], original_conclusion: string, source_quote: string, relationship: "consequential" | "unchanged" | "context_only" | "uncertain", change_type: "reversal" | "loss_of_support" | "major_qualification" | " material_magnitude_change" | "none", evidence_quote: string, reasoning: string, member_roles: [{source_insight_id: string, role: string}], validity_concern: string }], unmapped_reason: string }

61

Related documents

• The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search#133422
• AutoKD: Autonomous Knowledge Discovery#973947
• TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents#985283
• AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents#1201153
• Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills#1011135