ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering

ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering Alejandro Velasco✉

Daniel Rodriguez-Cardenas

Dipin Khati

William & Mary Williamsburg, USA svelascodimate@wm.edu

William & Mary Williamsburg, USA dhrodriguezcar@wm.edu

William & Mary Williamsburg, USA dkhati@wm.edu

David N. Palacio

Denys Poshyvanyk

Microsoft Redmond, USA davidnad@microsoft.com

William & Mary Williamsburg, USA dposhyvanyk@wm.edu

arXiv:2608.02323v1 [cs.SE] 3 Aug 2026

Abstract The scientific method has long guided empirical research in Software Engineering (SE), but the complexity of modern software systems often hinders its systematic application. This paper introduces ECLAIR, a causally grounded AI framework that integrates Large Language Models (LLMs) into every stage of the scientific process, from hypothesis generation to analysis and interpretation. ECLAIR treats LLMs as active scientific agents operating under the principles of causal inference, within a human-in-theloop design that safeguards against the risks of unsound automated reasoning. We demonstrate the framework through a case study examining how prompt design influences code generation accuracy in two LLMs. Results show that, for both models, instruction-style, longer few-shot, and signature-augmented prompts yield small negative causal effects on accuracy, illustrating how causal reasoning provides a principled foundation for explaining why software phenomena occur. This study presents the first causally grounded structured methodology for embedding LLMs within the scientific method in SE, designed around the epistemological demands of empirical SE research, establishing a basis for rigorous AI-assisted research.

CCS Concepts • Computing methodologies → Artificial intelligence; • Software and its engineering → Software creation and management;

Keywords AI4SE, Research Methods, Empirical Research, LLMs for Code ACM Reference Format: Alejandro Velasco, Daniel Rodriguez-Cardenas, Dipin Khati, David N. Palacio, and Denys Poshyvanyk. 2026. ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software

This work is licensed under a Creative Commons Attribution-NonCommercialNoDerivatives 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3834544

Engineering (ASE ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3832783.3834544

1

Introduction

Francis Bacon (1620) argued that knowledge should be built through systematic observation and reasoned experimentation. Centuries later, the scientific method remains the foundation for transforming questions into understanding. At its core lies the search for causal explanation: to explain a phenomenon is to discover what causes it a goal formalized in Pearl’s theory of causation through Structural Causal Models (SCMs) and the do-calculus [18], which provide a mathematical basis for reasoning about interventions and their effects. Through this lens, scattered observations become structured knowledge, and empirical data are transformed into coherent reasoning. In Software Engineering (SE), the scientific method has guided decades of empirical research, shaping how researchers study the software life cycle, how teams collaborate, and how causal relationships are identified within complex systems. Despite serving as a robust foundation for knowledge generation, the scientific method in SE is often constrained by the inherent complexity of the field. Modern software systems are large, dynamic, and collaborative, making them difficult to observe, instrument, and analyze consistently. Empirical studies often depend on fragmented datasets, inconsistent measurements, and ad hoc tools that hinder reproducibility. More fundamentally, SE research faces an epistemic challenge: connecting empirical evidence with the causal reasoning needed to explain it. This challenge is well documented: most empirical SE studies establish statistical associations rather than causal relationships, and few employ the formal identification strategies required to support causal claims [15, 21, 29]. As the field grows increasingly data-intensive and interdisciplinary, it remains difficult to identify causal factors, formulate testable hypotheses, and link them to the mechanisms that govern software behavior. This gap limits our ability to draw credible causal conclusions about which practices truly influence software engineering results. Large Language Models (LLMs) offer an opportunity to address this epistemic gap by supporting the reasoning processes that underpin empirical research. Trained in a vast corpus of code, documentation and scientific literature, LLMs can assist not only in synthesizing prior evidence, but also in formulating hypotheses and structuring empirical analyzes. Across domains ranging from cancer biology [2] (e.g., proposing drug hypotheses later validated in

ASE ’26, October 12–16, 2026, Munich, Germany

laboratory experiments) and psychology [26] to chemistry [7, 12], LLMs have demonstrated the ability to generate and test causal hypotheses at a scale comparable to trained researchers [17], suggesting that under an appropriate methodological framework, they can act as scientific agents that extend human reasoning in the formulation and testing of causal claims. This paper takes a step toward formalizing that potential within SE. We introduce ECLAIR (Empirical-Causal LLM-Augmented Inference for SE Research), a causally grounded AI framework that integrates LLMs into the scientific method, linking hypothesis generation with causal analysis through a transparent, human-in-theloop process. This human-in-the-loop design is deliberate, as fully delegating scientific reasoning to automated agents risks perpetuating unsound research practices [11], and ECLAIR instead positions human judgment as a necessary checkpoint at each critical stage (e.g., vetting hypotheses before experimentation, approving the causal model before effects are estimated). As illustrated in Fig. 1, ECLAIR builds upon the canonical scientific method by embedding causal reasoning and LLM assistance across its stages, transforming it into a structured process for empirical discovery. Rather than treating LLMs as passive instruments, ECLAIR positions them as scientific agents whose contributions are documented, constrained, and evaluated according to the principles of causal inference, enabling empirical research that is both scalable and scientifically rigorous. To the best of our knowledge, ECLAIR is the first framework to (i) operationalize Pearl’s theory of causation as an executable, phaseby-phase scientific method for SE, rather than a domain-agnostic hypothesis generator, (ii) pair LLM assistance with mandatory human checkpoints per phase against unsound automated reasoning, and (iii) ground hypothesis and experiment design in SE-specific artifacts (e.g., code, ASTs) instead of generic prompting. We demonstrate ECLAIR through a case study that examines how prompt engineering influences model performance in code generation. All data, code and experimental configurations are publicly available in our replication package [23]. Our contribution provides a foundation for using LLMs not only to automate research, but also to strengthen the reasoning process through which scientific discovery is achieved, grounded in the theory of causation.

2

The ECLAIR Framework For SE

ECLAIR is a human-in-the-loop framework that integrates LLMs into the scientific method for SE, based on the theory of causation [18] and established methodological guidelines [4]. Fig. 1 contrasts ECLAIR with the canonical scientific method across its eight sequential phases. For each phase below, we describe the researcher input given to the Scientific Agent and its expected output; exact prompt templates and a detailed checklist are provided in our replication package [23]. (1) Phenomenon Observation & Research Intent. The initial phase identifies SE phenomena requiring explanation, such as the influence of prompts on LLM code generation accuracy. This phase does not involve the Scientific Agent: the SE researcher selects a phenomenon, defines a research intention and question (optionally supported by a dataset), and applies expert judgment to prioritize problems of practical relevance and value [4].

Velasco et al.

(2) Knowledge Mapping. In this phase, the SE researcher selects a LLM (the Scientific Agent) to construct the research knowledge base. Input: the agent uses the research input, dataset, and description or samples oriented by the defined research intention and available data (e.g., types of code snippets). The Scientific Agent synthesizes the relevant literature, identifies limitations and assumptions of recent studies, and extracts patterns, variables, risks, and gaps from related datasets. Output: Scientific Agent generates an observation deduction with the theoretical context in a structured JSON file. The theoretical context captures the knowledge needed to formulate hypotheses about the SE phenomenon (i.e., phenomenon scope, observable variables, quality risks, limitations). To ground synthesis in citable sources rather than parametric memory alone, curbing hallucination, this phase uses Retrieval-Augmented Generation (RAG) and agentic search for backward/forward snowballing over the literature. (3) Generation of Causal Hypotheses. In this phase, observations become specific, testable causal hypotheses. Input: Scientific Agent uses phase (2) output and a prompt detailing hypothesis requirements (i.e., measurable and testable variables, avoid speculative and abstract claims [23]) . All prompts are structurally grounded in Pearl’s causal hierarchy [18], requiring the Scientific Agent to reason via why-based queries (e.g., “why would𝑇 change 𝑂?”), targeting the interventional/counterfactual rungs rather than associational what-is questions, and to express each hypothesis in treatment/outcome/confounder form consistent with the SCM vocabulary used in Phase (4). This follows the “Use Objective Designs” principle via Multiple Reasonable Hypotheses Testing (MRHT), considering established practices and competing alternatives [4]. Output: Scientific Agent generates a set of hypotheses from the given phenomenon and following the input instructions and restrictions. Given the risk of hallucination, each hypothesis is critically assessed by the SE researcher for soundness, validity, and feasibility before advancing to experimental design. (4) Causal Experiment Design. This phase ensures the credibility of the study design. Input: the SE researcher first selects the subset of hypotheses from Phase (3), then defines the relevant data constraints. The Scientific Agent generates a structural causal graph (SCM) specifying treatments, outcomes, and mediators. Output: a JSON-encoded SCM specification per hypothesis. The SE researcher then reviews and refines the proposed design, producing an experimental plan for each hypothesis. This reflects the scientific standard that experimental evidence, and meta-analyses thereof, provide the strongest validation [6, 30]. (5) Experimentation. This phase executes the causal design to generate the empirical corpus for causal analysis. Input: the SCM specification and the target dataset. The SE researcher conducts controlled experiments per hypothesis, applying the defined treatments and measuring the outcomes, mediators, and confounders specified in the causal graph. The Scientific Agent automates the experimentation workflow for consistency and reproducibility, producing logged model completions per treatment as output, while the SE researcher supervises and validates data completeness. (6) Causal Inference Analysis. This phase applies Pearl’s theory of causation [15, 18, 19] to estimate causal effects from experimental data. Input: the experimental corpus from Phase (5) and the SCM specification. Within ECLAIR, the Scientific Agent estimates

ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering

ASE ’26, October 12–16, 2026, Munich, Germany

ECLAIR: Scientific Method for Empirical Research in SE

deduction

State of the art

0

1

Make an SE Phenomenon Observation

Definition Research Intend and SE related question

deduction

Statistical analysis on existing data

7

8

Existing patterns and evidences Existing data gaps Measured variables Data risks

6 ATEs

Dissemination

Interpretation

Causal Inference Analysis

Causal 3 Hypotheses Generation

Related literature Existing limitations Existing assumptions Measured variables

Future work exploration For each H

Canonical Scientific Method for Empirical Research

2 Knowledge Mapping

Phenomenon - any observable fact, event, or occurrence, whether natural or extraordinary, that requires explanation

5 Results

Theoretical Framework Filtered Causal Hypothesis (H) SCM for each hypothesis, including outcomes, confounders, treatments, instruments, effect modifiers

Experimentation

0. Make an Observation

1. Define a Question

2. Gather Information from sources

5. Analyze the data

4. Test Hypothesis by performing experiments

3. Form explanatory hypothesis

6. Interpret The data 4

Causal Experiment Design

7. Publish results

Figure 1: Our ECLAIR framework (left) contrasted with the canonical scientific method (right) causal effects (e.g., the Average Treatment Effect (ATE)) for each tested hypothesis while controlling for confounding bias, output: ATE estimates and robustness/refutation results per method, following the principles of valid, transparent, and logically consistent inference [4]. (7) Interpretation. In this phase, causal results become scientific insight. Input: the ATE results from Phase (6) and related literature. The Scientific Agent synthesizes the findings and situates them within the existing literature, while the SE researcher critically evaluates their validity and relevance. Output: a consolidated interpretation that integrates validated findings and outlines open questions for future research. (8) Dissemination. This final phase emphasizes Bacon’s “dissemination of useful findings,” whereby conclusions must follow logically from evidence obtained through systematic observation. Taking Phase (7)’s interpretation as input, the SE researcher reports results transparently, ensures reproducibility, and submits the work to peer review.

3

A Study on Prompt Engineering

To demonstrate ECLAIR, we conducted a case study on the phenomenon of how prompt design influences code generation accuracy. In Phase (1), the SE researcher jointly fixed this phenomenon and a supporting dataset (detailed below), scoping the research question accordingly: RQ1 What is the prompt design impact on the model’s code generation accuracy? The Scientific Agent. We selected GPT-5 as the Scientific Agent since it is to date the latest and largest LLM released by OpenAI. Larger models are preferred because their parameter capacity enhances reasoning depth, factual recall, and stability in complex analytical tasks [1]. Dataset. We use the CodeText-Galeras[21] dataset, which pairs Python functions with natural language descriptions and structural metadata. The corpus aggregates samples from multiple opensource repositories and includes metrics such as token count, lines of code, AST depth, and documentation features. The combination of code and text enables the analysis of how prompt design affects code generation accuracy, as also explored in the dataset’s original study [21]. Models. We evaluate two instruction-tuned code generation models: Qwen/Qwen2.5-Coder-7B [5] (𝑀1 ) and codellama/CodeLlama7b-Python-hf [22] (𝑀2 ). Both are optimized for program synthesis

but differ in scope and specialization. 𝑀1 is trained in multilingual code corpora with strong reasoning capabilities, while 𝑀2 is a Python-focused variant of LLaMA. Both models share a comparable parameter count (7B), making their comparison controlled with respect to scale while contrasting in training strategy and domain specialization, which allows us to examine whether the causal effects of prompt design generalize across architectures or are model-specific. Like the dataset, 𝑀1 and 𝑀2 were fixed upfront rather than nominated during Experiment Design (§Discussion). Knowledge Mapping (Phase 2). We began by contextualizing the phenomenon through a literature synthesis conducted with the Scientific Agent. Recent studies [8, 10, 13, 25] show that instructionstyle prompts and function signatures can influence metrics such as pass@k and CodeBLEU, although prompt effects often depended on model configuration and benchmark design, with limited causal validation. Within the same phase, the Scientific Agent organized this evidence into a knowledge base by analyzing CodeText-Galeras, identifying measurable variables (e.g., complexity, AST depth, token counts), confounders, and data gaps. Notably, it flagged that the dataset does not directly pair prompts with accuracy scores, a gap that Experiment Design (Phase 4) resolved by applying prompt treatments synthetically; the remaining variables were suitable for causal modeling. Based on the mapped evidence, the Scientific Agent formulated sixteen causal hypotheses linking prompt characteristics to accuracy metrics such as CodeBLEU, Exact Match, and Compilation Success (see Appendix [23]). Among them, 𝐻 2 (adding function signatures improves CodeBLEU) and 𝐻 14 (prompt truncation decreases CodeBLEU) had been explored previously [21]. For empirical evaluation, we selected three representative hypotheses: 𝐻 1 (instructionstyle prompts improve CodeBLEU), 𝐻 11 (few-shot examples increase CodeBLEU), and 𝐻 13 (including function signatures benefits more complex functions). For each hypothesis, the Scientific Agent constructed a SCM defining treatments, outcomes, and confounders (see Table 1). In 𝐻 1 , the treatment was binary, with 𝑇0 representing code-delimited prompts (control) and 𝑇1 instruction-style prompts (intervention). In 𝐻 11 , the treatment was continuous, representing the number of few-shot samples (𝑇𝑥 ). In 𝐻 13 , the treatment was binary, where 𝑇0 excluded and 𝑇1 included function signatures. For 𝐻 1 and 𝐻 11 , structural and lexical features acted as confounders, while in 𝐻 13 , complexity was modeled as an effect modifier.

ASE ’26, October 12–16, 2026, Munich, Germany

Velasco et al.

Table 1: Causal Effects (ATEs) on CodeBLEU (i.e., RQ1 ) M1 [5] ATE

M2 [22] 𝜌 ATE

-0.575 -0.139 -0.163 -0.001 0.023 -0.055

-0.515 -0.126 -0.286 -0.028 -0.097 -0.055

Causal Experimental Settings Id

Interv.

Treatments

Confounders

𝐻 1 binary 𝑇0 : "{incomplete_method}" 𝑇1 : "{instruction} + {incomplete_method}" C 𝐻 11 continuous 𝑇𝑥 : "{instruction} + {x_samples} + {incomplete_method}" C 𝐻 13 binary 𝑇0 : "{docstring}" 𝑇1 : "{docstring} + {signature}" C \ {complexity}

𝜌

C = { complexity, #ast_levels, #ast_nodes, #ast_errors, #whitespaces, #tokens, vocab_size, #words, #identifiers, #loc, prompt_length, #prompt_tokens } . The outcome is CodeBLEU for all hypotheses; complexity is modeled as an effect modifier in 𝐻 13 and as a confounder otherwise. bold: − correlation; purple : − causal effect.

Following the experimental designs, the SE researcher executed the experiments and estimated causal effects using the DoWhy library [24]. For each hypothesis (𝐻 1 , 𝐻 11 , and 𝐻 13 ), the causal inference pipeline [15] computed ATE and performed robustness checks using placebo, random common cause, data subset and unobserved confounder tests. The results, summarized in Table 1, were consistent across both models and stable under SCM perturbations. The estimated effects were then interpreted to assess how prompt structure influenced model performance. For 𝐻 1 , the ATE was negative (−0.139 for 𝑀1 , −0.126 for 𝑀2 ), indicating that instruction-style prompts (𝑇1 ) reduced accuracy relative to code-delimited prompts (𝑇0 ). For 𝐻 11 , the effect was also negative (−0.001 for 𝑀1 , −0.028 for 𝑀2 ), suggesting that increasing few-shot demonstrations slightly decreased CodeBLEU. For 𝐻 13 , including function signatures (𝑇1 ) produced a small negative effect (−0.055 for both models). In all cases, robustness tests confirmed that the effects were stable and not vulnerable to confounding bias. RQ1 For both 𝑀1 and 𝑀2 , instruction-style, longer, and signatureaugmented prompts yield small negative causal effects on accuracy.

Discussion. Our results demonstrate the application of ECLAIR in guiding the SE researcher from hypothesis generation to causal validation, ensuring that assumptions, treatments, and confounders are explicitly defined and systematically evaluated. The analysis revealed small but consistent negative effects across all hypotheses tested, showing that, within the scope of our study, instructionstyle, longer, and signature-augmented prompts tend to reduce code generation accuracy for the two evaluated models. These findings challenge the expectation that adding structure or information to prompts improves model performance, instead suggesting that effectiveness depends on model architecture and training context. Unlike fixed causal pipelines, ECLAIR constructs structural models dynamically from observed phenomena and evidence synthesized by the Scientific Agent, improving transparency and validity. Designed to support researchers in answering causal questions that explain why phenomena occur, ECLAIR formalizes the reasoning required to identify and test causal mechanisms rather than describe correlations. As a simplification, the phenomenon, dataset, and candidate models were fixed jointly by the SE researcher in Phase (1), consistent with its definition (research intent may be supported by relevant datasets), rather than having Knowledge Mapping propose datasets or Experiment Design nominate models conditioned on the generated hypotheses. The present case study intentionally addresses a focused causal hypothesis to illustrate the framework’s end-to-end process; generalizability across more complex SE phenomena, larger datasets, and additional models remains an open direction, alongside a comparative evaluation between Scientific

Agent-generated outputs and those produced independently by SE researchers at each stage of the scientific process.

4

Related Work

Recent systems such as Denario [27] and Kosmos [14] demonstrate end-to-end scientific-discovery pipelines, from idea generation to paper drafting, across disciplines including astrophysics, biology, and chemistry [12, 17]. Google DeepMind’s AlphaEvolve [16] combines evolutionary search with Gemini-powered LLMs to autonomously discover novel algorithms. Wang et al. [28] survey AI’s role in scientific discovery, Reddy et al. [20] identify hypothesis generation as a key opportunity, and Zhou et al. [31] show LLMs can generate plausible hypotheses from literature, though without a formal causal structure or domain-specific methodology. None of these works addresses the epistemological demands of empirical SE research, where causal claims must be grounded in codecentric datasets and SE-specific confounders (e.g., AST complexity, token structure). Within SE, Baltes et al. [6] propose methodological guidelines for LLM-involving studies, formalizing roles such as annotators, judges, and synthesis agents [3, 9]. Unlike fully automated systems that risk perpetuating unsound research practices [11], ECLAIR positions LLMs as augmentative tools within a human-in-the-loop, causally grounded workflow that spans from hypothesis generation to validation. In contrast to prior work focused on task-specific assistance or domain-agnostic hypothesis generation, our approach integrates LLMs into the epistemic structure of the scientific method itself, tailored to the specific challenges of empirical SE research.

5

Conclusions & Future Plans

This paper presented ECLAIR, a causally grounded AI framework that integrates LLMs into the scientific method for SE. By formalizing how LLMs contributes to hypothesis generation, experimental design, and causal analysis, ECLAIR establishes a structured and transparent foundation for empirical research. The case study on prompt engineering demonstrated how causal reasoning can be operationalized in practice, automating analytical steps while preserving human oversight and interpretability. In future work, we plan to extend ECLAIR to additional SE domains, incorporate more advanced reasoning agents capable of critiquing causal assumptions, and expand the framework to support descriptive and temporal scientific questions beyond its current causal focus. Beyond SE, ECLAIR aspires to contribute to the broader goal of AI-assisted science by providing tools that explain why, how, and when complex software phenomena occur.

ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering

6

Data Availability Statement

All artifacts necessary to reproduce this study are publicly available in the repository at [23], including the implementation of ECLAIR, experimental scripts, prompts, and evaluation materials used in the case study. No proprietary or sensitive data were used.

References [1] 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/ [2] Abbi Abdel-Rehim, Hector Zenil, Oghenejokpeme Orhobor, Marie Fisher, Ross J. Collins, Elizabeth Bourne, Gareth W. Fearnley, Emma Tate, Holly X. Smith, Larisa N. Soldatova, and Ross King. 2025. Scientific hypothesis generation by large language models: laboratory validation in breast cancer treatment. Journal of The Royal Society Interface 22, 227 (June 2025), 20240674. Publisher: Royal Society. doi:10.1098/rsif.2024.0674 [3] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs Replace Manual Annotation of Software Engineering Artifacts? arXiv:2408.05534 [cs.SE] https://arxiv.org/abs/2408.05534 [4] J. Scott Armstrong and Kesten C. Green. 2022. The Scientific Method: A Guide to Finding Useful Knowledge. Cambridge University Press. doi:10.1017/ 9781009092265 [5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen Technical Report. arXiv:2309.16609 [cs.CL] https://arxiv.org/abs/2309.16609 [6] Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, Christoph Treude, and Stefan Wagner. 2025. Guidelines for Empirical Studies in Software Engineering involving Large Language Models. arXiv:2508.15503 [cs.SE] https: //arxiv.org/abs/2508.15503 [7] Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. 2023. Autonomous chemical research with large language models. Nature 624, 7992 (Dec. 2023), 570–578. Publisher: Nature Publishing Group. doi:10.1038/s41586-02306792-0 [8] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. arXiv:2304.05128 [cs.CL] https://arxiv. org/abs/2304.05128 [9] Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota. 2025. On the Effectiveness of LLM-as-aJudge for Code Generation and Summarization. IEEE Transactions on Software Engineering 51, 8 (2025), 2329–2345. doi:10.1109/TSE.2025.3586082 [10] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv:2204.05999 [cs.SE] https://arxiv.org/abs/2204.05999 [11] Fengqing Jiang, Yichen Feng, Yuetai Li, Luyao Niu, Basel Alomair, and Radha Poovendran. 2025. BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? arXiv:2510.18003 [cs]. doi:10.48550/ arXiv.2510.18003 [12] Quanliang Liu, Maciej P. Polak, So Yeon Kim, MD Al Amin Shuvo, Hrishikesh Shridhar Deodhar, Jeongsoo Han, Dane Morgan, and Hyunseok Oh. 2025. Beyond designer’s knowledge: Generating materials design hypotheses via a large language model. Acta Materialia 297 (Sept. 2025), 121307. doi:10.1016/j.actamat.2025.121307 [13] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651 [cs.CL] https://arxiv.org/abs/2303.17651 [14] Ludovico Mitchener, Angela Yiu, Benjamin Chang, Mathieu Bourdenx, Tyler Nadolski, Arvis Sulovari, Eric C. Landsness, Daniel L. Barabasi, Siddharth Narayanan, Nicky Evans, Shriya Reddy, Martha Foiani, Aizad Kamal, Leah P. Shriver, Fang Cao, Asmamaw T. Wassie, Jon M. Laurent, Edwin Melville-Green, Mayk Caldas, Albert Bou, Kaleigh F. Roberts, Sladjana Zagorac, Timothy C. Orr, Miranda E. Orr, Kevin J. Zwezdaryk, Ali E. Ghareeb, Laurie McCoy, Bruna Gomes, Euan A. Ashley, Karen E. Duff, Tonio Buonassisi, Tom Rainforth, Randall J. Bateman, Michael Skarlinski, Samuel G. Rodriques, Michaela M. Hinks, and Andrew D. White. 2025. Kosmos: An AI Scientist for Autonomous Discovery. arXiv:2511.02824 [cs]. doi:10.48550/arXiv.2511.02824

ASE ’26, October 12–16, 2026, Munich, Germany

[15] David Nader Palacio, Alejandro Velasco, Nathan Cooper, Alvaro Rodriguez, Kevin Moran, and Denys Poshyvanyk. 2024. Toward a Theory of Causation for Interpreting Neural Code Models. IEEE Transactions on Software Engineering 50, 5 (May 2024), 1215–1243. doi:10.1109/tse.2024.3379943 [16] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. 2025. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv:2506.13131 [cs]. doi:10.48550/arXiv.2506.13131 [17] Yang Jeong Park, Daniel Kaplan, Zhichu Ren, Chia-Wei Hsu, Changhao Li, Haowei Xu, Sipei Li, and Ju Li. 2024. Can ChatGPT be used to generate scientific hypotheses? Journal of Materiomics 10, 3 (May 2024), 578–584. doi:10.1016/j.jmat.2023.08.007 [18] Judea Pearl. 2009. Causality: models, reasoning, and inference. doi:10.1017/ CBO9780511803161 [19] Judea Pearl. 2018. Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution. arXiv:1801.04016 [cs]. doi:10.48550/arXiv. 1801.04016 [20] Chandan K. Reddy and Parshin Shojaee. 2025. Towards scientific discovery with generative AI: progress, opportunities, and challenges. In Proceedings of the ThirtyNinth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’25/IAAI’25/EAAI’25). AAAI Press, Article 3186, 9 pages. doi:10.1609/aaai.v39i27.35084 [21] Daniel Rodriguez-Cardenas, David N. Palacio, Dipin Khati, Henry Burke, and Denys Poshyvanyk. 2023. Benchmarking Causal Study to Interpret Large Language Models for Source Code. arXiv:2308.12415 [cs.SE] https://arxiv.org/abs/ 2308.12415 [22] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. arXiv:2308.12950 [cs.CL] https://arxiv.org/abs/2308.12950 [23] SEMERU Lab. 2025. ci4sesci: Causal Inference for Software Science. GitHub repository. https://github.com/WM-SEMERU/ci4sesci [24] Amit Sharma and Emre Kiciman. 2020. DoWhy: An End-to-End Library for Causal Inference. arXiv preprint arXiv:2011.04216 (2020). doi:10.48550/arXiv.2011.04216 [25] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 377, 19 pages. doi:10.52202/075280-0377 [26] Song Tong, Kai Mao, Zhen Huang, Yukun Zhao, and Kaiping Peng. 2024. Automating psychological hypothesis generation with AI: when large language models meet causal graph. Humanities and Social Sciences Communications 11, 1 (July 2024). doi:10.1057/s41599-024-03407-5 [27] Francisco Villaescusa-Navarro, Boris Bolliet, Pablo Villanueva-Domingo, Adrian E. Bayer, Aidan Acquah, Chetana Amancharla, Almog Barzilay-Siegal, Pablo Bermejo, Camille Bilodeau, Pablo Cárdenas Ramírez, Miles Cranmer, Urbano L. França, ChangHoon Hahn, Yan-Fei Jiang, Raul Jimenez, Jun-Young Lee, Antonio Lerario, Osman Mamun, Thomas Meier, Anupam A. Ojha, Pavlos Protopapas, Shimanto Roy, David N. Spergel, Pedro Tarancón-Álvarez, Ujjwal Tiwari, Matteo Viel, Digvijay Wadekar, Chi Wang, Bonny Y. Wang, Licong Xu, Yossi Yovel, Shuwen Yue, Wen-Han Zhou, Qiyao Zhu, Jiajun Zou, and Íñigo Zubeldia. 2025. The Denario project: Deep knowledge AI agents for scientific discovery. arXiv:2510.26887 [cs.AI] https://arxiv.org/abs/2510.26887 [28] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, Anima Anandkumar, Karianne Bergen, Carla P. Gomes, Shirley Ho, Pushmeet Kohli, Joan Lasenby, Jure Leskovec, Tie-Yan Liu, Arjun Manrai, Debora Marks, Bharath Ramsundar, Le Song, Jimeng Sun, Jian Tang, Petar Veličković, Max Welling, Linfeng Zhang, Connor W. Coley, Yoshua Bengio, and Marinka Zitnik. 2023. Scientific discovery in the age of artificial intelligence. Nature 620, 7972 (Aug. 2023), 47–60. doi:10.1038/s41586-023-06221-2 [29] Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Bjr̈n Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Springer. doi:10.1007/978-3-642-29044-2 [30] Yanbo Zhang, Sumeer A. Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, Saso Dzeroski, Jesper Tegner, and Hector Zenil. 2025. Exploring the role of large language models in the scientific method: from hypothesis to discovery. npj Artificial Intelligence 1, 1 (Aug. 2025), 14. doi:10.1038/s44387-025-00019-5 [31] Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. 2024. Hypothesis Generation with Large Language Models. In Proceedings of the 1st Workshop on NLP for Science (NLP4Science), Lotem Peled-Cohen, Nitay

ASE ’26, October 12–16, 2026, Munich, Germany

Calderon, Shir Lissak, and Roi Reichart (Eds.). Association for Computational Linguistics, Miami, FL, USA, 117–139. doi:10.18653/v1/2024.nlp4science-1.10

Velasco et al.

Received 2026-05-12; accepted 2026-07-02

Related documents

• PARNESS: A Paper Harness for End-to-End Automated Scientific Research with Dynamic Workflows, Full-Text Indexing, and Cross-Run Knowledge Accumulation#168411
• Observing the Conduct of Systematic Reviews with Generative AI Support: An Experience Report from a Graduate Software Engineering Course#1061962
• How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community#927947
• Making the Invisible Visible: A Framework for Reflective AI Use in Software Engineering Education#1129601
• Making the Invisible Visible: A Framework for Reflective AI Use in Software Engineering Education#1108823