Enhancing Understandability and Transparency of Research Software: Tracing Research to Code Adrian Bajraktari
Andreas Vogelsang
University of Cologne Cologne, Germany [email protected]
arXiv:2604.10793v1 [cs.SE] 12 Apr 2026
Abstract Modern research heavily relies on software. A significant challenge researchers face is understanding the complex software used in specific research fields. We target two scenarios in this context, namely long onboarding times for newcomers and conference reviewers evaluating replication packages. We hypothesize that both scenarios can be significantly improved when there is a clear link between the paper’s ideas and the code that implements them. As a timeand staff-saving approach, we propose an LLM-based automation tool that takes in a paper and the software implementing the paper, and generates a trace mapping between research ideas and their locations in code. Initial experiments have shown that the tool can generate quite useful mappings. ACM Reference Format: Adrian Bajraktari and Andreas Vogelsang. 2026. Enhancing Understandability and Transparency of Research Software: Tracing Research to Code. In Proceedings of 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-SERS ’26). ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
Software has not only become omnipresent in industry, society, and everyday life, but also in research [2]. Researchers nowadays heavily rely on software for their studies, experiments, simulations, and evaluations. In a survey [3] already conducted in 2014, more than 90% of participants reported relying on software in some way, and more than 60% reported they could not conduct further research in their field without software support. Software in research spans a wide range, from small one-shot scripts for a single paper to large infrastructures that last for several years and act as the backbone of the group’s research activities. The latter often introduces complexity. The projects are just too big to grasp in a workday or two. In a previous study [1], researchers reported that understanding the existing codebase first before producing original research takes a long time, especially for newcomers, e.g., new PhD students or bachelor’s/master’s theses, resulting in months of up-front preparation. These long times stem not only from the complexity of the projects themselves, but also from the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. ICSE-SERS ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
University of Duisburg-Essen Essen, Germany [email protected] fact that experienced researchers do not have enough time to guide newcomers through the projects. Thus, new scientific staff are left alone with the code, existing papers, and, at best, somewhat decent documentation. Another scenario in which understanding research software plays a crucial role, applying to large and small software alike, is faced by reviewers and artifact evaluators at conferences. Their role is to check whether the code provided in a replication package realizes what is stated in the paper. Besides many other obstacles [4] that make the process complex and lengthy, they are entirely unfamiliar with the code and the research presented in the paper, and have no one to ask about it. For a thorough software evaluation, reviewers should ideally verify that the code implements all claims, ideas, methods, simulations, and other aspects described in the paper. This is a time-consuming and challenging task, given the problems described before. While both scenarios affect different groups of researchers at various levels, the core problem of both is that it is hard to understand research software if you have not been heavily involved for a long time. To this end, we propose an LLM-based approach that extracts research ideas from papers and establishes trace links between these ideas and locations in associated code. With this, we aim to support understanding and transparency of research software, especially for the two scenarios mentioned above.
2
Approach
Our LLM-based tool takes the paper’s PDF and a git repository link as inputs and executes the following process steps: (1) Preprocess: pull the repository and transform the PDF of the paper into a string-based format1 that can be passed through the prompt to the LLM. (2) Create natural language representations (NLR) for each code block. Code bases often exceed the maximum context window size for LLMs. Thus, we first let the LLM create natural language summaries, which, in comparison, use fewer tokens for each “code block”2 . (3) Extract all research concepts from the paper. A research concept is anything stated in the paper that should also appear somewhere in the code, e.g., data preprocessing, experiments, calculations, methods, results, problems, insights, claims, etc., but original to the paper or a relevant basis needed for the paper’s contribution.
1 Using PyPDF2
2 a file, a class, a method, a cell in a Jupyter notebook, or a logically cohesive group of
code lines, e.g., in larger Python scripts with no real code structure
ICSE-SERS ’26, April 14, 2026, Rio de Janeiro, Brazil Concepts 4. Classifier Implementation - Baseline algorithms: keyword-driven classifier and logistic regression using tf-idf values. Code Summaries [code\Keyword_driven_approach\Baseline_keyword_driven_downsampled.ipynb::42-54] Defines a function to calculate TF-IDF for a given document and returns sorted terms and their matrix. […] [code\Logistic_regression\Baseline_logistic_regression_downsampled.ipynb::42-54] Defines a function to predict labels for test data using logistic regression and TF-IDF. Mappings 4. Classifier Implementation: - Baseline algorithms: - [code\Keyword_driven_approach\Baseline_keyword_driven_downsampled.ipynb::42-54] - [code\Logistic_regression\Baseline_logistic_regression_downsampled.ipynb::42-54]
Figure 1: Excerpt from one of the experiments for one particular 3-tuple of concept, code summary, and mapping.
(4) Create a trace link map between research concepts and code locations. For each research concept, the LLM returns all code locations that contribute to its implementation. For our experiments, we used GPT-4o to handle all three major steps (2–4). Steps 2 and 4 are done in one request to the LLM’s API each. Because of the size of many software repositories, transforming code into NLRs is not possible within a single context window, so the code is split into large chunks, with each chunk transformed in a separate request.
3
Preliminary Evaluation
We selected five research groups from different scientific disciplines that actively operate at least one larger research software project. From each, we selected five recent publications, totaling 25 test runs of our approach. In each iteration, we provide the respective paper as a PDF and the project’s GitHub link to the tool. We collect the artifacts from each run (code summaries, concepts, and mappings). To assess the quality of the results, we contacted the paper’s authors and plan to supply each with the artifacts for their paper to review them with respect to the following key observation criteria: Does code summarization accurately reflect the code’s intentions? (per code block). Possible challenges: too coarse-grained descriptions; misinterpretations of code; hallucinations by the LLM. Did the concept extractor extract all relevant research concepts? Possible challenges: Missing relevant concepts; irrelevant concepts. How well did the LLM trace the concepts to code? Possible challenges: Code not related to the concept; concept not implemented; wrong tracing. Afterwards, we will conduct semi-open interviews with the experts, using open-ended questions, to further assess the quality of the results. We also discuss their ideas for how the tool could assist them in their research.
3.1
Preliminary Experiences and Discussion
We conducted initial experiments using familiar research software, e.g., from our own catalog. These primarily stem from the field of software engineering research. The results showed that the tool can extract the main research concepts from the paper, but still
Adrian Bajraktari and Andreas Vogelsang
considers parts of the paper as main concepts that do not really contribute to the paper’s goals, e.g., some introductory definitions. It can summarize code in natural language, concisely and in accordance with a predefined format. It is also capable of recognizing the scientific construct behind an implementation, even when there are no comments indicating precisely what the code implements. It mapped research concepts to code coherently and identified concepts mentioned in the paper but not implemented in code, as well as code blocks that seem essential to the research context but are not mentioned in the paper.
3.2
Limitations
The current version of the tool has certain limitations, which we want to run through briefly. Currently, the tool uses NLRs rather than the code itself for mapping, which might leave the LLM with less information during the mapping process. Code that is not encapsulated in a structure (e.g., a method) is grouped into coherent code blocks by the LLM. Thus, the expressiveness of this grouping heavily depends on the LLM’s capabilities. During code summarization, large code bases are split into chunks that each fit the LLM’s context window. This might make the results less accurate in cases where code from another chunk is used,e.g., a method call. In these cases, the LLM has to "guess" what the missing code does.
4
Conclusion and Future Work
In this paper, we propose a novel approach to making research software more understandable and transparent to others via an LLM-based tool. It is far from finished, and there are plenty of possibilities to proceed. Beyond the interview study, we aim to incorporate the following aspects: • A quantitative study in the form of an extensive survey on the quality of the tool’s results among a large base of researchers. We aim for 200–300 participants. • A quantitative study over a large number of projects on general data of the tool, e.g., average number of extracted concepts, number of trace links found, size of NL code summaries, etc. • Include a RAG system to get more accurate results. • Extend the tool further for different use cases beyond the scope discussed in this paper.
References [1] Adrian Bajraktari, Michelle Binder, and Andreas Vogelsang. 2024. Requirements Engineering for Research Software: A Vision. In 2024 IEEE 32nd International Requirements Engineering Conference (RE). 423–431. doi:10.1109/RE59067.2024. 00050 [2] Michael Felderer, Ralf H. Reussner, and Bernhard Rumpe. 2020. Software Engineering und Software Engineering Forschung im Zeitalter der Digitalisierung. CoRR abs/2002.10835 (2020). arXiv:2002.10835 https://arxiv.org/abs/2002.10835 [3] Simon Hettrik. 2014. It’s impossible to conduct research without software, say 7 out of 10 UK researchers. Retrieved 24.10.2025 from https://www.software.ac.uk/blog/its-impossible-conduct-research-withoutsoftware-say-7-out-10-uk-researchers [4] Stefan Winter, Christopher S. Timperley, Ben Hermann, Jürgen Cito, Jonathan Bell, Michael Hilton, and Dirk Beyer. 2022. A retrospective study of one decade of artifact evaluations. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore) (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 145–156. doi:10.1145/3540250.3549172