arXiv CSopen access

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

arXiv:2604.10345v1 [cs.SE] 11 Apr 2026

MEHEDI SUN, William & Mary, USA ANTU SAHA, William & Mary, USA NADEESHAN DE SILVA, William & Mary, USA ANTONIO MASTROPAOLO, William & Mary, USA OSCAR CHAPARRO, William & Mary, USA Understanding the reasons behind past code changes is critical for many software engineering tasks, including refactoring and reviewing code, diagnosing and fixing bugs, and implementing new features. Unfortunately, locating and reconstructing this rationale can be difficult for developers because the information is often fragmented into partial pieces, inconsistently documented, and scattered across heterogeneous artifacts such as commit messages, issue reports, and pull requests. In this paper, we address this challenge in two major steps. First, we conduct an empirical study of 63 commits from five open-source Java projects to systematically analyze how nine rationale components (e.g., a change’s Goal, Need, and Alternatives) are distributed across artifacts associated with a commit. We find that rationale is highly fragmented: commit messages and pull requests primarily capture Goal, while Need and Alternatives are more often found in issues and pull requests. Other components are scarce but found in artifacts other than commit messages. Importantly, no single artifact type consistently captures all components, underscoring the need for cross-document reasoning and synthesis. Second, we introduce Argus, an LLM-based approach that identifies sentences expressing Goal, Need, and Alternatives across a commit’s artifacts and synthesizes them into concise rationale summaries to support code comprehension and maintenance tasks. We developed and evaluated Argus on the 63 commits and compared its performance against baseline variants. The best-performing version achieved 51.4% precision and 93.2% recall for rationale identification, while producing rationale summaries rated as accurate and correct. A user study with 12 Java developers further showed that these summaries were perceived as useful and helpful for tasks such as code review, documentation, and debugging. Our results highlight the need for multi-document reasoning in capturing rationale and demonstrate the potential of Argus to help developers understand and maintain software systems.

1

Introduction

Understanding the reasons behind past code changes is one of most frequent and important activities that software developers perform when developing and maintaining software systems [11, 17, 21, 32, 39, 42, 55, 57, 67]. This code change rationale provides essential context about why particular code design or implementation decisions were made, beyond describing the resulting code modifications themselves [11, 47, 57, 67]. Prior studies underline that understanding code change rationale is a critical requirement for developers to perform many software engineering tasks [11, 42], including refactoring and code review, diagnosing software and fixing bugs, documenting design and functionality, and reusing code for creating new features. This rationale allows developers to interpret the structure and behavior of the codebase, understand trade-offs considered during development, and make informed decisions when correcting defects, implementing enhancements, or designing new features [11, 17, 57, 67]. Without this context, developers risk introducing regressions, violating architectural constraints, or duplicating previous efforts, especially when they are unfamiliar with the project’s history or were not involved in earlier development phases [11, 17, 57]. Authors’ addresses: Mehedi Sun, William & Mary, Williamsburg, USA, [email protected]; Antu Saha, William & Mary, Williamsburg, USA, asaha02@ wm.edu; Nadeeshan De Silva, William & Mary, Williamsburg, USA, [email protected]; Antonio Mastropaolo, William & Mary, Williamsburg, USA, [email protected]; Oscar Chaparro, William & Mary, Williamsburg, USA, [email protected]. Manuscript submitted to ACM

1

2

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro Despite its importance, collecting and understanding rationale can be difficult for developers [11, 39, 42, 67]. In

many cases, rationale is not formally documented but instead informally noted or preserved in temporary media such as whiteboards or personal notes [30]. Even when recorded digitally, it is often documented informally, expressed inconsistently, and distributed across multiple software artifacts [57, 64]. Junior developers and newcomers, who might rely on senior colleagues for guidance, often lack easily accessible sources of rationale and are forced to reconstruct it from heterogeneous and unstructured artifacts [14, 24, 45, 57, 64], including issue reports, pull request, commit messages, code reviews, code comments, chat conversations, and meeting logs. These artifacts are created by different stakeholders, at different stages of the development process, and for different purposes. They vary widely in style, content, and length and often use inconsistent terminology to express similar ideas [19, 57, 68, 74]. Consequently, locating and reconstructing the rationale behind a code change often requires developers to navigate, interpret, and connect information distributed across multiple artifacts, which can be tedious, error-prone, and time-consuming [11, 57, 60, 60, 64]. Although prior research has explored methods for identifying rationale in software artifacts [12, 28, 29, 56, 72], existing approaches fall short in several key dimensions. First, they focus on detecting rationale content within a single artifact type, such as commit messages or issue reports, limiting their applicability in real-world projects where documentation is typically fragmented [35]. Second, they lack mechanisms to connect and integrate rationale fragments across documents, making it difficult for developers to obtain a coherent understanding of the reasoning behind a change [35, 58]. Third, they are unable to identify fine-grained rationale components (e.g., a change’s goal, underlying need, or alternative solutions considered)—information that developers often require and look for when making sense of software changes [11]. Finally, they do not support cross-artifact generation of concise, synthesized rationale summaries that developers can easily consume to facilitate software comprehension and maintenance tasks To address these challenges, this paper presents two main contributions. First, we conduct a comprehensive empirical study of how code change rationale components are expressed across multiple software artifacts. We manually analyze the artifacts associated with 63 real-world commits from five widely used Java open-source systems. These commits span a wide range of change types and sizes, contributed by different developers, and are linked to diverse artifacts that capture various development workflows used to implement and validate the changes. Using multi-coder qualitative analysis guided by an established rationale taxonomy [11], we identify seven fine-grained rationale components. These include the Goal of the change, the Need motivating it, Alternative solutions considered, and the Selected Solution. Our analysis reveals that rationale information is highly fragmented and inconsistently distributed across artifacts. Commit messages and pull requests primarily document a change’s Goal, while Need is mostly captured in issues and pull requests. Other components appear less frequently and are scattered across different artifacts, with pull requests and issues standing out as the most of diverse sources of rationale information. In essence, no single artifact type consistently captures all rationale components, which highlights the need for techniques that help developers retrieve and synthesize rationale information across artifacts. Second, building on these insights, we design and evaluate Argus, a novel LLM-based approach for fine-grained, multi-document rationale extraction and generation. Argus identifies the software artifacts associated with a code change (e.g., issue reports, pull requests, and code reviews) and performs two main tasks: (1) identifies sentences that express three rationale components (Goal, Need, and Alternatives), and (2) generates concise, coherent summaries of these component that integrate rationale information across documents. We experiment with multiple prompting strategies for Argus to guide the model in both rationale extraction and generation using the 63 studied commits. Employing a data-driven methodology using a development set of 13 commits, we identify the most effective prompting strategy (namely, task decomposition and few-shot exemplars with explanations) and evaluate its effectiveness on a Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

3

Fig. 1. A Motivating Example from the OkHttp Project[2] Commit 4c86085

ARGUS Generated Rationale

Commit Message: Drop ALPN support. ...

GOAL: Remove ALPN support from OkHttp’s Android Platform implementation.

File Name: okhttp/src/ ... /Platform.java - Method setAlpnProtocols = null; - Method getAlpnSelectedProtocol ... return new Android(...,setNpnProtocols, - getNpnSelectedProtocol,...Protocol); + getNpnSelectedProtocol); - if (setAlpnProtocols != null) { - setAlpnProtocols.invoke( ... );

NEED: Android’s ALPN support (introduced in 4.4) suffers a concurrency bug and its SSL_CTX_set_alpn_protos call is not threadsafe, leading to native crashes and segfaults in libssl.so; the underlying TLS-stack fix isn’t widely available ...

Class Javadoc: Platform.java#Platform /** * <p>ALPN ( ... ) is the successor to NPN. It * technical advantages over NPN. ALPN... * that release suffers a ... concurrency bug * we don't use it. ALPN will be supported ... */

Aritfacts:

ALTERNATIVES: Workarounds considered included early initialization of OkHttpClient via URL.setURLStreamHandlerFactory to avoid mixing AOSP HttpURLConnection and OkHttp, switching to SSL_set_alpn_protos instead of SSL_CTX_set_alpn_protos

Commit Message

Class Javadocs

Pull Request (PR)

PR #670: Drop ALPN support Comment#1: [INFO] Reactor Summary: [INFO] ...

Issue #647: segfault in /system/lib/... Comment#1: That said, we can probably do something to work-around in OkHttp ...

Issue #666: Native crash in SSL ... Comment#10: Yes, but the startHandshake code calls SSL_CTX_set_alpn_ proto which does not look thread-safe at all. Comment#12: Duplicate of #647. Workaround the problem by adding this code to run early in your program...

Issue

held-out test set of 50 commits. Our analysis reveals that Argus outperforms prompting baselines, achieving 51.4% precision and 93.2% recall for rationale identification, while generating rationale summaries rated as accurate and correct relative to reference summaries. We further evaluated the perceived usefulness of Argus in a user study with 12 Java developers, who assessed the generated rationale components for 17 commits. Participants reported that the synthesized rationale helped them understand why changes were made across different types of commits. They also indicated that the rationale could be useful in comprehending code changes and could be valuable for tasks such as code reviews, documentation, and debugging. In summary, our work makes the following contributions: (1) An empirical evidence showing that rationale components are inconsistently and incompletely distributed across artifact types, highlighting the need for cross-document rationale extractionand synthesis to support developers in understanding the reason behind code changes. (2) Argus, a novel LLM-based approach that identifies rationale information across artifacts and synthesizes summaries of each component. (3) A user study indicating that developers perceive the generated rationale components as useful for understanding code changes. (4) A publicly available dataset of fine-grained, multi-document code change rationale, along with a replication package containing source code, and infrastructure to support reproduction and further research [7]. 2

Background and Motivating Example

Code Change Rationale Components. A code change is a set of modifications (such as additions or removals) made to one or more code files in a software system, recorded as commits in a version control system. The rationale of a code change describes why the change was designed and implemented in a particular way. A code change’s rationale can be broken down into rationale components, each capturing a different aspect of the reasons for implementing the change (e.g., the need and benefits of the change or considered alternatives) [11]. Manuscript submitted to ACM

4

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro Our work follows the 15-component taxonomy of code-change rationale by Al Safwan et al. [11], developed through

a rigorous mixed-methods process, including literature review, practitioner surveys and interviews. This taxonomy captures the key types of rationale information that developers require when implementing and maintaining software. The components are categorized into four themes: the code change’s objective, design, execution (i.e., implementation), and evaluation. The components in this taxonomy overlap by design as some can apply in certain development contexts while others can apply in other contexts. We refer the reader to Al Safwan et al.’s paper [11] for the full taxonomy. Objective rationale components include the code change’s Goal (what the developers aimed to achieve) and Need (the reasons or motivations for the change.). Design components include Constraints (limitations or restrictions for implementing the change) and Alternatives (various implementations considered to achieve the Goal). Execution components include Time (timing-related reasons for implementing the change) and Modifications (reasons related to code modifications). Execution components include Side Effects (reasons related to mitigating a change’s side effects) and Maturity Stage (reasons related to how mature the change is). Problem and Motivating Example. Figure 1 illustrates a code change and the related rationale fragments scattered across multiple artifacts that explain why the change was made. The example shows a commit1 from OkHttp2 , a Java HTTP client for the Java Virtual Machine (JVM), Android, and Graal Virtual Machine (GraalVM). The commit removed logic in Platform.java that supported Application-Layer Protocol Negotiation (ALPN), a Transport Layer Security (TLS) extension enabling clients to inform servers which application-layer protocols they support. While commit message states the change’s Goal (“Dropping ALPN support”), the underlying motivations are not immediately clear. Artifacts linked to the commit, shown in the figure, provide richer context that helps explain the change. For instance, the Class Javadoc of Platform.java note that “Android 4.4 . . . suffers from a concurrency bug,” while Issues #6663 and #6474 include detailed comments about the problem: “the SSL_CTX_set_alpn_protos call is not thread safe” and causes a “segfault in . . . libssl.so.” Issue #666 also records alternative solutions that developers considered, including small code changes as workarounds until a permanent fix in Android became widely available. A developer seeking a comprehensive understanding of the commit’s rationale, including its Goal, Need, and Alternatives, would need to manually locate, examine, and interpret all these artifacts, especially when the original developers are not readily available (as in open-source projects). This process can be tedious and time-consuming, particularly for junior developers or newcomers unfamiliar with the project’s code and history. Our goal in this paper is to automatically process such artifact information and synthesize coherent rationale summaries—like those in Figure 1 (see Generated Rationale in the middle), produced by our tool, Argus—which developers can quickly read to support their development and maintenance tasks. 3

Investigating Code Change Rationale Components in Software Artifacts

We investigate how software artifacts (e.g., issue reports, pull requests, and code reviews) related to a commit capture different components of rationale. Guided by the taxonomy of Al Safwan et al. [11] (Section 2), our study examines the extent to which these artifacts provide rationale information that can be extracted, integrated, and presented to developers when understanding code changes. The study addresses the following research questions (RQs): RQ1 : How frequently rationale components are found in software artifacts? 1 https://github.com/square/okhttp/commit/4c86085429edbeef0a383941936ee7b64cc3805e 2 https://square.github.io/okhttp/ 3 https://github.com/square/okhttp/issues/666 4 https://github.com/square/okhttp/issues/647

Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

5

RQ2 : Which software artifacts contain more or less rationale component information? 3.1

Commit Collection

Our study builds on the dataset curated by Tian et al. [68], which consists of 1,649 manually selected commits from five large, widely used Java open-source software (OSS) projects: Spring Boot [10], a Java-based framework for building production-grade applications; Apache Dubbo [3], an RPC and microservice framework; OkHttp [6], an HTTP client for the JVM, Android, and GraalVM; JUnit4 [5], a unit testing framework; and Retrofit [8], an HTTP client built on top of OkHttp. These commits were sampled from a larger population of 41.8K commits collected from these projects’ development up to 2021. Tian et al. carefully curated this dataset to capture diverse commit message characteristics, including variations in how developers describe the change (what), the rationale (why), or both. We refer the reader to Tian et al. [68] for a detailed description of their data collection process. To focus our analysis on commits where rationale can be meaningfully interpreted, we applied additional filtering criteria. In particular, trivial edits often lack explicit rationale, while very large or non-atomic commits may involve multiple intertwined changes, making it difficult to associate rationale with a specific decision. We applied the following filtering criteria to the 1,649 commits: • Language Filtering: We retained only commits that modify at least one Java file as our analysis focuses on code-level rationale. This resulted in excluding 510 commits without Java file changes, • Atomicity: We excluded 52 non-atomic commits that bundle multiple unrelated changes, as such commits hinder the attribution of rationale to a single change. • Modification Size Filtering: We filtered commits based on the number of modified files and lines of code to exclude trivial edits and extremely large commits. Using the interquartile range (IQR), we removed 166 commits that were outliers in the number of modified files and 226 commits that were outliers in the number of changed lines of code, retaining commits within the typical range of modification sizes. Applying these criteria reduced the dataset to 830 candidate commits. From these, we randomly selected 63 commits, stratified across projects (12–13 commits per project), to ensure coverage across different systems while keeping the manual annotation effort tractable. The sampled commits cover a range of change characteristics, including variations in modification size, number of affected files, and number of related development artifacts, providing a diverse set of scenarios for analyzing how rationale is expressed across artifacts. While the filtering reduces the size of the dataset, it allows us to focus on commits where rationale is more likely to be present, attributable, and interpretable, which is essential for our qualitative analysis. In our replication package—publicly available [7]—we provide the full list of sampled commits along with the applied filtering criteria and stratification details to facilitate transparency and reproducibility. 3.2

Software Artifact Collection

Understanding the rationale behind code changes requires examining information that is distributed across multiple software artifacts. Prior researches on rationale source have shown that developers document rationale infomation in a wide range of artifacts, including commit messages [27], issue reports [57, 72], emails [29], code comments [54, 56]. Informal communication channels such as mailing lists [15, 63], discussions on forums and Q&A sites [73] and chat discussions [12] have also been shown to contain rationale, including arguments and design decisions, although such information is often unstructured and difficult to systematically extract and link to specific code changes. Manuscript submitted to ACM

6

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro Despite this diversity of sources, most existing work focuses on individual artifact types in isolation, leaving limited

understanding of how rationale is distributed, complemented, and reconstructed across artifacts. To address this gap, we adopt a multi-artifact perspective and systematically collect artifacts that capture different aspects of rationale information i.e., Goal, Need, Alternatives etc. We collected six types of artifacts linked to the 63 sampled commits: Issues, Pull Requests, Code Review, Code Comments (Javadocs & Inline Code Comments), and Commit Messages. These artifacts represent key stages of the software development process, including problem identification (Issues), design discussion and decision-making (Pull Requests and Code Review), implementation summaries (Commit Messages), and code-level intent and documentation (Code Comments). We focus on these artifacts because they are consistently available and publicly accessible across all five projects, as they are managed within GitHub, and they can be reliably linked to specific commits through explicit references or repository structure. Other potential sources of rationale, such as mailing lists, chat logs (e.g., Slack or IRC), and external discussions, were not considered in this study. Although prior work shows that these artifacts may contain rich rationale, they are often unavailable, incomplete, or not systematically linked to commits in the studied projects. This makes it difficult to collect them consistently and to establish reliable traceability between discussions and code changes. Focusing on GitHub-based artifacts therefore enables us to construct a consistent, traceable, and reproducible dataset. We distinguish between class-level Javadocs, method-level Javadocs, and inline code comments because they convey rationale at different levels of granularity and serve distinct documentation purposes. Class-level Javadocs typically describe high-level design intent, architectural roles, and overall responsibilities of a component. Method-level Javadocs provide more localized explanations, such as the behavior of individual methods, parameter semantics, and implementation constraints. In contrast, inline code comments are embedded directly within the code and often capture fine-grained, implementation-specific rationale. Separating these three forms of documentation enables us to analyze how rationale is expressed across different abstraction levels, from high-level design decisions to low-level implementation details. All five projects in the dataset are hosted on GitHub, which provides unified access to issues, pull requests, and code reviews. To establish accurate commit–artifact links, we combined heuristics, regular expressions, GitHub’s API, and manual validation. Specifically, we (i) searched commit messages for explicit references to issue or pull request identifiers using common patterns (e.g., “#”, “pull/”) and issue or PR urls, (ii) queried the GitHub API to identify mentions of commit hashes in issue and pull request discussions, and (iii) performed cross-artifact searches (e.g., identifying issues or pull requests referenced in commits, comments, titles, or discussion threads) to recover additional links, accounting for cases where the referenced issue/PR is not explicitly mentioned in the commit message but appears elsewhere in related artifacts. After identifying the related issues and pull requests, we used the GitHub API to systematically retrieve their detailed metadata, including descriptions, discussion threads. Javadocs and inline comments were extracted from modified Java files using the comment-parser [4] library with pattern-based heuristics, while code review comments were obtained from pull request discussions via the GitHub API. A single commit may be associated with multiple artifacts of the same type (e.g., multiple issues or pull requests), and artifacts may also reference each other. In total, we identified 339 artifacts across the 63 commits, providing a diverse and multi-granular dataset for subsequent annotation and analysis. 3.3

Artifact Validation and Annotation Process

We conducted an iterative multi-coder qualitative analysis to (1) validate that the identified issue reports and pull requests were truly associated with the commits, and (2) annotate the sentences in the artifacts according to the rationale Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

7

components they convey. Our process follows a consensus-based coding approach guided by the taxonomy of Al Safwan et al. [11]. We first parsed the textual content of all collected artifacts into sentences using SpaCy’s English transformer pipeline (en_core_web_trf), resulting in a total of 3,088 sentences across 339 artifacts associated with 63 commits. 3.3.1

Artifact Relevance Validation

Before annotating the rationale-containing sentences, two annotators independently evaluated whether each artifact was directly related to its corresponding commit. An artifact was considered relevant only if it provided meaningful context about the code change, rather than incidental or unrelated references (e.g., mentions in off-topic discussions). From the 339 artifacts collected across 63 commits, both annotators agreed that 44 artifacts were irrelevant, while 7 artifacts were flagged as irrelevant by only one of the annotators. All disagreements were subsequently resolved through consensus during annotation meetings. In total, 48 artifacts were deemed irrelevant and excluded from further analysis. These excluded artifacts contained 486 sentences, which were not considered for qualitative analysis. The remaining 2,602 sentences, drawn from relevant artifacts, were retained for rationale annotation. In contrast, for the automated rationale generation experiments described in section 5.2.2, we intentionally included irrelevant content to evaluate how our approach performs under realistic conditions where such artifacts may arise. We further examined this effect in the user study reported in section 6, analyzing how participants perceived the erroneous rationale introduced by irrelevant artifacts. 3.3.2

Rationale Annotation Process

Our goal is to identify sentences that express code change rationale and assign them to fine-grained components from Al Safwan et al.’s taxonomy [11] (e.g., Goal, Need,Alternatives, Validation etc.). We annotate at the sentence level where a sentence may express multiple rationale components. We conducted annotation over six iterative rounds, each involving 10–13 commits. Two annotators independently labeled the sentences in each round as follows: (1) Annotators labeled each sentence with one or more rationale components. Sentences that did not express rationale were left unlabeled. (2) After each round, annotators reviewed all conflicts and resolved them through discussion to reach consensus. (3) Following agreement, we summarized the identified rationale components for each commit into concise naturallanguage statements, which serve as ground truth for evaluating automated rationale generation. (4) This process was repeated across all six rounds until all commits were annotated. Codebook Development: A shared codebook guided the annotation process. It was initially derived from the definitions and examples provided in Al Safwan et al.’s taxonomy. Following each iteration of the annotation process, the codebook was refined to incorporate additional rules for identifying rationale components, particularly in response to annotator disagreements and sources of ambiguity. After each round, the annotators collaboratively updated the codebook by clarifying definitions, adding illustrative examples, and refining annotation guidelines to reduce subjectivity. This iterative refinement process improved consistency and minimized ambiguity across annotation rounds. The final version of the codebook is included in our replication package[7]. Contextual Grounding and Bias Mitigation: To ensure that sentences were accurately judged as reflecting rationale, and to minimize interpretation biases stemming from limited familiarity with the project’s code, the annotators followed a structured preparation and contextualization process. Before coding a commit’s artifacts, they first examined the code change itself, along with relevant surrounding code in the modified files, to establish the technical context of the modification. Next, they carefully reviewed all information contained in the associated artifacts (e.g., the full discussion Manuscript submitted to ACM

8

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro

and metadata in the issue reports) to capture the broader context of the commit. Only after this contextual review did the annotators proceed with sentence-level annotation. When necessary, they also consulted official project documentation, including web API references and code-level usages of modified APIs, to better understand their intended usage. This process ensured that annotations were not based on isolated sentences but grounded in the full technical and organizational context. It allowed annotators to determine (1) whether the artifacts were directly associated with the commit (rather than serving merely as cross-references, general discussions, or historical notes), and (2) whether a given sentence expressed rationale and, if so, which components of the taxonomy it represented. Ambiguous cases were resolved during reconciliation meetings held as part of the coding sessions. Rationale Component Selection: We initially conducted a pilot study on a stratified sample of 10 commits. During this phase, we observed that certain execution-related components (e.g., Committer, Time, Location, Modifications, and Explanation of Modifications) are directly obtainable from version-control metadata and therefore do not require interpretive annotation. This observation is also consistent with findings by Al Safwan et al. [11], whose participants identified these components as the easiest to locate. Additionally, the Benefits component was excluded due to its strong semantic overlap with Need, which made consistent distinction between the two difficult during annotation. From the 15 rationale components defined by Al Safwan et al. [11], we therefore excluded these components. Our analysis focuses on the remaining components that capture more interpretive, decision-oriented aspects of developer rationale. Inter-Annotator Agreement: We measured inter-annotator agreement using Cohen’s 𝜅 [22]. Across 6 annotation rounds, agreement reached 𝜅 = 0.905 for artifact relevance (indicating almost perfect agreement) and 𝜅 = 0.812 at the 5th iteration and 𝜅 = 0.786 at the final iteration for rationale component labeling (indicating substantial agreement). Agreement improved over successive rounds (from 0.584 to 0.786), reflecting increased alignment between annotators. Disagreements between annotators mainly arose from (1) implicit rationale-bearing sentences i.e., rationale spreading across sentences (2) components overlapping i.e., a single sentence expressing multiple rationale component information (3) differences in technical knowledge and subjective interpretation. Overall annotation accuracy averaged 99.4%, consistently remaining above 99.0% across all rounds, largely because most sentences contained no rationale information and both annotators consistently agreed on these cases. The final outcome of this process is a sentence-level, multi-label dataset of rationale components grounded in validated artifacts. This dataset serves as the foundation for our empirical analysis and the evaluation of our automated rationale extraction approach. 3.4

Results and Analysis

We analyzed 63 sampled commits comprising 127 modified code files (2.0 files per commit on average). For these commits, we initially collected 77 pull requests (Pull Requests) and 37 issue reports (Issues). After filtering out artifacts not directly related to the corresponding commits, 49 Pull Requests and 24 Issues were retained for analysis. Across all artifacts, we identified a total of 3,088 sentences. Following the artifact relevance validation process (Section 3.3.1), 48 artifacts (486 sentences) were deemed irrelevant and excluded. From the retained artifacts, we collected multiple sources of rationale. Specifically, from the modified files, we extracted 57 Class Javadocs (Class Javadocs), 16 Method Javadocs (Method Javadocs), and 68 code comments (Code Comments). Among the 49 Pull Requests, 14 included code review discussions. Overall, this resulted in 291 artifacts (4.6 per commit on average) used in our analysis. A detailed, project-wise distribution of all artifacts is presented in Table 1. Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

9

Table 1. Artifacts associated with the 63 sampled commits. Values in parentheses indicate the average per commit. Abbreviations: Changed Code Files (CCF), Class Javadocs (CJ), Method Javadocs (MJ), Code Comments (CC), Pull Requests (PR), Code Reviews (CR), and Issue Reports (IR). Project Dubbo Junit4 OkHttp Retrofit Spring-boot Total

Commits 13 13 12 13 12 63

CCF 18 (1.4) 18 (1.4) 40 (3.3) 23 (1.8) 28 (2.3) 127 (2.0)

CJ 11(1.4) 10(1.1) 13(1.9) 8(1.0) 15(1.4) 57(1.3)

MJ 4(1.0) 2(1.0) 5(1.3) 5(1.0) (0.0) 16(1.1)

CC 16(1.3) 7(1.0) 21(1.8) 17(1.5) 7(1.2) 68(1.4)

PR 15(1.5) 9(1.0) 11(1.0) 11(1.1) 3(1.0) 49(1.1)

CR 1(1.0) 3(1.0) 2(1.0) 7(1.0) 1(1.0) 14(1.0)

IR 6(1.2) 3(1.0) 4(1.3) 4(1.0) 7(1.2) 24(1.1)

Total 66(5.1) 47(3.6) 68(5.7) 65(5.0) 45(3.8) 291(4.6)

3.4.1 RQ1 : How Frequently Rationale Components are Found in Software Artifacts? We identified 7 rationale components in the analyzed artifacts, spanning 253 sentences and 62 of the 63 commits (Table 2). Among these, Goal (what the change achieves), Need (why the change was necessary), and Alternatives (considered alternatives) were the most frequently documented. Specifically, 146 sentences (57.7%) expressed Goal, 86 (34.0%) expressed Need, and 20 (7.9%) expressed Alternatives. In contrast, the remaining components were rare: Selected Alternative, Maturity Stage, Validation, and Side Effects appeared in only 1–6 sentences overall. We found no artifact content expressing Constraints or Dependency. At the commit level, artifacts associated with 61 of the 63 commits document the change’s Goal. The Need is documented for 34 commits (54.8%), indicating that nearly half of the commits lack an explicit statement of motivation in the analyzed artifacts. For most commits (54 or more), none of the remaining components were identified. Table 2. Rationale components found in software artifacts Component Goal Need Alternatives Selected Alternative Maturity Stage Validation Side Effects Total

Commit 61 (98.4%) 34 (54.8%) 9 (14.5%) 4 (6.5%) 1 (1.6%) 2 (3.2%) 2 (3.2%) 62

Artifact 114 (84.4%) 49 (36.3%) 9 (6.7%) 4 (3.0%) 1 (0.7%) 2 (1.5%) 2 (1.5%) 135

Sentence 146 (57.7%) 86 (34.0%) 20 (7.9%) 6 (2.4%) 1 (0.4%) 4 (1.6%) 2 (0.8%) 253

To examine whether the aggregate trends reported above are consistent across projects, we also analyzed rationale components at the system level (table 3). Overall, the same high-level pattern holds across all five repositories. In every system, Goal is the most frequently documented component and it appears in nearly all sampled commits unniformly. Need is the second most common component, but it also appears almost uniformly accross each repository. All remaining components unevenly distributed across systems but they are rare and very few in number for which we couldn’t derive any pattern for those components. For example, Alternatives and Selected Alternative appear only in a small number of commits in JUnit4, OkHttp & Spring-boot, while Maturity Stage and Side Effects are observed only once or twice. Overall, these results suggest that our aggregate findings are not driven by a single project; rather, the tendency for developers to document some rationale components, especially Goal and Need, much more often than others is consistent across systems. These findings partially align with Al Safwan et al. [11]. Their study, based on developers’ reported experiences, found that side effects and alternatives are among the most difficult rationale components to find, and that alternatives, selected alternative, constraints, and maturity stage are among the least frequently recorded. Our artifact-based results confirm this pattern: Alternatives and Selected Alternative were rarely documented, Maturity Stage was nearly absent, Manuscript submitted to ACM

10

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro Table 3. Project-wise distribution of commits containing each rationale component Component Goal Need Alternatives Selected Alternative Validation Maturity Stage Side Effects #Commits

JUnit4 11 7 1 1 1 0 0 12

Dubbo 13 7 0 0 1 0 0 13

Retrofit 13 7 0 0 0 0 0 13

OkHttp 12 6 4 2 0 0 1 12

Spring-boot 12 7 4 1 0 1 1 12

and Constraints did not appear at all. We also found Side Effects to be extremely uncommon. Taken together, these results suggest that the components developers report as hard to find are indeed sparsely documented in commit-related artifacts. The only sampled commit with no identifiable rationale component was JUnit4 commit cec4a6b[1]. This was a minimal change affecting a single file and adding only TODO-style comments, with an empty commit message and no associated discussion artifacts. As a result, the available artifact set did not provide enough contextual information to recover any rationale component. This suggests that very small commits with sparse surrounding discussion may leave little explicit rationale trace. We believe Goal, Need, and Alternatives are more frequently found because they are the components most naturally expressed in developer communication, such as commit messages, pull request descriptions, and issue discussions. By contrast, components such as Constraints, Dependency, Maturity Stage, and Side Effects are often left implicit unless they become especially salient during implementation or review. The JUnit4 commit[1] with no identifiable rationale illustrates this pattern: because the change was minimal and accompanied by no descriptive message or discussion, even basic rationale components such as Goal were absent. RQ1 Findings: Seven of nine rationale components are found in the software artifacts associated to the studied 63 commits. The most frequently documented components are the change’s Goal, Need, and Alternatives, found the artifacts of 61, 34, and 9 commits, respectively. The remaining components, Selected Alternative, Validation, Side Effects, and Maturity Stage are less frequently documented (found for 1-4 commits). 3.4.2 RQ2 : Which Artifacts Contain More or Less Rationale Component Information? We examined whether rationale is uniformly distributed across artifacts and projects. As Table 4 shows that Commit Messages, Issues, and Pull Requests are the richest sources of rationale, together accounting for 91.3% of all rationale content. Information about what a code change accomplishes (Goal) is primarily documented in Commit Messages (46.6%) and Pull Requests (41.8%). In contrast, information about the motivation behind code changes (Need) is most often recorded in Issues and Pull Requests, with only 16.3% of Need sentences appearing in Commit Messages. The components Alternatives and Selected Alternative occur exclusively in Issues, Pull Requests, and Code Reviews, which also capture some of the less frequent components. Pull Requests capture 36.4% of all rationale-bearing sentences and cover nearly all (5 of 7) components, making them the most diverse artifact for documenting rationale. Issues and Code Reviews follow as the second (5 of 7) and third (3 of 7) most diverse sources, respectively. Overall, these results indicate that developers and other stakeholders primarily rely on Pull Requests, Commit Messages, and Issues to document rationale across the 63 studied commits—a pattern consistently observed in all five software projects. Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

11

Table 4. Number of rationale sentences found for each artifact (with column-wise proportions). Artifacts

Goal

Need

Class Javadocs Code Comments Code Reviews Commit Messages Issues Method Javadocs Pull Requests

3 (2.0%) 2 (1.4%)

1 (1.2%) 1 (1.2%)

68 (46.6%) 7 (4.8%) 5 (3.4%) 61 (41.8%)

14 (16.3%) 39 (45.4%) 3 (3.5%) 28 (32.6%)

146 (100.0%)

86 (100.0%)

Total

Alternatives

Sel. Altern.

3 (15.0%)

3 (50.0%)

12 (60.0%)

1 (16.7%)

Validation

Side Effects

Maturity

Total 4 (1.6%) 3 (1.2%) 6 (2.4%) 80 (31.6%) 60 (23.7%) 8 (3.2%) 92 (36.4%)

1 (50.0%) 1 (100.0%)

5 (25.0%)

2 (33.3%)

4 (100.0%)

1 (50.0%)

20 (100.0%)

6 (100.0%)

4 (100.0%)

2 (100.0%)

1 (100.0%)

253 (100.0%)

Interestingly, all artifacts except Code Reviews include Goal and Need. Code Reviews instead primarily contains information about Alternatives, Selected Alternative, and Side Effects. This suggests that developers use Code Reviews mainly to document implementation-level rationale, focusing on decision-making and potential consequences, rather than high-level goals or motivations. We also compared the presence of rationale components in Commit Messages versus other sources, which may not be as readily accessible as commit messages for developers when understanding code changes. As shown in Table 5, Goal is captured in nearly all commit messages (61 of 63 commits). However, Need is far more often documented outside commit messages (24 commits) than inside them (10 commits). Other components are found exclusively in sources beyond commit messages.

Table 5. Commits where rationale components are found in commit messages (CM) vs. other artifacts.

Goal

Need

Alternatives

Sel. Altern.

Validation

Side Effects

Maturity

Only in others Only in CM In both

1 (1.6%) 19 (30.2%) 41 (65.1%)

24 (38.1%) 2 (3.2%) 8 (12.7%)

9 (14.3%) 0 (0.0%) 0 (0.0%)

4 (6.3%) 0 (0.0%) 0 (0.0%)

2 (3.2%) 0 (0.0%) 0 (0.0%)

2 (3.2%) 0 (0.0%) 0 (0.0%)

1 (1.6%) 0 (0.0%) 0 (0.0%)

Total

61 (96.8%)

34 (54.0%)

9 (14.3%)

4 (6.3%)

2 (3.2%)

2 (3.2%)

1 (1.6%)

In essence, commit messages primarily capture Goal, while the remaining six components are typically documented elsewhere. This means that developers inspecting only commits often miss information about Need and other components, and must search additional artifacts to retrieve it. This result motivates the automated cross-artifact identification and generation of rationale components. To assess how much rationale information each artifact contains, we compared the total number of sentences per source with the number coded for at least one rationale component (table 6). The density of rationale information varies widely. Commit Messages is by far the richest source, with 76.9% of its sentences containing rationale. However, this is the 2nd shortest source in terms of the number of sentences. Pull Requests and Method Javadocs follow with 15.8% and 9.5%, respectively. In contrast, Issues, Code Reviews, and Class Javadocs each contain fewer than 6% coded sentences, while Code Comment contributes almost none (0.36%). Manuscript submitted to ACM

12

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro

RQ2 Findings: Commit Messages and Pull Requests predominantly document the Goal of code changes, while Need is most frequently captured in Pull Requests and Issues. Pull Requests and Issues are the most diverse sources of rationale as they document 5-7 components. Most of the rationale components beyond Goal are documented in artifacts other than commit messages, which highlights the need for automated identification and generation of rationale across multiple sources.

Table 6. Proportion of sentences describing rationale in software artifacts.

Artifact

4

Total Sentences

Rationale Sentences

% of Rationale Sentences

Class Javadocs Code Comments Code Reviews Commit Messages Issues Method Javadocs Pull Requests

180 829 160 104 795 84 450

4 3 6 80 60 8 92

2.2% 0.4% 3.8% 76.9% 7.5% 9.5% 20.4%

Total

2,602

253

9.7%

Argus: Automated Extraction and Generation of Code Change Rationale

Our qualitative study (Section 3) showed that software artifacts associated to commits are rich sources of rationale. Commit messages frequently capture the change’s Goal but rarely include other components. Across artifacts, Goal, Need, and Alternatives are the most common components. As a result, developers often lack immediate access to the rationale needed to understand changes, motivating automated support for identifying and generating rationale across multiple sources. We introduce Argus, an LLM-powered approach for fine-grained Automatic Rationale extraction and Generation from mUlti-document Sources. Given a commit, Argus retrieves associated artifacts (e.g., issues and code reviews), analyzes their contents, and leverages LLM reasoning capabilities to identify sentences that express the change’s Goal, Need, and Alternatives. Powered by GPT-o4-mini, it then reasons and connects related content across artifacts to produce concise summaries for each component, which can support developers directly or serve as documentation of the rationale behind a change. Our goal in this paper is to evaluate the feasibility and effectiveness of fine-grained, multi-document rationale extraction and synthesis, rather than to maximize absolute performance using the strongest available LLM. We therefore selected GPT-o4-mini as it provides sufficient reasoning capability for this task while remaining cost-effective which enabled us to conduct extensive prompt development, repeated executions, and user evaluation within a realistic deployment setting. Argus ’s workflow, which in current form, supports the GitHub platform, consists of three modules, illustrated in Figure 2, which we detail next. Commit Message/ Code Diff

Artifact Retriever Commit URL

Code Diff

Developer

Pull Request/ Issues Reports

GitHub Search API

RegEx Artifact Repository

Heuristics

JAVA DOC

Javadocs/ Code Comment Code reviews

Rationale Extractor Prompt Sentence Parsing

Artifacts w/ labeled components

Code Diff Actual Sentence

+

GPT-4o-mini Reasoning

Prompt

Code Diff Goal: Goal Need Goal: Alternative

Fig. 2. Argus’s Architecture Manuscript submitted to ACM

Rationale Generator Component Description

Component Description

Actual Sentence

+

GPT-4o-mini Reasoning

Rationale Summaries Goal Need Need: Alternative

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

13

Artifact Retriever. This module takes as input a project commit (e.g., its URL) and outputs a set of associated artifacts, including their textual content parsed into sentences. It retrieves the commit message and diff, as well as linked issue reports, pull requests, and code reviews. As described in Section 3.2, artifacts are identified through a combination of regular expressions, GitHub API queries, and heuristic-based text matching, which detect artifact IDs in commit messages or commit references in the artifacts’ text. The retriever also parses commit data to extract Javadoc and inline comments from modified classes and methods. All textual content is segmented into sentences using SpaCy’s English transformer pipeline (en_core_web_trf) [9]. This multi-document retrieval step is essential because, as shown by our qualitative study (Section 3), rationale information is distributed across artifacts. Collecting from the most-likely sources associated to a commit ensures that Argus’s subsequent modules operate on a comprehensive set of potentially rationale-bearing artifacts. Rationale Component Extractor. This module automatically labels each artifact sentence extracted by the previous component according to the type of rationale it expresses (Goal, Need, and/or Alternatives). Argus focuses on these three categories because they are the most frequently documented in software artifacts. To perform the labeling, the module prompts the LLM to reason about the meaning of each sentence and assign one or more component labels, or none if the sentence conveys other information. The prompt follows a task decomposition approach [37], incorporating component definitions, sentences grouped by specific artifact, and few-shot exemplars with ground-truth labels and explanations. This design guides the LLM to interpret sentence semantics and apply fine-grained classification. section 5 provides details about our data-driven methodology, which experimented with different prompting strategies and context, to determine the most accurate approach to guide the LLM for this task. Rationale Component Generator. This module takes the sentences labeled by the Rationale Component Extractor and generates concise, human-readable summaries for each rationale component. To achieve this, the module prompts the LLM to reason over information collected from multiple sources, connect fragmented pieces, and synthesize them into a coherent description that captures the essence of each component. The prompt again follows a task-decomposition approach [37], incorporating component definitions, labeled sentences grouped by artifact, and few-shot exemplars with ground-truth labels. As detailed in section 5, we adopted a data-driven methodology, experimenting with different prompting strategies and contexts to guide the LLM toward producing accurate and correct rationale summaries. Although recent LLMs support large context windows (e.g., 200K tokens for GPT-o4-mini), providing all artifacts directly to the model is not necessarily optimal. Recent empirical studies [44, 50] show that LLM reasoning performance degrades as input length increases, even when the input remains within the model’s technical context limit. Prior work reports notable drops in reasoning accuracy for long prompts [44, 50] and demonstrates that models effectively utilize only a fraction of the provided context [41] , often failing to recall relevant information distributed across long inputs [71]. Consequently, simply concatenating all artifacts for a commit, such as issues, pull requests, code reviews, and documentation, can reduce the model’s ability to reason about the change rationale. Argus therefore adopts a two-stage design. The Rationale Extractor first identifies candidate rationale-bearing sentences across artifacts, which significantly reduces the input size and focuses the subsequent reasoning step on the most relevant information. This design improves controllability and allows the generation module to reason over a compact and semantically focused context rather than large heterogeneous artifacts. While the extractor may occasionally introduce false positives or miss some sentences, our evaluation shows that this modular design enables the model to achieve high recall(94%) while maintaining manageable context sizes for reasoning. Importantly, this design choice is motivated not by context-length limitations, but by the need to structure and filter heterogeneous artifact content. Separating extraction from generation also improves modularity, enables intermediate Manuscript submitted to ACM

14

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro

inspection and error analysis, and allows alternative extraction strategies (personalization) to be incorporated in the future. 5

Argus ’s Prompt Development and Evaluation

We adopted a data-driven methodology to develop prompts templates for Argus’s two major tasks: rationale component identification and generation. Our goal was to determine the most effective prompting strategies to guide the GPT-o4mini to accomplish both tasks. This was achieved using a subset of 13 commits and corresponding ground truth data (e.g., annotated artifact sentences). Additionally, we evaluated the best strategies, implemented in Argus, compared to baseline prompting approaches on the remaining 50 commits and associated ground truth data to assess the accuracy and correctness of Argus’s generated rationale components. We experimented with five prompt templates that implement well-known prompting strategies across Argus ’s two tasks: rationale component identification (three templates) and generation (two templates). The strategies include zero-shot, few-shot, and reasoning-based few-shot prompting, all combined with task decomposition prompting [37]. Importantly, in all experiments of this section, we included both artifacts relevant and irrelevant to the commits. This choice reflects real-world deployment of Argus, where the artifact retriever is expected to return both types. This design increases the likelihood of capturing code change rationale. With this in mind, we addressed the following research questions (RQs): RQ3 : What prompting strategies are most effective to identify rationale components in artifacts? RQ4 : What prompting strategies are most effective to generate rationale component summaries? RQ5 : How accurate is Argus’s rationale identification module? RQ6 : How accurate are Argus’s generated rationale summaries? 5.1

Dataset Construction

Prompt development and evaluation relied on two types of ground truth data: (1) manually labeled artifact sentences tagged with the rationale components they convey (for identification), and (2) manually written descriptions of the three rationale components Argus target: Goal, Need, and Alternatives (for generation). The annotated sentences were selected from the dataset built in our qualitative study (Section 3). The ground truth component descriptions were created by the same two annotators who participated in that study. One annotator synthesized clear and concise summaries for each commit by reviewing the commit diff, message, and annotated artifact sentences with Goal, Need, and Alternatives. The second annotator then reviewed the summaries for clarity, correctness, and accuracy compared to the labeled sentences. Disagreements were resolved by discussion until consensus was reached. This iterative, multi-coder approach was repeated in six rounds of 10–13 commits each, producing concise and reliable summaries while mitigating subjectivity and errors. We applied this methodology to all 63 commits collected in the qualitative study, then divided them into two subsets: 13 commits for prompt development (to answer RQ3 and RQ4 ) and 50 commits for evaluation (to answer RQ5 and RQ6 ). The 13 commits in the prompt development set are used exclusively for prompt formulation and qualitative refinement. The goal here to ensure prompt correctness rather than to learn dataset-specific patterns for which 13 commits were sufficient to observe recurring prompt-level issues. The split was random and stratified by project, rationale component coverage, and artifact diversity. Although both the prompt-development set (13 commits) and the evaluation set (50 commits) are relatively small, which limits statistical generalization, our objective is not population-level inference but to assess the feasibility of Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

Zero Shot

Few Shot

Code Change Sentences from Artifacts Output Format Exemplars

Reasoning-based Few Shot

Component Description

Few-Shot (CI-FS): Task description + instruction + input data + response format + two exemplar commits Reasoning-based Few-Shot (CI-RFS): Task description + instruction + input data + response format + two exampler commits + reasoning behind exemplar commit's sentences annotation

Task Summary

Zero Shot (CG-ZS): Task description + instruction + input data + response format

Input Description Instructions Few Shot

Input Description Instructions

Zero Shot

Zero-Shot (CI-ZS): Task description + instruction + input data + response format

Task Summary

15

Component Description

Few Shot (CG-FS): Task description + instruction + input data + response format + two exemplar commits

Code Change Sentences from Artifacts

Reasoning

Output Format Exemplar

(b) Rationale component generation (a) Rationale component identification Fig. 3. Structure of the developed prompts for both of Argus’s major tasks.

extracting and synthesizing fine-grained rationale across artifacts. The scale of the study is also shaped by the substantial manual effort required for dataset construction. Across the six annotation iterations, we spent approximately 52.8 hours on independent coding, 10.1 hours on reconciliation, and 20 hours on ground-truth rationale creation, for a total of about 82.9 human hours. Larger-scale evaluations are therefore an important direction for future work. Although the prompt development set is smaller, it still covers all five projects and includes a balanced mix of rationale components. It includes 13 commits with all rationale components represented (13 with Goal, 9 with Need, 4 with Alternatives) and 76 artifacts including Javadocs, Commit Messages, Pull Request etc.. The evaluation set contained 50 commits (48 with Goal, 25 with Need, 5 with Alternatives) and 263 artifacts across the same types. This variety allowed us to test Argus ’s performance across different projects, rationale components, and artifact sources. About 12% of sentences in development set and 17% in evaluation set came from irrelevant artifacts, helping refine prompts for real-world settings. 5.2

Prompt Development Methodology

Our prompt development process explored three strategies: zero-shot, few-shot, and reasoning-based few-shot prompting, all combined with task decomposition prompting [37]. For rationale identification, we created three templates (one per strategy), and for rationale generation we designed two templates (zero-shot and few-shot). Two of the 13 commits from the development set were selected as exemplars for few-shot templates chosen by annotators. We intentionally limit the number of few-shot exemplars (two commits) to reduce prompt length since each commits because on average each commits contains 41 sentences. Each prompt began with a base template that included standard components (e.g., task description, instructions, exemplars, input/output format). We executed the prompts, inspected errors on the 11 development commits, and revised the templates until performance improvements plateaued. For component identification, errors were categorized as false positives or false negatives by comparing LLM outputs with the ground truth. For component generation, errors were grouped according to weather the generated summaries lacked or contained extract information compared to the ground truth. Annotators examined these cases to derive adjustments including decision rules and prompt refinement. 5.2.1 Prompt Development for Rationale Identification. Rationale component identification (CI) is formulated as a multilabel classification task where artifact sentences are tagged with one or more components (Goal, Need, Alternatives) or none. We explored three prompting strategies, all using task decomposition [37] (see Figure 3a): (1) Zero-shot (CI-ZS): Manuscript submitted to ACM

16

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro

This prompt includes task summary, an input data description (the definition of the rationale components, the commit diff, and the sentences grouped by artifact), specific task instructions, the inputs, and the output format.(2) Few-shot (CI-FS): the prompt extended CI-ZS with two exemplar commits containing labeled and few unlabeled artifact sentences. (3) Reasoning-based Few-shot (CI-RFS): the prompt further extended the CI-FS prompt with explanations for why the exemplar sentences were tagged with specific rationale components. We developed this prompt in two steps. First, we used explanations behind each sentences’ of the example commits to better guide the model. Then, we analyzed the failed cases in that prompt and derived a set of rules to improve the prompt. Metrics and prompt execution. We measured prompt prediction accuracy by comparing predicted and ground truth labels for each component. This yielded true/false positives and negatives for each of the three rationale components. We computed overall accuracy by totaling true/false positives and negatives across components and then computing precision, recall, and F2 scores. Precision measures the fraction of predicted rationale sentences that are correct; recall measures the fraction of true rationale sentences that were identified; F2-score provides a balanced measure of accuracy via the harmonic mean between precision and recall, weighting recall twice as much as precision. We used F2-score because rationale-bearing artifact sentences are relatively rare (≈10% of all artifact sentences as seen in Section 3) so missing their identification is more detrimental than identifying non-rationale sentences. To account for nondeterministic model output, each prompt was run three times, and we report both metric consistency and average metrics across runs. 5.2.2 Prompt Development for Rationale Generation. For rationale generation (CG), Argus produces concise rationale descriptions, given sentences previously labeled as Goal, Need, and/or Alternatives. We designed two templates using task decomposition prompting [37] (see Figure 3b): (1) Zero-shot (CG-ZS): the prompt included the task summary, component definitions, specific task instructions, the inputs, and the output format. The inputs included the commit diff, and the labeled sentences grouped by artifact. (2) Few-shot (CG-FS): The prompt extends the CG-ZS prompt with two exemplar commits including their labeled sentences and ground truth rationale summaries. We did not employ reasoning-based few-shot prompting, as task decomposition provided sufficient guidance for generating concise summaries that reflect the content of the labeled sentences. Metrics and prompt execution. To evaluate summary quality, two researchers independently compared generated outputs with ground truth summaries at a semantic level, focusing on two dimensions: information coverage (IC)—the extent to which ground truth content was captured in the outputs, and extra information (EI)—content produced by the model but absent from ground truth. Both dimensions were rated on 5-point Likert scales, with disagreements reconciled through discussion and consensus. Overall inter-rater agreement, measured using weighted Cohen’s Kappa (𝜅) [23], was 0.88 indicating high annotator reliability. We used this index because the Likert scales are ordinal, and disagreements between distant scores (e.g., 1 vs. 5) are more severe than those between adjacent scores (e.g., 3 vs. 4). Disagreements were mostly due to subjective interpretation of the summaries, which were resolved through annotators’ discussion and consensus Although prompts were executed three times, generated summaries were highly consistent across runs. Embeddingbased cosine similarity (using OpenAI’s text-embedding-3-large model [53]) confirmed that outputs ranged from 0.810 to 0.958 with a median of 0.935 in terms semantic similarity. Given these results, we evaluated the generated summaries of the first run only. Since missing ground truth content is more detrimental than including additional but plausible details (as component identification may incorrectly label non-rationale artifact sentences), we weighted information coverage (IC) more Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

17

heavily than extra information (EI) by computing the F2 score [59] (which doubles the weight of coverage relative to extra information) as our overall measure of rationale generation accuracy.

5.3

Prompt Development Results

5.3.1 RQ3 : Prompt Development Results for Rationale Identification. We executed all three identification prompts three times using GPT-o4-mini [52]. To assess the consistency of predictions, we calculated Krippendorff’s alpha [40] to see the similarity between different runs. we found overall for Goal and nonrationale sentences, the model produce somewhat consistant responses (Goal = .88) but for Need (0.72) and Alternatives (.64) it was moderate. To address this variation, we adopted a majority-voting strategy: a sentence is assigned a label set 𝑋 (e.g., Goal and Need) only if the model produced the same outcome in at least two of the three runs. The intuition is that agreement across multiple executions increases confidence in the correctness of the prediction. Experiments on the development set (available in our replication package [7]) confirmed that the voting approach outperformed the average performance of the individual runs, yielding relative improvements of 7% in precision, 3% in recall and 4% in F2-Score. Based on these results, we implemented the voting approach for all three rationale identification prompts (Section 5.2.1) and report their performance in this section. Table 7a reports the performance of the three prompting strategies across components. Overall, reasoning-based few-shot (CI-RFS) achieves the highest accuracy, with 72.0% precision, 93.7% recall, and 88.3% F2-score. It substantially outperforms both the few-shot (CI-FS) and zero-shot (CI-ZS) strategies, which rank second and third, respectively. In practical terms, this means that CI-RFS correctly identifies about nine out of ten rationale-bearing sentences, and that roughly seven out of ten sentences it identifies as rationale indeed convey rationale. These results indicate that providing exemplars with explanations enables the LLM to better learn how to recognize rationale. In fact, when examining the relative improvement (RI) across strategies (Table 7b), we observe that the presence of explanations for the exemplars is the key factor driving CI-RFS’s superior performance, yielding gains over CI-FS of 16% in precision, 5% in recall, and 8% in F2-score (compared to CI-FS vs. CI-ZS improvements of 0%, 4%, and 3%, respectively). Table 7b reveals that combining both exemplars and their explanations leads to substantial gains in precision (16% RI of CI-FS vs CI-ZS) and recall (4% RI of CI-FS vs CI-ZS) , thus explaining the superiority of CI-RFS. Looking at performance by component in Table 7a, we observe that Goal is the easiest to identify, with CI-RFS achieving very high precision and recall (96%+). For Need and Alternatives, CI-RFS attains high recall but lower precision (59.5% and 50%, respectively). As discussed earlier, given the scarcity of rationale-bearing components, it is generally more beneficial for the model to maximize recall—ensuring that it captures as many rationale sentences as possible—even if this comes at the cost of precision. From this perspective, we consider CI-RFS’s performance acceptable for the task. 5.3.2 RQ4 : Prompt Development Results for Rationale Generation. Table 8 reports the results of the component generation prompts. Both variants (CG-FS and CG-ZS) produce similar rationale summaries for Goal and Alternatives in terms of information coverage (IC) and extra information (EI) relative to the ground-truth summaries. The main difference lies in the Need summaries generated by CG-FS, which cover moderately more information (4.8% RI) while introducing less (10.3% RI) extra content. (Note that EI is inversely defined: higher values indicate less extra information.) Overall, these results suggest that exemplars with explanations provide better guidance for the LLM, leading to summaries that are more accurate and faithful to the ground truth. Manuscript submitted to ACM

18

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro Table 7. Prompt development results for rationale identification (CI) (a) Rationale component generation (b) Relative Improvement (RI) between prompts Component Exp Precision Recall F2 Score CI-ZS 84.8% 90.3% 89.2% Component Comparison Precision Recall F2-score Goal CI-FS 87.1% 87.1% 87.1% Goal CI-FS vs CI-ZS 3% -4% -2% CI-RFS 96.8% 96.8% 96.8% Need CI-FS vs CI-ZS -1% 11% 7% CI-ZS 55.9% 79.2% 73.1% Alternatives CI-FS vs CI-ZS 9% 14% 12% Need CI-FS 55.3% 87.5% 78.4% CI-RFS 59.5% 91.7% 82.7% Overall CI-FS vs CI-ZS 0% 4% 3% CI-ZS 35.0% 87.5% 67.3% Goal CI-RFS vs CI-FS 11% 11% 11% Alternatives CI-FS 38.1% 100.0% 75.5% Need CI-RFS vs CI-FS 8% 5% 6% CI-RFS 50.0% 87.5% 76.1% Alternatives CI-RFS vs CI-FS 31% -13% 1% CI-ZS 62.1% 85.7% 79.6% CI-FS 62.2% 88.9% 81.9% Overall Overall CI-RFS vs CI-FS 16% 5% 8% CI-RFS 72.0% 93.7% 88.3%

Table 8. Prompt development results for rationale generation, CG (IC = Information Coverage; EI = Extra information)

Prompt CG-ZS CG-FS RI (CG-FS vs CG-ZS)

IC 4.7 4.6 -2.7%

Goal EI F2 3.6 4.5 3.7 4.4 1.8% -1.6%

IC 4.5 4.7 4.8%

Need EI 3.6 4.0 10.3%

F2 4.3 4.6 6.0%

Alternatives IC EI F2 4.6 2.6 4.0 4.7 2.5 4.0 1.5% -3.9% -0.2%

Avg. F2 4.3 4.3 1.4%

RQ3 and RQ4 Findings: Combining reasoning-based few-shot exemplars with task-decomposition prompting yields the best performance for Argus in both rationale identification and generation.

5.4

Argus’s Evaluation Methodology and Results

5.4.1 Methodology. To evaluate Argus ’s two main modules (rationale identification and rationale generation), we applied the best prompting strategy identified in the previous section (reasoning-based few-shot with task decomposition). The evaluation followed the same methodology as prompt development, including execution procedures and metrics, with one key difference: we used 50 commits with corresponding ground-truth data from the evaluation dataset (see section 5.1). The goal was to assess the accuracy and generalizability of Argus’s prompts, including the quality of the generated rationale summaries. As a baseline for rationale identification, we used the second-best prompt from the development phase, since it provides a direct comparison against an alternative prompting strategy for the same task. For rationale generation, because both candidate prompts performed similarly, we report only the accuracy of Argus’s generated summaries. To the best of our knowledge, no prior work has directly addressed the task of generating fine-grained, artifact-grounded rationale summaries from multiple software artifacts. We also conducted a cross-model sensitivity analysis using two additional recent models until February 2026: GPT-5.2 and Gemini-3-Flash. The selected models were required to (1) demonstrate strong code understanding capabilities and (2) effectively connect and reason between natural language and source code. To support this evaluation, we considered SWE-Bench Verified, a benchmark widely used by leading models such as Grok, Claude, Gemini, and OpenAI. SWE-Bench Verified assesses an LLM’s ability to resolve real-world software engineering issues sourced from GitHub, which inherently require both deep code comprehension and natural-language-to-code reasoning. The top-performing Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

19

Table 9. Argus’s rationale component identification performance on the evaluation dataset

Component Goal Need Alternatives Overall

Exp. Argus CI-FS Argus CI-FS Argus CI-FS Argus CI-FS

Precision 76.6% 72.0% 39.2% 37.1% 15.4% 14.5% 51.4% 47.8%

Recall 95.5% 93.6% 91.1% 87.5% 80.0% 90.0% 93.2% 91.5%

F2 Score 91.0% 88.3% 72.0% 68.8% 43.5% 44.1% 80.2% 77.3%

F2’s RI% 3.0% 4.7% -1.4% 3.7%

models on this benchmark were Claude Opus 4.5, Gemini 3 Flash, and GPT 5.2. Since Claude and GPT 5.2 both had similar performance we only considered GPT 5.2. We did not aim to provide an exhaustive benchmark across all available models; rather, our goal was to assess whether Argus’s prompting and pipeline design remain effective beyond GPT-o4-mini. We did not compare Argus against recent commit-message generation approaches such as OMG [46] because they target a related but different problem. Commit-message generation systems produce a single free-form summary of a change, sometimes including an explanation of why the change occurred, whereas Argus identifies artifact-grounded rationale sentences and synthesizes them into explicit components (Goal, Need, Alternatives, Selected Alternative, Validation etc.). Since our evaluation focuses on component-level rationale identification and generation, commitmessage generation outputs are not directly aligned with our ground-truth labels. Adapting such systems for componentlevel comparison is an important direction for future work. 5.4.2 RQ5 : Evaluation Results of Argus’s Rationale Identification Table 9 shows that Argus’s rationale identification module outperforms the baseline (CG-FZ) in terms of precision, recall, and F2-score, with overall relative improvements of 7.5%, 1.9%, and 3.7%. This trend holds for both Goal and Need, but not for Alternatives: here, Argus improves precision at the expense of recall. Still, in terms of F2-score, Argus performs on par with the baseline for Alternatives. We do not consider the recall drop particularly problematic, since it is explained by Argus missing only one Alternatives sentence out of ten. More importantly, compared to the baseline, Argus reduces false positives for Alternatives from 53 to 44. Taken together, these results indicate that Argus is more accurate than the baseline at identifying rationale sentences in artifacts. Comparing these evaluation results with the prompt development outcomes in Table 7a, we observe a consistent trend: Argus achieves somewhat lower recall but substantially higher precision across components, particularly for Alternatives. The baseline shows similar behavior for Need and Alternatives. Two main factors explain these findings. First, Argus misclassifies non-Alternatives sentences as Alternatives because they contain phrasing suggestive of alternatives. Second, the few-shot exemplars used in prompting may not sufficiently cover the range of ways alternatives are expressed in artifacts. We also investigated whether the inclusion of irrelevant artifacts increased the likelihood of misclassifications, particularly for Alternatives. However, we found that only 12 of the 311 sentences (3.9%) identified as rationale by Argus originated from irrelevant artifacts. This indicates that Argus is largely robust to noise introduced by the Artifact Retriever. 5.4.3 RQ6 : Evaluation Results of Argus’s Rationale Generation Table 10a shows that Argus’s generated summaries achieve medium levels of information coverage (IC) and extra information (EI) relative to ground-truth summaries, with overall IC and EI scores of 3.3 and 2.8. Compared with the prompt development results (Table 8), Goal summaries Manuscript submitted to ACM

20

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro

Table 10. Argus’s Generated Rationale Quality (IC = Information Coverage; EI=Extra Information; DC= Descarded Component) (a) Without missing components (b) With missing components

Component Goal Need Alternatives Overall

IC 4.5 3.4 2.1 3.3

EI 3.7 3.2 1.6 2.8

F2 Score 4.3 3.4 2.0 3.23

Component Goal Need Alternatives Overall

DC 2 (50) 9 (33) 4 (14) 15 (97)

IC 4.6 4.0 4.2 4.3

EI 3.8 4.3 2.8 3.6

F2 Score 4.4 4.1 3.8 4.1

are equally accurate, but Need and Alternatives summaries are noticeably less accurate. This drop is largely explained by cases where Argus generates Need or Alternatives summaries that should not exist at all (e.g., the ground truth contains no such summaries, yet the tool generates them). Although this may seem problematic, such cases are limited to a small subset of commits: 3 of 50 (6%) for Goal, 9 of 33 (27.3%) for Need, and 4 of 14 (21.4%) for Alternatives. If we exclude these commits and focus only on those where the model is expected to generate summaries, the picture changes substantially. As shown in Table 10b, IC and EI scores for Goal match the development results, while those for Need and Alternatives are only slightly lower than their development counterparts. This indicates high generalizability of Argus’s component generation prompt. Why does Argus generate summaries when it should not? As discussed earlier, this is not due to irrelevant sentences, since only 3.9% of sentences identified as rationale are actually irrelevant. Instead, the issue stems from the relatively high false positive rates of Argus ’s component extractor for Need and Alternatives, reflected in the low precision reported in Table 9. Addressing this will require strategies to reduce false positives, such as providing more diverse few-shot exemplars that better guide the LLM. Overall, these findings suggest that in practice developers should validate generated summaries as needed. One way to support this is by including references to the artifacts from which each summary was derived, enabling developers to quickly trace back to the original context. RQ5 and RQ6 Findings: Argus identifies Goal sentences in artifacts with high accuracy (76.6% precision and 95.5% recall), while its performance is lower for Need and Alternatives sentences. Nevertheless, Argus outperforms the baseline second-best prompting strategy by 8% in precision, 2% in recall, and overall 4% in f2-score, and remains robust to irrelevant artifact content. For rationale generation, Argus produces accurate Goal summaries that capture nearly all ground-truth information, whereas Need and Alternatives summaries are less accurate, primarily due to false positives from the rationale extractor. 5.4.4 Cross-Model Sensitivity Analysis Our goal here was to assess whether Argus’s behavior depends strongly on the underlying LLM, we also evaluated the best-performing prompts with two additional recent models, GPT-5.2 and Gemini-3-Flash, alongside GPT-o4-mini. We selected GPT-o4-mini as the primary model because it offers explicit reasoning capabilities at relatively low cost, which enabled extensive prompt development, repeated executions, and user evaluation within a realistic budget. The additional models were used to test whether our findings generalize beyond this initial choice. For rationale identification, we re-ran the best-performing identification prompt (CI-RFS with voting) on the evaluation set using all three models. Table 11 shows that the same qualitative pattern holds across models. In all three cases, Goal is the easiest component to identify, followed by Need, while Alternatives remains the most difficult. Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

21

All models achieve high recall for Goal and Need, indicating that rationale-bearing sentences for these components are broadly detectable across LLMs. However, GPT-o4-mini is consistently more precise than the other two models, especially for Goal and Need, which leads to higher F2-scores overall. For Alternatives, performance is substantially lower for all three models, suggesting that this component is intrinsically harder to distinguish from other rationale-bearing content. Overall, these results indicate that the rationale identification task is not tied to a single model, while also justifying our use of GPT-o4-mini as the primary model in the rest of the study. Table 11. Cross-model sensitivity analysis for rationale identification Component

Model

Precision

Recall

F2-score

Goal

GPT-o4-mini GPT-5.2 Gemini-3-Flash

76.64% 59.52% 54.40%

95.45% 90.91% 95.45%

90.99% 82.24% 82.94%

Need

GPT-o4-mini GPT-5.2 Gemini-3-Flash

39.23% 26.32% 24.88%

91.07% 89.29% 96.43%

72.03% 60.39% 61.22%

Alternatives

GPT-o4-mini GPT-5.2 Gemini-3-Flash

15.38% 17.65% 13.46%

80.00% 60.00% 70.00%

43.48% 40.54% 38.04%

For rationale generation, we applied the same best-performing generation prompt to the three models and manually evaluated the generated summaries using the same two dimensions described earlier: information coverage (IC) and extra information (EI). Table 12 reports the component-wise average ratings. The three models show broadly similar behavior across components. For Goal, all three models produce strong summaries with very similar IC scores. For Need, the models again behave similarly, with GPT-5.2 showing slightly higher average IC and EI scores than the other two models. For Alternatives, all three models obtain identical average scores, indicating that generation of this component is similarly challenging across models. Taken together, these results suggest that rationale generation quality is relatively stable across the tested models, with no major shifts in behavior comparable to the precision differences observed for rationale identification. Table 12. Cross-model sensitivity analysis for rationale generation Component

Model

N

Avg. IC

Avg. EI

Goal

GPT-o4-mini GPT-5.2 Gemini-3-Flash

50 50 50

4.52 4.62 4.58

2.28 2.46 2.10

Need

GPT-o4-mini GPT-5.2 Gemini-3-Flash

33 33 33

3.45 3.64 3.52

2.82 3.03 2.97

Alternatives

GPT-o4-mini GPT-5.2 Gemini-3-Flash

14 14 14

2.14 2.14 2.14

4.50 4.50 4.50

To ensure that these manual generation ratings were reliable, we also measured inter-rater agreement. Table 13 shows strong agreement overall and across models. Across all cross-model generation evaluations, exact agreement was 76.9%, with Cohen’s 𝜅 = 0.688 and quadratic weighted 𝜅 = 0.931, indicating substantial to near-perfect agreement depending Manuscript submitted to ACM

22

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro

on the weighting scheme. Agreement was strongest for GPT-o4-mini and somewhat lower, but still substantial, for GPT-5.2 and Gemini-3-Flash. This provides confidence that the observed generation trends are not artifacts of unstable human judgments. Table 13. Inter-rater agreement for manual evaluation of rationale generation across models. Model

Exact agreement

Cohen’s 𝜅

Weighted Cohen’s 𝜅

GPT-o4-mini GPT-5.2 Gemini-3-Flash

90.2% 70.6% 69.8%

0.869 0.598 0.591

0.980 0.909 0.906

Overall

76.9%

0.688

0.931

Overall, the results are only partially consistent across models. For rationale identification, all three models exhibit the same qualitative trend, Goal is easiest to identify, followed by Need, while Alternatives remains the most difficult, but GPT-o4-mini is clearly more precise and therefore more accurate overall. For rationale generation, by contrast, the three models produce broadly similar manual quality ratings across all components, suggesting that summary generation is more stable across models than sentence-level rationale identification. Taken together, these results suggest that Argus’s overall feasibility does not depend on a single LLM, although the choice of model meaningfully affects extraction accuracy, with GPT-o4-mini providing the best balance among the tested models. 6

Evaluating Argus’s Usefulness

We conducted a user study to understand whether Java programmers perceive Argus’s generated rationale summaries as useful and effort-saving for understanding change rationale and supporting other tasks. We also sought feedback on which features Argus should provide and how it could be integrated into developers’ workflows. We formulated the following research questions (RQs): RQ7: How useful do Java programmers find Argus’s generated rationale summaries? RQ8: What features and integration opportunities do Java programmers want for Argus? 6.1

Methodology

We recruited 12 Java programmers from our professional network and asked them to evaluate code changes along with the corresponding Argus-generated rationale summaries. Each participant examined three commits from a single project and were unfamiliar with the project’s code. Before the evaluation tasks, participants completed a short demographic and familiarity questionnaire. We intentionally recruited participants unfamiliar with the target projects because Argus is intended to support developers who need to understand past changes quickly (e.g., newcomers, external contributors, or engineers investigating unfamiliar code). Evaluating the tool with such users therefore reflects a common real-world usage scenario. For each assigned commit, participants were shown the commit message and a compact code diff alongside the Argus-generated rationale summaries (Goal, Need, and/or Alternatives). To ensure participants could still form informed judgments and to mitigate novelty-related bias, we collected free-text explanations for each rating, allowing us to distinguish perceived informational value from reactions driven primarily by novelty. Participants evaluated different commits to avoid repeated-exposure effects. Participants then rated the perceived usefulness and effort-saving value of the summaries for understanding the change rationale using 5-point Likert scales and explained their ratings through open-ended responses. Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

23

Table 14. Ratings for usefulness and effort-saving value of the generated rationale (N = 35 responses; 12 participants evaluating 2-3 commit each). Very low

Low

Neutral

High

Very high

Total

Usefulness

1 (2.9%)

3 (8.6%)

2 (5.7%)

10 (28.6%)

19 (54.3%)

35 (100.0%)

Rating 4.1

Effort Saving

2 (5.7%)

3 (8.6%)

0 (0.0%)

8 (22.9%)

22 (62.9%)

35 (100.0%)

4.2

Effort saving was defined as the participant’s perceived reduction in effort required to understand why the change was made when using the generated summary, compared to relying on the commit message and code diff alone. Prior to rating, participants were instructed to judge whether the summary would reduce the time and cognitive effort. For i.e., P11 mentioned ‘Without the answer, if I really had to, I would need to spend more time to understand why chunking is needed.’ After completing the evaluations, participants were asked which development tasks rationale summaries could support (multiple choice), which features they would like to see in Argus (open-ended), and how they envisioned integrating Argus into their workflow (open-ended). We randomly sampled 17 commits (three per project, except for one project with six) from the evaluation dataset, ensuring diversity in commit diff size, artifact coverage, generated summary components, and informativeness of commit messages. Detailed statistics for these dimensions are available in our replication package [7]. Each commit was reviewed by one to three participants. 6.2

Results

Demographics. Participants were mostly early- to mid-career programmers: seven reported 1–4 years of general programming experience, four had 5–9 years, and one had 10–19 years. Java experience followed a similar pattern, with three having under a year, seven with 1–4 years, and two with 5–9 years. In terms of industry experience, four had none, two had less than a year, and six had 1–4 years. Participants came from diverse domains, most notably education and research (5) and healthcare (4), with the rest spread across productivity software and other specialized areas. Seven were affiliated with academic or research institutions, while five worked in industry and two in open source. Rationale usefulness. As shown in Table 14, we collected 35 ratings for usefulness and 35 for effort-saving. Most responses (19 of 35) fell into strong agreement (19) and agreement (10) with the statement “The rationale summaries helped me understand the code change’s rationale.” Effort-saving ratings followed a similar trend. On average, participants rated usefulness at 4.1 and effort-saving at 4.2, indicating that Argus ’s generated summaries were perceived as valuable. Two main reasons for low ratings stand out: summary incompleteness due to lack or insufficient rationale information in artifacts, and lack of project knowledge to fully grasp the change’s rationale. The tasks participants most often identified (8-9 responses) as benefiting from Argus’s generated rationale were Understanding code design/implementation, Code reviews, Documenting code design/implemention, and Teaching programers about the code. Several participants reported that the summaries enhanced their understanding of implementation details. For example, one participant noted: “The commit message alone only told me that this change was trying to ’restore’ something [...] I didn’t know why this needed to be restored [...] The generated rationale [...] gave me insight into the need for this change" (P10). Participants also mentioned the generated rationale can be useful for code debugging (6 responses). Manuscript submitted to ACM

24

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro Argus features and integration. The most requested feature for Argus was the ability to provide references to

the sources of summary content. For example, P8 mentioned “Should be traceable (linked to the artifact)”. This feature is straightforward to implement, as Argus’s modules are already designed with traceability in mind. Participants’ main preferences for integration included using Argus as a lightweight, interactive IDE plugin (4 participants), within code review and pull request platforms (4), in version control systems to document commits (4), and as a browser extension for web-based programming (2). RQ7 and RQ8 Findings: Surveyed Java programmers found Argus’s rationale summaries useful and effortsaving for understanding change rationale. They highlighted that the summaries could support code reviews, documentation, teaching, and debugging. Participants would like to see Argus integrated into various development tools, including IDEs, code review platforms, and version control systems

7

Threats to validity

Construct validity. We adhered to rationale components’ taxonomy from prior studies throughout our annotation process. To address subjectivity in annotation and ground-truth creation, we applied a rigorous multi-coder methodology involving iterative coding sessions, discussion, and consensus to resolve disagreements. Across all annotation tasks, we observed high agreement according to established reliability indices, providing evidence for the quality of the process and resulting data. For rationale identification, we evaluated prompts using standard text classification metrics. For rationale generation, we adopted a multi-coder annotation approach to ensure that generated summaries were consistently and fairly compared against ground truth. Our study questions were formulated with neutral language, participants were trained with examples, and open-ended responses were manually annotated using card sorting. We also asked participants to explain their ratings to verify their judgments. Internal validity. The selection of commits and associated artifacts poses a threat to internal validity. To mitigate this, we ensured that the sampled commits used for analysis were sufficiently diverse to reflect real-world scenarios. The heuristic-based approach used to retrieve associated artifacts could influence the results. We employed data-driven testing to cover a variety of patterns in the data. Prompt development followed an iterative, data-driven methodology: failed cases were systematically analyzed, and prompts were refined based on evaluation metrics. To control for LLM hallucinations, we executed the developed prompts multiple times and measured prediction consistency. Regarding the user study we recognize that novelty bias (participants reacting positively to a new tool) and learning bias could influence subjective ratings. Two aspects of our design reduce these risks. First, participants were unfamiliar with the projects and reviewed the commit message and diff together with the summary, so ratings reflect the added value relative to the commit itself rather than pre-existing knowledge. Second, participants reviewed different commits and provided free-form explanations for each rating. These qualitative responses consistently pointed to concrete factual improvements rather than mere surprise at seeing an automatically generated text. External validity. Our results may not fully generalize beyond the selected Java projects, commits, artifacts, LLM, and participants. To strengthen generalizability, we included projects from different domains, commits of varying sizes and complexities, diverse types of artifacts, and programmers with a wide range of backgrounds. 8

Related Work

Rationale has been examined across many software engineering activities, including requirements elicitation, software architecture, design, and decision management [17, 31, 34, 69]. Various rationale models [13, 36] have been proposed Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

25

i.e. Tang et al. [66] proposed a taxonomy for architectural design rationale (e.g., Constraints, Assumptions, Tradeoffs, Costs, Benefits) and Burge et al. [16] designed a design rationale taxonomy, an extension of Decision Representation Language (DRL) proposed by Lee et al. [43], which includes Decision Problems, Alternatives, Arguments that contains Requirement, Assumption, Claim etc. However these rationale models focuses on high-level software design rationale, lacking the support for the rationale model for low-level code changes (code commit). Our research focus on rationale type that manifests in code changes that implement higher-level decisions [47]. Empirical work shows that software developers both value and depend on code change rationale to understand, maintain, and evolve systems [11, 62]. Dhaouadi et al. [26] created a Decision-Rationale graph from commit message to represent rationales behind the decisions made in code changes. But their model do not support all aspects of a rationale as Safwan et al. [62] showed that code change rationales can be decomposed to 15 components (Goal, Need, Benefit, Side Effect, and Alternatives etc.). Our work builds on these insights by targeting fine-grained components that developers look for and need in practice [11]. Rationale extraction and management. Proposed approaches to rationale management seek to capture design reasoning when stakeholders make decisions during software development. Frameworks and tools have been proposed to that end [25, 38, 48, 61]. One challenge of these approaches is the overheads of documentation the rationale as it requires organization culture and specific practices that align with this goal [66]. To alleviate overhead, several approaches have bee proposed to identify rationale information in software artifacts. Early automated extraction methods applied traditional ML (e.g., SVM [57], XGBoost [25]) to related documents around code changesthis line of work neither distinguishes component-level rationale nor integrates rationale information across heterogeneous artifacts. More recently, LLM-enabled pipelines broaden the scope of rationale extraction and downstream reasoning. For example, Zhao et al. [72] uses Llama2 and BERT language models to extract rationale information from issue report to enable program repair. Compared to these approaches, Argus addresses cross-document, fine-grained component labeling and synthesis, and systematically studies prompting strategies tailored to rationale identification and generation. Commit message generation and rationale. Research on commit-message generation [20, 49, 51, 70] typically optimizes concise descriptions of what changed However fewer works measure whether generated messages explain why. Casillo et al. [18] show commit messages often fail to capture rationale fully and are often insufficient and that external artifacts are needed. Argus differs by leveraging multiple sources (e.g., issues and pull request discussions and code review comments) and by (i) identifying rationale at the component level and (ii) synthesizing concise, component-oriented summaries across documents. Commit-message generation (CMG) and Argus address related but distinct problems. CMG primarily aims to produce a concise, human-readable summary of a code change (what changed) and in some variants [46, 70] a inferred explanation of why the change occurred, usually derived from the code diff and other limited context (i.e., method/class summary, pr/issue title etc.). By contrast, Argus focuses on extracting and synthesizing explicit rationale expressing sentences that are documented across multiple development artifacts (issues, pull requests, code reviews, Javadocs, etc.) and composing them into fine-grained rationale components (Goal, Need, Alternatives). Crucially, CMG “why” outputs are often generated end-to-end by an LLM from the diff and may therefore reflect the model’s plausibility judgement rather than concrete, artifact-grounded evidence. Argus instead constrains generation to sentences identified in artifacts, producing rationale that can be traced back to the original documents. This distinction is important because our empirical analysis shows that while commit messages are strong sources for Goals, other rationale components e.g., Need and Alternatives are more often present in issues and pull requests discussion (see Table 3 and Table 4). As such, CMG’s inferred “why” and Argus’s artifact-grounded rationale are not directly comparable tasks. Manuscript submitted to ACM

26

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro Besides commit messages and rationale components represent fundamentally different forms of documentation.

Commit messages are typically concise summaries that describe the change at a high level, often combining the description of what was changed with a brief explanation of why. In contrast, rationale components explicitly separate the reasoning behind a change into structured elements such as Goal, Need, and Alternatives, each capturing a distinct aspect of the decision-making process. These components are usually distributed across multiple development artifacts, including issues, pull requests, and code reviews, and therefore rarely appear together in a single commit message. As a result, a commit message generally captures only a subset of the information needed to reconstruct the full rationale of a change. Argus addresses this limitation by extracting and synthesizing rationale evidence across artifacts and organizing it into structured components, rather than relying on a single free-form summary. Cross-artifact information capture and composition. Prior studies show that design and implementation reasoning is frequently discussed in meetings and collaboration tools but only inconsistently captured for later reference [65]. Tool support for composing heterogeneous information can help developers answer complex code-related questions more efficiently [33], yet most existing solutions rely on manual selection and assembly of fragments. Our work contributes automated, LLM-based methods that locate rationale fragments across artifacts and compose them into coherent, developer-oriented summaries. 9

Conclusions

We analyzed 63 commits from five Java projects to examine how software artifacts capture code change rationale components. We found evidence of seven components, with Goal, Need, and Alternatives being the most common. Commit Messages and Pull Requests mainly document Goal, while Need is most often captured in Pull Requests and Issues. Since most rationale beyond Goal is spread across multiple artifacts, this highlights the need for automated support to identify and synthesize rationale information. To address this, we developed Argus, an LLM-powered approach that identifies rationale sentences across artifacts and generates summaries of a commit’s Goal, Need, and Alternatives. Evaluation on 50 commits shows that Argus accurately identifies Goal sentences and outperforms a prompting baseline, though performance is lower for Need and Alternatives. Generated Goal summaries closely match the ground truth, while summaries for Need and Alternatives are less reliable. A user study with 12 Java programmers confirmed that Argus ’s summaries are perceived as useful and effort-saving for understanding change rationale. Participants indicated Argus’s generation rationale can assist code reviews, documentation, teaching, and debugging. Data availability. We provide replication package containing data, source code, and infrastructure to facilitate the reproduction and validation of our results [7]. References [1] [n. d.]. JUnit4 commit cec4a6baf600b8dee3d1318c242a67b56874288a. https://github.com/junit-team/junit4/commit/ cec4a6baf600b8dee3d1318c242a67b56874288a. Accessed: 2026-03-27. [2] [n. d.]. OkHttp commit 4c86085429edbeef0a383941936ee7b64cc3805e. https://github.com/square/okhttp/commit/ 4c86085429edbeef0a383941936ee7b64cc3805e. Accessed: 2026-03-27. [3] 2025. Apache-Dubbo. https://dubbo.apache.org/. [4] 2025. Comment Parser. https://pypi.org/project/comment-parser/. [5] 2025. JUnit4. https://junit.org/junit4/. [6] 2025. OkHttp. https://square.github.io/okhttp/. [7] 2025. Online Replication Package. https://anonymous.4open.science/r/Fine-grained-Multi-Document-Extraction-and-Generation-of-Code-ChangeRationale-8BC6/README.md. [8] 2025. Retrofit. https://square.github.io/retrofit/. [9] 2025. Spacy Model: en_core_web_trf. https://spacy.io/models/en. Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

27

[10] 2025. Spring-Boot. https://spring.io/projects/spring-boot. [11] Khadijah Al Safwan, Mohammed Elarnaoty, and Francisco Servant. 2022. Developers’ need for the rationale of code commits: An in-breadth and in-depth study. Journal of Systems and Software 189 (2022), 111320. [12] Rana Alkadhi, Manuel Nonnenmacher, Emitza Guzman, and Bernd Bruegge. 2018. How do developers discuss rationale?. In Proceedings of the IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER’18). 357–369. [13] A.I. Anton. 1996. Goal-based requirements analysis. In Proceedings of the Second International Conference on Requirements Engineering. 136–144. https://doi.org/10.1109/ICRE.1996.491438 [14] Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 712–721. [15] Tingting Bi, Wei Ding, Peng Liang, and Antony Tang. 2021. Architecture information communication in two OSS projects: The why, who, when, and what. Journal of Systems and Software 181 (2021), 111035. [16] Janet E Burge and David C Brown. 2008. Software engineering using rationale. Journal of Systems and Software 81, 3 (2008), 395–413. [17] Janet E. Burge, John M. Carroll, Raymond McCall, and Ivan Mistrik. 2008. Rationale-Based Software Engineering. Springer Berlin Heidelberg. [18] Francesco Casillo, Antonio Mastropaolo, Gabriele Bavota, Vincenzo Deufemia, and Carmine Gravino. 2025. Towards Generating the Rationale for Code Changes. In 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC). IEEE Computer Society, 327–338. [19] Oscar Chaparro, Jing Lu, Fiorella Zampetti, Laura Moreno, Massimiliano Di Penta, Andrian Marcus, Gabriele Bavota, and Vincent Ng. 2017. Detecting missing information in bug descriptions. In Proceedings of the 2017 11th joint meeting on foundations of software engineering. 396–407. [20] Xiangping Chen, Yangzi Li, Zhicao Tang, Yuan Huang, Haojie Zhou, Mingdong Tang, and Zibin Zheng. 2024. ESGen: Commit Message Generation Based on Edit Sequence of Code Change. In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension. 112–124. [21] Mihai Codoban, Sruti Srinivasa Ragavan, Danny Dig, and Brian Bailey. 2015. Software history under the lens: A study on why and how developers examine it. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’15). 1–10. [22] Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46. [23] Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213. [24] Luis Fernando Cortés-Coy, Mario Linares-Vásquez, Jairo Aponte, and Denys Poshyvanyk. 2014. On automatically generating commit messages via summarization of source code changes. In 2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation. IEEE, 275–284. [25] Mouna Dhaouadi, Bentley Oakes, and Michalis Famelis. 2025. Automated Extraction and Analysis of Developer’s Rationale in Open Source Software. Proceedings of the ACM on Software Engineering 2, FSE (2025), 2548–2570. [26] Mouna Dhaouadi, Bentley James Oakes, and Michalis Famelis. 2022. End-to-end rationale reconstruction. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–5. [27] Mouna Dhaouadi, Bentley James Oakes, and Michalis Famelis. 2024. Rationale dataset and analysis for the commit messages of the Linux kernel out-of-memory killer. In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension. 415–425. [28] Mouna Dhaouadi, Bentley James Oakes, and Michalis Famelis. 2025. CoMRAT: Commit Message Rationale Analysis Tool. 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR) (2025), 831–835. https://api.semanticscholar.org/CorpusID:278777370 [29] Andrea Di Sorbo, Sebastiano Panichella, Corrado A Visaggio, Massimiliano Di Penta, Gerardo Canfora, and Harald C Gall. 2019. Exploiting natural language structures in software informal documentation. IEEE Transactions on Software Engineering 47, 8 (2019), 1587–1604. [30] Allen H Dutoit and Barbara Paech. 2001. Rationale management in software engineering. In Handbook of Software Engineering and Knowledge Engineering: Volume I: Fundamentals. World Scientific, 787–815. [31] Allen H Dutoit and Barbara Paech. 2001. Rationale management in software engineering. In Handbook of Software Engineering and Knowledge Engineering: Volume I: Fundamentals. World Scientific, 787–815. [32] Felipe Ebert, Fernando Castor, Nicole Novielli, and Alexander Serebrenik. 2019. Confusion in Code Reviews: Reasons, Impacts, and Coping Strategies. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering. 49–60. [33] Thomas Fritz and Gail C. Murphy. 2010. Using information fragments to answer the questions developers ask. In Proceedings of the ACM/IEEE 32nd International Conference on Software Engineering, Vol. 1. 175–184. [34] Fabian Gilson and Vincent Englebert. 2011. Rationale, decisions and alternatives traceability for architecture design. In Proceedings of the 5th European Conference on Software Architecture: Companion Volume. 1–9. [35] Daqing Hou, Chandan Raj Rupakheti, and H. James Hoover. 2008. Documenting and Evaluating Scattered Concerns for Framework Usability: A Case Study. In 2008 15th Asia-Pacific Software Engineering Conference. 213–220. https://doi.org/10.1109/APSEC.2008.39 [36] H. Kaiya, H. Horai, and M. Saeki. 2002. AGORA: attributed goal-oriented requirements analysis method. In Proceedings IEEE Joint International Conference on Requirements Engineering. 13–22. https://doi.org/10.1109/ICRE.2002.1048501 [37] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406 (2022). [38] Anja Kleebaum, Barbara Paech, Jan Ole Johanssen, and Bernd Bruegge. 2021. Continuous Rationale Visualization. In 2021 Working Conference on Software Visualization (VISSOFT). 33–43. https://doi.org/10.1109/VISSOFT52517.2021.00013 [39] Andrew J Ko, Robert DeLine, and Gina Venolia. 2007. Information needs in collocated software development teams. In 29th International Conference on Software Engineering (ICSE’07). 344–353. Manuscript submitted to ACM

28

Mehedi Sun, Antu Saha, Nadeeshan De Silva, Antonio Mastropaolo, and Oscar Chaparro

[40] Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications. [41] Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems 37 (2024), 106519–106554. [42] Thomas D. LaToza and Brad A. Myers. 2010. Hard-to-answer questions about code. In Evaluation and Usability of Programming Languages and Tools. Association for Computing Machinery, 1–6. http://doi.org/10.1145/1937117.1937125 [43] Jintae Lee. 1991. Extending the Potts and Bruns model for recording design rationale. In Proceedings-13th International Conference on Software Engineering. IEEE Computer Society, 114–115. [44] Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 15339–15353. [45] Jiawei Li and Iftekhar Ahmed. 2023. Commit message matters: Investigating impact and evolution of commit message quality. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 806–817. [46] Jiawei Li, David Faragó, Christian Petrov, and Iftekhar Ahmed. 2024. Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model. Proceedings of the ACM on Software Engineering 1, FSE (2024), 745–766. [47] Jenny T Liang, Maryam Arab, Minhyuk Ko, Amy J Ko, and Thomas D LaToza. 2023. A qualitative study on the implementation design decisions of developers. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 435–447. [48] Yan Liang, Ying Liu, Chun Kit Kwong, and Wing Bun Lee. 2012. Learning the "Whys": Discovering design rationale using text mining - An algorithm perspective. Comput. Aided Des. 44, 10 (Oct. 2012), 916–930. https://doi.org/10.1016/j.cad.2011.08.002 [49] Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. 2023. Cct5: A code-change-oriented pre-trained model. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1509–1521. [50] Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. 2025. Longreason: A synthetic long-context reasoning benchmark via context expansion. arXiv preprint arXiv:2501.15089 (2025). [51] Abhinav Reddy Mandli, Saurabhsingh Rajput, and Tushar Sharma. 2025. COMET: Generating commit messages using delta graph context representation. Journal of Systems and Software 222 (2025), 112307. [52] OpenAI. 2025. OpenAI o4-mini: Reasoning Language Model. https://en.wikipedia.org/wiki/OpenAI_o4-mini Accessed: 2025-09-11. [53] OpenAI. 2025. Text Embedding 3 Large. https://platform.openai.com/docs/models/text-embedding-3-large Accessed: 2025-09-11. [54] Luca Pascarella, Magiel Bruntink, and Alberto Bacchelli. 2019. Classifying code comments in Java software systems. Empirical Software Engineering 24, 3 (2019), 1499–1537. [55] Luca Pascarella, Davide Spadini, Fabio Palomba, Magiel Bruntink, and Alberto Bacchelli. 2018. Information needs in contemporary code review. Proceedings of the ACM on human-computer interaction 2, CSCW (2018), 1–27. [56] Pooja Rani, Sebastiano Panichella, Manuel Leuenberger, Andrea Di Sorbo, and Oscar Nierstrasz. 2021. How to identify class comment types? A multi-language approach for class comment classification. Journal of systems and software 181 (2021), 111047. [57] Sarah Rastkar and Gail C. Murphy. 2013. Why did this code change?. In Proceedings of the 35th International Conference on Software Engineering (ICSE’13). 1193–1196. [58] Michael Rath, Jacob Rendall, Jin L. C. Guo, Jane Cleland-Huang, and Patrick Mäder. 2018. Traceability in the wild: automatically augmenting incomplete trace links. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). Association for Computing Machinery, New York, NY, USA, 834–845. https://doi.org/10.1145/3180155.3180207 [59] C. J. Van Rijsbergen. 1979. Information Retrieval (2nd ed.). Butterworth-Heinemann, USA. [60] Tobias Roehm, Rebecca Tiarks, Rainer Koschke, and Walid Maalej. 2012. How do professional developers comprehend software?. In 2012 34th International Conference on Software Engineering (ICSE). IEEE, 255–265. [61] Benjamin Rogers, James Gung, Yechen Qiao, and Janet E. Burge. 2012. Exploring techniques for rationale extraction from existing documents. In Proceedings of the 34th International Conference on Software Engineering (Zurich, Switzerland) (ICSE ’12). IEEE Press, 1313–1316. [62] Khadijah Al Safwan and Francisco Servant. 2019. Decomposing the rationale of code commits: the software developer’s perspective. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 397–408. https://dl.acm.org/doi/10.1145/3338906.3338979 [63] Pankajeshwara Nand Sharma, Bastin Tony Roy Savarimuthu, and Nigel Stanger. 2021. Extracting Rationale for Open Source Software Development Decisions — A Study of Python Email Archives. In Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering (ICSE’21). 1008–1019. [64] Jonathan Sillito, Gail C. Murphy, and Kris De Volder. 2008. Asking and Answering Questions during a Programming Change Task. IEEE Transactions on Software Engineering 34, 4 (2008), 434–451. [65] Adriana Meza Soria, Taylor Lopez, Elizabeth Seero, Negin Mashhadi, Emily Evans, Janet Burge, and André Van der Hoek. 2024. Characterizing software maintenance meetings: Information shared, discussion outcomes, and information captured. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–13. [66] Antony Tang, Muhammad Ali Babar, Ian Gorton, and Jun Han. 2006. A survey of architecture design rationale. Journal of systems and software 79, 12 (2006), 1792–1804. [67] Yida Tao, Yingnong Dang, Tao Xie, Dongmei Zhang, and Sunghun Kim. 2012. How do software engineers understand code changes? an exploratory study in industry. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. Association for Manuscript submitted to ACM

Fine-grained Multi-Document Extraction and Generation of Code Change Rationale

29

Computing Machinery, 1–11. http://doi.org/10.1145/2393596.2393656 [68] Yingchen Tian, Yuxia Zhang, Klaas-Jan Stol, Lin Jiang, and Hui Liu. 2022. What makes a good commit message?. In Proceedings of the 44th International Conference on Software Engineering. 2389–2401. [69] A. van Lamsweerde. 2001. Goal-oriented requirements engineering: a guided tour. In Proceedings Fifth IEEE International Symposium on Requirements Engineering. 249–262. https://doi.org/10.1109/ISRE.2001.948567 [70] Yifan Wu, Yunpeng Wang, Ying Li, Wei Tao, Siyu Yu, Haowen Yang, Wei Jiang, and Jianguo Li. 2025. An Empirical Study on Commit Message Generation using LLMs via In-Context Learning. arXiv preprint arXiv:2502.18904 (2025). [71] Yuwei Zhang, Jayanth Srinivasa, Gaowen Liu, and Jingbo Shang. 2025. Attention reveals more than tokens: Training-free long-context reasoning with attention-guided retrieval. arXiv preprint arXiv:2503.09819 (2025). [72] Jiuang Zhao, Zitian Yang, Li Zhang, Xiaoli Lian, Donghao Yang, and Xin Tan. 2024. DRMiner: Extracting Latent Design Rationale from Jira Issue Logs. 2024 39th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2024), 468–480. [73] Xiyu Zhou, Ruiyin Li, Peng Liang, Beiqi Zhang, Mojtaba Shahin, Zengyang Li, and Chen Yang. 2025. Using LLMs in generating design rationale for software architecture decisions. ACM Transactions on Software Engineering and Methodology (2025). [74] Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schroter, and Cathrin Weiss. 2010. What makes a good bug report? IEEE Transactions on Software Engineering 36, 5 (2010), 618–643.

Manuscript submitted to ACM

Related documents

• Requirements After the First Edit: Mining Late Requirement Emergence and Rework in Real-World Coding-Agent Sessions#639748