Hybrid Source Code Change Impact Analysis

Hybrid Source Code Change Impact Analysis Katharina Stengg, Christian Macho, and Martin Pinzger

arXiv:2611.10958v1 [cs.SE] 7 Nov 2026

University of Klagenfurt Klagenfurt, Austria {katharina.stengg, christian.macho, martin.pinzger}@aau.at

Abstract—Whenever software developers modify a piece of code, they need to understand the impact of their changes on other code entities, for instance, to find out which other code entities need to be updated, too. Developers often use change impact analysis (CIA) in such situations. While several approaches for CIA were proposed, they still struggle with either low precision or low recall, thereby increasing manual effort for developers to verify the results. Furthermore, limitations in the used datasets and experimental setups can bias the results reported by many CIA approaches. In this paper, we first address the dataset limitations by manually labeling 1,931 changed code entities stemming from 263 commits of 22 open-source Java projects. Next, we introduce two variants of katIA, our code change impact analysis approach that combines static impact analysis with the reasoning capabilities of large language models (LLMs). The first variant uses a single prompt (katIA-SP) for refining an initial static impact set while the second variant uses three specialized agents (katIA-MA). The evaluation with our curated dataset shows that katIASP with Gemini 2.5 Flash achieved an F1-score of 22.4% when matching impacted entities by their fully qualified names (FQN) and 27.0% when matching them by their class and entity names (CE). The latter shows an improvement of 2.0 percentage points over the state-of-the-art. This score is further improved by katIAMA which, using Gemini 2.5 Flash, yields an FQN-matching F1-score of 29.1% and a CE-matching F1-score of 33.3%. While these improvements come at the cost of increased median generation time from 0.570 s to 11.214 s and median token usage from 1,235 to 6,462 tokens, the improved performance justifies the additional cost. The comparison of three LLMs using katIASP finds that Gemini 2.5 Flash is the most consistent model. These results indicate that combining static impact analysis with LLMbased refinement, particularly with the multi-agent variant, can improve code change impact analysis. Index Terms—Change Impact Analysis, Software Maintenance, Large Language Models, Multi-agent Systems, Empirical Software Engineering

I. I NTRODUCTION Software systems continuously evolve through changes to their source code [1]. Whenever software developers modify a piece of source code, they need to understand the impact of these changes on other parts of the system to ensure code functionality and quality. For example, changing a method’s sorting criterion may affect callers that rely on the previous ordering of the returned data, requiring updates to preserve the behavior. This research was funded in whole or in part by the Austrian Science Fund (FWF) 10.55776/P36698. For open access purposes, the author has applied a CC BY public copyright license to any author accepted manuscript version arising from this submission. The research was supported by the Austrian ministries BMIMI, BMWET and the State of Upper Austria in the frame of the SCCH COMET competence center INTEGRATE (FFG 892418).

According to Lehnert [2], developers face two fundamental questions when changing software: (1) which software entities need to be modified to implement the intended change (seed), and (2) which additional entities may be affected as a consequence of this modification (impacted). Change impact analysis (CIA) aims to identify such affected entities of a software system [2]. In practice, change impact analysis can support software maintenance tasks, such as bug fixing and new feature development. Kretsou et al. [3] found that developers benefit from knowledge about co-changing code parts, because identifying affected code parts often requires more effort than implementing the intended change itself. However, understanding the full impact of code changes still comes with a high workload and insufficient tool support, particularly during code review [4]. Furthermore, Yadavally and Nguyen [5] found that existing change impact analysis approaches often struggle with low precision or low recall, especially because not all code changes and their impacts are structurally connected. Existing CIA approaches rely on different sources of information, including syntactic dependency analysis of source code changes [6], co-evolution information [7], and, more recently, transformer-based code representations [8]. Motivated by the improvements achieved with large language models (LLMs) in software engineering [9]–[13], research has begun to investigate whether code change impact analysis can benefit from them, too. For example, Yadavally and Nguyen [5] presented R IPPLE, a two-phase LLM-based approach to predict the change impact from the changes in one method and the corresponding Jira bug report. While R IPPLE is a promising approach, several aspects of its methodology and evaluation can be improved. First, for determining the main change, which Yadavally and Nguyen [5] refer to as the seed edit location, they chose the structurally most central method as the seed change, which is an overly simplified heuristic. Other approaches, such as ATHENA [8], treat each changed method once as the seed change and the remaining changed methods as impacted entities. We argue that such heuristics might bias the evaluation of CIA approaches. Second, the dataset A LEXANDRIA [8] used by both R IPPLE and ATHENA does not cover added methods, changed fields and classes, and it does not account for language-specific features, such as method overloading in Java. For instance, R IPPLE matches impacted methods only by the source file name and the simple method name without parameters. To address these issues, we propose the dataset C ONSTANTINOPLE,

which extends A LEXANDRIA with field and class changes. C ONSTANTINOPLE comprises a total of 1,931 changed code entities. Each entity is represented by its fully qualified name and manually labeled as either seed, impacted, or non-related. Next, we introduce katIA, a hybrid approach that first uses static code analysis to provide an initial static impact set which katIA then refines using LLMs. For the LLM-based refinement, we explore two variants, first the refinement with a single prompt called katIA-SP and second, the refinement with three specialized agents, called katIA-MA. A preliminary version of C ONSTANTINOPLE and initial results for katIA are presented in an extended abstract [14], whereas this paper presents dataset construction, both variants of katIA, and the evaluation in detail. Compared to R IPPLE, katIA-SP and katIA-MA both receive the seed changes, the initial static impact set, and the bug report, whereas the change planner in R IPPLE receives only the bug report and the seed changes, and the reasoner receives the change intent, change plan and the candidate set within the dependence clusters (i.e., initial impact set). Furthermore, katIA does not rely on evolutionary coupling but focuses on the static code relationships of the seed changes. Finally, we do not restrict katIA-SP and katIA-MA to only refine the impact set within the clusters computed from the candidate set, but allow them to extend the initial impact set. We used C ONSTANTINOPLE to evaluate the two variants of katIA. First, we studied the performance and consistency of katIA-SP to answer RQ1: What is the performance of katIA-SP using static analysis and a single prompt to analyze the code change impact? We experimented with three LLMs and measured the performance with precision, recall, and F1-score. To compare with the state-of-the-art, we computed these metrics using CE-matching, based on class and entity names, and additionally using FQN-matching, based on fully qualified names. Furthermore, we repeated each experiment three times to measure the consistency of results. When using CE-matching, katIA-SP achieved F1scores of 27.0% with Gemini 2.5 Flash and 28.5% with Qwen3 Coder Plus. Gemini 2.5 Flash was the most consistent model, producing 99.62% consistent results. Furthermore, katIA-SP with Gemini 2.5 Flash and Qwen3 Coder Plus achieved F1score improvements of 2.0 and 3.5 percentage points over the F1-score reported for R IPPLE [5]. Second, we investigated whether decomposing the refinement task across three specialized agents improves katIASP to answer RQ2: To what extent does katIA’s multiagent variant (katIA-MA) improve change impact analysis compared to katIA-SP? For this experiment, we used only Gemini 2.5 Flash because of its high consistency and decent performance. The results show a further increase in precision, recall, and F1-score under FQN-matching, the latter reaching 29.1%. Under CEmatching, recall and F1-score increased, the latter reaching 33.3%, while precision decreased slightly. Compared to R IP PLE, the CE-matching F1-score is 8.3 percentage points higher. Overall, our results show that combining static code anal-

ysis with LLM-based refinement, especially in a multi-agent variant, can improve code change impact analysis. However, despite these improvements, change impact analysis remains a challenging problem. In summary, this paper makes the following contributions: 1) An extended, more detailed version of the A LEXANDRIA dataset, named C ONSTANTINOPLE, for evaluating code change impact analysis approaches, comprising manually labeled method, field, and class changes. 2) A detailed analysis of the manual labeling process. 3) katIA, our hybrid code change impact analysis approach that combines static code analysis with LLM-based refinement, considering a single prompt (katIA-SP) and a multi-agent variant (katIA-MA). 4) An evaluation of both variants and comparison to the state-of-the-art approach R IPPLE. The remainder of the paper is structured as follows: Section II discusses scientific work. Section III explains our dataset labeling process and presents the descriptive statistics of C ONSTANTINOPLE. Section IV presents our approach, including static code change impact set construction and LLMbased refinement. Section V reports our experimental setup and results, Section VI discusses the results and threats to validity, and Section VII concludes the paper. The supplementary material of this work can be found online1 . II. R ELATED W ORK This section discusses related work on change impact analysis and points out the research gaps addressed in this paper. A. Change Impact Analysis Several approaches to support developers in analyzing the impact of code changes were proposed. Ren et al. [6] designed and implemented C HIANTI, a change impact analysis tool for Java. C HIANTI receives two application versions and works on the atomic changes within them. The resulting change impact is the set of tests potentially affected by the changes performed. Furthermore, the authors reported the affecting changes, i.e., the changes that could alter test execution. Their empirical evaluation revealed that an average of 52% of the unit tests of Daikon were affected. Lehnert [2] proposed a taxonomy of impact analysis to help practitioners navigate and filter the variety of existing impact analysis approaches. The taxonomy comprises eight classification criteria, including scope, granularity, analysis style, and the available experimental results. Gethers et al. [15] proposed an approach for change request impact analysis that integrates history-based mining with dynamic analysis and information retrieval. They evaluated their approach on four opensource projects, yielding statistically significant improvements over the information retrieval-based impact analysis approach (IRCR). Dit et al. [16] presented I MPACT M INER, which implements the approach proposed by Gethers et al. [15]. 1 https://doi.org/10.6084/m9.figshare.32924534

Hanam et al. [17] addressed the problem of code change understanding in the context of code review by proposing semantic change impact relations and their tool S EMCIA which was evaluated on 2,000 JavaScript commits from three Node.js projects. They were able to decrease the number of false positives and the size of change impact sets. Finally, they conducted a user study where they found that their tool positively impacted both the speed and accuracy of the developers’ code review tasks. Huang et al. [18] proposed an approach to improve traditional change impact analysis by mapping historical change patterns from similar classes in other projects to boost the prediction accuracy of existing CIA tools. Zhou et al. [7] used evolutionary coupling to improve impact predictions. Evolutionary coupling refers to relationships where code entities tend to frequently change together, similarly to the co-evolution of production and test code. The authors discussed twelve types of software relationships that define the edges between code entities and can be used to model impact dependencies. Yan et al. [8] proposed ATHENA, a transformer-based ranked impact set prediction approach using program dependence graphs. They evaluated ATHENA using A LEXANDRIA, a dataset based on the set of untangled commits from Herbold et al. [19]. As mentioned before, the dataset contains no clear indication of the seed change. In their evaluation, each changed method of a commit was treated as a query method, i.e., seed change, once, while the other methods formed the corresponding ground-truth impact set. This led to 4,405 impact analysis tasks in the largest of the three different evaluation settings, which were evaluated using ranked set metrics. Yadavally and Nguyen [5] proposed R IPPLE, an intentaware, two-phase LLM-based approach to predicting impact sets from seed method changes and Jira bug reports. For the evaluation, they created a dataset comprising the changed methods of 100 commits from the A LEXANDRIA dataset represented by their source-file and method names. For each commit, they chose the method with the most direct and indirect dependencies, including call and class-member dependencies, as the seed change for their impact analysis task. Using GPT-4o, R IPPLE outperformed ATHENA achieving an average precision of 28.2%, recall of 36.3%, and F1-score of 25.0% for matching the source file name and the simple name of the method. Finally, they report the results from an ablation study, demonstrating the importance of incorporating both the change intent and the LLM planning phase into their method. Addressing the shortcomings of simple heuristics for determining the seed change, Stengg et al. [20] proposed a dataset extension of A LEXANDRIA, for which they labeled the seedimpact relationships of changed code entities. They evaluated the capabilities of the models GPT-5 and GPT-5-mini for CIA, obtaining median F1-scores of 0.0%, which shows the limitations of approaches that rely solely on simple prompts and LLMs.

B. Research Gap As mentioned before, existing CIA approaches, such as R IPPLE [5] or ATHENA [8], rely on datasets, such as A LEXAN DRIA , that have several limitations. First, they consist of Java method changes only and ignore new methods and changes of other code entities such as fields and classes. Second, they use simplified heuristics to determine the seed change [5] or use each method as the seed change once [8]. Third, they do not fully support method overloading. We argue that all three shortcomings can notably bias the results. In our dataset C ONSTANTINOPLE presented in the next section, we address these shortcomings and lay the foundation for a less biased evaluation of change impact analysis approaches. Furthermore, we propose katIA, a hybrid approach combining static code analysis and LLMs. Similar to R IPPLE, katIA begins with a static analysis phase to produce an initial candidate set. For R IPPLE, this set restricts LLM reasoning to a fixed dependency-enhanced candidate set, while katIA treats the initial static impact set as a starting point and uses the LLM to both filter existing entities and identify additional impacted entities. Furthermore, katIA relies solely on static code relationships, eliminating the need for evolutionary coupling and project history. We evaluate two variants of katIA, katIASP, which uses a single prompt for impact set refinement, and katIA-MA, which uses three agents for impact set refinement. III. C ODE C HANGE I MPACT DATASET This section presents the motivation, construction, and descriptive statistics of our dataset C ONSTANTINOPLE. It extends A LEXANDRIA, a benchmark for impact analysis published by Yan et al. [8]. A LEXANDRIA is based on a set of manually untangled commits proposed by Herbold et al. [19]. Yan et al. state that there are other datasets using manually untangled commits. However, these datasets either contain a limited number of commits or perform untangling at the file or commit level rather than the method level, which limits the applicability for method-level impact analysis. The A LEXANDRIA dataset contains 910 commits from 25 opensource Java projects, which is substantially more than other benchmarks so far. A. Limitations of A LEXANDRIA Despite being used for the evaluation of various state-of-theart CIA approaches such as ATHENA [8] and R IPPLE [5], we found that A LEXANDRIA has four main limitations, namely (1) limited entity coverage, (2) missing method signatures, (3) missing newly added methods, and (4) undefined seed changes. Limited entity coverage. It only covers method changes but ignores changes to other program entities, such as fields and classes. This limits the practical usability of the approaches as developers not only change methods. Missing method signatures. Method changes are represented by the method names without their parameters, which implies that approaches cannot precisely resolve overloaded methods.

Missing added methods. The dataset does not include newly added methods. Consequently, the approaches using it are limited to methods that already exist in the corresponding parent commit. Undefined seed changes. Yan et al. [8] state that the A LEXANDRIA dataset provides no clear indication of the seed change. Instead, in their approach each changed method of a commit is treated as a seed change once, while the remaining methods form the corresponding impact set. Yadavally and Nguyen [5] also used A LEXANDRIA but chose a different approach for determining the seed change. They used the method with the most direct and indirect call and classmember dependencies as the seed change. Both approaches solely rely on heuristics to identify the seed changes. To address these limitations, we extended the A LEXANDRIA dataset in two ways. First, in addition to method changes, we extracted changes to fields and classes, as well as newly added methods. For each entity, we used its fully qualified name2 , e.g. org.example.A.b(), including parameter types for methods to handle method overloading. Second, we manually labeled each changed code entity as either a seed, an impacted, or a non-related change. B. Dataset Construction Change impact analysis (CIA) aims to identify the software entities impacted by a change [2]. Accordingly, we annotated every changed production code entity either as seed, impacted, or non-related. Seed. A seed change is a code change made to fulfill the requirements derived from the corresponding Jira bug report. The set of all seeds for a bug report constitutes the bugfixing change as defined by Rodrı́guez-Pérez et al. [21]. Unlike the heuristics used in R IPPLE or ATHENA, a single commit may contain multiple seeds when several changed entities are involved in addressing the reported bug. Impacted. Impacted changes are triggered by the seed changes, not by the requirements of the reported bug. They are consequences, i.e., they do not directly address the bug report but become necessary because of the seed changes. For example, if a seed change modifies a method’s sorting criterion, its callers that rely on the previous ordering of the returned data must be modified to preserve the behavior. Non-related. Code changes that are neither seed nor impacted are non-related changes. These changes occur within the same tangled commit [22] but have no clear origin in or relationship to the bug report. Although Herbold et al. [19] untangled the dataset, we still found a portion of tangled changes during our dataset construction. To classify changed code entities into these categories, we grouped the data of A LEXANDRIA by repository, commit hash, and parent commit hash. This resulted in 918 commits. In contrast, the 910 commits reported by the original authors correspond to grouping by repository and parent commit hash only. We then extracted a list of changed files per commit and 2 https://docs.oracle.com/javase/specs/jls/se11/html/jls-6.html#jls-6.7

excluded test files based on standard directory and naming conventions. Finally, we filtered the commits to retain only those that modified between one and five Java source files for our dataset construction. We selected this range because at least one changed file is required to contain changed entities, while an upper bound of five changed files keeps manual labeling feasible. In the original dataset, only 94 of the 918 commits are not within this range. Similar to Yadavally and Nguyen [5], we extended our dataset with the issue ID from the corresponding commit message to link the commits to their Jira bug report. The majority of the bug reports (82.87%) refer to a single commit, while the maximum number of commits for a single bug report is eight. The latter occurs only once while 111 bug reports (14.62%) are referenced by exactly two commits, 13 by three, and five by four commits. We excluded commits referencing the same bug report from the dataset only if they contained identical code changes or a subset of changes of another commit in the dataset. Changes to comments, imports, and annotations were excluded. Also note that commits which did not contain at least one seed change and one impacted change were skipped, adhering to the criteria used in other works. To label the changed code entities, we implemented a helper tool based on Spoon [23] to extract the changed source code entities per commit. Furthermore, its user interface helps evaluators to inspect the changed code entities and the corresponding bug report. The labeled code entities are exported into a CSV file, where each row corresponds to a commit containing metadata and one JSON object per change label with further information, such as the fully qualified name. All evaluators used this tool for labeling. The labeling process was performed by the first author and two co-authors, all of them having several years of experience in Java and code analysis. The labeling comprised three phases. We initially sampled 420 commits to use 20 commits for the calibration phase and the remaining 400 for the actual dataset labeling phases. The calibration phase, where each evaluator labeled the same 20 commits, established a common understanding of the labeling guidelines and reduced ambiguities before the main labeling process. The shared phase produced the first 100 labeled commits of the dataset, for which the main author calculated the agreement among the evaluators to refine the guidelines and ensure consistent labeling. Finally, the individual phase was used to extend the dataset, as each evaluator labeled 100 individual commits. 1) Calibration Phase: First, each evaluator received the same set of 20 randomly sampled commits to be labeled independently. After that, they discussed their findings to establish common labeling guidelines and practices to define a comprehensive labeling strategy for further labeling. 2) Shared Phase: In the shared labeling phase, each evaluator labeled the same 100 randomly sampled commits that were not part of the calibration set. The first author collected the results from all evaluators and analyzed their pairwise overlap. Majority vote was applied, and we discussed labels with strong disagreements. The first author also improved the guidelines

with the learnings of this phase. TABLE I JACCARD SET SIMILARITIES PER EVALUATOR PAIR AND LABEL OF THE SHARED PHASE . Evaluator Pair

Seed

Impacted

Non-related

Overall

A–C A–B B–C

50% 51% 48%

51% 55% 47%

64% 74% 62%

55% 60% 52%

Average

50%

51%

67%

56%

Since each evaluator selected a label for each code entity, we measured the pairwise agreement using Jaccard set overlap similar to prior work on agreement for set-valued annotations [24]. We report the averages per evaluator pair and label in Table I. During this shared phase, the overall set overlap was 56%, with average pairwise overlaps ranging from 52% to 60%. The similar values among the author pairs suggest that differences did not occur between a specific evaluator pair. Furthermore, these values are not interpreted as intercoder agreement, but as analysis of the set overlaps within the shared labeling process. An overlap of 100% would mean that the sets reported by both authors were identical, whereas missing or additional code entities per label category result in lower values. We can observe that the agreement for nonrelated code changes is substantially higher than for seed or impacted changes. This indicates that it is easier to choose changes that neither contribute to a bug fix nor result from it than to distinguish between seed and impacted code entities. The distinction between seed and impacted code entities is not always clear from the changes alone but needs context and developer interpretation. One evaluator documented all label discussions and the final decisions for previously undecided labels. This resulted in the first 100 annotated commits of our dataset, comprising 71 labeled and 29 skipped commits, as they did not contain at least one seed change and one impacted change and therefore were excluded from our dataset. All evaluators agreed that previous discussions were valuable as preparation for the individual labeling phase and helped resolve ambiguities as the distinction between seed and impacted entities may allow different interpretations. 3) Individual Phase: In the individual labeling phase, each evaluator labeled another set of 100 commits that were in neither of the two previous labeling phases. The goal of this phase was to extend the dataset. C. C ONSTANTINOPLE Dataset Table II shows the descriptive statistics of the labeled dataset C ONSTANTINOPLE. It comprises 263 commits and 1,931 changed code entities across 22 projects. The number of changed code entities is computed as the sum of the entities labeled as seed changes (#S), impacted changes (#I), and nonrelated changes (#NR). For each commit, the dataset also contains the commit hash of the current commit and its parent,

TABLE II D ESCRIPTIVE STATISTICS OF THE LABELED DATASET C ONSTANTINOPLE . #C REFERS TO THE NUMBER OF COMMITS , #BR THE NUMBER OF BUG REPORTS IN J IRA , #S, #I, AND #NR LIST THE NUMBER OF CHANGED CODE ENTITIES LABELED AS SEED , IMPACTED , AND NON - RELATED , RESPECTIVELY.

Project

#C

#BR

#S

#I

#NR

ant-ivy commons-bcel commons-beanutils commons-codec commons-compress commons-configuration commons-dbcp commons-digester commons-io commons-jcs commons-lang commons-math commons-net commons-scxml commons-validator commons-vfs deltaspike giraph gora opennlp parquet-mr systemml

56 3 5 2 19 21 3 4 9 10 11 29 18 8 5 10 2 18 5 12 12 1

55 3 5 2 14 20 3 3 9 10 11 28 17 8 4 10 2 18 5 12 12 1

109 4 9 3 37 32 5 6 12 17 24 58 23 14 22 30 2 40 11 23 16 1

276 37 16 36 109 69 13 15 44 36 65 147 63 39 31 58 5 92 17 45 109 5

14 5 1 0 2 5 0 0 3 1 1 13 0 9 0 6 2 5 33 0 6 0

Total

263

252

498

1327

106

the number of changed production code files and code entities, the total number of added and deleted lines of code, the commit message, the Jira bug report key, title, description, and the labeled entities represented by their fully qualified names. C ONSTANTINOPLE is available in the supplementary material. During the phases of dataset construction, the evaluators labeled a total of 420 randomly sampled commits. The 20 samples used during the calibration phase were not considered for the final dataset. Among the 400 commits, they identified 37 bug reports for which multiple commits referenced the same Jira bug report key. For ten of these 37 bug reports, the corresponding commits contained no identical code changes, indicating that they addressed different aspects of the same bug report. For 20 bug reports, the commits contained identical code changes, while for seven bug reports, a later commit contained a subset of the code changes introduced by an earlier commit. To avoid duplicates, we retained only the first occurrence of each such commit and excluded the remaining occurrences from the dataset, resulting in 20 commits being removed from our final dataset. As an additional consistency check, we compared the labels of these commits. If commits contained a subset of the changes of another, we restricted this comparison to the shared code changes. We found that the code changes for 18 bug reports received identical labels, while four received partially identical labels with overlapping selected labels. The

Fig. 1. Overview of the katIA approach that, given the seed changes of a commit, first constructs an initial static code change impact set that then is refined either with a single prompt (katIA-SP) or three agents (katIA-MA).

commits of the remaining five bug reports were assigned different labels. During the labeling process, the evaluators identified 117 commits that did not contain at least one seed change and one impacted change. Similar to prior work that uses labeling heuristics to create sets containing at least one seed and one impacted entity, we excluded these 117 commits from the final version of C ONSTANTINOPLE. Together with the previously discussed 20 commits that reference the same bug reports, 137 of the 400 commits were excluded from C ONSTANTINOPLE, resulting in 263 labeled commits that we use for the evaluation of katIA. IV. KAT IA A PPROACH Figure 1 provides an overview of our proposed approach katIA. katIA consists of two phases. In the first phase, given a set of seed changes, static analysis is used to compute the initial static code change impact set. This set consists of code entities connected to the seed changes through usage relationships. For example, for a method, this set contains all its callers and callees. In the second phase, the initial static code change impact set is refined by using LLMs. The second phase exists in two variants, either using a single prompt (katIA-SP) or three agents (katIA-MA). In the following, we describe each phase of katIA in detail. A. Phase 1: Static Impact Set Construction To construct the initial static code change impact set, katIA first clones the corresponding project and checks out the respective parent commit. Next, it uses the static code analysis framework Spoon [23] to parse the project and to identify

the corresponding code entity of each given seed change. If a seed change is absent from the checked-out commit because the entity is newly added in the evaluated commit, the static analysis cannot resolve the corresponding code entity. Consequently, the analysis returns an empty impact set for that entity. Next, depending on the type of the code entity, katIA statically analyzes the dependencies to other code entities. katIA supports classes including interfaces, methods including constructors, initializer blocks, and fields. Concerning fields, katIA extracts read, write, and read-write relations, i.e., the static impact set for a field consists of a list of methods that access it. Concerning methods, katIA extracts all callers and callees. Classes can be used, extended, or implemented by other classes. katIA extracts incoming and outgoing dependencies, as it identifies the classes that use, extend, or implement the seed class, and the classes that the seed uses, extends, or implements. The initial static impact set of the seed changes in a commit consists of all dependent code entities identified by katIA. Currently, the static parser of katIA does not consider indirect dependencies between code entities, dynamic analysis during runtime, reflection-based accesses, or external libraries. Consider an example where methods a(), b(), and c() were changed, assuming that the seed change is in method a(). Method a() calls method b(), whereas c() calls a(). The corresponding initial static code change impact set computed by katIA contains org.example.A.b() and org.example.A.c(). B. Phase 2: Impact Set Refinement LLMs have been widely used to address software engineering challenges [25]. Regarding code change impact analysis, a recent approach of Yadavally and Nguyen [5] called R IPPLE showed that a combination of dependency analysis in the first phase and LLMs in the second phase can improve the performance of code change impact analysis. Motivated by R IPPLE, katIA also uses LLMs in the second phase to refine the initial static code change impact set computed by our static analysis. We provide two variants of the second phase: first, using a single prompt (katIA-SP), and second, using three specialized agents (katIA-MA). 1) Single Prompt Refinement: We carefully constructed the prompt according to the findings and guidelines of recent software engineering research [26]–[28]. The prompt template can be found in the supplementary material. It starts with a system prompt that tells the LLM that it is an expert in Java and in analyzing the impact of source code changes on other code entities in a project’s production code. Such impersonation has been shown to enhance the performance of LLMs [29]. The prompt then presents general background information. It states that the model receives information about a set of changes, called seed changes, that were made to fix a bug reported in Jira and an initial static impact set containing the fully qualified names of code entities statically connected to the seed changes. This general information is followed by the specific change impact analysis task. This task tells the model that an impacted code entity must be created or

modified as a consequence of the seed changes. Additionally, this part states when to remove or add a code entity to the initial static impact set and the final goal of the task, which is to refine the initial set to only contain impacted code entities. Next, the prompt contains all inputs (Jira bug report, seed changes, and initial static code change impact set), followed by explicit instructions on the output format and three example outputs. 2) Multi-Agent Refinement: The goal of katIA-MA is to improve katIA-SP by decomposing the single prompt into three separate subtasks. All agent and task definitions can be found in the supplementary material. Each subtask corresponds to a separate state of the change impact analysis process and is assigned to one specific agent. Analysis Agent. This agent is tasked to interpret the seed changes and the Jira bug report to extract evidence relevant to identifying non-seed Java code entities that likely need modification or creation as a consequence of the seed changes. It produces evidence for each entity of the initial static code change impact set, including their fully qualified name, an assessment of whether the entity should be modified as a consequence of the seed changes, and the supporting evidence. Similarly, it is tasked to produce this evidence for code entities that need to be created as a consequence of the seed changes. Refinement Agent. The structured evidence of the previous agent is provided to the refinement agent together with the seed changes and the initial static code change impact set. Similar to the task description of katIA-SP, this agent is instructed to remove existing code entities from the set and add missing ones when sufficient evidence was provided by the previous agent. Reporting Agent. The reporting agent is provided with the description of the expected output format and example outputs adhering to this format. This agent is responsible for reporting the output in the expected format. It is instructed to not alter the set received from the previous agent as its only responsibility is reporting the refined impact set in the expected output format. V. E VALUATION This section first presents the experimental setup used to evaluate katIA-SP and katIA-MA, followed by the results for our two research questions. RQ1 evaluates the single prompt refinement variant katIA-SP, while RQ2 evaluates the multiagent refinement variant katIA-MA. A. Experimental Setup We measured the performance and consistency of katIASP considering three different LLMs. Second, we used the LLM that provided the best balance between performance and result consistency to evaluate katIA-MA. We also compared the resource usage using this LLM in both variants of katIA. Table III lists the three LLMs used to evaluate katIA-SP. We selected a diverse set of models from three different model families, including both general-purpose and code-focused models. These API-hosted models are optimized mainly for

either inference efficiency or coding tasks while keeping inference costs low. We accessed all models through OpenRouter3 and pinned requests to a single provider per model to improve reproducibility. Based on token throughput per second, latency, uptime, and cost, we selected OpenAI for GPT-5.4 Mini, Alibaba Cloud Int. for Qwen3 Coder Plus, and Google Vertex (EU) for Gemini 2.5 Flash. This setup makes the evaluation independent of local computing resources. TABLE III S ELECTED LLM S , THEIR CONTEXT SIZE , THEIR O PEN ROUTER IDENTIFIERS , AND PINNED PROVIDERS .

Model

Context

OpenRouter Identifier

Provider

400K 1M 1M

openai/gpt-5.4-mini qwen/qwen3-coder-plus google/gemini-2.5-flash

OpenAI Alibaba Cloud Int. Google Vertex (EU)

GPT-5.4 Mini Qwen3 Coder Plus Gemini 2.5 Flash

Despite their advantages, LLMs have the fundamental problem of non-determinism, also for code generation [30]. To assess the consistency of the results while keeping the computational cost manageable, we repeated each experiment for each model and variant three times. Throughout all experiments, the temperature parameter was set to 0 to maximize output determinism. Similar to previous approaches, such as R IPPLE [5], we used set-based evaluation metrics to measure the performance because katIA outputs unordered impact sets. For a commit, we computed precision, recall, and F1-score, based on the ground truth impact sets from our curated dataset and the refined impact set output by katIA. Specifically, for each commit and metric, we computed the median across the three repetitions for each model. We then calculated the macroaverages across all commits for both FQN-matching and CEmatching. CE-matching only considers the declaring class and the code entity’s simple name without any parameters or further information. For example, the code entity with the fully qualified name org.example.A.b() gives the class-entity name A.b. We included CE-matching to allow the comparison with R IPPLE [5], as they represented their ground truth values as sourceFileSimpleName.methodSimpleName. Note that our CE-matching favored R IPPLE, because we considered class name and entity name, therefore also inner classes, while R IPPLE only considers top-level classes. We do not report ranking-based metrics (e.g. Hit@k), as our approach predicts an unordered set of impacted entities. B. Evaluation of katIA-SP Table IV lists the results of the three models and initial static impact set as an intermediate step, evaluated with both FQN-matching and CE-matching. We can observe that the two models with 1M context size, Gemini 2.5 Flash and Qwen3 Coder Plus outperform GPT-5.4 Mini for all metrics. CEmatching consistently outperforms FQN-matching, indicating 3 https://openrouter.ai

that the LLMs produce approximate results well but struggle to accurately identify complete fully qualified names, potentially because they lack access to the source code repository. For FQN-matching, the single prompt with all three LLMs improves precision compared to the impact set obtained only with static analysis. GPT-5.4 Mini increases precision by 2.9 percentage points, while the other models more than double the static analysis precision of 15.8%. Recall decreases by 3.7 percentage points for GPT-5.4 Mini and by 0.4 percentage points for Gemini 2.5 Flash, whereas Qwen3 Coder Plus improves the recall by 5.1 percentage points. Despite these differences, the single prompt approach with the three LLMs achieves higher F1-scores than the static analysis. For CE-matching, all models substantially improve precision over the static analysis, with Gemini 2.5 Flash achieving the highest precision of 39.9%. Recall also improves for all models except GPT-5.4 Mini, which exhibits a decrease of 2.0 percentage points compared to the static analysis. Overall, the single prompt F1-scores for two of the three LLMs are slightly higher than those achieved by R IPPLE, which reached a precision of 28.2%, a recall of 36.3%, and an F1-score of 25.0%. TABLE IV M ACRO - AVERAGED PRECISION (P REC ), RECALL (R EC ), AND F1- SCORE FOR FQN- MATCHING AND CE- MATCHING OF THE STATIC ANALYSIS , KAT IA-SP ACROSS THE THREE MODELS , KAT IA-MA USING G EMINI 2.5 F LASH , AND R IPPLE†. FQN-matching Configuration Static analysis katIA-SP (GPT-5.4 Mini) katIA-SP (Gemini 2.5 Flash) katIA-SP (Qwen3 Coder Plus) katIA-MA (Gemini 2.5 Flash) Ripple (GPT-4o)

†

CE-matching

Prec

Rec

F1

Prec

Rec

F1

15.8% 18.7% 31.9% 32.2% 32.9%

23.5% 19.8% 23.1% 28.6% 33.8%

14.6% 15.8% 22.4% 25.2% 29.1%

16.9% 22.1% 39.9% 36.4% 37.8%

24.9% 22.9% 26.7% 32.3% 38.0%

15.6% 18.5% 27.0% 28.5% 33.3%

–

–

–

28.2%

36.3%

25.0%

In addition to their predictive performance, we evaluated the consistency of the results obtained by the three LLMs. For each sample, we compared the impact sets from the three repetitions, irrespective of their order, and counted the number of commits for which all three impact sets were identical. GPT-5.4 Mini was the least consistent model, producing consistent results for only 57 out of the 263 commits. Qwen3 Coder Plus produced consistent results for 110 out of 263 samples. Gemini 2.5 Flash achieved the highest consistency with consistent results for 262 commits, whereas the only inconsistent commit was when the model produced an empty impact set for two of the three repetitions, while the third repetition’s result identified one correct impacted entity and missed the other three. The consistency analysis shows that 3 †The results of R IPPLE are taken from the paper of Yadavally and Nguyen [5]. The comparison is an approximation because our CE-matching additionally considers fields and classes, whereas R IPPLE predicts only method-level entities. Furthermore, R IPPLE uses the source-file name instead of the declaring class name and reports results for 100 commits, whereas katIA is evaluated on 263 commits. A full replication of R IPPLE was not possible due to missing files in the public repository at the time of our evaluation.

LLMs differ not only in predictive performance but also in result consistency, reinforcing recommendations for empirical studies using LLMs in software engineering [31]. We answer RQ1: What is the performance of katIASP using static analysis and a single prompt to analyze the code change impact? as follows: Compared to the static analysis results, all evaluated LLMs produce an impact set with higher precision and F1-scores. The best performance is achieved by Qwen3 Coder Plus, resulting in an F1-score of 25.2% for FQN-matching and 28.5% for CE-matching. Two of our three evaluated models outperform R IPPLE in terms of precision and F1-score. Qwen3 Coder Plus returned only 41.83% consistent results, while Gemini 2.5 Flash achieved comparable results of 22.4% and 27.0% F1-score, with 99.62% consistent results. GPT-5.4 Mini performed worst with a CEmatching F1-score of 18.5% and only 21.67% consistent results. C. Evaluation of katIA-MA Despite performing best, Qwen3 Coder Plus produced highly inconsistent results. Therefore, we used only Gemini 2.5 Flash to evaluate katIA-MA. The respective performance is depicted in the penultimate row in Table IV. Similar to the experiment with katIA-SP, we performed three repetitions and used the median result per commit to compute the performance metrics. As with katIA-SP, CE-matching achieves better performance across all metrics. Furthermore, katIA-MA with Gemini 2.5 Flash produced consistent answers across all samples. For CE-matching, recall and F1-score increase from 26.7% to 38.0% and 27.0% to 33.3%, whereas precision decreases from 39.9% to 37.8% compared to katIA-SP with Gemini 2.5 Flash. katIA-MA outperforms R IPPLE for all metrics in the CE-matching evaluation. For instance, katIA-MA’s F1-score is 33.3% while R IPPLE in its best setting achieved an F1score of 25%. Compared to katIA-SP, katIA-MA improves all FQN-matching metrics, increasing precision from 31.9% to 32.9%, recall from 23.1% to 33.8%, and F1-score from 22.4% to 29.1%. Overall, we observe that katIA-MA particularly improves the recall while reaching similar or slightly better precision than the other approaches. In addition to performance metrics and result consistency, we report token usage, cost, and generation time obtained from the respective OpenRouter logs. Table V lists the median and quartiles ([Q1 , Q3 ]) across the 263 commits for katIA-SP and katIA-MA, both using Gemini 2.5 Flash and the median values among the three repetitions. In comparison to katIA-SP, katIAMA increased median token usage by 5.2×, median generation time by 19.7×, and costs by 13.0× observed using the pricing options available at the time of the experiments (July 2026). It shows that katIA-MA comes with higher token usage, and thus, higher cost and generation time due to its three model calls instead of katIA-SP’s single prompt approach. Despite that increase, both time and cost are within an acceptable range while improving both performance and result consistency.

TABLE V R ESOURCE USAGE OF KAT IA-SP AND KAT IA-MA WITH THE MODEL G EMINI 2.5 F LASH . M EDIAN AND THE FIRST AND THIRD QUARTILES ([Q1 , Q3 ]) WERE EXTRACTED ACROSS ALL 263 COMMITS USING THEIR MEDIAN OVER THE THREE RESPECTIVE REPETITIONS . A LL VALUES WERE EXTRACTED FROM THE CORRESPONDING O PEN ROUTER LOGS . Variant

katIA-SP

Generation Time

Token Usage

OpenRouter Cost

(s)

(×103 )

(US cents)

0.570 [0.444, 0.738] 1.235 [1.023, 1.774] 0.048 [0.036, 0.064]

katIA-MA 11.214 [8.298, 14.301] 6.462 [4.953, 8.836] 0.626 [0.472, 0.807]

With these results, we answer RQ2: To what extent does katIA’s multi-agent variant (katIA-MA) improve change impact analysis compared to katIA-SP? as follows: katIAMA outperforms the single prompt approach katIA-SP and achieves higher performance metrics than the approach R IP PLE. Compared to katIA-SP and using Gemini 2.5 Flash, katIA-MA increases the FQN-matching F1-score from 22.4% to 29.1% and the CE-matching F1-score from 27.0% to 33.3%. Despite the increased performance, katIA-MA needs substantially more resources than katIA-SP. Nevertheless, these values stay within a reasonable range. VI. D ISCUSSION This section discusses potential implications of the results and threats to the validity of our dataset C ONSTANTINOPLE, approach katIA, and evaluation. A. Implications Our results show that the combination of static code analysis and LLMs benefits code change impact analysis. The consistent increase in precision across the experiments indicates that the LLMs primarily act as filters of the initial static impact set. The static analysis includes entities connected through program dependencies, even when they are not relevant for the given seed changes. By additionally considering the seed changes, their specific source code changes, and the corresponding bug report, LLMs can remove candidates that are structurally related but unlikely to need modification as a consequence of the seed changes. However, such filtering introduces a trade-off between precision and recall as filtering false positives increases precision, but it may also wrongly remove impacted entities and thereby reduce the recall. The results achieved by katIA-MA indicate that performing impact set refinement across three specialized agents, each responsible for exactly one task, achieves similar precision but higher recall. To exactly map the performance and consistency improvements to the agentic setup, an ablation study needs to be performed, as our results do not report whether the improvement is due to the setup, decomposed prompt, structured intermediate results, or additional model calls. In addition to predictive performance, our experiments demonstrated that result consistency is an important criterion when evaluating LLMs for code change impact analysis. While Qwen3 Coder Plus and Gemini 2.5 Flash both achieved

comparably high F1-scores, the results of Qwen3 Coder Plus were less consistent, reducing the practical usability of the approach. Despite improving the state-of-the-art in code change impact analysis, the absolute performance values indicate that it remains a challenging problem. Even katIA-MA, which performed best in our experiments, missed a large number of impacted code entities, highlighting the complexity of code change impact analysis. B. Threats to Validity We discuss threats to external and internal validity, covering both the generalizability of our findings and potential bias in our study design and execution. 1) External Validity: Our curated dataset C ONSTANTINO PLE faces several potential threats to external validity. First, the projects contained in the dataset comprise only opensource Apache Java projects. Hence, our results might not generalize to other programming languages or projects outside the Apache ecosystem. This limitation stems from using the set of projects used in A LEXANDRIA, a well-established dataset for code change impact analysis. While we cannot remove this limitation, we improved the data quality of A LEXANDRIA by proposing our curated dataset C ONSTANTINOPLE. Another threat is that the considered projects are not uniformly represented in the dataset, as we randomly sampled commits from the projects rather than enforcing an equal number of commits per project. This imbalance might bias the results towards the projects with a large number of commits. However, Table II shows that the majority of projects have between 5 and 20 commits, and only one project has 56 commits. We randomly sampled 400 commits from the 918 in A LEXANDRIA and finally obtained a dataset that contained 263 manually labeled commits. This provides a margin of error of 5.11% within a 95% confidence level, which we deem acceptable. We excluded commits without both seed and impacted code entities as keeping them in our dataset might introduce a selection bias. Similar works also followed this criterion, as they always considered a minimum of two changed methods and then applied their heuristics to determine the respective seed change. Furthermore, we considered only commits with one to five changed production code files, which does not cover large commits. However, in the original dataset, 824 of 918 commits are within this range, and only 94 are larger commits. Therefore, our restriction excludes a relatively small proportion of commits which we consider acceptable. We also did not include files other than Java production code files in our analysis. We excluded both test files and configuration files. Consequently, our approach cannot identify impacts outside the production code. Providing information on Java test files or meta-files, such as XML files, might improve the detection of change impact. Finally, we performed the experiments with three LLMs and the results might not generalize to all LLMs. Many LLMs exist, and the rapid developments reveal new versions frequently. We mitigated this threat by selecting models from

three different model families, including both general-purpose and code-focused models. 2) Internal Validity: One potential threat to internal validity concerns the labeling of the commits. We mitigated this threat by performing the labeling with three experts and in three phases: a calibration phase, a shared phase, and an individual labeling phase. The first two phases were used to evaluate and refine the rules and guidelines then used for labeling the commits in the individual phase. We considered the informal agreement in the shared phase sufficient to proceed with the individual labeling. Since there was no dedicated overlap in the individual phase, we did not measure an agreement for this share of data. However, the previous labeling phases helped to reduce potential errors. The prompt is the fundamental part of any LLM call, and therefore represents a potential source of bias. To mitigate the risk of introducing a prompt bias, we carefully designed all prompts used in this study according to findings and guidelines of recent software engineering research [25], [26], [31]. In addition, LLMs have many hyperparameters that can lead to different results. We tested all LLMs using their default hyperparameter settings and a temperature parameter set to 0, to mitigate this threat. Furthermore, we addressed LLM non-determinism in our experiments by performing three repetitions per experiment to assess consistency in the results. We also addressed potential model version drift by providing the model identifier and provider to improve reproducibility. Another threat to internal validity is the comparability of our work to recent LLM-based code change impact analysis approaches, such as R IPPLE [5]. We performed a best-effort comparison by aligning our evaluation with their more coarsegrained matching strategy. We compared their reported average performance metrics with an approximation of ours, using simple class and entity names rather than file and entity names. This favors R IPPLE because our entity-name matching also considers inner classes, while the file name used by R IPPLE does not. Furthermore, we provide an evaluation based on FQN-matching, which also addresses method overloading. Finally, R IPPLE considers only method changes and one seed change, whereas katIA also considers field and class changes, newly added methods, and multiple seed changes, which, in our opinion, represents a more realistic setting because not all bugs are fixed by changing one method. VII. C ONCLUSION AND F UTURE W ORK Code change impact analysis is a fundamental activity in software development. However, this activity is still often performed manually by software developers because existing approaches struggle with either low precision or low recall. Moreover, appropriate benchmarks are missing to thoroughly evaluate change impact analysis approaches. Addressing the problem with the benchmark, we first presented C ONSTANTINOPLE, a curated dataset consisting of 263 commits from 22 open-source Java projects and a total of 1,931 manually labeled changed source code entities. By

labeling the changed entities as either seed, impacted, or nonrelated, we provide a more realistic ground truth for code change impact analysis, compared to prior work [5], [8]. Next, we presented our approach katIA that combines static impact analysis with the reasoning capabilities of LLMs. Regarding the integration of LLMs into our approach, we studied two variants, one using a single prompt called katIASP and one using three specialized agents called katIA-MA. Using C ONSTANTINOPLE, we evaluated both variants, and, in addition to the classical performance metrics, precision, recall, and F1-score, we also considered the consistency of the results by repeating each experiment three times. The results for katIA-SP show that the LLM refines the initial static impact set. It improves the FQN-matching F1score from 14.6% to 22.4% with Gemini 2.5 Flash and to 25.2% with Qwen3 Coder Plus when considering the fully qualified name of code entities. Furthermore, it achieved CEmatching F1-scores 2.0 and 3.5 percentage points higher than the score reported for R IPPLE [5]. Gemini 2.5 Flash produced consistent results and was used for evaluating katIA-MA. The results for katIA-MA using Gemini 2.5 Flash showed a further increase in precision, recall, and F1-score under FQN-matching, with the latter reaching 29.1%. Under CEmatching, the F1-score reached 33.3%. Overall, the results show that katIA-MA with Gemini 2.5 Flash further improves the performance while keeping inference time and costs low and that the consistency of predictions should be considered and reported by software engineering studies involving LLMs. Despite these improvements, code change impact analysis remains a challenging problem. In future work, we plan to further improve katIA-MA by providing access to the full source code and equipping it with specific tools [32]. We also plan to extend C ONSTANTINOPLE with more commits from more projects and enrich our labels with change-type information based on the taxonomy of Fluri et al. [33]. Furthermore, future work should investigate the factors that caused katIAMA’s performance improvement through ablation studies and analyze the conversation logs for further information about the contribution of the individual agents and their patterns. ACKNOWLEDGMENT We used OpenAI Codex to assist with coding, literature search, and polishing of the prompts and the paper. The authors reviewed and refined the generated material and take full responsibility for the final manuscript. R EFERENCES [1] M. M. Lehman, “Programs, life cycles, and laws of software evolution,” Proceedings of the IEEE, vol. 68, no. 9, pp. 1060–1076, 1980. [2] S. Lehnert, “A taxonomy for software change impact analysis,” in Proceedings of the 12th International Workshop on Principles of Software Evolution and the 7th Annual ERCIM Workshop on Software Evolution, ser. IWPSE-EVOL ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 41–50. [Online]. Available: https://doi.org/10.1145/2024445.2024454 [3] M. Kretsou, E.-M. Arvanitou, A. Ampatzoglou, I. Deligiannis, and V. C. Gerogiannis, “Change impact analysis: A systematic mapping study,” Journal of Systems and Software, vol. 174, p. 110892, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S016412122030282X

[4] A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in 2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 712–721. [5] A. Yadavally and T. N. Nguyen, “From seed to scope: Reasoning to identify change impact sets,” in Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, ser. ICSE ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 400–412. [Online]. Available: https://doi.org/10.1145/3744916.3773265 [6] X. Ren, F. Shah, F. Tip, B. G. Ryder, and O. Chesley, “Chianti: a tool for change impact analysis of java programs,” in Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, ser. OOPSLA ’04. New York, NY, USA: Association for Computing Machinery, 2004, p. 432–448. [Online]. Available: https://doi.org/10.1145/1028976.1029012 [7] D. Zhou, J. Zhang, P. Yu, and W. Guo, “Enhancing change impact prediction by integrating evolutionary coupling with software change relationships,” in Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ESEM ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 49–60. [Online]. Available: https://doi.org/10.1145/3674805.3686668 [8] Y. Yan, N. Cooper, K. Moran, G. Bavota, D. Poshyvanyk, and S. Rich, “Enhancing code understanding for impact analysis by combining transformers and program dependence graphs,” Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643770 [9] A. Khare, S. Dutta, Z. Li, A. Solko-Breslin, R. Alur, and M. Naik, “Understanding the effectiveness of large language models in detecting security vulnerabilities,” in 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), 2025, pp. 103–114. [10] L. Fruntke and J. Krinke, “Automatically fixing dependency breaking changes,” Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3729366 [11] C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY, USA: Association for Computing Machinery, 2024, p. 819–831. [Online]. Available: https://doi.org/10.1145/3650212.3680323 [12] Z. Yang, C. Gao, Z. Guo, Z. Li, K. Liu, X. Xia, and Y. Zhou, “A roadmap for modern code review: Challenges and opportunities,” ACM Trans. Softw. Eng. Methodol., vol. 35, no. 9, Aug. 2026. [Online]. Available: https://doi.org/10.1145/3800963 [13] C. Macho, K. Stengg, and M. Pinzger, “Agent-based dependencyrelated build repair,” in 2026 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2026, pp. 1–6. [14] K. Stengg, C. Macho, and M. Pinzger, “katIA: Hybrid source code change impact analysis,” in Proceedings of the 1st International Workshop on Agentic AI for Next-Generation Software Development, ser. AgenticDev ’26. New York, NY, USA: Association for Computing Machinery, 2026, p. 177–178. [Online]. Available: https: //doi.org/10.1145/3843282.3844420 [15] M. Gethers, B. Dit, H. Kagdi, and D. Poshyvanyk, “Integrated impact analysis for managing software changes,” in 2012 34th International Conference on Software Engineering (ICSE), 2012, pp. 430–440. [16] B. Dit, M. Wagner, S. Wen, W. Wang, M. Linares-Vásquez, D. Poshyvanyk, and H. Kagdi, “Impactminer: a tool for change impact analysis,” in Companion Proceedings of the 36th International Conference on Software Engineering, ser. ICSE Companion 2014. New York, NY, USA: Association for Computing Machinery, 2014, p. 540–543. [Online]. Available: https://doi.org/10.1145/2591062.2591064 [17] Q. Hanam, A. Mesbah, and R. Holmes, “Aiding code change understanding with semantic change impact analysis,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2019, pp. 202–212. [18] Y. Huang, J. Jiang, X. Luo, X. Chen, Z. Zheng, N. Jia, and G. Huang, “Change-patterns mapping: A boosting way for change impact analysis,” IEEE Transactions on Software Engineering, vol. 48, no. 7, pp. 2376– 2398, 2022. [19] S. Herbold, A. Trautsch, B. Ledel, A. Aghamohammadi, T. A. Ghaleb, K. K. Chahal, T. Bossenmaier, B. Nagaria, P. Makedonski, M. N. Ahmadabadi, K. Szabados, H. Spieker, M. Madeja, N. Hoy, V. Lenarduzzi, S. Wang, G. Rodrı́guez-Pérez, R. Colomo-Palacios, R. Verdecchia, P. Singh, Y. Qin, D. Chakroborti, W. Davis, V. Walunj,

H. Wu, D. Marcilio, O. Alam, A. Aldaeej, I. Amit, B. Turhan, S. Eismann, A.-K. Wickert, I. Malavolta, M. Sulı́r, F. Fard, A. Z. Henley, S. Kourtzanidis, E. Tuzun, C. Treude, S. M. Shamasbi, I. Pashchenko, M. Wyrich, J. Davis, A. Serebrenik, E. Albrecht, E. U. Aktas, D. Strüber, and J. Erbel, “A fine-grained data set and analysis of tangling in bug fixing commits,” Empirical Software Engineering, vol. 27, no. 6, p. 125, Jul. 2022. [Online]. Available: https://doi.org/10.1007/s10664-021-10083-5 [20] K. Stengg, C. Macho, and M. Pinzger, “A dataset and preliminary study of using gpt-5 for code-change impact analysis,” 2025. [Online]. Available: https://arxiv.org/abs/2512.19481 [21] G. Rodrı́guez-Pérez, G. Robles, A. Serebrenik, A. Zaidman, D. M. Germán, and J. M. Gonzalez-Barahona, “How bugs are born: a model to identify how bugs are introduced in software components,” Empirical Software Engineering, vol. 25, no. 2, pp. 1294–1340, Mar. 2020. [Online]. Available: https://doi.org/10.1007/s10664-019-09781-y [22] M. Dias, A. Bacchelli, G. Gousios, D. Cassou, and S. Ducasse, “Untangling fine-grained code changes,” in 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), 2015, pp. 341–350. [23] R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, and L. Seinturier, “Spoon: A library for implementing analyses and transformations of java source code,” Software: Practice and Experience, vol. 46, no. 9, pp. 1155–1179, 2016. [Online]. Available: https://onlinelibrary.wiley. com/doi/abs/10.1002/spe.2346 [24] R. Passonneau, “Measuring agreement on set-valued items (masi) for semantic and pragmatic annotation,” in Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC), may 2006. [Online]. Available: https://doi.org/10.7916/ D8PV6TTT [25] Y. Sasaki, H. Washizaki, J. Li, N. Yoshioka, N. Ubayashi, and Y. Fukazawa, “Landscape and taxonomy of prompt engineering patterns in software engineering,” IT Professional, vol. 27, no. 1, pp. 41–49, 2025. [26] G. Wang, Z. Sun, S. Ye, Z. Gong, Y. Chen, Y. Zhao, Q. Liang, and D. Hao, “Do advanced language models eliminate the need for prompt engineering in software engineering?” ACM Trans. Softw. Eng. Methodol., vol. 35, no. 8, Jul. 2026. [Online]. Available: https://doi.org/10.1145/3771933 [27] Q. Ma, W. Peng, C. Yang, H. Shen, K. Koedinger, and T. Wu, “What should we engineer in prompts? training humans in requirement-driven llm use,” ACM Trans. Comput.-Hum. Interact., vol. 32, no. 4, Aug. 2025. [Online]. Available: https://doi.org/10.1145/3731756 [28] L. Fan, J. Liu, Z. Liu, D. Lo, X. Xia, and S. Li, “Exploring the capabilities of llms for code-change-related tasks,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 6, Jul. 2025. [Online]. Available: https://doi.org/10.1145/3709358 [29] L. Salewski, S. Alaniz, I. Rio-Torto, E. Schulz, and Z. Akata, “In-context impersonation reveals large language models'strengths and biases,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 72 044–72 057. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2023/file/e3fe7b34ba4f378df39cb12a97193f41-Paper-Conference.pdf [30] S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 2, Jan. 2025. [Online]. Available: https://doi.org/10.1145/3697010 [31] S. Baltes, F. Angermeir, C. Arora, M. M. Barón, C. Chen, L. Böhme, F. Calefato, N. Ernst, D. Falessi, B. Fitzgerald, D. Fucci, J. He, C. Treude, M. Kalinowski, S. Lambiase, D. Russo, M. Lungu, C. M. Montes, L. Prechelt, P. Ralph, R. van Tonder, and S. Wagner, “Guidelines for Empirical Studies in Software Engineering involving Large Language Models,” Empirical Software Engineering, 2026. [Online]. Available: https://arxiv.org/abs/2508.15503 [32] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 68 539–68 551. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2023/file/d842425e4bf79ba039352da0f658a906-Paper-Conference.pdf

[33] B. Fluri, M. Würsch, M. Pinzger, and H. Gall, “Change distilling: Tree differencing for fine-grained source code change extraction,” IEEE Transactions on Software Engineering, vol. 33, no. 11, pp. 725–743, 2007.

Related documents

• The Gap Between Open Access and Open Science in Education Research. A Repository-Based Analysis of Data, Code, and Materials Sharing in Open-Access Papers#409868
• Growth#1179746
• The Code Book#1222795
• Managing change#1238347
• Managing change#1225456