arXiv:2605.26100v1 [cs.SE] 25 May 2026
Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models Bar Weiss∗
Antonio Abu-Nassar
Viterbi Faculty of Electrical and Computer Engineering Technion, Haifa, Israel [email protected]
IBM Research Haifa, Israel [email protected]
Adi Sosnovich
Karen Yorav
IBM Research Haifa, Israel [email protected]
IBM Research Haifa, Israel [email protected]
Abstract Code review is a critical practice in software engineering, yet the growing scale and frequency of code patches in modern projects, together with the widespread adoption of AI code assistants, make manual review increasingly challenging. Identifying the types of changes within a patch, such as renames, moves, or logic modifications, can substantially improve review efficiency by enabling prioritization, filtering, and automation. However, existing LLMbased approaches to code review have largely focused on summarization and comment generation, leaving structured code reviews underexplored. In this paper, we present a systematic study of using large language models (LLMs) for taxonomy-based labeling of code changes in a code patch. We introduce a two-stage pipeline that assigns labels to diff hunks and then refines them to capture structural relationships and semantic attributes, such as rename propagation and type changes. Our approach employs few-shot prompting to produce language-agnostic and customizable labels, without the engineering overhead of traditional static-analysis pipelines. We evaluate four LLMs across multiple context configurations on a manually curated benchmark of natural and synthetic patches. Our best configuration achieves up to 84% recall and 81% precision, with high accuracy in extracting relational and attribute metadata. These results suggest that LLM-based labeling can effectively complement static analysis by enabling flexible, multilingual, and automationfriendly code review workflows.
CCS Concepts • Software and its engineering → Maintaining software.
Keywords Code Review, Refactoring, Large Language Models, Classification
1
Introduction
Code review is an essential activity in software engineering: having human peers inspect changes helps catch defects early, ensures consistent standards, and fosters shared knowledge among team members [15]. Yet, as codebases grow and code patches become more frequent, review throughput and quality increasingly become bottlenecks. We use the term code patch to denote a set of code changes. In practice, such patches are often obtained from pull ∗ Work done while interning at IBM Research.
2026-05-26 02:42. Page 1 of 1–13.
Static Analysis
LLM
Language Agnostic
No
Yes
Development Effort
Very High
Low
Maintenance
Medium
Low
Provable Guarantees
Yes
No
Label Customization
Medium
High
Accuracy High ? Table 1: Static analysis and LLM properties comparison.
requests (PRs), but our formulation is independent of the surrounding review workflow. Recent advances in Large Language Models (LLMs) have transformed code-related tasks, including code completion, bug detection, and summarization [3, 10, 18]. In the context of code review, LLMs are increasingly used to summarize patches, generate review comments, and explain changes [5, 13, 16]. While these approaches aim to reduce cognitive load and improve productivity, they primarily produce free-text outputs, which are difficult to integrate into automated workflows or analytics pipelines. One promising direction is to support both human understanding and automation by labeling diff hunks according to a well-defined taxonomy of change types (e.g., rename, move, interface change, style change). Such structured output enables downstream postprocessing, supports tailored workflows, and integrates naturally with dashboards and analytics systems. Traditionally, change-type identification relies on static analysis of program structure, offering strong accuracy and formal guarantees within supported domains. However, these methods are often language-specific, costly to extend, and expensive to maintain. By contrast, an LLM-based approach promises greater flexibility: it can be largely languageagnostic (assuming the model has sufficient exposure to the language), easier to adapt to new taxonomies, and faster to deploy. This flexibility comes at the cost of reduced formal guarantees and potentially less predictable accuracy, motivating a careful empirical study of its capabilities and limitations. This paper presents a study of LLMs for structured labeling of diff hunks in patches. Specifically, we investigate whether LLMs can: (1) assign taxonomy-based labels to individual diff hunks; (2)
Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav
capture structure-aware relationships across hunks (e.g., renaming a function argument and all of its usages); and (3) extract semantic attributes (e.g., old/new identifier names for renaming changes, old/new types for retyping changes). To evaluate our approach, we assembled a manually curated benchmark of PRs and assessed four LLMs across multiple retrieval-context strategies. Our best configuration achieves 84% recall and 81% precision. Overall, our method complements classical static-analysis techniques by trading formal guarantees for faster adoption, multilingual support, and easier customization, while retaining structured outputs that enable practical downstream automation. We conclude by discussing tradeoffs, limitations, and opportunities for extending this framework to semantic and intent-level change classification. 1
2
Related Work
Traditional approaches to identifying fine-grained code changes rely on static analysis of program structure. These techniques typically operate on Abstract Syntax Trees (ASTs) and apply rulebased algorithms to detect refactoring and structural edits. Representative tools include RefactoringMiner [17], GumTree [6], and ChangeDistiller [7]. RefactoringMiner, for example, introduced a threshold-free AST-based matching technique capable of identifying 15 common refactoring types directly from commit histories, without requiring projects to be built. These tools demonstrate the effectiveness of static analysis for structured change detection, but remain constrained by language-specific implementations, fixed taxonomies, and the engineering effort required to adapt them to new ecosystems. LLMs have demonstrated strong capabilities in code-related tasks such as completion, bug detection, and summarization [3, 10, 18]. Code-oriented assistants like Codex [4] and Claude Code [1] leverage large-scale training on code corpora to enable few-shot and zero-shot reasoning. In the context of code review, LLMs have been applied to PR summarization, comment generation, and reviewer assistance [5, 13, 16]. While these efforts highlight the potential of LLMs to support review workflows, they primarily produce freetext outputs rather than structured representations of code changes. To the best of our knowledge, no prior work has systematically evaluated LLMs for taxonomy-based labeling of diff hunks in code patches. Existing change-classification tools rely mainly on static analysis, achieving high precision for refactoring detection but remaining language-specific and difficult to customize. In contrast, our work investigates whether LLMs can provide structured, language-agnostic labels for code edits, including cross-hunk relationships (e.g., rename propagation, code moves) and semantic attributes (e.g., old/new identifiers), opening a new direction for automation-friendly code review.
3
Figure 1: Example of a diff hunk with identified labeling.
Labeling Code Changes
In this paper, we study the problem of labeling code changes. We define a set of label types T , each type describing a supported category of code change (the full taxonomy is detailed in Appendix A). Using git diff, we decompose a patch into a set of diff hunks H , where each hunk corresponds to a contiguous block of 1 Additional experimental details and prompt templates are available in the an
anonymized appendix at https://figshare.com/s/a254f611ba26b4da18a2.
(a)
(b)
Figure 2: Illustration of hunk relationships in (a) rename and (b) move label types.
changes between the old and new versions of the code. Our goal is to assign to each diff hunk in the patch all the label types from T that describe the changes in it. Formally, a labeling instance is defined as a tuple, ℓ = (𝑖, ℎ, 𝑡, 𝑝, 𝑎), where 𝑖 ∈ N is an integer index, ℎ ∈ H is the diff hunk the labeling is attached to, 𝑡 ∈ T is the label type, 𝑝 ∈ N is the index of the parent labeling instance, and 𝑎 is a set of string attributes. For most label types, 𝑝 and 𝑎 are unused and are therefore set to default values; their role is described in detail later. The output of our system is the set of all labeling instances, denoted by L. This formulation allows diff hunks to remain unlabeled, indicating that none of the label types in T adequately describes the changes in the hunk. Importantly, we do not assume that T exhaustively covers all possible change types. Overall, we wish to assign to each diff hunk ℎ ∈ H a (possibly empty) set of labels, as illustrated in Figure 1. Static analysis tools can provide information about relationships between edits for certain classes of code changes. We therefore extend our formulation to capture what we term structure-aware labels, which explicitly encode relationships between related code changes. Figure 2 illustrates this concept. For a rename label, the labeling corresponding to the declaration change references the labelings corresponding to the renamed usages. Similarly, for a move label, the labeling representing code removal is linked to the labeling representing code addition at the new location. These relationships are encoded via the 𝑝 field of the labeling tuple, which stores the index of the parent labeling object, and is set to 0 when no such relationship exists. In addition, for rename and retype labels, we extract attributes describing the change itself. For rename operations, we record the kind of renamed entity (VAR, CLASS, PACKAGE, METHOD, ATTRIBUTE, or PARAMETER), along with the old and new names. For retype operations, we extract the name of the retyped 2026-05-26 02:42. Page 2 of 1–13.
Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models
Figure 3: Illustration of our system’s pipeline. element and its old and new types. These attributes are stored in the 𝑎 field, for example 𝑎 = ("VAR", "old_name", "new_name").
4
Proposed Approach
Our objective is to develop an LLM-based tool for labeling patches that can be readily integrated into larger review and analysis pipelines. As outlined in Section 1, we adopt a zero-shot and few-shot prompting strategy, following prior work on large language models and in-context learning [3, 10, 18]. The pipeline is illustrated in Figure 3. We will explain its stages throughout this section. Each prompt provided to the LLM includes (i) the label taxonomy definitions, (ii) a task description, (iii) examples for more complex labels, and (iv) a required JSON output schema, together with a stream of diff hunks to be labeled. For each diff hunk, we augment the input with its file name and limited local context—specifically, five non-empty lines of code preceding and following the hunk (Figure 3A). The model’s JSON response is subsequently parsed into structured output objects for downstream processing. We decompose the overall problem into two sequential tasks: (1) Label Assignment, where the LLM assigns zero or more label types to each diff hunk based on its content, and (2) Structure Extraction, where relationships among labeled hunks are identified. We refer to the component handling the first task as the Labeler, and to the component performing the second as the Label Refiner (or Refiner for short). This decomposition serves several purposes. First, it breaks a complex task into manageable subproblems, improving robustness and simplifying prompt design. Second, it enforces a unified output schema for all labels identified in the Labeler stage. Third, it allows us to vary the context window of the Labeler independently of the Refiner, enabling controlled experimentation with token-budget trade-offs. Lastly, this enables an additional use for the Refiner, correcting errors from the Labeler that can originate from a lack of context in the Labeler. 2026-05-26 02:42. Page 3 of 1–13.
4.1
Labeler
As described above, the Labeler assigns label types to individual diff hunks. It operates by prompting an LLM with task instructions and label definitions (see Appendix B.1), followed by a stream of diff hunks. For each hunk ℎ ∈ H , the model returns a set of label types 𝑇 (ℎ) ⊆ T (Figure 3B). These subsets are subsequently converted into the labeling set representation L defined in Section 3 (Figure 3C), with the fields 𝑝 and 𝑎 initialized to default values and later refined by the Refiner for label types that require that. Constructing the diff hunk stream introduces a fundamental context trade-off. Providing more hunks supplies the LLM with a richer global context, which can improve label selection; however, longer contexts also increase the burden of filtering irrelevant information, potentially degrading performance. This effect has been observed in both controlled studies [9, 11, 19] and real-world LLM applications [20]. To study this trade-off, we evaluate three Labeler operating modes: (1) per-hunk inference, (2) per-file inference, and (3) per-patch inference, corresponding to increasing context lengths. Our results empirically demonstrate the impact of this trade-off.
4.2
Refiner
The Refiner serves two primary purposes: (1) assigning the 𝑝 and 𝑎 fields to labeling instances, and (2) correcting label types that may have been misassigned by the Labeler due to limited local context. Because labeling instances across files can be related, the Refiner processes the entire patch in a single inference. To bound context length, we restrict its input to diff hunks containing labelings that either require relational fields (rename, retype, move) or may conceal a missed structure-aware label (logic, unlabeled; Figure 3D). Accordingly, the Labeler is instructed to conservatively assign the logic label to suspected rename, retype, or move operations when context is insufficient. Identifying relationships between labeling instances also requires scope awareness. To capture this without substantially increasing context, we include the git diff hunk header before each
Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav
diff hunk as an approximation of the scope (Figure 3E). The Refiner is then prompted with task instructions and label definitions (Appendix B.2) to assign the 𝑝 and 𝑎 fields (Figure 3F). When a single diff hunk contains multiple rename or retype operations, the LLM returns a list of attributes in the 𝑎 field, which is subsequently split into distinct labeling instances.
5
Benchmark
We collected PRs to act as code patches from SWE-bench Multilingual [21] and SWE-PolyBench [14] and manually labeled them according to the change types present in each diff hunk. To ensure coverage of all label types, we additionally constructed and labeled a set of fabricated PRs. The dataset was split into development and test sets: the development set was used for prompt tuning, while the test set was reserved exclusively for the final evaluation reported in this paper. The test set contains 6 natural and 7 fabricated PRs, comprising 95 diff hunks and 142 labeling instances in total. Most PRs are written in Java; however, to reflect the language-agnostic nature of our LLM-based approach (Section 1), we also include Python PRs (two in the development set and one in the test set). Additional details about the benchmark are provided in Appendix C.1.
5.2
Mode
Cost [I/O Tokens]
IoP
IoGT
SWE
hunk file patch
1437/77 808/60 428/51
0.63 0.66 0.7
0.62 0.62 0.63
Llama
hunk file patch
1306/111 803/81 430/62
0.64 0.65 0.67
0.68 0.7 0.68
Sonnet
hunk file patch
1503/116 946/87 526/76
0.65 0.76 0.68
0.72 0.79 0.71
Gemini-3
hunk file patch
1436/863 901/373 492/477
0.64 0.81 0.68
0.68 0.84 0.7
Evaluation
We evaluate our system of a benchmark dataset created in this work. We compare our results on four different LLMs (two open-source and two frontier models): (1) SWE-Agent-LM-32B [21], (2) Llama4-Maverick-17B-128E-Instruct [12], (3) Claude Sonnet 4.5 [2], and (4) Gemini-3-Pro-Preview [8]. We abbreviate these models to SWE, Llama, Sonnet, and Gemini-3, respectively.
5.1
LLM
Table 2: Labeler results for different LLMs and run modes. The cost is presented in input/output tokens used, divided by the number of diff hunks.
and the relative ranking of types changes significantly across LLMs. We discuss these trends in Section 6.
Labeler Results
We evaluate the Labeler and Refiner separately in order to isolate their respective contributions. For the Labeler, we compare the predicted label type set 𝑇 (ℎ) for each diff hunk ℎ ∈ H against the manually annotated ground-truth labels 𝑇𝐺𝑇 (ℎ) from our benchmark. We measure agreement via the intersection of the predicted and ground-truth label sets and define the following two metrics: 1 ∑︁ |𝑇 (ℎ) ∩ 𝑇𝐺𝑇 (ℎ)| Avg-IoP(𝑇 ,𝑇𝐺𝑇 ) = , (1) |H | |𝑇 (ℎ)|
Figure 4: Precision versus recall per label type for the Labeler. Results are shown for Gemini-3 in per-file mode.
ℎ∈ H
Avg-IoGT(𝑇 ,𝑇𝐺𝑇 ) =
1 ∑︁ |𝑇 (ℎ) ∩ 𝑇𝐺𝑇 (ℎ)| . |H | |𝑇𝐺𝑇 (ℎ)|
5.3 (2)
ℎ∈ H
Intersection over Predictor (IoP) measures the fraction of predicted labels that are correct, while Intersection over Ground Truth (IoGT) measures the fraction of ground-truth labels recovered by the Labeler. These correspond to per-hunk precision and recall, respectively; we use the IoP/IoGT terminology to avoid ambiguity with later analyses. Table 2 reports Labeler performance across LLMs and context modes (per-hunk, per-file, and per-patch). We further analyze label difficulty using per-type precision and recall, defined as the fraction of correct predictions and the fraction of recovered ground-truth instances, respectively. Figure 4 shows these metrics for each label type under the best-performing configuration (Gemini-3, per-file mode; Table 2), with additional results in Appendix C.2. Performance varies substantially across label types,
Refiner Results
The evaluation of the Refiner focuses on the correctness of the attribute field 𝑎 and the parent field 𝑝 in the labeling objects. We define a parent-match between a predicted and a ground-truth labeling object when both reference the same parent diff hunk. For each relevant label type (rename and move), we count—per diff hunk—the number of matching parent references. We then normalize this count by the number of predicted labeling objects of that type to obtain a precision score, and by the number of ground-truth labeling objects to obtain a recall score. We evaluate the attribute field 𝑎 in a similar manner for rename and retype labels. Specifically, we count the number of matching attributes between predicted and ground-truth labeling objects within the same diff hunk, and normalize to obtain precision and recall scores. Since each labeling object contains three attributes, we divide by 3 to get a score in the range [0, 1]. Results for different LLMs and run 2026-05-26 02:42. Page 4 of 1–13.
Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models
modes are reported in Table 3. For the sake of brevity, we present here the results only for the frontier models (Sonnet and Gemini-3) and provide the full results in Appendix C.3. LLM / Mode
Label
Attribute
Parent
Precision
Recall
Precision
Recall
Sonnet / hunk
rename retype move
0.92 0.8 -
0.85 0.67 -
0.92 1.0
0.85 0.56
Sonnet / file
rename retype move
1.0 1.0 -
0.85 0.67 -
1.0 0.93
0.85 0.72
Sonnet / patch
rename retype move
1.0 1.0 -
0.85 0.67 -
1.0 0.82
0.85 0.5
Gemini-3 / hunk
rename retype move
0.97 0.92 -
0.9 0.61 -
1.0 1.0
0.92 0.61
Gemini-3 / file
rename retype move
0.72 0.92 -
0.67 0.61 -
1.0 0.93
0.92 0.78
Gemini-3 / patch
rename retype move
1.0 1.0 -
0.69 0.5 -
1.0 0.82
0.69 0.5
Table 3: Refiner results on capturing structure and attributes for different LLMs and run modes, with split precision and recall values
6
Discussion
We now discuss the results presented in Section 5. As shown in Table 2, Gemini-3 achieves the highest overall performance. This improvement, however, comes at a cost: its output token usage is significantly higher than that of the other models, reaching up to 7.5× that of the second-best model, Sonnet. This increase is largely due to the reasoning field in the response schema (Appendix B), where Gemini-3 provides more detailed explanations, which may partly account for its improved accuracy. For all models except SWE, the per-file inference mode performs best, highlighting a trade-off between context length and labeling accuracy. Consistent with prior work [11, 19], our results suggest that intermediate context sizes offer a better balance between available context and model focus. The per-type results in Figure 4 indicate that many label types can be detected reliably, though performance varies substantially across LLMs (Appendix C.3). For most types, precision and recall are correlated; however, for several types, these metrics diverge, creating opportunities for selective downstream use. Label types with high recall but lower precision capture most true instances but also generate false positives, which can be filtered by users or removed through post-processing. Conversely, label types with high precision but lower recall identify predominantly correct instances while missing some true cases; nevertheless, such labels can still 2026-05-26 02:42. Page 5 of 1–13.
be valuable, for example, by highlighting representative changes during codebase exploration or review. Regarding structure extraction, the results in Table 3 show that relational structure can indeed be captured by LLMs. In particular, Gemini-3 operating in the per-file mode identifies the parent field with very high precision and slightly lower recall. For the attribute field, precision is likewise very high, indicating that when the model identifies a rename or retype instance, its attributes are typically extracted correctly. Figure 4 further shows that, at the Labeler stage, per-hunk rename and retype detection achieves perfect precision and recall. In contrast, attribute recall remains lower than precision at the Refiner stage. This gap suggests that most errors stem not from incorrect attribute extraction, but from failures to correctly split or enumerate multiple labeling instances within a diff hunk, resulting in fewer extracted instances than required. Overall, the results demonstrate strong performance, particularly for frontier models. While the approach is not error-free, some degree of imperfection is acceptable in the context of code review. When higher accuracy is required for specific change types, a hybrid strategy can be adopted—using LLM-based labeling for most types while relying on static analysis for a small set of critical label types.
7
Conclusions and Future Work
We presented a novel LLM-based framework for labeling diff hunks in pull requests. By producing structured outputs, our approach enables systematic evaluation of LLM-based code review methods and supports seamless integration into automated review and analysis pipelines. We further showed that simple cross-hunk relationships, such as renames and code moves, can be captured using our two-stage labeling and refinement pipeline. This work also opens several directions for future research. First, the current label taxonomy T can be extended beyond primarily syntactic change types to incorporate semantic and intent-level categories, such as performance optimizations, bug fixes, or vulnerability mitigations. Evaluating the reliability of LLMs in identifying such changes and determining the pipeline adaptations required to support them remains an open question. Second, our reliance on git diff hunks may limit granularity, as individual hunks often contain multiple distinct changes. Automatically decomposing large hunks into smaller, semantically coherent units could improve interpretability and labeling accuracy. Finally, we see substantial potential in integrating our structured outputs into hybrid frameworks that combine LLM reasoning with classical static analysis, enabling more robust automated code review systems. We view this work as an initial step toward structured, machine-interpretable LLM-based code review beyond free-text feedback.
References [1] Anthropic. 2024. Claude Code. AI-assisted coding tool based on Claude models. Accessed: 2026-01-12. [2] Anthropic. 2025. Claude Sonnet 4.5. Large language model used for AI-assisted coding and writing. Accessed: 2026-01-12. [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. [4] Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).
Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav
[5] Codedog Project. 2024. Codedog: AI-Powered Pull Request Summarization and Review Suggestions. https://github.com/codedog-ai/codedog. Accessed: 2024-11-30. [6] Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014. Fine-grained and accurate source code differencing. In ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014. 313–324. doi:10.1145/2642937.2642982 [7] Beat Fluri, Michael Würsch, Martin Pinzger, and Harald C. Gall. 2007. Change Distilling: Tree Differencing for Fine-Grained Source Code Change Extraction. IEEE Transactions on Software Engineering 33 (2007), 725–743. [8] Google DeepMind. 2025. Gemini 3 Pro. Large multimodal language model. Accessed: 2026-01-12. [9] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv preprint arXiv:2404.06654 (2024). [10] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213. [11] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. [12] Meta AI. 2025. Llama 4 Maverick 17B 128E Instruct. https://ai.meta.com/llama/ Instruction-tuned variant of the Llama 4 Maverick 17B model. [13] Palos Publishing. 2024. Leveraging LLMs for Feedback Summarization in Code Reviews. Palos Tech Insights (2024). https://palospublishing.com/llms-forfeedback-summarization-in-code-reviews/ [14] Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al. 2025. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703 (2025). [15] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern code review: a case study at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice. 181–190. [16] Weisong Sun et al. 2024. Source Code Summarization in the Era of Large Language Models. arXiv preprint arXiv:2407.07959 (2024). https://arxiv.org/abs/2407. 07959 [17] Nikolaos Tsantalis, Matin Mansouri, Laleh M. Eshkevari, Davood Mazinanian, and Danny Dig. 2018. Accurate and Efficient Refactoring Detection in Commit History. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). ACM, New York, NY, USA, 483–494. doi:10.1145/3180155.3180206 [18] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021). [19] Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025 (2023). [20] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [21] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025. SWE-smith: Scaling Data for Software Engineering Agents. arXiv:2504.21798 [cs.SE] https://arxiv.org/abs/2504.21798
2026-05-26 02:42. Page 6 of 1–13.
Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models
A
Supported Code Change Types
Here we describe the label types set T supported by our system and their description as it appears in the prompt we provide to the LLM.
(1) Documentation - adding new or changing existing comments or descriptions. Also include explicit edits of .txt, .md or similar files. (2) Testing - changes to testing code. (3) Output Handling - changes to code that handles stdout, stderr, writes to output files, print statements, etc. (4) Retype - changing the type of a variable or attribute. examples: changed int to bool, as a consequence conditions are different, changed int to long, return type of a method returns base class rather than inherited class. (5) Code move - moving code from one location to another, this label should be added at the diff hunk where the code was removed and where it was added. examples: replacing a chunk of code with a function that runs the same code. Moving code from one file to another. (6) Style change - changes that modify the appearance of the code or the writing style but not the abstract syntax tree (AST). examples: move { from same line to line below, change comment style from // to /* */, split long lines, aligning and indentation (when the indentation does not matter), and other cosmetic changes. (7) Logging - everything related to logging, initializing the logger, summarizing the log, writing to the log, etc. (8) Rename - only changes to the name of a variable, method, attribute, class, parameter or package. (9) Error handling - changes that affect when an error or warning is raised or what happens when they are raised. examples: changes in the try-catch block logic, changes in exception types. (10) Logic change - any change that modifies the application execution, for example modifies the control flow or results in different application behavior. If you suspect that a diff hunk might be renaming, retyping, or code_move but you lack context to decide, label it as logic_change. (11) Internal interface change - The interface of a class or a package are all the publicly accessible elements. Interface changes are changes to the declarations of said elements. The word internal refers to elements that are internal to the application but not internal to a certain file or class. examples: changing methods between being Public or Private, modifying public method declarations or public attributes. (12) External interface change - changes to the interface itself or user interfaces, the program’s external API, command line interface, etc. examples: adding or modifying CLI arguments.
B.1
Labeler per-hunk prompt You are an experienced programmer reviewing pull requests on a large GitHub repository. You are given a diff hunk from a single pull request. Choose the labels that best describe the changes made in the hunk from the following list: {label_types} {specific_instructions} Here are some examples to clarify the instructions: {examples} {hunk_format_instructions} {json_format_request} Here is the diff hunk and some context: {input_stream}
Now we describe the per-file and per-patch prompt structure. Note that they are the same and will differ only in the length of the input stream.
Labeler per-patch/file prompt You are an experienced programmer reviewing pull requests on a large GitHub repository. You are given a stream of diff hunks from a single pull request. For each diff hunk, choose the labels that best describe the changes made in the hunk from the following list: {label_types} {specific_instructions} ———- Examples (do not respond to these diff hunks) ———{examples} ——– End of Examples ———{stream_format_instructions} {json_format_request} Here is the diff hunk stream with some context (respond to all of these): ———— Diff Hunk Stream ————{input_stream} ———— end of Diff Hunk Stream —————
B.2
Prompts
This appendix describes the prompt used for the Labeler and Refiner for the different context modes (per-hunk, per-file, per-patch). 2026-05-26 02:42. Page 7 of 1–13.
Refiner Prompt
Here, we provide the strcture of the Refiner’s prompt. For it as well we have placeholder values provided in Appendix B.3.
B.3 B
Labeler Prompt
We describe the structure of the Labeler’s Prompt for each of the context modes. We put in purple brackets value placeholders that are filled for the prompts for better readability. The values are given in Appendix B.3. Start with the per-hunk prompt structure:
Placeholder values
Here we present in table, all the placeholder values from the prompts described in Appendices B.1 and B.2. The {stream_format_instructions} and {refiner_stream_format_instructions} placeholders are too long to fit in a table and hence we provide them separately. The
Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav
Refiner prompt You are an experienced programmer reviewing pull requests on a large GitHub repository. You are given a stream of diff hunks from a single pull request that were labeled by another reviewer. Your task is to provide additional details on the labels. Here is the description of the labels. {label_types} For the following labels you should provide attributes and a parent field with the following meaning: {parent_and_attributes_instructions} Note that the parent label might appear after its children in the stream. The logic change label might be too broad, you may decide to replace it with a more specific label using the updated_type field in the response and provide attributes and parent_id accordingly. The parent_id must have the same label type as the one pointing to it. If you decide to keep the logic change type return the same type provided in the response. You should refer to names of attributes of a class using "<class_name>.<attribute_name>". And to names of arguments of a method using "<method_name>.<argument_name>". The return value of a method "<method_name>.return". {refiner_stream_format_instructions} {json_format_request} Here is the diff hunk stream with some context (respond to all of the labels): ———— Diff Hunk Stream ————{input_stream} ———— end of Diff Hunk Stream —————
rest of the values are provided in Table 4 for the Labeler and Table 5 for Refiner specific placeholders. {stream_format_instructions}. Ensure the output is valid JSON as it will be parsed using `json.loads()` in Python. It should be in the schema: <json> { "response_dict": { "<diff_hunk_idx1>": { "reasoning": "<reasoning1>", "label_names": "[<label1.1>, <label1.2>, ...]" }, "<diff_hunk_idx2>": { "reasoning": "<reasoning2>", "label_names": "[<label2.1>, <label2.2>, ...]" }, "<diff_hunk_idx3>": { "reasoning": "<reasoning3>", "label_names": "[<label3.1>, <label3.2>, ...]" }, "<diff_hunk_idx4>": { "reasoning": "<reasoning4>",
"label_names": "[<label4.1>, <label4.2>, ...]" } } } </json> Where <diff_hunk_idx1> is the integer index of the diff hunk, as it is in the diff hunk stream. And there should be an entry in the response_dict for each diff hunk in the stream. The label names should be exactly the same as in the list above. Example response: <json> { "response_dict": { "3": { "reasoning": "The diff hunk describes a change in the documentation and declares a new public method.", "label_names": [ "internal_interface_change", "documentation" ] }, "4": { "reasoning": "The method my_func was renamed to your_func.", "label_names": ["renaming"] }, "5": { "reasoning": "The change in the diff hunk does not match any of the defined labels.", "label_names": [] } } } </json> {refiner_stream_format_instructions}. Ensure the output is valid JSON as it will be parsed using ‘json.loads()‘ in Python. It should be in the schema: <json> { "response_dict": { "<label_id1>": { "reasoning": "<reasoning1>", "updated_type": <updated_type1>, "attributes": [<attribute1.1>, <attribute1.2>, <attribute1.3>], "parent_id": <parent_id1> }, "<label_id2>": { "reasoning": "<reasoning2>", "updated_type": <updated_type2>, "attributes": [<attribute2.1>, <attribute2.2>, <attribute2.3>], "parent_id": <parent_id2> }, "<label_id3>": { 2026-05-26 02:42. Page 8 of 1–13.
Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models
Placeholder {label_types} {specific_instructions}
{examples}
{hunk_format_instructions}
{json_format_request}
Prompt Segment List of label types and descriptions as they are presented in Appendix A in the following format: label_name: {label}, description: {description} You may choose more than one label if necessary, a diff hunk might contain more than one type of change. Compare the diff hunk against all labels in the labels list, and select all the labels that fit, considering all the types of changes in the diff hunk. The labels do not cover all possible code changes. Therefore, if none of the label describe the change in the diff hunk you should decide to assign no labels, i.e., return an empty list of labels. Pay attention to the file name and its suffix. In file code.py: Example diff hunk: import pandas as pd import matplotlib.pyplot as plt + import numpy as np + import json + import os Example response: "reasoning": "The diff hunk describes a change that does not fit any of the label types in the list, therefore the response is none of the labels, an empty list." "label_names": [] Ensure the output is valid JSON as it will be parsed using `json.loads()` in Python. It should be in the schema: <json> { "reasoning": "<reasoning1>", "label_names": "[<label1.1>, <label1.2>, ...]" } </json> The label names should be exactly the same as in the list above. Example response: <json> { "reasoning": "The diff hunk describes a change in the documentation and declares a new public method.", "label_names": ["internal_interface_change", "documentation"] } </json> The output should be only a JSON object according to the schema described above without additional text. For each diff hunk output every field only once! Output only a valid JSON object! Do not include quotation marks inside the reasoning string as they will make the JSON invalid, for example: Don’t say in the print statement print("hello"), instead say print(hello) or simply don’t quote such statements. Do not start the JSON with ```json or end with ```, instead use <json> at the beginning and </json> at the end of the JSON.
Table 4: Placeholder fields used in the Labeler’s prompt templates.
"reasoning": "<reasoning3>", "updated_type": <updated_type3>, "attributes": [], "parent_id": <parent_id3> }, "<label_id4>": { "reasoning": "<reasoning4>", "updated_type": <updated_type4>, "attributes": [<attribute4.1>, <attribute4.2>, <attribute4.3>], "parent_id": <parent_id4> }, } } </json> 2026-05-26 02:42. Page 9 of 1–13.
Where <label_id> is the integer index of the label, as it is in the input stream. And there should be an entry in the response_dict for each label in the stream. The label names should be exactly the same as in the list above. Example response: <json> { "response_dict": { "1002": { ""reasoning": "The orignal label fits this change becuase...", "updated_type": "LOGIC_CHANGE", "attributes": [], "parent_id": "0" }, "3000": { "reasoning": "This label is a consequence
Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav
of the RENAME label with id 5002.", "updated_type": "RENAME", "attributes": ["METHOD", "my_func", "your_func"], "parent_id": "5002" }, "5001": { "reasoning": "Code from this diff hunk was moved to diff hunk 8. as mentioned by label 8000", "updated_type": "CODE_MOVE", "attributes": [], "parent_id": "8000" }, "5002": { "reasoning": "The name of the method my_func was renamed to your_func in its declaration. ", "updated_type": "RENAME", "attributes": ["METHOD", "my_func", "your_func"], "parent_id": "0" }, "7001": { "reasoning": "This label is a consequence of the RENAME label with id 5002.", "updated_type": "RENAME", "attributes": ["METHOD", "my_func", "your_func"], "parent_id": "5002" }, "9003": { "reasoning": "This label is a consequence of the RENAME label with id 5002.", "updated_type": "RENAME", "attributes": ["VAR", "x", "y"], "parent_id": "5002" }, "8000": { "reasoning": "Code from diff hunk 5 was moved here as mentioned by label 5001", "updated_type": "CODE_MOVE", "attributes": [], "parent_id": "0" }, 9002: { "reasoning": "the variable x was changed from type int to long", "updated_type": "RETYPE", "attributes": ["x", "int", "long"], "parent_id": "0" } } } </json>
Input Stream Format. This is a formatting of the diff hunks processed in this call to the LLM. It changes based on the input but it has the following structure. The Labeler’s per-hunk stream follows the format below: In file <filename>: Code above the diff hunk: ``` <5 lines before hunk> ``` Diff hunk content: Header <header>: ``` <diff hunk> ``` Code below the diff hunk: <5 lines after diff hunk> And the per-file input stream has the following format: In file <file_name>: Diff hunk number <diff_hunk_index_1>: ``` <5 lines before hunk 1> <diff_hunk_1> <5 lines after hunk 1> ``` . . . Diff hunk number <diff_hunk_index_k>: ``` <5 lines before hunk k> <diff_hunk_k> <5 lines after hunk k> ``` The per-patch format simply is a concatenation of the per-file format. The Refiner’s stream follows the format below: In file <filename>: Diff hunk number <diff hunk_index_1> in scope <scope_1>: Labeled as: Type: <type_1.1>, ID: <label_id_1.1> . . . Type: <type_1.m>, ID: <label_id_1.m> ``` <5 lines before hunk 1> <diff_hunk_1> <5 lines after hunk 1> ``` . . . And this format continues for all files and diff hunks.
C
Evaluation Appendix
Here, we provide additional details on the evaluation in Section 5. 2026-05-26 02:42. Page 10 of 1–13.
Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models
Placeholder {parent_and_attributes _instructions}
Prompt Segment RENAME: parent_id - each rename is either a change in the declaration or a change to the usage i.e., a consequence of the change in the declaration. If the change is in the declaration this is the root of the change and the parent should be 0. If the change is a consequence the parent should contain the label id of the declaration change. attributes - for each rename we require three fields with the following order: The type of element that was renamed, one of the following: VAR, ATTRIBUTE, METHOD, CLASS, PARAMETER, PACKAGE. The original name. The new name. Example: if there are two renames described by a label the attributes should be: ["VAR", "my_var", "your_var", "CLASS", "MyClass", "YourClass"] RETYPE: parent_id - always 0 attributes - for each retype we require three fields with the following order: The name of the retyped element. The original type. The new type. Example: if there are two retypes described by a label the attributes should be: ["x", "float", "complex", "res", "list[str]", "dict[str, str]"] CODE_MOVE: parent_id - Where the code was added the parent is 0 and where it was removed it should be the label id of the label that describes the addition of the matching code. attributes - None, return an empty list [] LOGIC_CHANGE: parent_id - always 0 attributes - None, return an empty list []
Table 5: Placeholder fields used in the Refiner’s prompt template.
(a)
(b)
Figure 5: Histogram of label occurrences in our benchmark’s (a) validation set and (b) test set.
C.1
Benchmark Statistics
Here, we present additional details on the benchmark we constructed for this paper. As mentioned in Section 5.1, we split the data into development and test sets. The development set contains 3 natural JAVA PRs, 2 fabricated Python PRs. The histogram of label 2026-05-26 02:42. Page 11 of 1–13.
type appearances in the development set and test set are shown in Figure 5. Importantly, all label types are represented in both sets and we made an effort to keep the type frequencies as similar as possible.
Bar Weiss, Antonio Abu-Nassar, Adi Sosnovich, Karen Yorav
LLM / Mode
Label
Attribute
C.2
Parent
Precision
Recall
Precision
Recall
SWE / hunk
rename retype move
0.67 0.52 -
0.62 0.61 -
0.92 0.0
0.85 0.0
SWE / file
rename retype move
0.52 0.75 -
0.52 0.6 -
0.56 0.1
0.56 0.06
SWE / patch
rename retype move
1.67 0.8 -
1.54 0.67 -
0.58 0.625
0.54 0.28
rename retype move
0.63 0.67 -
0.77 0.44 -
0.81 0.375
rename retype move
0.58 0.67 -
0.49 0.56 -
rename retype move
0.63 0.43 -
Sonnet / hunk
rename retype move
Sonnet / file
Additional Labeler Results
Here, we present additional results for the Labeler. We present the IoP and IoGT of the Labeler without the type update by the refiner in Table 6. For almost all modes there is a slight improvement becuase of the additional refiner step, but the trend of per-file being the best performing mode remains. And Gemini-3 is still the best performing model. LLM
Mode
Cost [I/O Tokens]
IoP
IoGT
SWE
1.0 0.17
hunk file patch
1437/77 808/60 428/51
0.63 0.64 0.67
0.62 0.62 0.61
Llama
1.0 0.44
0.85 0.22
hunk file patch
1306/111 803/81 430/62
0.58 0.64 0.67
0.63 0.69 0.69
Sonnet
0.44 0.5 -
0.33 0.64
0.23 0.39
hunk file patch
1503/116 946/87 526/76
0.62 0.75 0.68
0.67 0.77 0.71
Gemini-3
0.92 0.8 -
0.85 0.67 -
0.92 1.0
0.85 0.56
hunk file patch
1436/863 901/373 492/477
0.60 0.79 0.69
0.62 0.83 0.71
rename retype move
1.0 1.0 -
0.85 0.67 -
1.0 0.93
0.85 0.72
Sonnet / patch
rename retype move
1.0 1.0 -
0.85 0.67 -
1.0 0.82
0.85 0.5
Gemini-3 / hunk
rename retype move
0.97 0.92 -
0.9 0.61 -
1.0 1.0
0.92 0.61
Llama / hunk
Llama / file
Llama / patch
Gemini-3 / file
rename retype move
0.72 0.92 -
0.67 0.61 -
1.0 0.93
0.92 0.78
Gemini-3 / patch
rename retype move
1.0 1.0 -
0.69 0.5 -
1.0 0.82
0.69 0.5
Table 7: Refiner results on capturing structure and attributes for different LLMs and run modes, with split precision and recall values
Table 6: Labeler results for different LLMs and run modes. The cost is presented in input/output tokens used, divided by the number of diff hunks.
We also present here the per-type precision recall plots for additional run modes in Figure 6. We show results for Gemini-3 and Sonnet in per-hunk and per-file modes. While some label types appear to have consistent results (rename, retype, docs).
C.3
Refiner Results
We present here the Refiner results for all models in Table 7. We further present the per-type precision vs. recall plots for the frontier models in all inference modes (per-hunk, per-file and per-patch) in Figure 6. As mentioned in the main text, the performance of each type for the different models and inference modes varies significantly. This is especially true for label types with less instances in the benchmark’s test set (external interface, error handling, unlabeled, log). This fact makes it difficult to draw conclusions on the behavior of each type, and a larger scale experiment is required before deciding how the handle errors of the different types.
2026-05-26 02:42. Page 12 of 1–13.
Beyond Summaries: Structure-Aware Labeling of Code Changes with Large Language Models
(a)
(b)
(c)
(d)
(e)
(f)
Figure 6: Per-type precision recall plots for (a) Gemini-3 per-hunk, (b) Gemini-3 per-file, (c) Sonnet per-hunk, and (d) Sonnet per-file.
2026-05-26 02:42. Page 13 of 1–13.