arXiv CSopen access

XAgent: eXecution-guided Agentic AI for Effective Localization and Resolution of GitHub Issues

· arxiv_cs

arXiv CS · Papers · License: Open Access

software-architecture software-engineering testing

software engineering, software architecture, testing

XAgent: eXecution-guided Agentic AI for Effective Localization and Resolution of GitHub Issues Hieu Huynh · Patanamon Thongtanunam · Michael Fu · Bach Le · Kla Tantithamthavorn

arXiv:2609.09769v1 [cs.SE] 9 Sep 2026

Received: date / Accepted: date

Abstract Agentic AI has enabled capabilities in leveraging Large Language Models (LLMs) to autonomously resolve repository-level GitHub issues. However, due to the reliance on limited static description of issues, existing agentic approaches suffer from incorrect localization and incomplete validation. Solely relying on this information can bias LLM reasoning toward the narrow scope of the issue description, leading to incomplete patches that fail to address the underlying issue. In this paper, we present XAgent, an execution-guided agentic framework that analyzes dynamic behavior and additional program context to localize and validate issues. The experimental results on the SWEbench-lite dataset demonstrate that XAgent outperforms other existing approaches, achieving a resolve rate of 62.0% and a function localization accuracy of 72.8%, while maintaining cost efficiency. Our analysis further shows that XAgent successfully resolves 7 additional issues that the top existing Hieu Huynh School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia E-mail: [email protected] Patanamon Thongtanunam School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia E-mail: [email protected] Michael Fu School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia E-mail: [email protected] Bach Le School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia E-mail: [email protected] Kla Tantithamthavorn Monash University, Clayton, Australia E-mail: [email protected]

Huynh et al.

baselines fail to address. This work highlights a shift from static, descriptionoriented patch generation toward dynamic execution-guided issue resolution, opening new opportunities for LLM-based coding agents to achieve more robust and generalizable software maintenance. Keywords Agentic AI · Coding Agents · Differential Analysis · Code Execution

1 Introduction Agentic AI has demonstrated potential to address various software engineering tasks including resolving real-world GitHub issues (e.g., AutoCodeRover (Zhang et al. 2024), Agentless (Xia et al. 2025), ExpeRepair (Mu et al. 2025), SWE-agent (Yang et al. 2024a), and CoSil (Jiang et al. 2025)). Such agentic approaches leverage Large Language Models (LLMs) as agents to autonomously perceive the input environments (e.g., GitHub repositories), reason (e.g., generating a coding plan), and autonomously perform actions (e.g., open files, run bash scripts) to achieve an ultimate goal (e.g., resolve a GitHub issue at a repository-level). Typically, such approaches follow a three-stage workflow: (1) reproduction, which recreates the reported buggy behavior by generating test cases that verify the described issue; (2) localization, which identifies the relevant files, functions, and code regions responsible for the buggy behavior; (3) patch generation & validation, which produces candidate fixes for localized components and evaluates whether the proposed patch resolves the issue through execution of the reproduction and regression test. However, existing agentic approaches are still ineffective in resolving GitHub issues. For example, ExpeRepair (Mu et al. 2025), which is #1 in the SWEBench-lite benchmark dataset can correctly localize 70.7% of the buggy functions and correctly resolve only 60.3% of the GitHub issues, which is still far from perfect. We suspect that the ineffectiveness has to do with the overreliance on static issue descriptions for the localization and the test validation stages in the existing agentic approaches. In particular, we identified the following two major limitations. – Limitation 1: Incorrect Localization. The localization stage of the existing approaches (Yang et al. 2024a; Wang et al. 2024b; Mu et al. 2025; Xia et al. 2025) relies on the semantic similarity of keywords that appear in the issue descriptions and the files and functions within the whole repository (i.e., keyword search). However, as issue descriptions are generally noisy or incomplete, the sole-reliance on static issue descriptions may bias the LLMs toward functions that are irrelevant or incorrect to the given issue description, leading to an incorrect localization by the existing approaches. – Limitation 2: Incomplete Validation. Similarly, the test validation stage of the existing approaches (Xia et al. 2025; Mu et al. 2025) solely uses the issue description to generate a set of validation tests without considering

XAgent: eXecution-guided Agentic AI for GitHub Issues

the broader program context. As a result, the validation process often overfits the narrow scope defined by the issue description, causing the generated patches to pass the validation test but fail to address the underlying issue. These limitations make existing approaches prone to producing patches that overfit to the described issue (i.e., overfitting patches (Smith et al. 2015; Le et al. 2018)), ultimately restricting their effectiveness in real-world scenarios. In this paper, we present XAgent, an execution-guided agentic framework for GitHub issue resolution that analyzes dynamic behavior during program execution to localize functions and leverages program context to generate more reliable validation tests. Inspired by differential analysis (Zeller 2009), XAgent identifies the buggy functions by comparing failing and passing executions of the program. Specifically, it performs localization by identifying suspicious functions that appear only in failing execution but not in the passing one. To prevent the validation test from overfitting to the issue description, we develop a context-aware test augmentor that leverages additional program context (e.g., relevant files and functions) to broaden the scope of validation beyond what is described in the issue. This enables XAgent to generate edgecase and related-component tests, ensuring that the generated patches address the underlying issue. Experimental results show that XAgent achieves state-of-the-art performance on the SWE-bench-lite dataset, surpassing top baselines with a resolve rate of 62.0%. XAgent also demonstrates highly effective bug localization, achieving SOTA function localization accuracy of 72.8%, while maintaining high efficiency at a cost of approximately $1.56 per issue, which is 37% cheaper than the prior SOTA approach (Mu et al. 2025). Additionally, our analysis reveals that XAgent uniquely resolves 7 complex instances that other top approaches (i.e., ExpeRepair (Mu et al. 2025), Refact Agent (Refact.ai 2025), SWE-Agent (Yang et al. 2024a)) failed to fix. The results show that XAgent can accurately localize and cost-effectively generate correct patches to resolve the underlying issues. This work highlights a shift from static, descriptionoriented patch generation toward dynamic, execution-guided issue resolution, opening new opportunities for LLM-based agents to achieve more robust and generalizable software maintenance. Novelty & Contributions. To the best of our knowledge, we are the first to propose: – XAgent: A novel execution-guided agentic framework for GitHub issue resolution that shifts from static, description-oriented approaches to a dynamic, execution-guided and context-aware approach. – A novel execution-guided localization method using differential analysis (buggy vs. non-buggy) and tree edit distance to localize bugs by analyzing execution divergence, effectively addressing the limitation of existing static bug localization. – A context-aware test augmentor that generates broader validation tests targeting edge cases, effectively extending validation coverage beyond the issue description.

Huynh et al.

– A comprehensive empirical evaluation on SWE-Bench-Lite shows that XAgent sets a new state of the art, achieving the highest Resolve Rate (62.0%) and function-level localization accuracy (72.8%) while reducing cost by 37%. Furthermore, XAgent successfully resolves 7 issues that top existing baselines (e.g., SWE-Agent, EXPEREPAIR, Refact) failed to address. – We conduct an ablation study to verify our key design for XAgent, demonstrating that the execution-guided localization and context-aware test augmentor together contribute up to 9% relative improvement in the overall performance.

2 Background & Related Work Background. Software issue resolution at the repository level is a core software maintenance activity that requires understanding, localizing, and fixing real-world bugs in complex codebases based on issue descriptions provided by users. Given an issue report, developers typically reproduce the problem to confirm the issue, localizing the bug, implement a patch, and validate the fix through testing. Recent advances in LLMs have significantly transformed this process by serving as powerful reasoning and code generation components in software engineering, enabling a shift from manual workflows toward increasingly autonomous solutions. To further enhance effectiveness, agentic AI has been adopted to enable LLMs to interact directly with code repositories through tool use, allowing them to iteratively plan, execute, and adapt their strategies based on environmental feedback. Existing agentic methods for repository-level issue resolution typically organize the workflow into three main stages: (1) reproduction, (2) localization, and (3) patch generation and validation. Below, we outline how LLMs are used at each stage of software issue resolution within a code repository. Reproduction is the stage for reproducing the behavior as users described in the issue description. Typically, existing approaches (Jiang et al. 2025; Yang et al. 2024a; Mu et al. 2025; Xia et al. 2025; Yu et al. 2025; Wang et al. 2024a; Nashid et al. 2025; Kang et al. 2023) leverage an LLM to generate the reproduction test based on the issue description and the code context of the repository. This reproduction test then executes the program to verify the existence of the reported issue. Localization aims to identify the specific files and functions to be fixed (Jiang et al. 2025; Chakraborty et al. 2025). Existing work (Yang et al. 2024a; Wang et al. 2024b; Mu et al. 2025; Xia et al. 2025) relies on LLMs to extract keywords from issue descriptions and search for files and functions to be fixed within the whole repository. For example, AGENTLESS (Xia et al. 2025) adopts a hierarchical localization strategy by extracting keywords from the issue description, identifying the top-K relevant files, then narrowing down to candidate classes, functions, and finally specific edit locations. AutoCodeRover (Zhang et al. 2024) also performs localization by keyword search, then further refine it by leveraging the spectrum-based fault localization technique if test

XAgent: eXecution-guided Agentic AI for GitHub Issues

suites are available in the repository. LocAgent (Chen et al. 2025) localizes bugs by using keywords from the issue description to search through the repolevel code graph. OrcaLoca (Yu et al. 2025) analyzes the error trace provided in the issue description to identify the relevant files and functions. PatchPilot (Li et al. 2025) enhances localization by leveraging the execution information from the reproduction test. Despite the promising results, these existing works heavily relied on the limited information in the issue description and the reproduction test. Limitation 1: The limited information in the issue description and the reproduction test may bias the LLMs toward the failure scenario, leading them to misidentify the buggy function. Patch Generation & Validation is the stage when an LLM produces patch candidates for the localized functions and validates whether the patch resolves the issue. Typically, existing approaches (Yang et al. 2024a; Wang et al. 2024b; Mu et al. 2025; Li et al. 2025; Yu et al. 2025) iteratively generate the patches for the localized functions and validate them against the validation tests. These validation tests include the reproduction test to verify that the reported issue has been resolved, as well as regression tests to ensure that the fix does not adversely affect existing functionality. To effectively achieve this task, many approaches leverage agentic method (Yang et al. 2024a; Wang et al. 2024b; Zhang et al. 2024; Mu et al. 2025) which enable an LLM autonomously to select actions and adjusts its plan. For example, SWEagent (Yang et al. 2024a) and OpenHands (Wang et al. 2024b) equip LLMs with tools to interact directly with the coding environment, such as viewing files, searching code, and editing source files, enabling them to plan and decide actions based on feedback from the validation test results. EXPEREPAIR (Mu et al. 2025) further improves the method by integrating a Dual-Memory System that allows LLMs to retrieve knowledge accumulated from previous runs. In addition, they expand validation tests by generating additional tests from original reproduction tests using different hyper-parameters. Despite the effectiveness of the agentic method in generating patches, the validation process is solely based on the issue description without considering the broader program context. Limitation 2: The validation process often overfits narrow scopes defined by the issue description, causing the generated patches to pass the validation test but fail to address the underlying issue.

3 Motivating Examples 3.1 Example 1 Fig 1 illustrates the issue Matplotlib-25433 in the SWE-bench-lite dataset. This issue occurs when a user creates a plot with a slider that controls a

Huynh et al.

Issue Description - Matplotlib-25433: Using clf and pyplot.draw in range slider on_changed callback blocks input to widgets. When using a RangeSlider's on_changed callback to clear and redraw a figure, all widget inputs become unresponsive. The same redraw operation works correctly when triggered by a Button's on_clicked callback. def on_slider_change(val): plt.clf() create_dummy_widgets() plt.draw() slider = RangeSlider(...) slider.on_changed(on_slider_change) slider.set_val((2, 8)) slider.set_val((3, 9)) plt.show() ↑ Trigger bug

🐞

(a) Reproduction Script (Buggy) Buggy test: ├── init RangeSlider Localized by ├── on_changed existing methods ├── set_val │ └── on_slider_change │ ├── clf Actual buggy │ │ ├── delaxes func. │ │ └── clear │ ├── create_dummy_widgets │ └── draw ...

❌

✅

🐞

def on_slider_change(val): create_dummy_widgets() plt.draw() slider = RangeSlider(...) slider.on_changed(on_slider_change) slider.set_val((2, 8)) slider.set_val((3, 9)) plt.show()

(b) Reference Non-buggy test: Script (Non-buggy) ├── init RangeSlider ├── on_changed ├── set_val │ └── on_slider_change │ │ │ │ │ │ │ ├── create_dummy_widgets │ └── draw ... (d) Non-buggy Execution Trace

Fig. 1: Matplotlib-25433 Issue.

dynamic value in the graph. The slider is associated with a callback function (i.e., an on slider change handler) that is triggered whenever the user drags the slider. The problem occurs when this callback calls clf() to clear and redraw the figure: the clf()) function removes all plot elements, including the slider currently being used, without releasing its mouse event. As a result, Matplotlib continues to wait for input from a slider that no longer exists, causing the application to freeze and ignore further mouse interactions. The issue was fixed by modifying the delaxes function (called by clf()) to release all active mouse events before clearing the figure, ensuring that interactive plots remain responsive. Existing approaches. Refact (Refact.ai 2025) modifies the on changed as their LLM reasoning tends to follow the keyword on changed mentioned in the issue description. On the other hand, other approaches (Mu et al. 2025; Yang et al. 2024a; Yu et al. 2025) localize the set val function as this function triggers the error in the reproduction script. However, the actual bug resides in the delaxes function, which lies deeper in the runtime execution traces and is not explicitly mentioned in the issue description. Indeed, the functions can be efficiently localized based on a differential strategy (Zeller 2009): ”code that is executed only in failing runs is more likely

XAgent: eXecution-guided Agentic AI for GitHub Issues

to contain the defect than code that is always executed”. This inspires us to focus on contrasting failing behavior against a passing one. In other words, rather than solely focusing on the buggy scenario (Fig 1a), a non-buggy scenario (Fig 1b) that demonstrates the correct and expected behavior could also provide important information for bug localization. Specifically, when we compare the execution traces between these two scenarios, specific functions that are invoked exclusively in the buggy scenario will be highlighted. Hence, these functions are potentially buggy. For example, Fig 1c-1d presents simplified buggy and non-buggy traces. We can clearly see that set val and on changed functions appear in both traces, only clf, delaxes, and clear functions appears uniquely in the buggy trace. These functions precisely match the actual buggy function identified by the developer. Moreover, this comparison between the buggy and non-buggy execution also helps reduce the search space of the functions. Specifically, there are 756 functions in the buggy trace and 758 functions in the nonbuggy trace. Comparing the two traces reveals only 15 functions that are uniquely invoked in the buggy trace. This supports our intuition that the behavioral differences between the buggy and non-buggy scenarios might reduce the search space significantly and could reveal the functions where the issue is likely located. Key Idea 1: Execution-guided Localization. Comparing buggy and non-buggy executions could reveal functions with divergent behaviors, which are likely potentially buggy functions.

3.2 Example 2 Fig 2 presents another sample issue from the SWE-bench dataset, seaborn2848, which is a regression bug in the Seaborn visualization library. The bug occurs when drawing pairplot: users provide a dataset containing multiple categories but configure the plot to display only a subset of them. In previous versions, the library handled the unselected categories by rendering them as transparent. However, in the buggy version, the program crashed with an uncaught exception. The correct behavior is to map these unselected categories to a transparent color code (0, 0, 0, 0) rather than treating the lookup failure as a data type error. The fix involves modifying the HueMapping. lookup single function in the oldcore.py file. Existing approaches. Leading agentic approaches (Refact.ai 2025; Yang et al. 2024a; Mu et al. 2025; Yang et al. 2025) primarily use the issue description to generate validation tests to assess the correctness of their patches. Such a validation method could lead to the symptom-fix trap, where agents address the surface failure without resolving the underlying issue. In this example, the issue explicitly reports a failure in pairplot. As a result, existing approaches (Refact.ai 2025; Yang et al. 2024a; Mu et al. 2025; Yang et al. 2025) interpret the task narrowly as “prevent pairplot from crashing.” Con-

Huynh et al. Issue Description - Seaborn-2848: In version 0.11.1, `pairplot` fails if `hue_order` is a subset of the `hue` column values. This worked in versions earlier than 0.11. >>> import seaborn as sns >>> iris = sns.load_dataset("iris") >>> # hue column has 3 species; we only want 2 >>> sns.pairplot(iris, hue="species", hue_order=["setosa", "versicolor"])

This raises a `TypeError` (previously, unused points were filtered silently):

>>> TypeError: ufunc 'isnan' not supported for input types and the inputs could not be safely coerced

(a) Add. Context: pairplot in repo

Implementation

(b) Reproduction Script

of (c) Additional validation test for plot types

Fig. 2: Seaborn-2848 Issue. cretely, they only focus on the internal function plot bivariate, which is the core engine behind pairplot. Hence, their validation tests are limited to verifying whether the reported pairplot failure is resolved. Consequently, these patches fail human-written validation tests, which also evaluate the behavior of scatterplot, a related plot type. As shown in the relevant code context in Fig 2a, pairplot is not an isolated component: other plotting functions (i.e., scatterplot and histplot) depend on a shared logic. This suggests that if pairplot exhibits a given failure, validation should also test whether similar issues arise in these related plots. This supports our intuition that we should consider the broader context of the program to find relevant information that is needed to validate. Key Idea 2: Context-Aware Validation Test. Issue descriptions often offer only a limited view of the problem, leading to generated test cases that address specific symptoms rather than the underlying issue. To overcome this, leveraging program context when generating validation tests will enhance the coverage, ensuring that edge cases and similar functions are properly tested and that the fix generalizes beyond the initial symptom.

4 XAgent: eXecution-guided Agentic AI 4.1 Overview In this paper, we introduce XAgent, an execution-guided agentic framework for GitHub issue resolution that analyzes dynamic behavior during program execution for localization and leverages program context to generate more reliable validation tests. Based on Key Idea 1 (Section 3.1), we introduce the

XAgent: eXecution-guided Agentic AI for GitHub Issues

🛠

Function X Function Y Function D Function Z

Fig. 3: An overview of XAgent method.

execution-guided localizer that compares the failing and passing execution of the program to identify functions that potentially contain the issue. This directly addresses Limitation 1, overcoming the constraints of static issue descriptions that often fail to surface the deeper context needed for accurate bug localization, as demonstrated in Example 1. To extend validation coverage beyond the issue description addressing Limitation 2, we develop the context-aware validation test augmentor. Based on Key Idea 2 (Section 3.2), our validation test augmentor uses additional contextual information from related files and functions to generate comprehensive validation tests, improving code coverage, as demonstrated in Example 2. Fig 3 illustrates the overall workflow of XAgent, which consists of three phases: reproduction, localization, and patch & validation. In the reproduction phase 1 , the issue description is provided to the reproduce agent, which is responsible for generating two scripts: one that captures the buggy behavior and another that represents the non-buggy behavior (see examples in Fig. 1c and 1d). After generating these scripts, the agent validates them by comparing their outputs against the expected results described in the reported issue. If the issue is confirmed, XAgent will proceed the next phase. In the localization phase 2 - 3 , the reproduction scripts are executed to collect dynamic execution traces, which are represented as call trees (see examples in Fig. 1c and 1d). These two call trees are then analyzed using a tree edit distance algorithm (Pawlik and Augsten 2016) to identify suspicious functions that uniquely appear in the failing trace. The identified functions are then ranked by the embedding similarity ranker 3 based on the issue description and function names to prioritize the most likely buggy functions. The process ends with the patch generation & validation phase 4 - 5 . In this phase, the Context-aware Validation Test Augmentor 4 generates additional validation

Huynh et al.

tests based on the context of related files and functions, enabling a more thorough evaluation of the bug. The set of validation tests and the ranked list of functions are then passed to the patch agent 5 . This agent will iteratively identify the final buggy function, create a candidate patch, and validate it until the generated patch passes all the validation tests.

4.2 Reproduction During this phase, the reported issue description is passed to the reproduction agent, which generates a reproduction script and a reference script. The reproduction script is designed to trigger the unexpected behavior (buggy) described in the issue report, thereby confirming the presence of the bug. The reference script, in contrast, demonstrates the intended correct behavior (nonbuggy) of the target feature, showing that the program executes as expected without errors. Together, these scripts establish a baseline of correct functionality and a concrete example of failure. To guide the process, XAgent explicitly instructs the reproduction agent to first carefully read and interpret the issue description. The agent then constructs the reproduction script to capture the erroneous behavior described in the issue. After that, it generates the reference script with a structure similar to the reproduction script but introduces minimal input differences to reflect the intended correct behavior of the feature. These small variations are essential, as large differences may introduce unrelated factors that affect execution traces and add noise, making it harder to identify the root cause during call tree comparison. By enforcing this similarity constraint, the agent produces clean and comparable scripts that improve the reliability of downstream analysis. Once both scripts are generated, the reproduction agent executes them and validates their outputs against the behaviors described in the issue report. If the reference script runs successfully and the reproduction script exhibits the expected erroneous behavior, the reproduction is considered successful. Otherwise, the agent regenerates the scripts while incorporating the current results. Finally, both scripts are submitted to the localization phase of the pipeline. 4.3 Localization Once the reproduction agent submits its generated scripts, XAgent performs localization to identify the potential buggy functions. Particularly, XAgent executes the two scripts using a tracing module, which records function calls, including the caller–callee relationships and any associated traceback errors. From this execution trace, we construct call trees, where each node corresponds to a function, and each child node represents a callee of its parent. Let T1 = (V1 , E1 ) and T2 = (V2 , E2 ) represent the call trees for the buggy and non-buggy executions, respectively, where V1 and V2 are the sets of nodes

XAgent: eXecution-guided Agentic AI for GitHub Issues

(functions) in T1 and T2 , and E1 ⊆ V1 × V1 and E2 ⊆ V2 × V2 are the edge sets representing caller–callee relationships. To capture the behavioral differences between the two call trees, we apply the All Path Tree Edit Distance (APTED) algorithm (Pawlik and Augsten 2016, 2015, 2011). APTED is an efficient algorithm for computing the edit distance between two trees. It produces a set of node mappings that explain how one tree can be transformed into the other, while minimizing the number of edit actions (i.e., delete, insert). The output mappings fall into three categories. Given the tree edits from a buggy tree to a non-buggy tree (T1 → T2 ) and the node mappings (u, v) ∈ M where u ∈ V1 and v ∈ V2 , each category is defined as below: – Identical : u and v are matched without any modification. – Deletions: (u, ϵ) ∈ M , node u is deleted from the buggy tree. – Insertions: (ϵ, v) ∈ M where node v is inserted to the buggy tree. For localization, we focus on nodes (functions) that appear uniquely in the buggy execution (deleted nodes). We denote them as the potential buggy functions S = {deleted nodes}. Additionally, we incorporate knowledge from the issue description and the reproduction phase. Specifically, functions explicitly mentioned in the issue report or highlighted by the reproduction agent are often highly relevant to the bug. If such functions are not included in S, we add them to ensure they are included in downstream analysis. Embedding Similarity Ranker. To improve efficiency and prioritize the most suspicious candidate functions, we rank functions in S by their similarity to the issue description using UniXcoder (Guo et al. 2022). We encode the issue description, each function body, and each function name with the UniXcoder tokenizer and encoder, then obtain a single vector for each by mean-pooling the last hidden state and applying L2 normalization. We compute cosine similarities between the issue vector and the vectors of the function body and name to derive a relevance score, sort S by this score to obtain S ′ , and retain the top K functions for subsequent steps. 4.4 Patch Generation & Validation After localizing the potential buggy functions, XAgent proceeds to the final phase of patch generation and validation. To help the agent more thoroughly evaluate the generated patches, we extend the validation beyond the two reproduction scripts by generating additional validation tests as motivated in Section 3.2. To this end, we introduce the context-aware validation augmentor in XAgent, which generates validation tests based on the program context beyond the code context described in the issue report. By doing so, they help ensure that the patch agent not only fixes the reported bug but also maintains overall functionality and correctness across related components. Context-aware Validation Augmentor. To generate additional validation tests, the validation augmentor leverages program context extracted directly from the reproduction agent’s trajectory.

Huynh et al.

Broadly speaking, an agent trajectory is a structured record of the agent’s interaction with the environment, consisting of iterative thought–action– observation tuples (e.g., reasoning traces, tool invocations, and returned outputs). For our reproduction agent, the trajectory captures how the agent navigates the repository, inspects code snippets, and how different files and functions relate to the issue description. This process surfaces a filtered set of context—i.e., files and functions already deemed relevant by the agent’s problem-solving process—providing a strong foundation for validation. We then leverage this trajectory as task-specific context to generate additional validation tests through two steps. First, we prompt an LLM to analyze the recorded trajectory to extract program context—specifically, the files and functions the reproduction agent found relevant to the issue description. Then, we prompt an LLM to generate an additional validation script based on (1) the extracted program context, (2) the original reproduction scripts, and (3) the issue description. As all relevant context has been obtained, there is no need to use tools to extract additional program context, so we use zero-shot prompting rather than an iterative, tool-enabled agent. We reuse the context gathered during reproduction rather than letting the validation augmentor independently navigate the codebase, which eliminates redundant exploration and substantially reduces both token consumption and computational cost. Patch Generation. Finally, XAgent generates the patch based on the ranked list of functions S ′ identified by our execution-guided localizer and the issue description based on the following three steps: First, the patch agent executes our five validation tests (i.e., three context-aware validation and two reproduction scripts) and existing regression tests to ensure that the environment is correctly set up and that the bug is reproducible. Second, the agent is guided to iterate through each ranked function in S ′ . For each function, the agent determines whether it is related to the issue. If so, it attempts to generate a candidate patch; otherwise, it moves on to the next function in S ′ . Third, once a patch candidate is generated, the agent reruns our five validation tests to check that the fix resolves the issue as well as the regression tests to confirm that the fix does not adversely affect existing functionality. If the patch fails to address the issue, the agent receives feedback from the environment and attempts to regenerate the patch. This process continues until either the generated patch successfully passes the validation and regression tests or a predefined limit is reached.

5 Experimental Setup 5.1 Research Questions To evaluate XAgent, we formulate the following three RQs: (RQ1) Effectiveness and Efficiency: How accurate and cost-efficient is XAgent in bug localization and issue resolution?

XAgent: eXecution-guided Agentic AI for GitHub Issues

(RQ2) Ablation Study: How does each key component of XAgent contribute to its overall performance? (RQ3) Resolve Rate vs. Issue Complexity: How does the resolve rate change across different levels of issue complexity?

5.2 Benchmark To address our research questions, we adopt the widely used benchmark SWEbench-Lite (Jimenez et al. 2024), specifically designed to evaluate the ability of LLMs to resolve real-world software engineering tasks at the repository level. Unlike traditional code generation benchmarks such as HumanEval (Chen et al. 2021) or MBPP (Austin et al. 2021), which focus on isolated functions, issues in SWE-bench-lite involve repository-level challenges that require understanding of code dependencies across multiple files and comprehensive reasoning about software behavior. This benchmark provides a publicly available leaderboard, enabling direct comparison with other state-of-the-art methods. We select this dataset to ensure consistency with prior studies (Yu et al. 2025; Yang et al. 2024a; Jiang et al. 2025; Mu et al. 2025) and to maintain computational efficiency, as it already includes standardized evaluation results. SWE-bench-Lite consists of 300 real GitHub issues (including bug reports and feature requests) drawn from 12 popular open-source Python repositories. Each instance includes an issue description and the corresponding repository, where the goal is to modify the relevant source code to resolve the issue without prior knowledge of the issue location.

5.3 Experimental Setup In this experiment, we used Claude Sonnet 4 as the backbone LLM because of its strong coding capabilities and competitive performance on SWE-bench tasks. This is also consistent with the top method (i.e., ExpeRepair), making our choice consistent with prior SOTA methods and ensuring fairness in comparison. We limit the number of suspicious functions to K = 20, as this threshold covers 80% of cases in our experiments (Table 3) while remaining manageable for the LLM to search. As the reproduction and patch agents autonomously perform actions (i.e., view and edit files, execute terminal commands, and inspect diffs after modifications), we set the action limits to control the experimental costs. Particularly, action limits are 100 for the reproduction agent and 200 for the patch agent. For the validation augmentor, we set the temperature parameter to 0.4, 0.6, and 0.8 to generate three varied validation tests to increase the variation in generation. For the other tasks, we configured the temperature parameter to match their task requirements: 0 for reproduction to ensure consistency and adherence to the issue description, and 0.5 for patch generation to maintain a balance between creative problem-solving and code accuracy (Renze 2024).

Huynh et al.

5.4 Baselines We select nine methods representing the top-performing methods on the SWEbench-lite leaderboard: ExpeRepair (Mu et al. 2025) (Claude 4 Sonnet), Refact Agent (Refact.ai 2025), SWE-Agent (Yang et al. 2024a) (Claude 4 Sonnet), DARS (Aggarwal et al. 2025), KGCompass (Yang et al. 2025), CodeFuse (Tao et al. 2025), OpenHands (Wang et al. 2024b), Composio (com 2026-01-28), and OrcaLoca (Yu et al. 2025). These methods hold the highest rankings on the leaderboard. Each method has officially submitted its results, and we reuse their public leaderboard submissions for comparison.

6 Experimental Results 6.1 RQ1: How accurate and cost-efficient is XAgent in bug localization and issue resolution? Approach. To address this RQ, we compare XAgent against 9 state-ofthe-art baselines introduced in Section 5.4 on 300 test instances from the SWE-bench-lite benchmark. To evaluate the bug localization of our XAgent, we use Precision, Recall, and F1-score, which provide a fine-grained assessment of how accurately an approach identifies true buggy locations while penalizing unnecessary edits. Specifically, let S denote the set of files or functions modified in the groundtruth patch, and S ′ those modified by the approach. We define T P = |S ′ ∩ S|, F P = |S ′ \ S|, and F N = |S \ S ′ |. The corresponding metrics are computed P TP 2P R as P = T PT+F P , R = T P +F N , and F 1 = P +R . These metrics are applied at both the file and function levels. An F1 score of 1 indicates that the predicted buggy files (or functions) exactly match those modified in the golden patch, reflecting a perfect bug localization. This formulation addresses limitations of prior evaluations (Xia et al. 2025; Yu et al. 2025), which rely on a superset match criterion that deems a localization correct if all ground-truth locations are edited, even when many irrelevant files or functions are also modified. Such a criterion can overestimate performance by rewarding high recall despite low precision, or penalize minimal yet correct fixes that omit unnecessary edits. In contrast, our formulation explicitly captures the trade-off between localization precision and edit redundancy. To evaluate the issue resolution of our XAgent, we follow prior works (Yang et al. 2024a; Zhang et al. 2024; Mu et al. 2025) and use the %Resolved metric, which measures the proportion of GitHub issues that an approach can successfully resolve. Specifically, a software issue is considered resolved if the generated patch (i) can be correctly applied to the target codebase, and (ii) passes all associated unit tests, especially the fail-to-pass tests that verify the original defect has been fixed. These unit tests are written by human developers to ensure correctness and are kept hidden from the model to prevent test data leakage.

XAgent: eXecution-guided Agentic AI for GitHub Issues

(a) Resolved Instances

(b) Perfect Localized Instances

Fig. 4: Overlapping Analysis. Table 1: Performance on SWE-bench-lite dataset evaluation. The best results for each metric are bolded. File

Method

%Resolve Cost Token

XAgent ( -4)

62.0 (186)

1.56

426.5k 85.1

60.3 (181) 60.0 (180) 56.7 (170) 46.0 (138) 44.0 (132) 41.7 (125) 41.0 (123) 41.0 (123) 39.0 (117)

2.49 1.18 0.20 1.77 -

713.3k 379.3k -

ExpeRepair ( -4&o4m) Refact ( -3.7&o4m) SWE-Agent ( -4) KGCompass ( -3.5) CodeFuse (CGM) OpenHands ( -3.5) Composio ( -3.5&o1) OrcaLoca ( -3.5) Moatless ( -3.5)

: Claude Sonnet models. o4m: GPT-o4-mini.

Function

89.7 86.6 72.6

84.7 88.0 78.3 93.6 78.6 89.6 79.0 79.0 60.7 63.3 45.4 82.6 78.4 80.3 80.6 80.7 59.0 82.4

85.7 83.0 81.5 79.0 61.6 54.4 79.0 80.6 66.3

76.9 72.8

70.3 74.8 64.6 77.9 67.6 76.5 65.5 62.3 28.0 28.5 36.3 62.8 62.5 63.3 66.9 64.0 46.3 63.2

70.7 67.7 69.0 63.3 27.7 42.9 61.8 64.8 51.2

: Rank 1–4.

To evaluate the cost-efficiency of our XAgent for both bug localization and issue resolution tasks, we measure both LLM API cost and token usage. For baseline methods, we report the total monetary cost (in USD) as stated in their original papers; when such information is unavailable, it is marked as “–”. Token usage includes all input and output tokens exchanged with the LLM, including system prompts, user inputs, model responses, and tool calls. For baselines, token counts are recomputed from their released trajectories to ensure fair comparison. Results. Table 1 presents the bug localization and issue resolution results of XAgent and 9 other methods on the SWE-bench-lite benchmark across different metrics. Our XAgent achieves the best overall performance across both tasks, with file- and function-level localization F1 scores of 86.6% and 72.8%, and the highest issue resolution rate of 62%. In particular, XAgent reduces token usage by nearly 40% compared to the prior state of the art, resulting in an overall cost reduction of approximately $300 across all instances. This improvement is primarily due to avoiding LLM calls during localization and eliminating the need for generating and selecting multiple

Huynh et al.

candidate patches, which together save about $1 per instance. These results demonstrate that XAgent achieves higher repair accuracy while being substantially more cost-efficient than the existing best approach. In terms of bug localization, XAgent achieves the best performance across all key metrics, with file-level Precision and F1 scores of 85.1% and 86.6%, and function-level Precision and F1 scores of 72.6% and 72.8%, respectively. While Refact Agent achieves the highest recall at both file and function levels, it achieves substantially lower precision, indicating a tendency to modify many non-buggy locations. This behavior reflects a common pattern across baseline methods, which exhibit higher recall than precision, suggesting that they often over-approximate the buggy region by editing more files or functions than necessary. In contrast, XAgent achieves a more balanced trade-off, maintaining high recall while significantly improving precision, thereby reducing redundant edits. Fig 4(b) presents an overlap analysis between our XAgent and the three top-performing baselines—ExpeRepair, Refact Agent, and SWE-Agent—based on perfect localization, defined as instances achieving an F1 score of 100%. Our XAgent correctly localizes the largest number of instances (181) and uniquely identifies the correct buggy locations in 11 cases that none of the other methods handle correctly. In comparison, Refact Agent, ExpeRepair, and SWE-Agent achieve 10, 10, and 1 unique instances, respectively. We further analyzed these 11 cases and found that most successes (7/11) stem from our differential analysis, which effectively captures behavioral differences between failing and passing executions. The remaining cases benefit from our XAgent’s exploratory reasoning during the reproduction phase. Although Refact Agent identifies 10 unique buggy locations, it achieves the lowest total number of perfectly localized instances (142) among the top methods, largely due to its tendency to modify more files or functions than necessary. Overall, these results demonstrate that XAgent provides more accurate and reliable bug localization at both the file and function levels than existing approaches. The improved localization accuracy also translates into better issue resolution performance. As a result, our XAgent resolves 186 issues, outperforming ExpeRepair, Refact Agent, and SWE-Agent by 5, 6, and 16 instances, respectively. Fig 4(a) presents an overlap analysis between our XAgent and the three top-performing baselines on issue resolution. We observed that other baselines often fail to localize the bug at the function level or generate fixes that only address the symptom described in the issue report. In contrast, human developers typically identify underlying causes and implement solutions that generalize beyond specific reported issues. A representative example that only XAgent is able to fix correctly is matplotlib-24265, which describes a behavioral bug in the Matplotlib repo, where attempting to load the seaborn-colorblind style using plt.style. library raises a KeyError. This style has been deprecated since version 3.6.1; however, instead of raising a deprecation warning, the program crashes. The correct fix involved renaming the style file to seaborn-v0 8-colorblind

XAgent: eXecution-guided Agentic AI for GitHub Issues

and ensuring that accessing the old seaborn-colorblind key now raises a Matplotlib DeprecationWarning rather than an error. To validate the fix, developers added a unit test to confirm that (1) the new seaborn-v0 8-colorblind style loads successfully, and (2) accessing the deprecated seaborn-colorblind key triggers the appropriate warning. However, the three baseline methods generate patches that address only the former, neglecting the required warning behavior and thus producing incorrect fixes. In contrast, XAgent begins patch generation by executing both reproduction and validation tests to understand how the program behaves in failure cases and in expected usage scenarios before generating any patch. During validation, XAgent tests both plt.style.library[\seaborn-colorblind"] and plt.style.use(\seaborn-colorblind"), an alternative access path for the same style. Through this process, XAgent observes that the former raises a KeyError, whereas the latter executes successfully and emits a deprecation warning. This discrepancy guides XAgent to inspect the underlying code paths, revealing that the warning logic is already implemented in plt.style.use(). XAgent therefore applies the same to plt.style.library[], resulting in a correct patch that passes all unit tests. This example illustrates how XAgent’s strategy of deriving diverse validation tests from reproduction cases enables it to uncover subtle behavioral inconsistencies and generate correct fixes that resolve the root cause. Overall, these results show that XAgent provides more accurate and reliable issue resolution by enhancing the coverage of the bug instead of addressing only the specific symptoms described in the issue report. Answer to RQ1. XAgent achieves the best overall performance on SWEbench-Lite, with the highest bug localization accuracy and issue resolution rate, while reducing both monetary cost and token usage compared to the prior top-performing method (i.e., ExpeRepair). This demonstrates that XAgent provides accurate, efficient, and comprehensive issue resolution.

6.2 RQ2: How does each key component of XAgent contribute to its overall performance? Approach. To answer this RQ, we investigate the two key components in our XAgent: execution-guided bug localizer and context-aware validation test augmentor. Referring to the workflow illustrated in Fig 3, we introduce two variants of XAgent: – Vanilla Agentic LLM : A baseline agentic approach without the use of our execution-guided bug localizer ( 2 and 3 ) and context-aware validation test augmentor ( 4 ). This variant follows the same design as the prior agentic approach, SWE-Agent, employing a single LLM agent to reproduce the issue, localize the bug solely based on the information provided in the issue description, and generate a patch accordingly. The generated patch

Huynh et al.

Table 2: Contribution of Key Components. Method Vanilla Agentic LLM + Bug Loc. + Bug Loc. + Val. Test (XAgent)

%Resolve (∆) 56.7 58.0 (↑ 1.3) 62.0 (↑ 5.3)

Func. F1 (∆) 69.0 73.5 (↑ 4.5) 72.8 (↑ 3.8)

is then validated using the existing test suite, without any additional test augmentation. – Vanilla Agentic + Execution-guided Bug Localizer : This variant extends the Vanilla Agentic LLM by incorporating the execution-guided bug localizer ( 2 and 3 ). Then, the LLM agent generates a patch, which is subsequently validated using the reproduction and regression tests without any additional test augmentation. This variant isolates and quantifies the contribution of the execution-guided bug localization component. – Vanilla Agentic + Execution-guided Bug Localizer + Validation Test Augmentor (XAgent): This variant further incorporates the contextaware validation test augmentor ( 4 ) to form our XAgent approach. After identifying suspicious functions, the LLM agent generates additional validation tests based on the context of the localized files and functions. It then generates a patch and validates it against both the original and the augmented tests. This variant quantifies the additional benefit of validation test augmentation beyond execution-guided localization. In addition, we conduct an ablation on the Embedding Similarity Ranker ( 3 ) by altering the ranking method: – Embedding Similarity Ranker : This ranker is used by XAgent to prioritize candidate functions based on their semantic similarity to the issue description using UniXcoder and compute cosine similarity between their embeddings. Functions are then ranked according to this similarity score. – Commit History Ranker : Prior work (Hata et al. 2012; Hoang et al. 2020) argued that functions modified more frequently in recent history are more likely to contain bugs. Hence, we rank the candidate functions based on their change frequency over the past year, producing an ordered list used for subsequent localization. – Proximity-based Ranker : Inspired by OrcaLoca (Yu et al. 2025), this ranker leverages simple proximity-based signals derived from execution traces. Specifically, it prioritizes functions that reside in the same file as suspicious keywords, appear along descendant paths in the trace, or are closer to the root of the execution tree. These signals are combined to score and rank candidate functions for bug localization. We assess ranking quality using a Top-K metric, which measures whether the ground-truth buggy function appears within the top K ∈ {10, 20} ranked candidates. Results. Table 2 presents the results of XAgent and its two variants. When incorporating our execution-guided bug localizer into the vanilla agentic

XAgent: eXecution-guided Agentic AI for GitHub Issues

Table 3: Ablation of Ranking Methods. Ranker Embed. Sim. (XAgent) Commit History Proximity-based

Top-10 75.7 73.3 74.3

Top-20 79.0 73.3 76.3

LLM, we observe a clear improvement in F1-score. Specifically, our bug localizer increases the function-level F1 score by 4.5%, achieving the highest Function Localization F1 of 73.5%. In addition, the %Resolved metric improves by 1.3%, corresponding to four more successfully resolved issues compared to the vanilla agentic LLM variant. We also observe that, with our execution-guided bug localizer, the agent uses 13% fewer tokens than the vanilla agentic LLM. Consequently, LLM usage and associated costs are reduced. These results demonstrate the effectiveness of the execution-guided bug localizer in accurately identifying buggy functions. For agentic LLM + bug localizer + validation test augmentor (XAgent), we observed the best overall performance, achieving a %Resolved score of 62.0%. Compared to Agentic LLM + Bug Localizer , this represents an improvement from 58.0% to 62.0%, corresponding to 12 additional resolved issues attributable to the introduction of our validation test augmentor. The slight decrease of function-level F1 score (by 0.7%) is due to the inclusion of additional validation tests that encourage broader code modifications, mildly reducing precision. Overall, these results demonstrate that our validation test augmentor plays a critical role in improving issue resolution by evaluating patches against a richer set of test cases, enabling more generalizable fixes. In terms of ranking strategies, Table 3 presents the Top-10 and Top-20 localization performance of the three ranking methods: Embedding Similarity Ranker , Commit History Ranker , and Proximity-based Ranker . The embedding similarity ranker used in XAgent achieves the highest Top-10 and Top-20 accuracy. Despite the common intuition of past tendency (Hata et al. 2012; Hoang et al. 2020), the commit history ranker often fails when issues originate from older or long-stable code. As a result, it cannot reliably capture the semantic intent or reasoning reflected in issue descriptions. The proximitybased ranker, on the other hand, prioritizes functions closer to the root of the execution trace. However, in many cases, the true fault lies deeper in the call chain, causing this proximity-based heuristic to misrank relevant functions. Overall, these results indicate that semantic similarity between candidate functions and the issue description provides a reliable signal for ranking functions in XAgent.

Huynh et al.

1.0

XAgent ExpeRepair Refact Agent SWE-Agent

0.8 0.6

(a) Code Hunks

(b) Changed Functions

(80 ) 11 >

11 2) 0( 5-1

[25] 1-4 (63) (10 8)

0) (23 [11]

un ks [0- (37) 0] (7)

s( un k 2h

(19 un k 1h

73 )

0.4 0)

Resolve Rate

Fig. 5: Resolve Rate across different complexity metrics.

Answer to RQ2. Our ablation study shows that each component of XAgent contributes to performance improvement. Execution-guided localization improves localization accuracy and reduces cost. Validation test augmentor increases issue resolution rates and semantic similarity provides effective ranking. These components collectively enable XAgent to achieve strong performance in both accuracy and efficiency for automated issue resolution.

6.3 RQ3: How does the resolve rate change across different levels of issue complexity? Approach. To address this RQ, we analyze the issue resolution capability of our XAgent approach and the three top-performing baselines (ExpeRepair, Refact Agent, and SWE-Agent) with different levels of issue complexity. Following prior work that characterizes bug complexity by the structure and effort required for repair (Böhme and Roychoudhury 2014; Xin et al. 2024), we define issue complexity by the amount of human effort needed to fix the issues, using three metrics: (1) how many code hunks are produced in the ground-truth (human-written) patch (#Hunks), (2) how many functions are modified (#Functions), and (3) how many lines are removed and added (#Lines). Larger values in these metrics indicate higher issue complexity, as they reflect greater human effort required for fixing the bug. We then report the %Resolved rate across different levels of issue complexity to analyze how resolution performance varies as complexity increases. Results. Fig 5 presents the relationship between the %Resolve and the complexity of instances on the three metrics (#Hunks, #Functions, and #Lines). Fig 5(a) illustrates the relationship between %Resolve and the number of hunks (#Hunks) in patches. The majority of instances (190 out of 300) contain a single hunk, followed by 73 instances with two hunks and 37 instances with three hunks. All evaluated methods perform best on single-hunk cases, where our XAgent achieves the highest %Resolved rate at 70%, while SWEAgent performs the worst at 64%. For those instances requiring two hunks,

XAgent: eXecution-guided Agentic AI for GitHub Issues

Refact Agent attains the highest resolution rate at 54%, followed closely by our XAgent at 51%, with SWE-Agent again ranking lowest at 47%. For more complex cases involving three hunks, our XAgent again achieves the best performance with a %Resolved rate of 44%, whereas both ExpeRepair and Refact Agent drop to 35%. These results confirm that XAgent remains effective when resolving issues that require changes across multiple code hunks. Fig 5(b) presents the relationship between the %Resolved and the number of modified functions (#Functions). We categorize the 300 instances into three groups: no function modified, one function modified, and two to five functions modified. Seven instances involve no function-level changes, as the fixes affect only global configurations (e.g., imports or global variables); all four studied methods successfully resolve these cases. Among the 230 instances requiring changes to a single function, our XAgent approach achieves the highest %Resolved at 67%, followed by Refact Agent, ExpeRepair, and SWE-Agent. For the remaining 63 instances that involve modifications to two to five functions, our XAgent approach again achieves the highest %Resolved rate at 41%, tying with Refact Agent and outperforming ExpeRepair and SWE-Agent. These results confirm that XAgent remains effective when resolving issues that require modifications across multiple functions. Finally, Fig 5(c) illustrates the relationship between the %Resolved and the number of changed lines (#Lines). We found that as the number of modified lines increases, the %Resolved steadily decreases. Across all ranges of line changes. Nevertheless, our XAgent approach consistently achieves the highest resolution performance. These results demonstrate that XAgent scales effectively to fixes requiring changes to a larger number of lines of code. Answer to RQ3. As issue complexity increases, issue resolution performance generally declines across all methods. However, in the most complex cases—larger numbers of hunks, modified functions, and lines—XAgent consistently achieves the highest resolve rate among the evaluated approaches, indicating stronger effectiveness in challenging issue resolution.

7 Discussion In this section, we further discuss the performance of XAgent.

7.1 How much search space can be reduced by our execution-guided bug localizer? As discussed in Section 3.1, comparing buggy and non-buggy executions could reduce the search space of buggy functions. Hence, we further analyze the ability of XAgent in reducing the function search space. Table 4 presents the size

Huynh et al.

Table 4: Number of functions per issue. Strategy The whole repo buggy scenario only ∆ buggy & non-buggy (ours)

Percentage

46.0%

41.0%

40 10.0%

Max 33,618 1,049 637 (↓39%)

62.0%

56.7%

Min 702 1 1

9.7%

8.3%

Avg 20,769 234 52 (↓78%)

Med 5,946 128 11 (↓91%)

% Resolved % Exact Match 40.7% 39.0%

8.3%

7.3% 6.9% 0 nt ss ca XAgent Agentless Moatless KGCompa SWEAge OrcaLo Fig. 6: Similarity of generated patches to human-written patches. of the search space based on 3 strategies: the whole repository, the functions that are executed in the buggy scenario, and the functions that uniquely executed in the buggy scenario, but not in the non-buggy scenario. By searching the whole repository, the search space for identifying buggy functions can span a maximum of 33,618 functions (20,000 functions on average). If we consider only functions executed in the buggy scenario, the search space reduces to an average of 234 functions. However, 234 functions still represent a substantial context length for an LLM to analyze effectively. Our execution-guided bug localizer further refines this by identifying functions that are uniquely executed in the buggy scenario. This reduces the search space to an average of only 52 functions and a median of just 11 functions. This highlights the significant reduction of search space by execution-guided bug localizer. 7.2 How similar are XAgent’s patches to human-written patches? As we use Claude Sonnet 4 as our LLM backbone, we investigate whether XAgent’s performance stems from architectural innovation or potential memorization of ground-truth patches. Similar to prior work (Yang et al. 2024b), we analyze the exact match rate between XAgent-generated patches and the golden (human-written) patches. The exact match rate measures the percentage of instances where the generated patch is identical to the ground truth patch (ignore comments and empty lines). Fig 6 presents the %Resolved and %Exact match for XAgent and five baseline methods (Yang et al. 2025; Yu et al. 2025; Yang et al. 2024a; Xia et al. 2025; Antoniades et al. 2025). XAgent maintains an exact match rate of only 8.3%, lower than KGCompass, SWE-Agent, and OrcaLoca. Upon manual inspection of the exact match cases, we found that most of them involve simple edits of only 1-2 lines of code. These simple cases naturally have fewer solution alternatives, making exact matches

XAgent: eXecution-guided Agentic AI for GitHub Issues

more likely regardless of the approach used. For more complex bugs requiring multi-line or multi-function change, XAgent consistently generates alternative solutions that differ from human-written patches but achieve functional correctness. These findings indicate that XAgent does not rely on memorizing ground-truth patches, but instead produces functionally correct solutions, particularly for complex bugs with multiple valid fixes.

7.3 How long does XAgent take to resolve an issue? Execution time is an important factor for understanding the practical behavior of our agentic issue resolution approach. Our new components (i.e., the executed-guided bug localizer and context-aware validation test) may cost additional computation time. On average, XAgent takes approximately 8.8 minutes to resolve an issue. The workflow is dominated by two agentic phases: reproduction (181.8s, 34.5%) and patch generation (240.8s, 45.6%), which together account for 80.1% of the total time. This behavior is expected for agentic issue resolution approaches, as these phases involve iterative LLM interactions with tools and execution environments. In contrast, our execution-guided localization component (including execution tracing, call tree comparison, and function ranking) collectively consumes only 49 seconds (9.3% of total time). This demonstrates that our localization approach is highly efficient, adding minimal overhead while providing accuracy improvements. The test augmentation phase takes 56 seconds (10.6%), which is reasonable given that it generates multiple validation tests to ensure comprehensive patch evaluation. These results indicate that both the execution-guided bug localizer and the context-aware validation test augmentor in XAgent are computationally efficient, contributing meaningful performance improvements without significantly increasing overall runtime.

8 Limitations SWE-Bench-lite may contain under-specified issues and flaky tests. While SWE-Bench-verified offers manually-verified 500 instances, most baselines use SWE-Bench-lite. Hence, we focus on SWE-Bench-lite for a fair comparison in our experiment (ver 2026-01-28). Nonetheless, these two datasets share an overlapping subset of 93 instances. Hence, we examine the generated patches of our XAgent and TRAE (Team et al. 2025) (i.e., the state-of-the-art agent on the SWE-bench-verified leaderboard). We find that XAgent and TRAE both achieve the same %Resolve of 82% ( 76 93 in this subset.) Notably, TRAE is an ensemble method that combines the reasoning of multiple models (Claude Sonnet 4, Opus 4, Sonnet 3.7, and Gemini 2.5 Pro), which may require a higher

Huynh et al.

cost than ours.1 This analysis provides evidence of comparable performance of our XAgent between the two datasets. The effectiveness of XAgent relies on the quality of its agent-generated reproduction scripts. While our agent successfully generates buggy and nonbuggy scripts pairs for 93% of evaluated issues. The failure may come from under-specified issues or feature requests. The remaining 7% must fall back to LLM localization. Furthermore, while techniques like spectrum-based fault localization (Jones et al. 2002; Abreu et al. 2007) are more thorough, they often require comprehensive test suites that are rarely available for newly reported bugs (Zhang et al. 2024). Therefore, XAgent can only leverage existing available information in the code to generate reproduction scripts. LLMs may exhibit inherent randomness in their outputs, particularly at higher temperature settings, which can affect reproducibility. Following prior work (Aggarwal et al. 2025; Mu et al. 2025; Yang et al. 2025), we use low temperature values of 0 and 0.5 for reproduction and patch generation to ensure reproducibility. Nonetheless, we use a high temperature of 0.8 for the validation augmentor to generate more diverse validation tests. As LLMs may exhibit inherent randomness in this setting, we assess result consistency by repeating the validation generation three times at a temperature of 0.8. Based on cosine similarity over text embeddings generated by OpenAI’s text-embedding-3-large model, the generated validations show an average pairwise similarity of 91.2% (SD = 0.038) across the runs. These results indicate that, despite the higher temperature, the model still produces reasonably consistent outputs. To further support reproducibility, we make our complete implementation and experimental setup publicly available at (xag 2026-01-28).

9 Conclusion This work introduces XAgent, an execution-guided agentic framework for GitHub issue resolution that advances automated software issue resolution by integrating dynamic program execution with context-aware test generation for patch validation. By comparing failing and successful executions, XAgent localizes the exact buggy function accurately. In addition, the context-aware test augmentation enhances validation coverage and reduces overfitting to issue descriptions. Our empirical evaluation on the SWE-bench-lite dataset shows that XAgent achieves a resolve rate of 62.0% and a function localization accuracy of 72.8%, outperforming SOTA approaches (e.g., ExpeRepair, Refact Agent, and SWE-Agent), while maintaining a 37% lower cost. These findings demonstrate how moving beyond static descriptions toward execution-guided reasoning enables LLM-based agents to deliver more reliable and workflow-aligned automated software maintenance. 1 Unfortunately, TRAE did not provide the actual cost of generation. Hence, we cannot compare the cost.

XAgent: eXecution-guided Agentic AI for GitHub Issues

10 Data Availability The source code and experimental details are publicly available at an anonymous GitHub repository (xag 2026-01-28).

Declarations Funding No funding was received to assist with the preparation of this manuscript. Competing interests The authors have no competing interests to declare that are relevant to the content of this article. Ethics approval Not applicable.

References (2026-01-28) Composio. URL https://github.com/ComposioHQ/composio (2026-01-28) Swe-bench-verified. URL https://openai.com/index/ introducing-swe-bench-verified/ (2026-01-28) Xagent. URL https://github.com/xagent-se/XAgent Abreu R, Zoeteweij P, Van Gemund AJ (2007) On the accuracy of spectrum-based fault localization. In: Testing: Academic and industrial conference practice and research techniques-MUTATION (TAICPART-MUTATION 2007), IEEE, pp 89–98 Aggarwal V, Kamal O, Japesh A, Jin Z, Schölkopf B (2025) Dars: Dynamic action resampling to enhance coding agent performance by adaptive tree traversal. arXiv preprint arXiv:250314269 Antoniades A, Örwall A, Zhang K, Xie Y, Goyal A, Wang WY (2025) SWE-search: Enhancing software agents with monte carlo tree search and iterative refinement. In: The Thirteenth International Conference on Learning Representations, URL https: //openreview.net/forum?id=G7sIFXugTX Austin J, Odena A, Nye M, Bosma M, Michalewski H, Dohan D, Jiang E, Cai C, Terry M, Le Q, et al. (2021) Program synthesis with large language models. arXiv preprint arXiv:210807732 Böhme M, Roychoudhury A (2014) Corebench: Studying complexity of regression errors. In: Proceedings of the 2014 international symposium on software testing and analysis, pp 105–115 Chakraborty P, Alfadel M, Nagappan M (2025) Blaze: Cross-language and cross-project bug localization via dynamic chunking and hard example learning. IEEE Transactions on Software Engineering Chen M, Tworek J, Jun H, Yuan Q, de Oliveira Pinto HP, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G, Ray A, Puri R, Krueger G, Petrov M, Khlaaf H, Sastry G, Mishkin P, Chan B, Gray S, Ryder N, Pavlov M, Power A, Kaiser L, Bavarian M, Winter C, Tillet P, Such FP, Cummings D, Plappert M, Chantzis F, Barnes E, Herbert-Voss A, Guss WH, Nichol A, Paino A, Tezak N, Tang J, Babuschkin I, Balaji S, Jain S, Saunders W, Hesse C, Carr AN, Leike J, Achiam J, Misra V, Morikawa E, Radford A, Knight M, Brundage M, Murati M, Mayer K, Welinder P, McGrew B, Amodei D, McCandlish S, Sutskever I, Zaremba W (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2107.03374 Chen Z, Tang R, Deng G, Wu F, Wu J, Jiang Z, Prasanna V, Cohan A, Wang X (2025) Locagent: Graph-guided llm agents for code localization. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 8697–8727

Huynh et al.

Guo D, Lu S, Duan N, Wang Y, Zhou M, Yin J (2022) Unixcoder: Unified cross-modal pretraining for code representation. URL https://arxiv.org/abs/2203.03850, 2203.03850 Hata H, Mizuno O, Kikuno T (2012) Bug prediction based on fine-grained module histories. In: 2012 34th international conference on software engineering (ICSE), IEEE, pp 200– 210 Hoang T, Kang HJ, Lo D, Lawall J (2020) Cc2vec: Distributed representations of code changes. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, pp 518–529 Jiang Z, Ren X, Yan M, Jiang W, Li Y, Liu Z (2025) Cosil: Software issue localization via llm-driven code repository graph searching. arXiv preprint arXiv:250322424 Jimenez CE, Yang J, Wettig A, Yao S, Pei K, Press O, Narasimhan K (2024) Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, URL https://arxiv.org/abs/2310.06770, 2310.06770 Jones JA, Harrold MJ, Stasko J (2002) Visualization of test information to assist fault localization. In: Proceedings of the 24th international conference on Software engineering, pp 467–477 Kang S, Yoon J, Yoo S (2023) Large language models are few-shot testers: Exploring llmbased general bug reproduction. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, pp 2312–2323 Le XBD, Thung F, Lo D, Le Goues C (2018) Overfitting in semantics-based automated program repair. In: Proceedings of the 40th International Conference on Software Engineering, Association for Computing Machinery, New York, NY, USA, ICSE ’18, p 163, DOI 10.1145/3180155.3182536, URL https://doi.org/10.1145/3180155.3182536 Li H, Tang Y, Wang S, Guo W (2025) Patchpilot: A cost-efficient software engineering agent with early attempts on formal verification. URL https://arxiv.org/abs/2502.02747, 2502.02747 Mu F, Wang J, Shi L, Wang S, Li S, Wang Q (2025) Experepair: Dual-memory enhanced llm-based repository-level program repair. URL https://arxiv.org/abs/2506.10484, 2506.10484 Nashid N, Bouzenia I, Pradel M, Mesbah A (2025) Issue2test: Generating reproducing test cases from issue reports. arXiv preprint arXiv:250316320 Pawlik M, Augsten N (2011) Rted: a robust algorithm for the tree edit distance. arXiv preprint arXiv:12010230 Pawlik M, Augsten N (2015) Efficient computation of the tree edit distance. ACM Transactions on Database Systems (TODS) 40(1):1–40 Pawlik M, Augsten N (2016) Tree edit distance: Robust and memory-efficient. Information Systems 56:157–173, DOI https://doi.org/10.1016/j.is.2015.08.004, URL https://www. sciencedirect.com/science/article/pii/S0306437915001611 Refactai (2025) Refact.ai: Ai coding agent for software development. https://refact.ai/, accessed: 2026-01-18 Renze M (2024) The effect of sampling temperature on problem solving in large language models. In: Findings of the association for computational linguistics: EMNLP 2024, pp 7346–7356 Smith EK, Barr ET, Le Goues C, Brun Y (2015) Is the cure worse than the disease? overfitting in automated program repair. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA, ESEC/FSE 2015, p 532–543, DOI 10.1145/2786805.2786825, URL https://doi.org/10.1145/2786805.2786825 Tao H, Zhang Y, Tang Z, Peng H, Zhu X, Liu B, Yang Y, Zhang Z, Xu Z, Zhang H, Zhu L, Wang R, Yu H, Li J, Di P (2025) Code graph model (cgm): A graph-integrated large language model for repository-level software engineering tasks. URL https://arxiv. org/abs/2505.16901, 2505.16901 Team TR, Gao P, Tian Z, Meng X, Wang X, Hu R, Xiao Y, Liu Y, Zhang Z, Chen J, Gao C, Lin Y, Xiong Y, Peng C, Liu X (2025) Trae agent: An llm-based agent for software engineering with test-time scaling. arXiv preprint arXiv:2507.23370, URL https://arxiv.org/abs/2507.23370, 2507.23370 Wang X, Gao P, Meng X, Peng C, Hu R, Lin Y, Gao C (2024a) Aegis: An agentbased framework for general bug reproduction from issue descriptions. arXiv preprint

XAgent: eXecution-guided Agentic AI for GitHub Issues

arXiv:241118015 Wang X, Li B, Song Y, Xu FF, Tang X, Zhuge M, Pan J, Song Y, Li B, Singh J, et al. (2024b) Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:240716741 Xia CS, Deng Y, Dunn S, Zhang L (2025) Demystifying llm-based software engineering agents. Proceedings of the ACM on Software Engineering 2(FSE):801–824 Xin Q, Wu H, Tang J, Liu X, Reiss SP, Xuan J (2024) Detecting, creating, repairing, and understanding indivisible multi-hunk bugs. Proceedings of the ACM on Software Engineering 1(FSE):2747–2770 Yang B, Ren J, Jin S, Liu Y, Liu F, Le B, Tian H (2025) Enhancing repository-level software repair via repository-aware knowledge graphs. arXiv preprint arXiv:250321710 Yang J, Jimenez CE, Wettig A, Lieret K, Yao S, Narasimhan K, Press O (2024a) Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37:50528–50652 Yang Z, Zhao Z, Wang C, Shi J, Kim D, Han D, Lo D (2024b) Unveiling memorization in code models. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp 1–13 Yu Z, Zhang H, Zhao Y, Huang H, Yao M, Ding K, Zhao J (2025) Orcaloca: An llm agent framework for software issue localization. arXiv preprint arXiv:250200350 Zeller A (2009) Why programs fail: a guide to systematic debugging. Morgan Kaufmann Zhang Y, Ruan H, Fan Z, Roychoudhury A (2024) Autocoderover: Autonomous program improvement. In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp 1592–1604

Record · ID 673601 · SHA-256 9899cce5a6c387e7
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.