SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents Niful Islam, Muhammad Anas Raza, Mohammad Wardat∗
arXiv:2604.17699v1 [cs.SE] 20 Apr 2026
Department of Computer Science and Engineering Oakland University Rochester, Michigan, USA {islam3,mraza,wardat}@oakland.edu
Abstract
1
Large Language Models (LLMs) have transformed software development and AI applications. While LLMs are designed for text processing, LLM agents extend this capability by enabling autonomous actions, tool use, and multi-step task completion. As this field grows, developers face new challenges in debugging these complex systems. To address this challenge, we present the first empirical study on bug fix patterns in LLM agents. We study buggy posts and code snippets from three platforms: Stack Overflow, GitHub, and HuggingFace Forums. We examine their fix patterns, the components where fixes are applied, and the programming languages and frameworks involved. Furthermore, we introduce AgentDefect, the first benchmark dataset for bugs in LLM agents. The dataset contains 37 runtime buggy instances along with fixed code and test files. Finally, we present SelfHeal, a multi-agent system designed to fix bugs in LLM agents. The system leverages two independent ReAct agents: the fix agent and the critic agent. These agents use tools that provide both internal knowledge (fix rules) and external knowledge (web search) to propose and validate fixes. Our evaluation shows that SelfHeal with Gemini 3 Pro as the backbone LLM outperforms both baseline and state-of-the-art approaches by a significant margin.
With improvements in Deep Learning (DL) and Large Language Models (LLMs), their applications span diverse disciplines, including healthcare [62], software engineering [60], and cybersecurity [39]. In software engineering, there is an increasing shift toward Agentic AI, in which autonomous agents perform complex tasks such as reasoning and tool manipulation [21]. This shift introduces unique hurdles that current engineering practices are still evolving to address, primarily due to the inherent non-determinism of these systems. AI agents are composed of multiple tightly coupled components, including reasoning modules, memory mechanisms, planning strategies, and external tool interfaces [21], where errors in any component can propagate across the entire execution pipeline, leading to cascading failures. This interdependence, combined with the black box and non-deterministic nature of LLMs, makes debugging substantially more challenging than traditional software, as faults are often non-local, temporally distributed, and difficult to reproduce. Furthermore, the ecosystem remains in a nascent stage. It is characterized by new frameworks with small communities and a lack of standardized taxonomies or comprehensive documentation. New frameworks related to agents and their versions are released every month. For instance, LangChain, the most popular library for constructing LLM agents, was released in late 2022 and has released over 45 new versions in 2025 and over 65 versions in 2024 [74]. LlamaIndex, the second most popular library, released over 60 versions in 2025 and over 140 versions in 2024 [22]. Each version introduces new features and changes. Therefore, developers often fail to adapt to the new changes and receive less community support since fewer developers are involved in the field. Prior research has established various taxonomies for software bugs, ranging from general software systems [19] to specific deep learning artifacts and attention-based mechanisms [37, 43]. While recent studies have begun to evaluate the performance of agents on coding benchmarks and trace their autonomous workflows [23, 76], these evaluations primarily focus on agent effectiveness rather than agent correctness. To date, no empirical study has systematically investigated fix patterns that can be used to fix bugs within agentic systems. Furthermore, although several frameworks have been proposed that employ agent-based architectures for automated bug fixing in conventional software systems [76] [81] [71], these approaches treat agents as reliable repair tools rather than faulty software artifacts. As a result, no dedicated framework exists for diagnosing and repairing bugs specific to agentic systems. Although initial efforts attempt to define agent defects by analyzing discrepancies between developer logic and generated content [55], the
CCS Concepts • Do Not Use This Code → Generate the Correct Terms for Your Paper; Generate the Correct Terms for Your Paper; Generate the Correct Terms for Your Paper; Generate the Correct Terms for Your Paper.
Keywords Large Language Model, Bug Fix, Fix Pattern, Agentic AI ACM Reference Format: Niful Islam, Muhammad Anas Raza, Mohammad Wardat. 2026. SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents. In Proceedings of 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026) (EASE). ACM, New York, NY, USA, 12 pages. https: //doi.org/XXXXXXX.XXXXXXX ∗ Corresponding author. Email: [email protected]
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE, Glasgow, United Kingdom © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX 1
Introduction
literature still lacks a comprehensive methodology for fixing realworld agent bugs reported by practitioners across platforms such as community forums, version control repositories, and model hubs. To address these gaps, we present the first comprehensive study on bug fix patterns in LLM agents. Following prior works [37, 40], we have collected posts and code snippets from Stack Overflow, GitHub commits, GitHub issues, and HuggingFace Forums, and analyzed the bug fix patterns within them. In addition to the fix pattern, our analysis also includes the following dimensions: the specific component of the LLM agent where each fix was applied, the programming languages and frameworks used, the rationale behind selecting each fix pattern, and the buggy and fixed code patches (where provided by users). When fixes involved framework version issues, we also documented the buggy and corrected version requirements. Secondly, we curated the first benchmark dataset on runtime bugs in LLM agents, AgentDefect, containing 37 executable buggy code samples. Each instance includes the buggy code, the user’s one-sentence intent, the corresponding fixed code, a test file, a readme file explaining execution instructions, and a requirements file with library versions. Additionally, we present SelfHeal, a comprehensive framework for analyzing and repairing bugs in agentic systems. SelfHeal employs two independent ReAct agents: a fix agent that generates repairs and a critic agent that validates them. Both agents are empowered by internal and external knowledge sources. Through careful design, SelfHeal outperforms both baseline and state-of-the-art (SoTA) approaches. Overall, the primary contributions of the research works are as follows:
Overflow posts. For GitHub and HuggingFace Forums, we applied the same process described in [40] and obtained 51 instances from GitHub issues, 129 from GitHub commits, and 85 from HuggingFace Forums with no duplicate entries. We studied a total of 930 unique buggy instances from the three platforms and analyzed their fix patterns.
2.2
We began the annotation process using fix patterns from existing taxonomies of software and data science bugs [20, 56]. After studying these taxonomies, we adopted fix patterns that were present in LLM agent-related issues, while patterns not observed in our dataset were excluded. We then annotated the fix patterns in our dataset and identified eight patterns not covered by prior studies. For these newly identified patterns, we derived labels using an open card sorting method, which we added for labeling. Definitions of all fix patterns are provided in Section 2.3. In addition to fix patterns, we annotated the component where the fixes are applied, following the four-component framework for LLM agents described by prior works [40, 55]. We also recorded the programming language and framework used to build each agent. For code-related issues, we documented the buggy and fixed source code along with output or error messages. For library version issues, we recorded the buggy and fixed requirements when the buggy version is specified. Each annotation included a one-line rationale explaining the fix pattern selection and cited external resources when solutions were found outside the post. Notably, to identify bug fixes and their patterns, we analyzed verified answers and replies from Stack Overflow and HuggingFace Forums, before-and-after code changes from GitHub commits, and associated pull requests for GitHub issues. When posts in Stack Overflow or HuggingFace Forums lacked verified answers, we searched for fixes from external sources and documented these external resources. For the data annotation, we followed previous literature [37] where two PhD students independently labeled fix patterns and components. We measured inter-annotator agreement using Cohen’s kappa coefficient after every 10% of the Stack Overflow dataset was annotated. When annotators disagreed, an expert assisted in reaching a consensus. Programming languages and frameworks were determined from Stack Overflow tags or GitHub repository metadata and annotated collaboratively. For the smaller GitHub and Huggingface datasets, agreement was measured after completing the full annotation. Final kappa scores for the Stack Overflow dataset were 0.983 for fix patterns and 0.808 for components. For GitHub commits, scores were 0.953 and 0.919; for GitHub issues, 1.0 and 0.812; and for HuggingFace forums, 0.973 and 0.973 for fix patterns and components, respectively. The complete table of kappa scores after each interval is provided in the supplementary material [42]. When multiple fix patterns were required to resolve a single bug, we created separate entries for each pattern. This resulted in 89 additional entries in the Stack Overflow dataset, 5 in the GitHub commits dataset, and 1 in the HuggingFace forums dataset. In total, it resulted in 1025 entries for all the datasets combined.
• We have conducted the first empirical study on bug fix patterns in LLM-based agents. We also provide the curated dataset with code patches and component-level annotations that can be used by future studies. • We present AgentDefect, the first benchmark dataset of runtime bugs in LLM agents. The dataset contains 37 executable instances, each with buggy and fixed code, user intent, test files, execution instructions, and versioned dependencies. • We present SelfHeal, a multi-agent system for fixing bugs in agentic systems. It combines a fix agent and a critic agent, both empowered by internal and external knowledge sources, to iteratively repair and validate buggy code. • Our proposed solution significantly outperforms baseline and SoTA approaches on the newly developed dataset.
2
Empirical Study
We followed previous studies on LLM agents [40], to collect and annotate the dataset for our empirical study. The data collection and annotation process is explained in the subsequent sections.
2.1
Data Annotation
Data Collection
Following prior studies [37, 40], we sourced our dataset from three platforms, namely Stack Overflow, GitHub (issues and commits), and HuggingFace Forums. For Stack Overflow we searched for posts containing keywords such as LangChain, LlamaIndex, Semantic Kernel, CrewAI, AutoGen, OpenAI, and Ollama and found 2,734 instances. We manually removed irrelevant posts (i.e., posts that do not discuss LLM agents), posts with insufficient descriptions, and posts without a determined solution that was either verified in the post or publicly available, and retained 665 unique annotated Stack
2.3
Description of Bug Fix Patterns
To systematically evaluate the repair capabilities of various agentic systems, we categorize observed fixes into a taxonomy of 23 distinct 2
patterns. Fix patterns documented in prior literature on traditional software systems were adopted and cited accordingly [20] [56], while patterns not identified in existing work were excluded from the taxonomy. The description of each fix pattern is listed below.
2.3.11 Change Reference (CR). Fixes incorrect references to internal modules, classes, or external namespaces. Such errors typically arise from refactoring, file relocation, or inconsistent naming and result in unresolved symbols at runtime or import time. Due to rapid developments in the field, modules often relocate to different packages. This requires developers to update their import statements (e.g., [12]). 2.3.12 Use Different Module (UDMo). Replaces an existing software component with an alternative module that provides equivalent functionality but improved compatibility, stability, or performance for the target task. This pattern reflects higher-level design substitution rather than localized fixes (e.g., [1]). 2.3.13 Change External Resources (CER). Reconfigures dependencies on external resources, including API endpoints, authentication credentials, environment variables, or remote services. These changes are not related to the agents (e.g., [8]).
2.3.1 Add New Attribute (ANA). Augments a function or method call by introducing explicit arguments that replace insufficient default values or satisfy previously unmet functional requirements. This pattern typically arises when implicit assumptions encoded in defaults fail under specific execution contexts, requiring the caller to pass configuration parameters directly [20] (e.g., [6]). 2.3.2 Remove Attribute (RA). Augments a function or method call by removing explicit arguments so the call uses default values or avoids unsupported parameters [20] (e.g., [30]). 2.3.3 Addition of Precondition Check (AOPC). Implements defensive programming constructs by inserting conditional checks that validate system state, environmental assumptions, or input invariants before execution proceeds. This prevents invalid states from propagating deeper into the program and reduces the likelihood of runtime failures [56] (e.g., [6]). 2.3.4 Change Version (CV). Modifies dependency or package version specifications to restore compatibility between interacting libraries or frameworks. This pattern addresses failures caused by deprecated APIs, breaking changes, or version-specific regressions that emerge during library evolution [61] (e.g., [28]). 2.3.5 Install Library (IL). Resolves execution failures originating from missing external dependencies by explicitly installing or declaring required libraries. This pattern reflects environment-level repairs rather than code-level logic changes and is common in deployment or reproduction scenarios [44] (e.g., [29]) 2.3.6 Addition of Operations (AOO). Introduces previously omitted functional steps into the execution pipeline, such as data loading, preprocessing, or intermediate transformation stages. In complex systems like RAG architectures, this often involves inserting embedding generation or retrieval components necessary for downstream tasks [56] (e.g., [3]). 2.3.7 Removal of Operations (ROO). Eliminates obsolete, redundant, or logically incorrect code segments that contribute unnecessary computation or interfere with correct execution. This pattern streamlines control flow and can improve both correctness and efficiency [56] (e.g., [2]). 2.3.8 Change Data Type (CDT). Corrects mismatches between expected and actual data types in variable assignments or function calls. By aligning data representations with API or language constraints, this pattern prevents type-related runtime exceptions and semantic errors [20] (e.g., [10]). 2.3.9 Change Prompt (CP). Refines the structure, phrasing, or formatting of prompts provided to Large Language Models in order to improve output relevance, consistency, or task alignment. This includes modifying instruction wording or adding clarifying context (e.g., [65]). 2.3.10 Fix Syntax (FS). Addresses violations of programming language grammar, including missing delimiters, incorrect indentation, or malformed expressions. These fixes are necessary to restore parsability and enable successful compilation or interpretation [20] ( e.g., [70]).
2.3.14 Change Function (CF). Replaces an incorrect function with the correct one that provides the intended functionality. This occurs when developers initially select the wrong function or API for a specific task. [56] (e.g., [17]). 2.3.15 Add Exception Handling (AEH). Encapsulates error-prone code regions within try-except or equivalent constructs to improve system robustness. This pattern allows failures to be caught, logged, or recovered from gracefully instead of terminating execution abruptly [56] (e.g., [4]). 2.3.16 Fix Attribute Name (FAN). : Resolves inconsistencies or typographical errors in attribute or parameter names so that they align with the definitions provided by the underlying API or object interface. These fixes are often subtle but critical for correct binding [20](e.g., [9]). 2.3.17 Change Parameter Value (CPV). : Updates literal constants or variable values passed to functions to better match expected operational ranges or logic constraints. Unlike data type changes, this pattern preserves type correctness while adjusting semantics [20] (e.g., [16]). 2.3.18 Change Input Data (CID). : Cleans, restructures, or reformats malformed or incompatible external data to ensure that downstream components can process it correctly. This pattern focuses on data integrity rather than code modification (e.g.,[80]). 2.3.19 Add Input Data (AID). : Resolves invocation or execution failures by supplying required input data that was previously missing or implicitly assumed. This includes mandatory configuration files, parameters, or runtime artifacts [26]. 2.3.20 Use Different Model (UDM). Enables a model-level substitution, such as switching to a different LLM, to satisfy constraints related to performance, cost, availability, or compatibility. This pattern reflects strategic adaptation at the system level (e.g., [47]). 2.3.21 Change Parameter Order (CPO). Reorders arguments in function calls to align with the expected positional signature, preventing logical errors caused by incorrect argument binding [75]. 2.3.22 Move Code to Different Scope (MCTDS). Corrects errors caused by incorrect indentation or improper code placement within control flow structures (e.g., [64]). 2.3.23 Fix Data Access (FDA). Corrects the logic governing the retrieval, indexing, or slicing of structured output data (e.g., [46]). 3
(a) Stack Overflow
(b) GitHub
(c) HuggingFace Forums
Figure 1: Fix pattern distribution across different data sources.
2.4
Empirical Analysis
Addition of Operations is most common in Stack Overflow, Change Parameter Value dominates GitHub, and Change External Resources is most frequent in HuggingFace Forums.
Our empirical analysis is conducted to answer the following research question. • RQ1 (Frequency): What are the common fix patterns in LLM agents, and how frequently do they occur? • RQ2 (Variation): Do the fix patterns in agentic systems and their distributions differ from those in traditional software systems? • RQ3 (Spread): What is the distribution of fix patterns across different libraries? • RQ4 (Challenges): What are the challenges in fixing bugs in LLM agents?
2.4.2 Variation: The distribution of fix patterns in agentic systems diverges fundamentally from established paradigms in traditional software engineering and deep learning by shifting from deterministic logic and structural optimization toward semantic orchestration and workflow augmentation. While traditional fixes in software systems are primarily characterized by repairs to deterministic logic, such as implementing precondition checks [56], resolving system interaction bugs involving file management [50], or implementing path resolution protocols [50], and DL systems focus on data-centric adjustments, most notably Data Dimension (18.8% in Stack Overflow) [38] and Network Connection (17.8% in Stack Overflow) [38] to ensure model convergence, agentic AI introduces a unique class of behavior-centric repairs. As evidenced by empirical data, the high prevalence of Addition of Operations (13.1% in Stack Overflow) and Add New Attribute (9.9%) indicates that agentic failures often stem from missing functional components or inadequate tool-calling protocols rather than broken syntax or tensor misalignment. This represents a significant departure from DL fix patterns where logic is embedded within weights and architecture [38]; in agentic systems, the logic resides in the ‘Agentic Flow,’ where swapping a library via Use Different Module is often a more frequent solution than swapping the core LLM engine. Furthermore, the emergence of Change External Resources as a dominant pattern in specialized forums (23.3% in HuggingFace) highlights an environmental dependency unique to agents that must dynamically manage local model hosting and API continuity. This complexity mirrors the high maintenance costs found in early DNN development due to library versioning, where versioning-related fixes reached 17.6% in GitHub [38], but introduces a distinct layer of orchestrational autonomy and cross-platform interaction challenges [50]. Lastly, although prompts play a critical role in LLM based systems and changing prompts represents a unique fix pattern for agentic systems, our study indicates that only a small number of fixes (5-7%) require prompt modification.
2.4.1 Frequency: Figure 1 presents the distribution of fix patterns across Stack Overflow, GitHub, and HuggingFace Forums data sources. As shown in the figure, Addition of Operations (AOO) is the most common fix pattern on Stack Overflow, Change Parameter Value is the most common on GitHub, and Change External Resources is the most frequent on HuggingFace Forums. These differences stem from the nature of each platform. On GitHub, agents are typically already developed and require parameter tuning for optimization or adaptation to specific use cases. On HuggingFace Forums, the prevalence of Change External Resources reflects the fact that HuggingFace Hub provides LLMs for use in agentic systems. Since these models must be downloaded locally, they introduce additional complexity in configuring external resources. Therefore, users prefer to discuss these problems in HuggingFace Forums with the specialized community there. Nevertheless, the high frequency of AOO and ANA type fix patterns in the Stack Overflow dataset indicates that many bugs arise from missing or mishandled operations. This highlights the need for more comprehensive documentation and best practices in agent development. Additionally, the higher number of fix patterns related to version change (CV) indicates the rapid development in agentic libraries, which forces developers to frequently update their code to maintain compatibility, a trend also found in the early stages of DNN development [38]. Lastly, the lower frequency of data-related fix patterns (e.g., AID) compared to those in DNN [38] is due to the fact that building agents generally do not require fine-tuning LLMs. Rather, agents use pretrained LLMs, which makes them less dependent on input data.
4
(a) Stack Overflow
(b) GitHub
(c) HuggingFace Forums
Figure 2: Fix pattern distribution across frameworks. tuning methods [79], these approaches are not integrated with mainstream libraries and consequently get neglected by practitioners. Achieving optimal agent performance requires systematic tuning of embedding models, LLMs, and chunking parameters, a computationally expensive process since SoTA LLMs contain billions of parameters. However, because frameworks lack built-in support for these optimization techniques, particularly cost-efficient tuning approaches, developers must either avoid optimization altogether or implement custom tuning solutions on their own. Version Instability: Secondly, due to rapid changes in library versions, developers frequently encounter breaking changes to functionality or API references that disrupt existing implementations and require code modifications to restore compatibility (e.g. [5] and [7]). These changes often cause agent crashes or introduce new bugs that were not present in prior versions, which is reflected in the increase of the change version type fix patterns. Consequently, developers must navigate a tradeoff between using newer versions with additional features and maintaining stable implementations with older, more reliable versions. Opaque Behavior: Lastly, the black box nature of LLMs makes certain bugs difficult to debug or reproduce. For instance, in a Stack Overflow post [15], an agent fails to answer questions about previous conversations, which suggests a memory issue. However, further investigation reveals that the problem stems from how the LLM interprets different prompts rather than an actual memory failure. The question ‘What’s my name?’ fails to retrieve information from chat history, while a rephrased version ‘Do you know my name?’ successfully accesses the same information. This behavior varies across different LLM providers, making it difficult for developers to diagnose and localize whether issues originate from the framework, the model, memory, or the prompt design. In this situation, applying fixes to an LLM agent becomes challenging.
The distribution of fix patterns in agentic systems diverges from traditional and deep learning paradigms by prioritizing semantic orchestration and workflow augmentation over deterministic logic and structural optimization. 2.4.3 Spread: Figure 2 presents the distribution of fix patterns across libraries. While low-frequency fix patterns (AEH, AID, and CPO) appear in only one library, high-frequency patterns like Change Version (CV) and Change References (CR) occur predominantly in LangChain across all data sources. This is unsurprising given LangChain’s rapid release cycle, with over 45 new versions in 2025 and over 55 versions in 2024 [74]. The most common fixes involve pinning specific library versions in dependency files and updating import statements when modules shift between packages across versions. Developers also frequently need to replace deprecated API calls with their updated equivalents based on migration guides. Additionally, data access fixes (FDA), though less common, appear primarily in custom agent implementations. Developers building custom agents often need to add parsing logic to extract structured data from LLM responses or implement error handling to access nested attributes in API responses, tasks that library-based implementations handle automatically through built-in parsers and response objects. Fix patterns are skewed, with version and reference changes dominating LangChain and data access fixes in custom agents. 2.4.4 Challenges: To understand the challenges developers face in fixing agentic code, we explored the bugs fixes and listed some of the major issues in this field. Feature Gaps: Since many parts of this field are still underdeveloped, agent frameworks and libraries often miss needed features such as custom filters, embedding support, or sophisticated integration with external tools. Frameworks’ reliance on pre-built modules makes it difficult to implement highly customized algorithms or complex data processing flows that forces developers to build outside the framework’s abstraction. For instance, as of January 2026, LangChain supports basic indexing workflows, but retrieval logic is usually built by wiring components together. Developers therefore have to write custom tools, build wrappers, or extend base classes to meet their specific needs, which leads to a higher number of Addition of Operations–type fix patterns (e.g., [13] and [14]). There is also a significant gap between research and implementation, particularly in agent optimization. Although research has explored agent
Developers struggle to fix AI agent code because frameworks lack needed features, libraries change too frequently breaking existing code, and AI models behave unpredictably making bugs hard to diagnose. As developers continue to face these challenges, the situation becomes worse when issues fail to produce correct output without any error logs. In this study, to assist developers, we build a multiagent solution that automatically fixes bugs in LLM agents. The solution is described in the Section 3. 5
a) Fix Pattern Rule Generation Description 1
Fix Pattern 1
Fix 1
Fix Pattern 2
Rational 1
Fix Pattern 3
Description 2
Fix Pattern 4
Training Set
Summary 2.1 LLM
Rule 2
Fix 2
Summary 2.2
LLM
Set of Rules
Rational 2
LLM
b) Fixed Code Generation I want help diagnosing and fixing... Code Compare
Fixed Code
Validate API Validate Format
Fix Rules
Critic Agent
Fix Agent
Web Search
Code Intent Buggy Code
Figure 3: Workflow of the proposed multi-agent approach.
2.5
tools exist for traditional software, they lack support for agentspecific breaking changes in memory management, tool execution, and prompt handling, or for detecting LLM API interaction issues. This forces manual conflict resolution, making automated solutions for agentic systems a critical research gap.
Implications
The prevalence of specific fix patterns across support forums suggests a shifting paradigm in the maintenance of agentic software. The dominance of Change Parameter Value (CPV) type fix patterns highlights a critical need for specialized mutation testing frameworks tailored for agentic systems. Traditional testing fails to account for the high sensitivity of stochastic parameters (e.g. [11]). Although many research works have leveraged LLMs for conducting mutation testing [66], few studies have explored applying mutation testing to LLM agents. However, agentic systems require high computational costs to operate. This makes large-scale testing expensive. There is also a gap between research and implementation in cost-efficient testing approaches. While mutation testing techniques exist, they have not been adapted for the resource constraints of agentic systems. Few studies have focused on developing cost-efficient testing mechanisms for agentic systems, indicating a research gap that warrants further investigation. Similarly, the high frequency of Change External Resources (CER) type fix patterns in HuggingFace Forums highlights the necessity of standardized protocols for integrating open-source LLMs. Each open-source LLM has different API structures, input formats, and deployment requirements. For example, models like Llama [67], Qwen [18], and DeepSeek [31] require different configuration approaches for hosting and inference. Switching between LLM providers requires minimal code changes only when using unified APIs, but multi-backend libraries often face dependency bloat. Additionally, switching between tools presents substantial integration challenges. Although some solutions exist to address this issue [24], they have not been widely adopted that forces developers to write custom integration code for each model and tool. The prevalence of Change Version fixes highlights the need for intelligent conflict resolver tools that identify dependency mismatches and recommend verified versions. While version migration
3
Approach
Figure 3 provides an overview of the proposed solution, SelfHeal, which consists of two main steps. In the first step, fix pattern rules are extracted and subsequently used in the second step to generate fixes. In this context, a rule is a concise natural language description that specifies the conditions under which a particular fix pattern should be applied and how the code should be modified accordingly. The following sections describe these steps in detail.
3.1
Fix Pattern Rule Generation
The rule generation process comprises two main stages, as depicted in Figure 3. During the first stage, we generate individual summaries for each post associated with a given fix pattern. As shown in the diagram, for fix pattern 2, we process all associated Stack Overflow posts (two examples are shown) using an LLM. The summarization process takes as input the problem description (comprising the Stack Overflow post’s title and body), the applied fix (either the corrected code or updated version), and the rationale provided by human annotators during the data annotation phase. The LLM processes this input to generate a concise one-line summary capturing both the problem and its solution. This process is repeated for every fix pattern in our taxonomy. Given the limited context window of LLMs, these summaries enable us to aggregate information from all posts associated with a fix pattern, a constraint that would otherwise prevent the inclusion of complete posts in the subsequent stage. In the second stage, for each fix pattern, we collect all individual summaries and perform a comprehensive analysis to generate a unified rule. This rule encapsulates the general occurrence patterns of bugs and the corresponding fix strategies. 6
Figure 3 illustrates this two-stage process for fix pattern 2. The same procedure is applied across all fix patterns to generate the complete set of rules that guide the agent in subsequent steps. The generated rules provide a summary of each fix pattern and describe the issues in the category as well as the methods to apply to fix the issues. For rule generation, we used GPT 5 Mini, as it ranked as the best model for summarization according to the benchmark as of January 2026 [58].
3.2
fixed code, which is later used by the critic agent for evaluation and stored in the file if it passes the agent’s assessment. Once the fix agent generates a complete fixed code, the critic agent is invoked with the original buggy code and the test code (generated by the authors in Stage 4.2). Following the fix agent, the critic agent has access to specific tools that it can invoke. The tools available to the critic agent are presented in Section 3.2.2. 3.2.2 Critic Agent Tools. The critic agent has access to three specific tools for its decision-making process. Code Compare: This tool compares the differences between the buggy code and the fixed code generated by the fix agent in the previous step. It uses the Python library difflib [59] to identify and explain the changes, which the critic agent then evaluates in subsequent steps. Validate API: Given the constantly evolving nature of the agentic field, validating the current functionality of APIs is essential. This tool uses web search to verify the usage of specific functions, methods, or API components. It searches for available parameter and return type documentation online and confirms whether the API is correctly invoked in the appropriate context. Like the fix agent, this tool excludes the source website from search results to prevent data leakage. Validate Format: This tool ensures the fixed code format aligns with the buggy code format so the test file can execute properly. For instance, if the test file requires a function named run_agent, this tool verifies that the code includes the correct formatting and structure, such as ensuring the agent returns output through a function rather than printing it directly. It also ensures the fixed code can be executed on the test file without any modifications. With the help of the available tools, the critic agent generates a verdict (accept or reject) along with supporting reasoning. If the critic agent decides to reject, the fix agent analyzes the reasoning and generates another version of the fixed code, which is then passed back to the critic agent. This iterative loop between the two agents continues for a maximum of three iterations. If the critic agent rejects the third version of the fixed code, the last code generated by the fix agent is returned as the final output.
Fixed Code Generation
The proposed SelfHeal agent consists of two independent ReAct agents [77] as the core component. A ReAct agent is an LLM-based agent that combines reasoning with actions to solve tasks. It operates through an iterative cycle that alternates between reasoning and action. The agent begins by reasoning about the problem to determine the optimal next step, then executes actions (such as tool invocations) that generate observations. These observations inform the subsequent reasoning step that creates a feedback loop that continues until the task is resolved. In the proposed solution, the first agent, the fix agent, generates a fix, while the second agent, the critic agent, evaluates the fix produced by the first agent. The approach begins by passing the complete buggy code, along with the code intent generated in Step 4.2, to the fix agent. The agent analyzes the buggy code and attempts to create a corrected version using specific tools at its disposal. Details of the tools available to the fix agent are presented in Section 3.2.1. 3.2.1 Fix Agent Tools. The fix agent has access to four tools that support rule lookup and external information retrieval during the fix generation process. List Fix Patterns: This tool returns the names of all available fix patterns when the fix agent needs to address a problem by following the fix rules. Our study identified twenty-three fix patterns for LLM agents. If we include all pattern rules directly in the prompt, it would drastically increase token usage and costs. The agent may not need all these rules, so we provide only pattern names first through this tool. The agent can then selectively retrieve specific rules by passing the fix pattern name through the Fix Pattern Rule tool when it decides they are needed. Fix Pattern Rule: After the agent receives the pattern names from List Fix Patterns, this tool (Fix Pattern Rule) retrieves the complete rule for a specific pattern. The rule explains what causes the issue and how to fix it. The agent analyzes this rule and then decides whether to modify code, request another pattern’s rule, or continue without fixes. This two-step approach first lists names (through List Fix Patterns) and then fetches rules on demand (through Fix Pattern Rule). It minimizes unnecessary token consumption while it provides targeted guidance when needed. Web Search: This tool allows the agent to search the internet for specific information using the Google Search API to retrieve relevant results. Since the evaluation dataset was collected from Stack Overflow and HuggingFace Forums, the tool filters out results from the corresponding source site (i.e., Stack Overflow results are excluded for Stack Overflow dataset instances, and HuggingFace results are excluded for HuggingFace dataset instances) to prevent potential data leakage. Submit Fix Code: This tool is invoked when the fix agent has successfully generated a corrected version of the code. It saves the
4
Evaluation
We analyze our proposed solution to answer the following research question. • RQ5 (Effectiveness): How effective is our proposed approach at localizing and fixing bugs in agentic systems? • RQ6 (Cost): What is the average cost and time required by our solution? • RQ7 (Ablation): What is the individual impact of fix-pattern –extracted rules and the web search on the effectiveness of the proposed system? • RQ8 (Comparison): How effective is our approach compared to existing solutions?
4.1
7
Experimental Setup
The experiment was conducted on a MacBook Air with an M3 processor and 16 GB RAM. For invoking LLMs, we leveraged the OpenAI API (for GPT-5.2) and OpenRouter API (for Gemini 3 Pro and Claude Sonnet 4). For web search, we used SerpAPI for Google Search. Lastly, we used LangChain [53] and LangGraph [48] with Python programming language for building the solution.
4.2
pattern: while Gemini 3 Pro and Claude Sonnet 4 tied on line-level localization, Gemini 3 Pro led in both function and component levels. GPT 5.2 scored the lowest overall.
Real-World Benchmark Dataset
While our empirical analysis, presented in Section 2, focused mainly on fix patterns, the collected dataset also includes buggy code, fixed code, error messages for crash bugs, and output logs for bugs producing wrong output. To better understand the nature of failures in real world settings, we further analyze the dataset. Our analysis on the Stack Overflow dataset shows that although crash bugs occur over ten times more frequently than bugs producing wrong output, the latter are significantly more difficult to debug. Many crash bugs can be debugged by analyzing the crash report, which helps developers identify the failure point [54, 72]. Additionally, some LLM-powered solutions have been utilized for automatically fixing crash bugs in software systems [35]. By contrast, wrong or missing output provides little diagnostic information that makes it ambiguous which component caused the failure. To address this concern, we present the first benchmark dataset, AgentDefect, containing bugs in LLM agents. We collected buggy posts from Stack Overflow and HuggingFace forums that generate incorrect output. Since these posts generally contain partial code focused on the buggy segment, we added minimal additional code while preserving the user-provided segment intact to reproduce the bug. We also generated a fixed version of the code following the accepted answer and created a test file to validate the fix. We further provided a markdown file explaining how to run the code and a requirements file containing the necessary libraries and their appropriate versions. Since all code segments are executable and produce no error messages, additional context about the user’s intention is required. To address this, we generated a one-line code intent describing what the user expects the code to perform by passing the post’s title and body to an LLM (GPT 5 Mini), which was then verified by a human. We selected GPT 5 Mini due to its performance in summarization tasks [58]. The collected AgentDefect dataset consists of bugs found in the two most popular libraries, namely LangChain [53] and LlamaIndex [52], as these two libraries account for 60.1% of the total bugs found in the Stack Overflow dataset. In total, we collected 37 code entries, with 36 sourced from Stack Overflow and 1 from HuggingFace forums. Additionally, we ensured that the component where the bug occurred spans across all four components of LLM agents, with a maximum of 19 out of 37 bugs occurring in the tools. In summary, each buggy code entry includes the buggy code, fixed code, test code, a markdown file with execution instructions, a requirements file listing the prerequisite libraries to install, and a code intent explaining what the user wants the code to perform.
4.3
SelfHeal with Gemini 3 Pro achieved the highest performance by solving 22 of 37 bugs in the AgentDefect dataset. 4.3.2 Cost. Figure 4 presents a comparison of time and cost consumption across different LLMs and approaches. The diagram shows an inverse correlation between the performance and both time and cost. In SelfHeal, Gemini 3 Pro, with the highest performance, consumes an average of 322.72 seconds before reaching a solution, while the worst performer, GPT-5.2, consumes the lowest time of just 41.77 seconds. Similarly, Gemini 3 Pro consumes an average of 0.4442 USD for fixing a buggy code segment, while GPT-5.2 takes 0.0492 USD. Claude Sonnet 4 occupies a middle position with respect to both performance and cost. On average, the model incurs a cost of 0.0759 USD per buggy code segment and requires 43.40 seconds to generate a fix. There is an inverse correlation between performance and both time and cost. The highest-performing model consumes the most time and incurs the highest cost. 4.3.3 Ablation. To measure the contribution of each component, we evaluated our solution by removing two data sources: internal and external. First, to assess the contribution of the internal data source, we removed the fix rules (fix pattern list and fix pattern rule tool) from the fix agent, allowing it to rely only on the internet search for external data. This resulted in an 18.92% performance drop. Next, we removed the web search tool from the fix agent and the validate API tool from the critic agent (which uses web search in the backend), resulting in a 13.51% performance drop. These results demonstrate the model’s reliance on both internal and external data sources. Furthermore, in three instances, the fix agent generated incorrect fixes that were detected by the critic agent, which then led to successful corrections. Table 2 illustrates the performance of different models and settings in detail. Since Gemini 3 Pro produced the best result, the ablation study was conducted using that LLM. Removing internal data sources degrades the repair performance by 18.92%, compared to 13.51% for external sources. 4.3.4 Comparison. We compared our approach against a zero-shot baseline and the SoTA SWE-Agent [76], which has been shown to outperform prior methods on software engineering benchmarks. Although the original SWE-Agent study leveraged GPT-4 Turbo and Claude 3 Opus, we adopt GPT-5.2 as the base model to ensure a fair comparison. Table 2 presents a detailed comparison of the approaches. The results show that the proposed method significantly outperforms both the baseline and SWE-Agent, with an average repair rate improvement of 21.72% over the baseline and 16.21% over SWE-Agent. Although Gemini 3 Pro yields the best performance in SelfHeal, Claude Sonnet 4 attains the highest repair rate among zero-shot approaches. This suggests that the performance gains of Claude Sonnet 4 stem mainly from the strength of the base model rather than external tools when compared with the three LLMs used. This pattern is also reflected in the number of attempts, as the LLM often produces a correct fix on the first attempt and needs
Result Analysis
4.3.1 Effectiveness. To evaluate SelfHeal, we tested it on the newly curated AgentDefect dataset. Following prior work [71], we report the resolution rate and localization accuracy at three levels of granularity: line, function, and LLM agent component. We selected three LLMs for evaluation: Claude Sonnet 4, Gemini 3 Pro, and GPT 5.2, representing the best-performing models for agentic coding as of December 2025 across three providers, based on SWE Benchmark results [45] [68]. Table 1 presents the results. SelfHeal equipped with Gemini 3 Pro achieved the highest resolution rate by correctly fixing 22 out of 37 buggy code segments, followed by Claude Sonnet 4 with 21 and GPT-5.2 with 20. Localization followed a similar 8
RP
Table 1: Comparison between the proposed approach, baseline, and SoTA. Complete comparison shared in [42] SelfHeal Zero-Shot SWE-Agent GPT-5.2 Gemini 3 Pro Claude Sonnet 4 GPT-5.2 Gemini 3 Pro Claude Sonnet 4 GPT-5.2 LI FN CP RP LI FN CP RP LI FN CP RP LI FN CP RP LI FN CP RP LI FN CP RP LI FN CP
76906469 79753835 79497660
✓ × ×
✓ × ×
✓ × ×
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ × ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
× × ×
× × ×
× × ×
× ✓ ✓
× × ×
✓ × ×
✓ ✓ ✓
✓ ✓ ✓
✓ × ×
✓ ✓ ×
✓ ✓ ✓
✓ ✓ ✓
× × ×
× × ×
✓ × ×
✓ × ✓
Total
20
24
29
33
22
27
31
34
21
27
29
33
12
18
22
24
13
19
29
30
15
22
29
32
14
18
27
30
Post ID
Abbreviations: RP: Repair; LI: Line Identification; FN: Function identification; CP: Component identification; SelfHeal: Proposed Approach.
(a) Average cost comparison
(b) Average time comparison
Figure 4: Performance comparison of different approaches, including (a) SWE Agent, (b) Zero-shot, (c) No Fix Rules, (d) No Web Search, and (e) SelfHeal. Table 2: Performance comparison of bug identification and repair using different approaches. App. M. Repair Line Fn. Comp. Attmp. SWE-A
G5.2 G5.2 Zero-shot G3P CS4 NFR G3P NWS G3P NCA G3P G5.2 SelfHeal G3P CS4
0.3784 0.3243 0.3514 0.4054 0.4054 0.4595 0.5135 0.5405 0.5946 0.5676
0.4865 0.4865 0.5135 0.5946 0.5676 0.5135 0.7297 0.6486 0.7297 0.7297
0.7297 0.5946 0.7838 0.7838 0.7568 0.8378 0.8378 0.7838 0.8378 0.7838
0.8108 0.6486 0.8108 0.8649 0.8919 0.8649 0.9189 0.8919 0.9189 0.8919
additional information from the external environment. In rapidly developing fields like LLM agents, relying solely on an LLM’s training knowledge is insufficient, as it quickly becomes obsolete. Therefore, our solution provides additional information through internal (fixrule) and external (web search) sources, which enhances the agent’s capabilities in conjunction with the critic agent that critiques the code and provides additional feedback. Nonetheless, this solution incurs a higher cost and time when using the best-performing LLM. Although the proposed solution achieved noteworthy performance, outperforming both baseline and state-of-the-art approaches, it has several limitations that can be addressed in future studies. First, the AgentDefect dataset contains only 37 instances, smaller than other benchmarks used in software engineering tasks such as the 40 instances in the buggy DNN code benchmark [69]. This limited size is due to LLM-based agents for code debugging being a relatively new field, making collection of real-world buggy code that does not crash challenging. Additionally, it consists of bugs from only the two most popular libraries. Second, although the benchmark contains complete executable code, SelfHeal does not execute code during debugging to leverage dynamic features, instead relying on an internal dataset of fix rules and internet search. While executing code and extracting dynamic features would significantly increase solution time, future work can explore this trade-off. Lastly, the solution has not been evaluated on repository-level codebases, unlike tools such as SWE-Agent [76], AutoCoderover [81], and requires modifications before adoption for fixing bugs in large-scale agentic systems with multiple files.
3.78 1.00 1.00 1.00 1.30 1.24 1.32 1.35 1.32 1.05
App.: Approach; M.: Model; G5.2: GPT-5.2; G3P: Gemini 3 Pro; CS4: Claude Sonnet 4; Fn.: Function; Comp.: Component; Attmp.: attempts.; SWE-A: SWE-Agent; NFR: No fix rules; NWS: No Web Search; NCA: No Critic Agent.
fewer attempts afterward. In contrast, SWE-Agent requires more attempts on average to reach a solution. SelfHeal outperforms both baseline and SoTA approach by a significant margin.
4.4
Discussion
5
Overall, SelfHeal demonstrated noteworthy performance in both localizing and repairing buggy agentic code. This higher performance can be attributed to the careful design of the architecture. While existing agentic systems like SWE-Agent [76], AutoCoderover [81], and Agentless [71] also employ step-by-step reasoning and feedback processes, their integrated tools do not provide LLMs with
Related work
This section reviews existing work on software bugs in deep learning, the evolution of Agentic AI in software engineering, and prior research on agent-based bug repair. Study on Software bug: Several studies investigated software bug types across traditional systems and modern deep learning 9
applications. Early work by Catolino et al defined a taxonomy of common software bugs and introduced an automated classification model[19]. As machine learning systems became prevalent, comprehensive analyses showed data and logic bugs from incorrect parameters were most common[37], leading to a validated taxonomy based on practitioner interviews[36]. Du et al. [25] introduced a framework using language model embeddings to automatically classify deep learning bug reports. Jahan et al. [43] presented a taxonomy of faults in attention-based neural networks, showing many failures are unique to attention mechanisms. Han et al. [32] studied bugs in foundation language models and found that dependency and API issues are the main causes of crashes. Pan et al. [57] analyzed bugs from LLM-based code translation and identify translation-specific error categories. Yu et al. [78] analyzed 308 bugs in large scale LLM training systems and identify common causes, challenges in debugging. Islam et al. [40] conducted the first large-scale study on bugs in LLM agents. Xue et al. [73], on the other hand, studied the bugs in LLM agent building frameworks. However, no study has analyzed bug-fix patterns specifically in agentic systems. Given the rapid development of LLM agents, we conducted the first study on bug fix patterns in this field. Agentic AI and SE. Recent work has increasingly explored the role of Agentic AI in advancing software engineering practices. Liu et al. [51] provided a comprehensive systematic survey on LLMbased agents in Software Engineering, categorizing them by both SE applications and agent architectures. Building on this foundation, Yang et al. [76] demonstrated that providing LLMs with a custom interface via SWE-agent leads to state-of-the-art results in autonomous code repair on the SWE-bench dataset. Zhang et al. [34] proposed MLE-Agent, an autonomous assistant that streamlines ML development through a combination of research tools and automated debugging. Beyond single-agent systems, He et al. [33] examined the role of multi-agent LLM systems in software engineering and outlined a path toward scalable, autonomous development. Terragni et al. [63] explored the future of human-AI synergy in coding while highlighting the technical challenges ahead. Furthermore, Liang et al. [49] presented the RepoCod benchmark to evaluate Python generation, demonstrating that current models often fail on full-scale software tasks. Agent-Based bug Repair. Recent studies have systematically examined agent-based approaches for automated bug repair. Deshpande et al. [23] introduced a dataset and error taxonomy for evaluating agentic workflow traces, showing that current LLMs failed significantly at automated debugging. Complementing this empirical analysis, Epperson et al. [27] introduced an interactive debugging tool that enables developers to visualize complex histories and reset agent messages to resolve common challenges. From a defect analysis perspective, Ning et al. [55] presented a systematic study and static analysis tool to detect agent defects by analyzing discrepancies between developer logic and LLM-generated content. They benchmarked real-world agent issues and found that SoTA LLM agents struggle to resolve agent-specific bugs. To address this gap, we curated the first benchmark on runtime bugs in LLM agents and developed SelfHeal, an LLM agent capable of fixing bugs in agentic systems. Overall, this study addresses two major research gaps in LLM agents. First, the lack of empirical analysis on bug fix patterns in
LLM agents, and second, the limited ability of SoTA LLM agents to fix bugs in agentic systems.
6
Threats to Validity
Internal Validity: A potential threat arises from the correctness of the annotated dataset, as fix patterns related to agents involve complex and stochastic behaviors. We mitigate this risk by employing two independent annotators with experience in agent development, measuring agreement with Cohen’s Kappa coefficient, and resolving all conflicts through discussion with an expert. Another threat concerns the correctness of SelfHeal’s implementation, which we address through independent code review by two authors and validation of experimental results. Finally, since the benchmark dataset was curated from publicly available sources and SelfHeal integrates web search, there is a risk of data leakage during internet searches. We mitigate this by restricting web search to the source website of each benchmark instance and verifying through ablation studies that the agent’s fixing ability does not depend solely on external sources. We have also ensured the instances selected for the AgentDefect dataset are excluded while generating the fix rules. External Validity: A threat to generalizability stems from benchmark representativeness. We mitigate this by curating data from GitHub, Stack Overflow, and HuggingFace Forums, which cover diverse real-world agent deployments. Another threat relates to dataset validity, as unverified fixes may not reflect realistic scenarios. We ensure each instance includes an accepted answer, validated solution, or external reference. For externally sourced fixes, both annotators independently verified correctness and documented the references in the dataset. Finally, to reduce LLM stochasticity, we compare our approach against baselines under identical settings.
7
Conclusions and Future Work
This study presents an empirical analysis of bug-fix patterns in LLM agents, comparing these patterns across different software domains, and provides actionable implications for the community. To facilitate further research on fixing bugs in LLM agents, we prepared an executable benchmark dataset, AgentDefect, containing runtime bugs with buggy code, fixed code, and corresponding test cases. We also built a multi-agent system, SelfHeal, for automatically fixing bugs in LLM agents. Our analysis reveals that the proposed solution outperforms both the baseline and state-of-theart approaches by a noticeable margin. Despite these contributions, this study has several limitations that future research can address. First, the AgentDefect dataset contains a relatively small number of instances and can be extended by exploring additional libraries beyond LangChain and LlamaIndex, as well as incorporating other data sources. Second, the proposed solution relies on static analysis for bug detection. Future studies can integrate dynamic analysis to determine whether runtime data improves bug-fixing performance. Finally, SelfHeal currently operates on single files containing buggy instances. Future work can extend the approach to handle repository-level codebases with multiple interdependent files.
8
Data Availability
The fix pattern dataset and benchmark dataset (AgentDefect), along with results and code, are publicly available in the site [42] and GitHub [41]. 10
References
[25] Xiaoting Du, Zhihao Liu, Chenglong Li, Xiangyue Ma, Yingzhuo Li, and Xinyu Wang. 2024. LLM-BRC: A large language model-based bug report classification framework. Software Quality Journal 32, 3 (2024), 985–1005. [26] Madeline Endres, Georgios Sakkas, Benjamin Cosman, Ranjit Jhala, and Westley Weimer. 2019. Infix: Automatically repairing novice program inputs. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 399–410. [27] Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang (Eric) Zhu, and Saleema Amershi. 2025. Interactive Debugging and Steering of Multi-Agent AI Systems. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 156, 15 pages. doi:10.1145/3706598.3713581 [28] GitHub. 2015. Running non stop in Colab · Issue #60 · shroominic/codeinterpreterapi — github.com. https://github.com/shroominic/codeinterpreter-api/issues/60. [Accessed 23-01-2026]. [29] GitHub. 2024. [BUG] No module named ’uvloop’ – Issue #623 – kyegomez/swarms. https://github.com/kyegomez/swarms/issues/623. Accessed 23 January 2026. [30] GitHub. 2024. LLamaSharpEmbeddings Exception: EmbeddingMode must be true. · Issue 343 · tryAGI/LangChain — github.com. https://github.com/tryAGI/ LangChain/issues/343. [Accessed 23-01-2026]. [31] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). [32] Junxiao Han, Guanqi Wang, Jiakun Liu, Lingfeng Bao, Xing Hu, Jinling Wei, and Shuiguang Deng. 2025. A Comprehensive Study of Bug Characteristics on Foundation Language Models. In 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge). 257–268. doi:10.1109/ Forge66646.2025.00037 [33] Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Transactions on Software Engineering and Methodology 34, 5, Article 124 (May 2025), 30 pages. doi:10.1145/3712003 [34] Lei Zhang Huaizheng Zhang*, Yizheng Huang*. 2024. MLE-Agent: Your Intelligent Companion for Seamless AI Engineering and Research. https://github.com/MLS ysOps/MLE-agent. [35] Yuchao Huang, Junjie Wang, Zhe Liu, Mingyang Li, Song Wang, Chunyang Chen, Yuanzhe Hu, and Qing Wang. 2025. One Sentence Can Kill the Bug: AutoReplay Mobile App Crashes From One-Sentence Overviews. IEEE Transactions on Software Engineering 51, 4 (2025), 975–989. doi:10.1109/TSE.2025.3535938 [36] Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE ’20). Association for Computing Machinery, New York, NY, USA, 1110–1121. doi:10.1145/3377811.3380395 [37] Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. 2019. A comprehensive study on deep learning bug characteristics (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 510–520. doi:10.1145/ 3338906.3338955 [38] Md Johirul Islam, Rangeet Pan, Giang Nguyen, and Hridesh Rajan. 2020. Repairing deep neural networks: fix patterns and challenges. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE ’20). Association for Computing Machinery, New York, NY, USA, 1135–1146. doi:10.1145/3377811.3380378 [39] Mohammad Saiful Islam, Mohamed Sami Rakha, William Pourmajidi, Janakan Sivaloganathan, John Steinbacher, and Andriy Miranskyy. 2025. Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 377–388. doi:10.1109/ICSE-SEIP66354.2025.00039 [40] Niful Islam, Ragib Shahriar Ayon, Deepak George Thomas, Shibbir Ahmed, and Mohammad Wardat. 2026. When Agents Fail: A Comprehensive Study of Bugs in LLM Agents with Automated Labeling. arXiv preprint arXiv:2601.15232 (2026). [41] Niful Islam, Muhammad Anas Raza, and Mohammad Wardat. 2026. "SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents" - GitHub. https: //github.com/Laboratory-software-Innovation/SelfHeal. Accessed: 2026-04-19. [42] Niful Islam, Muhammad Anas Raza, and Mohammad Wardat. 2026. "SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents" - Site. https: //sites.google.com/view/selfheal/home. Accessed: 2026-01-23. [43] Sigma Jahan, Saurabh singh Rajput, Tushar Sharma, and Mohammad Masudur Rahman. 2026. Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networks. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE 2026). [44] Li Jia, Hao Zhong, Xiaoyin Wang, Linpeng Huang, and Xuansheng Lu. 2021. The symptoms, causes, and repairs of bugs inside a deep learning library. Journal of Systems and Software 177 (2021), 110935. [45] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve
[1] 2023. fix pfdreader · mt7180/quaigle@c29f047 — github.com. https://github .com/mt7180/quaigle/commit/c29f047546876af 0812cc4f 06adf7a09048058f 0. [Accessed 23-01-2026]. [2] 2023. fix: span topo · jina-ai/langchain-serve@762626d — github.com. https: //github.com/jina-ai/langchain-serve/commit/762626d588167cbe8bacc10d2fa8a 02a9c417b8b. [Accessed 23-01-2026]. [3] 2023. How to see the Embedding of the documents with Chroma (or any other DB) saved in Lang Chain? — stackoverflow.com. https://stackoverflow.com/ques tions/76379440/how-to-see-the-embedding-of-the-documents-with-chromaor-any-other-db-saved-in. [Accessed 23-01-2026]. [4] 2023. LangChain: Querying a document and getting structured output using Pydantic with ChatGPT not working well — stackoverflow.com. https://stackove rflow.com/questions/76822673/langchain-querying-a-document-and-gettingstructured-output-using-pydantic-with. [Accessed 23-01-2026]. [5] 2023. LangChain textttModuleNotFoundError: No module named ’langchain’. https://stackoverflow. com/questions/76726419 [6] 2023. Trying to create vectors and chunked data using Azure Cognitive Search/Azure AI Search. https://stackoverflow.com/questions/77646675/tryingto-create-vectors-and-chunked-data-using-azure-cognitive-search-azure-ai. Accessed: 2026-01-23. [7] 2023. TypeError: issubclass() arg 1 must be a class when importing LangChain in Flask. https://stackoverflow.com/questions/76313568 [8] 2023. Using Vicuna + langchain + llama_index for creating a self hosted LLM model — stackoverflow.com. https://stackoverflow.com/questions/76067104 /using-vicuna-langchain-llama-index-for-creating-a-self-hosted-llm-model. [Accessed 23-01-2026]. [9] 2024. Fix bug with empty tool list (#225) · startino/aitino@e68077f — github.com. https://github.com/startino/aitino/commit/e68077f1da17d0f16c25d13800c90900 1b216325. [Accessed 23-01-2026]. [10] 2024. Fix hf generate for llama3.2 (#12497) · intel/ipex-llm@7d27f13 — github.com. https://github.com/intel/ipex-llm/commit/7d27f134ddd094ef49b3dd71487261c 452d46056. [Accessed 23-01-2026]. [11] 2024. Getting error when using memory with chain: TypeError: Object of type Member is not serializable. https://stackoverf low.com/questions/79313470 /getting-error-when-using-memory-with-chain-typeerror-object-of -typemember-is. Accessed: 2026-01-23. [12] 2024. ModuleNotFoundError: No module named ’langchain_openai’ — stackoverflow.com. https://stackoverflow.com/questions/77782167/modulenotfounderrorno-module-named-langchain-openai. [Accessed 23-01-2026]. [13] 2025. How can I match the token count used by BGE-M3 embedding model before embedding? https://stackoverflow.com/questions/79753835 [14] 2025. How to create custom columns when creating embeddings using LlamaIndex in Postgres (with pgvector extension)? https://stackoverflow.com/questions/7949 7660 [15] Aapolaris. 2025. Langchain-based model memory. https://stackoverflow.com/qu estions/79776520/langchain-based-model-memory Stack Overflow question. [16] AIBTC. 2024. fix: update callbacks and output format for chat msgs · aibtcdev/aiagent-crew@e6c341d — github.com. https://github.com/aibtcdev/ai-agentcrew/commit/e6c341d49362c60f 05e1cf ea1d6f 9149e732bc8c. [Accessed 23-01-2026]. fix: ‘requires_dependencies‘ import (#3763) · argilla[17] Argilla. 2023. io/argilla@dcafd79 — github.com. https://github.com/argilla- io/argilla/c ommit/dcafd79f02715abf9ba492ae8fd7dcc0173e3107. [Accessed 23-01-2026]. [18] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023). [19] Gemma Catolino, Fabio Palomba, Andy Zaidman, and Filomena Ferrucci. 2019. Not all bugs are the same: Understanding, characterizing, and classifying bug types. Journal of Systems and Software 152 (2019), 165–181. [20] Wei-Hao Chen, Jia Lin Cheoh, Manthan Keim, Sabine Brunswicker, and Tianyi Zhang. 2025. Towards Understanding Fine-Grained Programming Mistakes and Fixing Patterns in Data Science. Proceedings of the ACM on Software Engineering 2, FSE, Article FSE082 (June 2025), 23 pages. doi:10.1145/3729352 [21] Sadia Sultana Chowa, Riasad Alvi, Subhey Sadi Rahman, Md Abdur Rahman, Mohaimenul Azam Khan Raiaan, Md Rafiqul Islam, Mukhtar Hussain, and Sami Azam. 2026. From language to action: a review of large language models as autonomous agents and tool users. Artificial Intelligence Review (2026). [22] LlamaIndex Contributors. 2026. Llama Index Python Package. https://pypi.org/p roject/llama-index/#history. Accessed: 2026-01-23. [23] Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. 2025. TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv preprint arXiv:2505.08638 (2025). [24] Peng Ding and Rick Stevens. 2025. Unified Tool Integration for LLMs: A ProtocolAgnostic Approach to Function Calling. arXiv preprint arXiv:2508.02979 (2025). 11
[67] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [68] Vellum AI. 2025. LLM Leaderboard 2025. https://www.vellum.ai/llm-leader board?utm_source=google&utm_medium=organic. Accessed: 2026-01-21; leaderboard of public benchmark performance for LLMs, updated 15 Dec 2025. [69] Mohammad Wardat, Wei Le, and Hridesh Rajan. 2021. DeepLocalize: Fault Localization for Deep Neural Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 251–262. doi:10.1109/ICSE43902.2021 .00034 [70] whoabuddy. 2024. hotfix: remove async from crew execution · aibtcdev/aibtcdevbackend@62f3f3a — github.com. https://github.com/aibtcdev/aibtcdev-backend/ commit/62f3f3aacc67f40833641c6251b6fbf50fc5d5eb. [Accessed 23-01-2026]. [71] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proc. ACM Softw. Eng. 2, FSE, Article FSE037 (June 2025), 24 pages. doi:10.1145/3715754 [72] Yang Xu, Chao Liu, Yong Li, Qiaoluan Xie, and Hyun-Deok Choi. 2023. A Method of Component Prediction for Crash Bug Reports Using ComponentBased Features and Machine Learning. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 773–777. doi:10.1109/ SANER56733.2023.00089 [73] Ziluo Xue, Yanjie Zhao, Shenao Wang, Kai Chen, and Haoyu Wang. 2025. A Characterization Study of Bugs in LLM Agent Workflow Orchestration Frameworks. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) (Seoul, Korea, Republic of). IEEE Press, 3369–3380. doi:10.1109/ASE63991.2025.00278 [74] Baskar Y, Harrison Chase, and LangChain Contributors. 2026. LangChain Python Package. https://pypi.org/project/langchain/#history. Accessed: 2026-01-23. [75] Boyang Yang, Haoye Tian, Jiadong Ren, Hongyu Zhang, Jacques Klein, Tegawendé F. Bissyandé, Claire Le Goues, and Shunfu Jin. 2026. MORepair: Teaching LLMs to Repair Code via Multi-Objective Fine-Tuning. ACM Transactions on Software Engineering and Methodology 35, 2, Article 38 (Jan. 2026), 38 pages. doi:10.1145/3735129 [76] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 50528–50652. doi:10.52202/079017-1601 [77] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=WE_vluYUL-X [78] Xiao Yu, Haoxuan Chen, Feifei Niu, Xing Hu, Jacky Wai Keung, and Xin Xia. 2025. Towards Understanding Bugs in Distributed Training and Inference Frameworks for Large Language Models. arXiv preprint arXiv:2506.10426 (2025). [79] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2024. AgentTuning: Enabling Generalized Agent Abilities for LLMs. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, Bangkok, Thailand, 3053–3077. doi:10.18653/v1/ 2024.findings-acl.181 [80] Xiaoyun Zhang. 2024. fix #22 (#23) · LittleLittleCloud/Agent-ChatRoom@a8fe2e4 — github.com. https://github.com/LittleLittleCloud/Agent-ChatRoom/commit/a8 fe2e481e486fdabe1b004fcbe2242af2fafc1b#diff-b3074baecba2094a4e7ff0b44b56 bf33d749eb69db93c929b4d0ab2c8173d034R80-R238. [Accessed 23-01-2026]. [81] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (Vienna, Austria) (ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 1592–1604. doi:10.1145/3650212.3680384
real-world github issues? arXiv preprint arXiv:2310.06770 (2023). [46] Yuya Kakui. 2024. Merge pull request #202 from kyaukyuai/fix/agent · kyaukyuai/gpt-all-star@1803420 — github.com. https://github.com/kyauk yuai/gpt- all- star/commit/1803420f fd2186686705b0461c7d8e638c7eb579. [Accessed 23-01-2026]. [47] Aimen Kerrour. 2025. Fixed bug and use gpt-4o-mini · kaymen99/Upwork-AIjobs-applier@97b1158 — github.com. https://github.com/kaymen99/Upwork-AIjobs-applier/commit/97b115891cd678a0b15a08ff40de7996d289b1bf. [Accessed 23-01-2026]. [48] LangChain Developers. 2026. LangGraph: Build and Orchestrate Stateful Agents. https://www.langchain.com/langgraph. Accessed: 2026-01-25. [49] Shanchao Liang, Nan Jiang, Yiran Hu, and Lin Tan. 2025. Can Language Models Replace Programmers for Coding? REPOCOD Says ‘Not Yet’. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vienna, Austria, 24698– 24717. doi:10.18653/v1/2025.acl-long.1204 [50] Di Liu, Yanyan Yan, Hongcheng Fan, and Yang Feng. 2024. Mining Fix Patterns for System Interaction Bugs. In Proceedings of the 15th Asia-Pacific Symposium on Internetware (Macau, China) (Internetware ’24). Association for Computing Machinery, New York, NY, USA, 367–376. doi:10.1145/3671016.3671398 [51] Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2026. Large Language Model-Based Agents for Software Engineering: A Survey. ACM Trans. Softw. Eng. Methodol. (2026). doi:10.1145/37 96507 [52] LlamaIndex. 2022. LlamaIndex. https://www.llamaindex.ai/. Accessed: 2026-0121. [53] Vasilios Mavroudis. 2024. LangChain. Preprints.org (2024). doi:10.20944/preprin ts202411.0566.v1 [54] Marcos Medeiros, Uira Kulesza, Roberta Coelho, Rodrigo Bonifacio, Christoph Treude, and Eiji Adachi Barbosa. 2024. The Impact Of Bug Localization Based on Crash Report Mining: A Developers’ Perspective. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice (Lisbon, Portugal) (ICSE-SEIP ’24). Association for Computing Machinery, New York, NY, USA, 13–24. doi:10.1145/3639477.3639730 [55] Kaiwen Ning, Jiachi Chen, Jingwen Zhang, Wei Li, Zexu Wang, Yuming Feng, Weizhe Zhang, and Zibin Zheng. 2026. Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents. IEEE Transactions on Software Engineering 52, 3 (2026), 1074–1093. doi:10.1109/TSE.2026.3658554 [56] Kai Pan, Sunghun Kim, and E James Whitehead Jr. 2009. Toward an understanding of bug fix patterns. Empirical Software Engineering 14, 3 (2009), 286–315. [57] Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 82, 13 pages. doi:10.1145/3597503.3639226 [58] ProLLM. 2026. Summarization Leaderboard. https://www.prollm.ai/leaderboar d/summarization?language=afrikaans,brazilian+portuguese,english,polish&l evel=advanced,basic Evaluates an LLM’s ability to accurately summarize long texts from diverse sources. [59] Python Software Foundation. 2025. difflib — Helpers for computing deltas. https: //docs.python.org/3/library/difflib.html Accessed: 2026-01-15. [60] Muhammad Anas Raza and Mohammad Wardat. 2025. Graph neural network for fault localization in sequence-based models. Empirical Software Engineering 30, 5 (2025), 119. [61] Talia Ringer, RanDair Porter, Nathaniel Yazdani, John Leo, and Dan Grossman. 2021. Proof repair across type equivalences. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation (Virtual, Canada) (PLDI 2021). Association for Computing Machinery, New York, NY, USA, 112–127. doi:10.1145/3453483.3454033 [62] Tony Rousmaniere, Simon B Goldberg, and John Torous. 2026. Large language models as mental health providers. The Lancet Psychiatry 13, 1 (2026), 7–9. [63] Valerio Terragni, Annie Vella, Partha Roop, and Kelly Blincoe. 2025. The Future of AI-Driven Software Engineering. ACM Transactions on Software Engineering and Methodology 34, 5, Article 120 (May 2025), 20 pages. doi:10.1145/3715003 [64] thedigitalworkplace. 2024. Fix bug in process event in worker runtime (#492) · thedigitalworkplace/Autogen@1ba7a68 — github.com. https://github.com/thedi gitalworkplace/Autogen/commit/1ba7a681a93a1308e608dcfedee2f29a6dab4a76 #diff-e5e45bd0c38c1fd4a21ca55a40b0df2161922f15d285de1bdadbe7d110aa109f R458-R446. [Accessed 23-01-2026]. [65] thedigitalworkplace. 2025. Fix chess sample (#4932) – thedigitalworkplace/Autogen commit 52c2a70. https://github.com/thedigitalworkplace /Autogen/commit/52c2a70e95df2006e0094e96ad192243148ec4bb. Accessed 23 January 2026. [66] Frank Tip, Jonathan Bell, and Max Schäfer. 2025. LLMorpheus: Mutation Testing Using Large Language Models. IEEE Transactions on Software Engineering 51, 6 (2025), 1645–1665. doi:10.1109/TSE.2025.3562025 12