IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
1
Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports
arXiv:2605.17561v1 [cs.SE] 17 May 2026
Mahmut Furkan Gön* , Emre Dinç* , Tevfik Emre Sungur* , and Eray Tüzün
Abstract—Context: Issues faced when using software are reported in the form of bug reports. However, many bug reports are invalid, meaning they do not require code changes, and are resolved with a no-code fix. Manually determining the root cause of the invalid bug reports and providing actionable resolutions by the customer support cause a serious waste of resources. Objective: Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification, and perform experiments to test the accuracy of various approaches on invalid subclassification and no-code fix generation. We study how different configurations perform on a gold-standard benchmark we have created. Method: Using a manually curated benchmark for higher quality analysis, we experimented with vanilla LLMs, Retrieval Augmented Generation, and agentic web search to identify invalid subclasses and generate no-code fixes. We evaluated the results against manually labeled ground truth data that includes the invalid subclass and no-code fixes from the original bug reports. We measured subclass detection performance with weighted F1Score, and assessed no-code fix suggestions using BERTScore and Judge LLM success rates. Results: For subclassification, retrieval augmented generation achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0.65 and agentic web search at 0.64. At the subclass level, performance peaks at 0.85 F1 for Non-reproducibility and 0.79 for Feature Request and Question, while Wrong Version remains the most challenging with scores between 0.00 and 0.29. For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68.9%, compared to 64.4% for RAG applications and 64.9% for vanilla LLMs, with subclass-level peaks of 87.4% for Working as Designed and 72.2% for Question. Conclusion: This study shows that retrieval augmented generation improves invalid subclassification performance, while no-code fix generation benefits more from agentic web search, indicating that different components are effective for different tasks. The generated no-code fixes demonstrate the potential to automate customer support workflows, reduce manual triage effort, and provide immediate, actionable guidance to users within bug tracking systems. Index Terms—Bug Report, Invalid Bug Reports, Invalid Bug Report Subclassification, No-Code Fixes, LLM Evaluation, LLM Agents, Customer Support Agents, Retrieval Augmented Generation
I. I NTRODUCTION Software maintenance is an essential aspect of software development. Software maintenance costs more than 2 trillion USD * These authors contributed equally to this work. All authors are with the Department of Computer Engineering, Bilkent University, Ankara, Türkiye. Emails: {furkan.gon, emre.sungur,}@bilkent.edu.tr, [email protected], [email protected] 1 https://github.com/brave/brave-browser/issues/25057 2 https://github.com/brave/brave-browser/issues/26323
annually [1]. Most open-source and proprietary projects address software maintenance through bug reports in bug tracking platforms such as Jira or GitHub Issues [2], [3]. End users create bug reports that reflect their problems, such as implementation issues, crashes, configuration problems, or misunderstandings about the software product. These reports are then reviewed by project owners or developers, who investigate and resolve the reported problems. From the software maintainer’s perspective, bug reports can be categorized as either valid bug reports that require modifications to the source code or invalid bug reports that do not require changes to the source code, hence not inherently a software defect [4], [5], [6]. A considerable portion of submitted bug reports are labeled as invalid. For example, in the Eclipse project, about 22% of the 121,855 reported bugs were classified as invalid [4]. While invalid bug reports usually do not require changes to the source code, they may still highlight issues within the product. In an analysis of 7,000 issue reports across five open-source projects, 33.8% of reports initially labeled as valid bugs were actually misclassified [7]. As a result, 39% of source files marked as defective were found to have never contained an actual bug [7]. Moreover, one study found that resolving invalid bug reports can take nearly as long as resolving valid ones, 16 and 18 days on average across two different software products [8]. There might be several root causes why a bug report can be invalid, including a misunderstanding of an intended feature [5], [9], [10], [11], user-side configuration problems [5], [9], [10], [11], or using a wrong version of the software product [5], [10], [11]. The customer support team typically identifies the root causes of invalid bug reports [12]. After identifying the root causes, the customer support team resolves these invalid bug reports with no-code fixes, fixes that do not require changes to the source code. In this study, we argue that each potential root cause may require distinct no-code fixes. For instance, a user configuration error could be resolved by finding the required configuration on the user’s machine (see Figure 1a). In contrast, an external system dependency issue could be resolved by explaining that the issue originates from an upstream external issue (see Figure 1b). The customer support team can also consider these root causes while providing nocode fix solutions to the reporters. Early detection and resolution of invalid bug reports significantly reduce the workload of all stakeholders involved in the bug tracking lifecycle, including users, customer support, project managers/team leads, and developer teams. By identifying invalid bug reports early, the customer support team is relieved of the repetitive task of manually reviewing, classifying, and responding to them. Instead, they can conc
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
(a) Faulty Configuration Issue3
(b) External System Dependency Issue4
Fig. 1: Invalid Bug Report Examples with Different Root Causes entrate on verifying valid reports and reviewing automatically generated no-code fixes. Once the valid reports are classified, they are forwarded to the team leads, who then assign them to the appropriate developers. Except for a few edge cases, team leads usually prioritize valid issues that are more likely to require code changes [13]. Consequently, developers can devote their time to genuine software bugs without having to determine whether a bug report actually requires source code modification. However, this may still leave reporters uncertain about how to resolve the invalid bug reports. Incorporating no-code fixes directly addresses this gap by guiding reporters toward the appropriate solution. As a result, it enhances user experience by improving interaction satisfaction and fostering continued engagement. Some earlier studies have applied different machine learning (ML) algorithms [8], [4] to detect invalid bug reports. Some others used deep learning (DL) models [14], [15] for the same task. More recent studies have applied large language models (LLMs) [16], while others have implemented hybrid solutions that combine DL and LLM approaches [17] to detect invalid bug reports. In addition to basic validity detection, several studies have empirically investigated the root causes behind invalid bug reports [5], [10], [11], [9], [7]. However, these works are limited because they rely solely on manual analysis rather than proposing automated methods for invalid sub-categorization by root causes. Although early studies exist, they do not go beyond detecting invalid bug reports or merely discussing the root causes of them. Invalid bug reports need to be detected, categorized by potential root causes, and resolved with no-code fixes. To the best of our knowledge, no previous study has automatically detected the root causes of invalid bug reports and generated
2
no-code fixes to resolve them. In our previous work [18], we employed a hybrid approach, using multiple ML models and feeding their results to a judge LLM to determine the validity of the bug report. The objective of this study is to further develop an automated framework for subclassifying invalid bug reports by root cause and resolving them using no-code fixes. We do not include validity classification in this paper as it would shift the focus of the paper, when we want to focus deeply on invalid bug report subclassification and no-code fix suggestion. The study also aims to evaluate and compare different approaches to achieve these objectives and validate their effectiveness. To achieve these objectives, we address the following research questions: RQ1: How can invalid bug reports be subclassified according to their root causes? With this research question, our aim is to identify the root causes of invalid bug reports automatically. RQ1.1: How do LLMs perform in subclassification of invalid bug reports? With this research question, we aim to evaluate how different LLMs perform on the subclassification task of invalid bug reports based on their root causes. RQ1.2: Does context engineering affect the invalid bug report subclassification performance? With this research question, we aim to evaluate the effects of retrieval augmented generation (RAG) and agentic web search on the subclassification of the invalid bug reports with LLMs. RQ2: How can invalid bug reports be resolved with no-code fixes? With this research question, our aim is to develop automated approaches to the crucial step of generating no-code fixes. RQ2.1: How do LLMs perform in generating no-code fixes without subclass knowledge as prior information? With this research question, we aim to measure the performance of LLMs without the subclass information. RQ2.2: Does utilizing the root cause subclass information of invalid bug reports affect the no-code fix generation performance? With this research question, we aim to examine the impact of using the root cause subclassification of invalid bug reports (RQ1) on the generation of no-code fixes. RQ2.3: Does context engineering affect the no-code fix generation performance? With this research question, we aim to evaluate the effects of RAG and web search tool utilization applications on no-code fix generation. To address these research questions, we designed a multistep evaluation framework called IssueSupport that leverages LLMs to subclassify invalid bug reports and generate nocode fixes. We first established a baseline (vanilla) setting, evaluating the inherent capabilities of both proprietary and open-source LLMs to map bug reports to predefined seven invalid subclasses: External System & Dependency Issues, Faulty Configuration, Feature Request, Non-reproducible, Question, Working as Designed, and Wrong Version. Then, IssueSupport subsequently suggests actionable no-code fixes. To isolate the impact of subclassification on the quality of
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
generated resolutions (RQ2.1 vs RQ2.2), we conducted an ablation study that requested no-code fixes entirely without subclass priors, later comparing these outcomes with generations informed by the root cause mappings. Building upon the vanilla baseline, we implemented an RAG pipeline to investigate the role of external context in improving classification performance (RQ1.1) and no-code fix quality (RQ2.3). Our RAG system dynamically retrieves relevant developer wiki articles and similar historically invalid bug reports utilizing dense vector search. We utilize the framework introduced by an agentic web search paper to test the impact of web search on LLM performance. To ensure robust evaluation across all experimental paradigms, the quality of the generated no-code fixes was automatically assessed using BERTScore [19] and a Judge LLM to determine semantic and functional alignment with human-annotated ground truths. This study contributes to the field by introducing an automated system for invalid bug subclassification into previously defined invalid subclasses and further generating no-code fixes after subclassification, which was not available in the previous studies. To the best of our knowledge, there is no particular benchmark that explicitly subclasses invalid bug report content by the root cause. We also introduce a novel benchmark comprising invalid bug reports, their invalid subclasses, and the respective no-code fixes (if any). This dataset serves as ground truth and a benchmark for our study and possible future studies in this domain. Our study has the following contributions: 1) A comprehensive root cause based invalid bug report subclass taxonomy and subclassification methodology. 2) A novel approach to generate no-code fixes to resolve invalid bug reports. 3) A comprehensive evaluation of the effect of prior information on the invalid bug report subclassification task. 4) A comprehensive evaluation of different experimental settings on the no-code fix generation task. 5) A new benchmark curated for no-code fix suggestions to subclassified invalid bug reports. The remainder of this paper is organized as follows. Section II reviews the relevant literature and discusses prior studies on invalid bug reports and their classification, as well as other points concerning no-code fix generation, highlighting the research gap. Section III presents the curated benchmark and the methodology for the IssueSupport framework, detailing the design and workflow. Section IV reports the experimental results and the performance evaluation of different experimental settings. Section V provides a detailed discussion of the findings and their implications. Section VI presents the possible threats to validity. Finally, Section VII concludes the paper by summarizing the key outcomes of the obtained results and suggesting directions for future work. II. R ELATED W ORKS AND BACKGROUND In this section, we review the existing literature relevant to our study, establishing the foundational background on bug report validity and the root causes of invalid bug reports. We also examine the modern application of LLMs, with and without
3
the additional context provided by RAG or web search. We inspect their usage in software maintenance and customer support workflows, highlighting the critical research gap in automated resolution for invalid bug reports. A. Bug Report Validity and Automated Triaging The high volume of incoming bug reports necessitates robust automated triaging systems. Herzig et al. [7] demonstrated that a significant percentage of bug reports are fundamentally misclassified by users, often acting as feature requests or routine tasks, which severely impacts the reliability of bug prediction models and wastes developer time. To mitigate this manual overhead, early empirical studies explored supervised [20], unsupervised [21], and semi-supervised [22] ML algorithms to automatically categorize bug reports. As the field advanced, researchers adopted DL to capture the semantic complexities of natural language descriptions [23], [24], [15]. However, treating bug validity purely as a binary classification stops short of addressing what aspects of the reports are missing for them to be valid. Recognizing this, further studies focused on extracting explanatory patterns. He et al. [14] proposed a DL-based approach to simultaneously determine validity and extract these patterns, while transformer architectures like BERT [25] have been leveraged for similar rationale extraction. Building on the need to make these explanations actionable for the reporter, our previous work, Judge the Votes [18], introduced a system that goes beyond mere classification by providing concrete, automated suggestions back to the user, based on the nature of their submission. Unfortunately, these studies primarily treat validity as a routing or binary classification problem, stopping short of addressing the specific actions required once a report is deemed invalid. B. Subclassification of Invalid Bug Reports Understanding why a bug report is invalid is also crucial, as the root cause might dictate the subsequent resolution strategy. A primary cause of invalidity is duplication, which has been extensively addressed by specialized word embedding models [26] and DL techniques [27] designed to retrieve similar historical reports. Beyond duplicates, reports are frequently marked invalid because they stem from distinct root causes such as user misunderstandings, faulty environment configurations, or they are feature requests mistakenly reported as bugs. In order to define these root causes, researchers have conducted empirical analyses and proposed various taxonomies to subclassify invalid bug reports. Sun [9] attributed many invalid bug reports to testing errors or a lack of system knowledge. Similarly, Su et al. [10] found that 45% of invalid defects are actually ”Working as Designed.” Herzig et al. [7] further highlighted this expectation gap, showing that over 33% of reported bugs are simply misclassified feature requests or documentation issues. Panichella et al. [11] found that GitHub ”wontfix” issues are often just support questions or out-of-scope requests. Laiq et al. [5] used topic
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
4
TABLE I: Root Causes of Invalid Bug Reports Mentioned by Source Sources Identified Common Root Causes
Laiq et al. (2023) [5]
Su et al. (2017) [10]
Panichella et al. (2021) [11]
Sun (2011) [9]
Faulty Configuration & Environmental Setup
Faulty Configuration
Wrong Settings
Configuration Problem on the User Side
Wrong Configuration Parameters
Non-reproducibility
Non-reproducible
Irreproducible Defects
Not Replicable Bug
Reproducing Not Successful
Working as Designed
Working as Expected
Working as Designed
Wrong Usage of Functionality
Misunderstanding on Functionality
Does Not Comply with User Expectations
Feature Request
New Requirement
Feature Requests
Feature Requests
Feature Request
Wrong Version - Already Fixed
Wrong Version
Problem Already Fixed with the New Version
Build was Not New Enough to Contain the Fix
Versions are Open to Backport
Error in External Systems
Issues Caused by Third Party Libraries
External System & Dependency Issues
Limitations and Behaviors of External Constraints
Question
modeling (LDA) to extract broader patterns, identifying nonreproducibility, faulty environments, and user inexperience as key drivers of invalid bug reports. The overview of the mentioned invalid bug report root causes per source are displayed in Table I. Unfortunately, none of the previous studies focused on fully automated invalid subclass detection. As bug tracking paradigms evolve into the Generative AI era [12], subclassification has become a prerequisite for routing invalid bug reports away from human developers and toward intelligent support agents.
Herzig et al. (2013) [7]
Question
detect fabricated, AI-generated bug reports in bounty programs. Furthermore, tools like ImproBR [30] use RAG pipelines alongside LLMs to automatically detect missing execution steps in bug reports and dynamically generate improved, reproducible instructions. While these advanced systems significantly reduce the maintenance burden by contextualizing and refining codelevel bugs, they largely ignore the distinct workflow required for providing direct answers to invalid bug reports, such as configuration problems or conflicting user expectations. D. Agentic Web Search for Information Retrieval
Integrating web search agents addresses the siloed nature of traditional maintenance tools by treating the live web as State-of-the-art LLMs have revolutionized automated bug dynamic external memory [31], [32]. Unlike RAG systems that triage by enabling sophisticated reasoning over technical rely on a static knowledge base, agentic web search, which is documentation and bug reports. For instance, the Judge The often framed as Agentic Deep Research [32], uses autonomous Votes framework [18] evaluated the performance of LLMs in feedback loops to verify user-reported symptoms against realvalidity classification, leveraging model voting mechanisms time external sources such as upstream changelogs, third-party and comparisons between similar issues to determine report issue trackers, and community discussions. By going beyond legitimacy. Despite these capabilities, LLMs often hallucinate internal wikis, these agents can uncover “External Dependency” when lacking project-specific context, leading to inaccurate or or “Wrong Version” root causes that static systems would miss, generic resolutions. providing the depth of context needed for large-scale software To address these limitations, recent studies have heavily products. adopted RAG to ground LLM outputs in repository realities. Recent frameworks further show that technical triage often By retrieving relevant documents before generation, RAG requires multi-step retrieval rather than a single query. Systems ensures that the model’s responses are anchored in the specific such as WebExplorer [33] and BrowseMaster [34] scale tool codebase and history of a project. For example, RAG4Tickets usage and employ specialized agent architectures to support [28] employs semantic embeddings to retrieve historical bug long-horizon reasoning on the web. In particular, WebExplorer reports, passing them as context to an LLM to generate context- enables agents to iteratively refine queries and navigate complex aware ticket resolutions. web structures, interactively following links and filtering RAG has also been applied to assess the quality and developer discussions to reconstruct the chain of events behind authenticity of the reports themselves. A 2026 study [29] a software failure or to clarify the true cause of a bug report utilized RAG combined with zero-shot reasoning to successfully beyond the project’s own knowledge base. C. LLMs and RAG in Software Maintenance
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
We specifically utilize WebThinker [31] as our foundational framework due to its ”Think-Search-and-Draft” architecture, which allows a Large Reasoning Model (LRM) to interleave its internal chain-of-thought with deep navigation of web elements. This approach prevents the reasoning process from becoming fragmented, as the agent can autonomously bridge knowledge gaps in real-time before drafting a final response. By integrating a Deep Web Explorer module, WebThinker empowers models to autonomously search and extract information from the web when encountering knowledge gaps during the reasoning process [31]. This methodology ensures that the generated nocode fixes are grounded in evolving technical truths rather than static training data. By leveraging such an agentic architecture, the system can provide reporters with actionable workarounds that are validated against the current state of the software’s broader dependency landscape and give suggestions based on external resources. E. Bridging Customer Support and Bug Tracking with No-Code Fixes
5
Recent large-scale datasets, such as GitBugs [39], provide over 150,000 bug reports with resolution metadata, which greatly supports tasks such as duplicate detection and automated triaging. However, datasets like GitBugs do not explicitly label bug report content by invalid subclasses, nor do they assess the semantic correctness or user-helpfulness of the actual textual responses provided to reporters. The lack of a standardized benchmark for evaluating AI-generated workarounds limits the development of automated no-code fix systems. This gap motivates the creation of our novel benchmark to systematically evaluate how well AI can generate accurate, safe, and helpful no-code fixes for invalid bug reports. III. M ETHODOLOGY In this section, we explain the curated taxonomy of invalid subclasses, the IssueSupport Benchmark creation steps, and the IssueSupport methodology. A. Invalid Bug Report Taxonomy Based on Root Cause
The literature review leads us to a distinct taxonomy As LLM capabilities expand, their role is shifting from based on root causes for invalid bug reports. Unlike generic passive classification tools to active resolution assistants. In labels, IssueSupport Benchmark prioritizes the root causes of a standard bug tracking framework, valid bug reports are invalidity. The final clusters of invalid subclasses are External routed to developers. However, invalid bugs, often stemming System & Dependency Issues, Faulty Configuration, Feature from user misunderstandings, faulty configurations, or external Request, Non-reproducible, Question, Working as Designed, system dependencies, traditionally require time-consuming and Wrong Version. The mapping definitions of each invalid customer support intervention. This dynamic closely mirrors subclass per source are given in Table I. As detailed in the challenges faced in consumer-facing applications, where a Table II, the taxonomy organizes these reports by three primary massive volume of user-reported issues are not actual system dimensions: failures, but rather requests for usage workarounds or policy • Invalid Subclass (Root Cause): Represents the root cause clarifications [35]. The fundamental nature of these consumerof the invalidity. facing queries is similar to the subclasses of invalid bug • Description: Outlining the specific symptoms or traits of reports in software engineering; both scenarios involve users the report. struggling with the intended design or configuration rather • High-Level No-Code Fix Guideline: Identifies actionable, than encountering a source code defect. Because consumer non-invasive solutions such as configuration guidance or issues and invalid bug reports share these underlying root workarounds. causes, they naturally necessitate a similar resolution process. By applying this taxonomy to the real-world bug reports, Recent benchmarks like ECom-Bench [36] evaluate the capacity we can effectively distinguish between the root causes, such as of multimodal LLM agents to resolve real-world customer configuration mismatches, conflicting user expectations, and support issues by simulating user interactions and providing external dependencies. This structured approach ensures that immediate solutions. In app-centric ecosystems, instead of the final benchmark is not merely a collection of invalid bug silently dismissing an invalid bug report, the AI agent should reports, but a categorized dataset optimized for automated act as an advanced support bot [12], which actively resolves resolution by invalid subclass. the issue by generating actionable no-code fixes, such as step-by-step UI configuration guides, explanations of design choices, or workaround instructions. Applying this customer B. IssueSupportBenchmark support resolution paradigm directly to software engineering 1) Data Selection & Fetching: To form a clear picture of allows organizations to automate the handling of invalid bug our approach, we focused on distinguishing between valid and reports, which might improve first-contact resolution rates while invalid bug reports. Since the primary objective of this paper keeping developers strictly focused on valid bug reports, which is to subclassify invalid bug reports by root cause and provide are source code defects. no-code fixes, we targeted user-facing open-source repositories on GitHub. F. The Need for a No-Code Fix Benchmark Before detailing our dataset, it is important to clarify the While extensive benchmarks exist for evaluating LLMs on terminology used in this ecosystem. In the context of GitHub, source code generation and program repair, such as SWE- GitHub Issues correspond directly to bug reports. Throughout bench [37] and Livecodebench [38], there is a notable gap in this paper, we treat GitHub Issues as the primary unit of analysis benchmarking the generation of natural language no-code fixes. for bug reports, and we use the two terms interchangeably.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
6
TABLE II: Taxonomy of Invalid Bug Reports: Root Causes, Descriptions, High-Level No-Code Fix Guidelines and Examples Invalid Subclass (Root Cause) External System & Dependency Issues Faulty Configuration & Environment Setup
Description
High-Level No-Code Fix Guideline
Example
Failure triggered by defects, outages, or hard limitations in third-party systems or hardware outside the reporters’ control. Incorrect user-side setting, parameter, or local environment mismatches before the testing.
Direct the user to the relevant thirdparty system and explain that the issue originates from an external dependency. Explain why the problem arises. Describe the required configuration and how to apply the necessary modifications. Inform the user that the report is a feature request rather than a bug. If there is a workaround solution, explain it. Then, decide whether to implement the requested feature and notify users accordingly. Ask the user for a detailed description, Steps to Reproduce (S2R), Expected Behavior (EB), Observed Behavior (OB), and complete environmental details if any of them are missing. If they are already known, but the bug exhibits intermittent behavior, suggest trying again at a different time. Provide a solution if available, or inform the user if the question cannot be answered. Explain the system or specific feature in natural language.
R: Brave icon is missing from Notification Ads after macOS update. M: It is an upstream issue origins from Chromium. [40] R: Tabs aren’t visible or clickable after going full screen mode. M: Enabling Always show toolbar in full screen option will work. [41] R: Like Chrome & Firefox, can Brave also have a development version? M: We have already Nightly, Beta and Dev versions. No need to proceed. [42]
Feature Request
User requests new functionality or enhancements rather than identifying a fault in the existing code.
Nonreproducibility
Cannot recreate the bug due to missing steps, race conditions, or one-time occurrences.
R: Crash happened in Brave Ads page. M: Blocked until we get more information regarding crashes. [43]
R: What format does Brave use to store date/time for Ads feature? M: Unix Epoch, thanks. [44] Working as R: The browser keeps getting restarted Designed after each update. I lose all my tabs. (Conflicting M: Changes only reflect when browser Expectation) is restarted. This current behavior is expected. [45] Wrong Version Issue exists in an outdated or unsup- Inform the user that the issue has R: While setting up Brave Wallet, (Already Fixed) ported version but is already resolved already been resolved in a newer the recovery phase shows duplicate in a newer release. version and kindly request that the options, indicating a posible bug. user updates to that version. M: It was already resolved in version 1.35.x. [46] Note: The reporter (R) and the maintainer (M) actors have been referred in abbreviations at the Example column. Question
The report is an inquiry seeking help or discussion rather than a functional defect. Software functions according to specifications, but the user perceives the correct behavior as a defect due to conflicting expectations.
To ensure high data quality and relevance, we established subset, with their explanations and counts, is detailed in strict static criteria for selecting our target repository: Table III. • Popularity: The repository must have at least 20,000+ 2) Evaluation Benchmark: The Evaluation Benchmark is a stars, ensuring it is a widely used and active project. curated subset of invalid bug reports designed to provide a high• Volume: The repository must contain 40,000+ bug reports quality foundation for our analysis. While the initial extraction to provide a statistically significant dataset. from the Brave repository yielded 8,289 reports, their native • Labeling: Crucially, the repository must actively use a labeling system often reflects implementation-specific details set of labels that explicitly categorize invalid bug reports. rather than the fundamental reasons for invalidity. To bridge this After conducting research based on these parameters, we gap, we implemented a curation pipeline that re-evaluates these identified the Brave browser repository5 as the ideal candidate. reports based on the previously defined root-cause oriented It not only met our volume and popularity thresholds but also invalid subclass taxonomy. The comprehensive workflow for maintained a rigorous labeling system for subclassifying invalid this benchmark curation and manual labeling methodology is issues, making it highly suitable for our classification goals. illustrated in Figure 2. By mining the Brave repository with the GitHub API, we a) Automated Filtering: Starting with 8,289 invalid bug constructed a comprehensive dataset comprising 38,789 closed reports, we have decided to filter out some of the Brave-native bug reports (captured as of January 19, 2026). Following the invalid labels mentioned in Table III, as some of those labels’ 6 criteria specified in the Brave Browser wiki , we identified objectives are not aligned with this paper’s goals. Here are the 8,289 reports as invalid based on labels that the project reasons for the filtered-out labels. considers invalid. The distribution of labels within the invalid closed/duplicate: Duplicate reports were removed as they 5 https://github.com/brave/brave-browser typically point to existing tickets rather than requiring unique 6 https://github.com/brave/brave-browser/wiki/ Issue-with-missing-milestones no-code fixes.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
7
TABLE III: Invalid Bug Report Labels, Descriptions, and Distribution in Brave Browser Repository [47] Label closed/duplicate closed/invalid closed/stale closed/not-actionable closed/wontfix closed/works-for-me closed/no-milestone question support closed/workaround closed/fixable-by-custom-rules Total
Description Issue has already been reported. Not an issue with the browser. Possibly opened by mistake or there could be user error. Issue is no longer relevant, often due to inactivity or deprecated functionality. Closed because there is no way to reproduce the error or there is no clear action the team can take. Working as intended. The reported behavior will not be fixed. Issue was closed because nobody was able to reproduce. No longer an issue. Not fixed by a particular release or milestone. Seeking inquiries or clarifications. Addition/edits to articles on support website. Issue has been resolved through a workaround and does not require a client-side fix. Issue is fixable using custom rules or filter lists rather than code changes.
Count 2285 2083 1647 985 925 712 297 194 66 3 1 8289
Manual Annotation Three Subsets
Automated Filtering
5 Issue Label Filtering
1
closed/duplicate closed/stale
Brave Invalid Issue Dataset
2
Time Filtering
3
Individual Annotation Tasks
4 Random Sampling
created after 2021
95% CI, 5% margin error
closed/no-milestone
6
support invalid subclass succesfull no-code fix failed no-code fix
Fig. 2: Overview of Evaluation Benchmark Curation Workflow
closed/stale: Stale reports were excluded because they do not clearly indicate whether the reported issue is valid or not. These reports typically require additional investigation or clarification, but the discussion was discontinued due to prolonged inactivity. As a result, the issue became irrelevant over time. Importantly, this irrelevance does not imply that the report is invalid; rather, the comment thread is incomplete, making it impossible to determine the actual validity of the issue. closed/no-milestone: Reports labeled as closed/nomilestone were excluded because they were not closed due to the invalidity of the reported issue. Based on our analysis, these reports describe valid issues that were eventually fixed in a subsequent release but were not assigned to a specific milestone, such as in the case of hot-fixes. Therefore, these reports cannot be considered explicitly invalid or attributed to user-side errors. support: Reports labeled as support were excluded because they do not contain discussions related to issue resolution or root cause analysis. Instead, maintainers typically redirect the reporter to the support team without providing further technical details or insights into the requested action. As a result, these reports not only lack sufficient information
but also do not provide the necessary context to determine whether the issue is invalid. After these exclusions, the filtered dataset consists of 4,463 invalid bug reports. Furthermore, among these 4,463 invalid bug reports, we decided to keep the reports created after and including 2022 (6 years in until the first created issue report in 2016). Main reasons to apply this filter are: Rich Knowledge Base and Discussion History: Over several years, the Brave community has generated a vast corpus of technical discussions and community-driven solutions. By focusing on more recent reports, we leverage this established knowledge base, which provides the depth of context necessary to identify and reconstruct recurring no-code fixes. • Project Maturity and Maintenance Phase: As the project reached a mature maintenance phase, it adopted standardized reporting templates and established a stable code base. During this stage, it is observed that the majority of reports are created by members of the external community rather than internal maintainers. This demo•
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
graphic represents our primary target audience, as they are more likely to submit reports that, while technically invalid, can be resolved through no-code fixes.
8
do not contain sufficient information to reach a definitive classification. c) Developer Workflow: Reserved for reports that are opened for internal maintenance tasks that only The filtered dataset consists of 1,404 invalid bug reports. include the details, such as the task description and b) Random Sampling: To establish a high-quality “gold acceptance criteria. standard” for our evaluation benchmark, we conducted a While our curation pipeline aims to isolate invalid manual annotation process on a statistically significant subset bug reports, including these categories ensures the of the data. Using Cochran’s sample size formula [48], we benchmark remains a transparent and representative randomly selected 302 reports from the total pool of invalid reflection of the repository’s reporting environment. bug reports (N = 1, 404). This sampling strategy ensures 2) Comment-Level Extraction: a 95% confidence level with a 5% margin of error. This sampled subset allows for a robust assessment of both invalidity • No-Code Fix Identification: We isolated individual subclassification and the extraction of proposed no-code fixes. comments that proposed no-code fix suggestions to the reported issue. Unlike the report-level classification, c) Manual Annotation Task Assignment: The manual which is mutually exclusive, this is a multi-label task; validation of the 302 sampled invalid bug reports was conducted a single bug report may contain multiple distinct by three authors using a rotating-pair configuration. The dataset comments identified as potential no-code fixes. was partitioned into three subsets, with each subset assigned • Solution Verifiability: To evaluate the reliability of these to a different pair of authors for independent review. This interventions, we extracted both successful and failed distribution resulted in each author assessing two-thirds of suggestions based on the following criteria: the total sample. In cases where the two primary annotators assigned conflicting labels, the third author who had not – Successful Fixes: Comments where the proposed participated in the initial assessment of that specific subset workaround or clarification was explicitly validated. reviewed the report to make the final determination. Evidence of success includes: (1) direct acknowld) Manual Annotation: Before finalizing the rules and edgment from the reporter via text or reaction (e.g., steps of the manual evaluation, the three authors initially a thumbs-up emoji), (2) the reporter closing the manually annotated 7 random invalid bug reports independently. issue immediately following the suggestion, or (3) a Then, the labeling reasoning and perspectives have been maintainer closing the issue while citing the specific discussed to finalize the manual annotation keypoints for a comment as the resolution, such as ’Issue closed robust annotation process. based on this workaround...’. – Failed Fixes: Comments containing suggestions During the manual analysis, we considered several key data that were attempted but ultimately rejected. These points in the analysis process: are identified by subsequent feedback from the • Issue Metadata: We examined the title and description reporter or maintainers indicating the solution was for clarity and quality of the demonstration of the issue, ineffective, such as ’This did not work’. alongside the original labels assigned by developers and f) Inter-rater Reliability: We evaluated the agreement temporal events (e.g., the duration between comments) as between the annotators using Cohen’s kappa (κ) for invalid captured by the GitHub API. subclass annotation and the Jaccard similarity coefficient for • Content and Artifacts: We evaluated the comment threads for technical quality and relevance, including at- the unstructured no-code fix comment extraction. The resulting tachments like screenshots, screen recordings, and GitHub- scores were κ = 0.5732 (moderate agreement) for invalid specific reactions (e.g., thumbs up, rocket emojis) which subclass annotation. For no-code fix comment extraction, we observed a Jaccard similarity of Js = 0.7322 for successfull nosignal community consensus. code fixes and Jf = 0.9338 for failed no-code fixes, confirming e) Annotation Granularity: The manual annotation was high agreement on the selection of the no-code fix content. conducted at two levels: g) Final Benchmark & Observations: Following this 1) Report-Level Classification: process, we established a gold-standard evaluation benchmark • Subclassification of Invalids: Each report is assigned containing both verified invalid subclasses and no-code fixes. a single invalid subclass according to the taxonomy The final invalid subclass distribution is given in Table IV. defined in Table II, identifying the specific root cause Based on the labeling results, 114 (37.75%) of the sampled of its invalidity. bug reports do not belong to the pre-defined invalid subclass • Outlier Annotation: To ensure the integrity of the taxonomy, where 60 (19.87%) of these bug reports are labeled benchmark and account for real-world edge cases, we as valid, 23 (7.62%) of them are labeled as no conclusion and introduced three outlier labels: 2 (0.66%) of them were labeled as duplicate. This distribution a) Valid: Assigned to reports that, upon manual review, highlights the practical inconsistency between the invalid label are found to represent actual software defects definitions and final label annotation. requiring a code change. Furthermore, the benchmark details the distribution of nob) No Conclusion: Reserved for reports where the code fixes across the invalid subclasses. Out of the 188 validly provided content and subsequent comment threads categorized invalid bug reports, a total of 159 successful and
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
Subclass Working as Designed Feature Request Nonreproducibility External System & Dependency Issues Faulty Configuration & Environment Setup Question Wrong Version
9
Count
Count Percentage
Successful No-Code Fix Count
Failed No-Code Fix Count
48
15.89%
48
0
41 35
13.58% 11.59%
26 21
0 0
20
6.62%
20
3
19
6.29%
19
2
13 12
4.30% 3.97%
13 12
2 0
Base Annotations Total
188
62.25%
159
7
Valid Developer Workflow No Conclusion Duplicate
60 29
19.87% 9.60%
– –
– –
23 2
7.62% 0.66%
– –
– –
Outlier Annotations Total
114
37.75%
–
–
Total
302
quote: ’yep, that is a dupe’ even though the only invalid Brave label the bug report had was ’closed/invalid’.
One of the sub-patterns observed inside the no conclusion subset is that the issue remains stale for a subsequent time and then the particular feature related to the bug report gets deprecated in later updates. Example: In issue #210118 , the reporter claims that the IPFS feature inside the Brave Browser has flyout menus and by quote ’flyout menu items should have an ellipsis, as they require further user input’, which was reported in February 10, 2022. However, the issue remains stale until September 2, 2024 without any kind of clarification or suggestion about the issue. In September 2, 2024; the collaborator ’vadimstruts’ responds by quote ’The IPFS local node and scheme has been deprecated.’ Since the particular bug report remained stale for nearly 2.5 years without any fix suggestions or clarifications; no one can possibly understand whether the issue is valid or invalid.
100.00%
159
7
TABLE IV: Manual Annotation Results for the Sampled Invalid Bug Reports
A common pattern among Brave maintainers is the creation of ”follow-up” issues that lack self-contained context. Example: Issue #258429 addressed an 81% failure rate in crash dump uploads on the Brave Stable channel caused by an incorrect sampling probability. Although a patch was deployed and the issue is closed on October 20, 2022, issue #2614810 was opened that same day as a direct follow-up. That issue particularly identified that the original problem persisted during frequent, consecutive crashes. However, the child issue relied entirely on the parent issue for context, by quote, ’This is follow-up for #25842.’.
The poor maintenance choice is that the child issue is 7 failed no-code fixes were identified. Notably, there is a inherently coupled with the parent issue, and the good practice pronounced need for no-code resolutions in specific subcate- should have been the parent issue being re-opened after another gories. Excluding Non-reproducibility and Feature Request, the edge-case occurs. remaining invalid subclasses demonstrate a critical demand for succesfull no-code fixes, as their definitions depend on how the The other pattern is about the ground-truth no-code fixes. no-code fix resolves the issue. However, Feature Request and Non-reproducibility invalid subclasses are inherently detectable Some of the no-code fix comments left by maintainers tend to by the bug report content only. The External System, Faulty reference another bug report without sufficient explanation on Configuration, and Question subclasses not only required a what the final conclusion is. high volume of successful no-code fixes, but also accounted Example: Issue #3351111 opened an issue about the ”Select and Search” for all 7 of the failed no-code fix attempts. This phenomenon function of the ”Lazy Chrome” extension, which stopped working after they underscores the potential complexity of providing actionable, updated to Brave version 1.59. However, the same issue has been opened in non-code guidance for these specific operational, environmental, the Lazy Chrome repository12 and the main solution discussions took part in and design-related issues. that issue report. As a result, in the Brave issue report, the author responds, Particular patterns and outliers across the bug reports have by quote, ’Figured out due to change in brave’s extension permission control emerged after manual annotation and constructive discussions frog1014/lazy chrome#22’. between the authors. As an example of the outlier pattern, the manually annotated two duplicate bug reports do not have the original Brave issue label closed/duplicate. Example: In issue #240897 , the contributor ’srirambv’ directly asks if the reported issue is by quote: ’Dupe of #23898?’ and the author responds by 7 https://github.com/brave/brave-browser/issues/24089
The curated benchmark serves as a valuable asset for the software engineering research community. By providing a ground truth set of invalid bug reports paired with their specific no-code solutions, it enables the rigorous evaluation of automated classification models and solution recommendation systems, facilitating the development of tools that can alleviate the maintenance burden in open-source projects.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
10
Fig. 3: Overview of IssueSupport Methodology
C. IssueSupport Methodology We experimented with four distinct methodologies: (1) Vanilla LLM Pipeline, (2) Vanilla LLM Pipeline Without Prior Invalid Subclass Information, (3) RAG Pipeline, and (4) Agentic Web Search Pipeline. The tested methodologies return one invalid subclass and one suggested no-code fix, except for (2), and differ based on the tools and sources they use. Figure 3 presents the overall methodology. 1) Prompt Configurations: All mentioned approaches utilize zero-shot prompting strategy, consisting of two distinct components: a system prompt and a user prompt. The system prompt constructs the task’s context by formally defining both an invalid bug report and a no-code fix. It instructs the model to classify the given report into a specific invalid subclass and generate an appropriate no-code fix. To guide this reasoning, the prompt incorporates the subclass descriptions and high-level no-code fix guidelines outlined in Table II, except for approach (2). Finally, the model is strictly instructed to return its response in a structured JSON format with the given three fields: classification (except for (2)), reasoning, and no code fix. Since approach (2) is for ablation study purposes, 8 https://github.com/brave/brave-browser/issues/21011 9 https://github.com/brave/brave-browser/issues/25842
it lacks subclass descriptions and guidelines for the no-code fixes in the system prompt. Conversely, the user prompt serves strictly as the data input layer, injecting only the raw markdowns of title and body of the bug report for vanilla LLM pipelines, and additionally retrieved information for RAG and agentic web search pipelines. The complete base system and user prompt templates are available in the prompt.py file of our replication package13 . 2) Vanilla LLM Pipeline: The Vanilla LLM pipeline serves as our primary zero-shot baseline. In this setup, we have evaluated three proprietary and three open-source LLMs. These LLMs were selected based on their Intelligence Index from Artificial Analysis14 . This index evaluates core capabilities, such as reasoning and knowledge, by aggregating results from diverse datasets. According to the leaderboard from March 25, 2026, selected proprietary models are Gemini 3.1 Pro Preview, GPT-5.4 (xhigh), and Claude Opus 4.6 (max), and the open-source models are GLM-5 [49], MiniMax M2.7 [50], and Kimi 2.5 [51]. In this setting, we picked the bestperforming open-source and proprietary LLMs based on their overall vanilla invalid subclassification performances for use in other experimental setups. 3) Without Prior Invalid Subclass Information: To rigorously quantify the significance of our proposed taxonomy and structured guidelines, we formulated the Without Prior Information
10 https://github.com/brave/brave-browser/issues/26148 11 https://github.com/brave/brave-browser/issues/33511
13 \url{https://figshare.com/account/articles/32228832?file=64399383}
12 https://github.com/frog1014/lazy chrome/issues/22
14 https://artificialanalysis.ai/methodology/intelligence-benchmarking
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
11
ablation pipeline. This setup mirrors the Vanilla LLM pipeline; reconstructed from the issue creation date, which we discuss however, all invalid subclass priors are deliberately removed as a threat to validity. Tool observations were concatenated from the system prompt. The LLM is neither provided with in chronological order and passed as retrieved context to the the definitions of the seven invalid subclasses nor the tailored downstream classifier/generator, which then produced the final guidelines for generating no-code fixes. Instead, the model is invalid subclass and no-code fix. All language-model calls in the browsing orchestrator were prompted generically as an “expert software engineer” and issued through a single OpenAI-compatible client configured must deduce a no-code fix resolution relying solely on its to OpenRouter [53]. For the OpenRouter-Kimi configuraintrinsic, pre-trained knowledge. tion, orchestration used moonshotai/kimi-k2.5 with 4) RAG Pipeline: The RAG pipeline is designed to ground temperature 0.7 and max_tokens = 8192, while the final the LLM’s deductive reasoning in historical project data and classifier used the same backbone with temperature 0.2, project wiki. We populated two distinct vector databases: max_tokens = 8192, and JSON-object response formatone containing historical bug reports and another con15 ting when supported by the endpoint. Gemini-only agentaining official Brave Wiki documentation . We utilized 16 tic runs used google/gemini-3.1-pro-preview, with the zembed-1 text embedding model, selected from the 17 google/gemini-2.5-flash for query generation, page Agentset embedding models leaderboard , to generate vector embeddings. These embeddings are stored in LanceDB18 , compression, and evidence summarization. chosen for its efficient metadata filtering, which is critical D. Evaluation for our chronological data leakage prevention. To evaluate the effectiveness of the proposed methodologies, When evaluating a new bug report, the system uses the we executed each pipeline in three independent trials to report’s title and body as the query for a dense vector search. We reduce the variance that can be caused by the probabilistic employ a two-stage retrieval process: first, the system retrieves nature of LLMs [54]. We kept the backbone LLMs at their the top k = 20 most semantically similar candidates from default temperature settings to take advantage of their inherent each database. To strictly prevent data leakage, historical bug reasoning capabilities, avoiding the constraints of the zeroreports are chronologically filtered using a metadata constraint temperature configuration. (created_at < current_created_at) to ensure only The evaluation is structured to address our two primary prior issues are retrievable. research questions: These initial candidates are then processed by the 19 In RQ1, we assess the performance of the invalid subclassizerank-2 cross-encoder re-ranking model, also selected from the Agentset leaderboard20 . The reranker refines the fication task using the weighted F1-Score, which provides a selection to the top k = 5 most relevant contexts. These reliable measure given the imbalanced distribution of invalid highest-ranked contexts are appended to the LLM prompt, subclasses. For each bug report, the final classification label is providing the contextual data the model needs to formulate its determined by majority vote across the three independent runs. In RQ2, in order to evaluate the semantic and functional final classification and no-code fix. utility of the generated no-code fixes, we employ a dual-metric 5) The Agentic Web Search Pipeline: To incorporate dyapproach. For each bug report, the final classification label is namic external evidence, we implemented an agentic web determined by aggregating the results of the three independent search pipeline loosely inspired by WebThinker-style reason– runs. search–observe loops [31]. Given a bug report title, body, and • BERTScore: This metric provides a token-level semantic creation date, the pipeline first constructs a compact research similarity analysis. We compute this metric by comparing brief containing the issue context and the information need. model outputs specifically against the successful groundAn orchestrator LLM then performs a bounded informationtruth fix, since semantic closeness to the failure case is gathering loop: it proposes natural-language search queries, not the target objective. retrieves web results through the Serper Search API [52], • Judge-LLM Evaluation: To assess practical alignment, removes the canonical original issue page to mitigate direct we utilize an LLM-as-a-judge framework. The judge role leakage, and summarizes the retained evidence with source is fulfilled by google/gemini-3.1-pro-preview, URLs. When useful, the agent can also request a page visit; selected for its strong instruction-following and reasoning fetched page content is compressed into the same evidence capabilities. As illustrated in Figure 4, the Judge LLM is format. presented with a contrastive three-part prompt consisting The search trajectory is constrained to limit cost, latency, of the predicted fix, a successful ground-truth no-code and noise. In our experiments, the outer loop was capped at 35 fix, and a failed ground-truth no-code fix. This allows the turns and at most three search actions; page visits were treated judge to perform a qualitative comparative assessment, as part of the same bounded trajectory. Search results were determining whether the model’s suggestion aligns with obtained from the live web index at experiment time, rather than successful resolution patterns or mimics known failure 15 https://github.com/brave/brave-browser/wiki modes. 16 https://zeroentropy.dev/articles/introducing-zembed-1-the-worlds-best-multilingual-text-embedding-model/
17 https://agentset.ai/embeddings
IV. R ESULTS 19 https://zeroentropy.dev/articles/zerank-2-advanced-instruction-following-multilingual-reranker/ We structurally organize our experimental findings to ad20 https://agentset.ai/rerankers dress our primary research questions regarding the invalid 18 https://www.lancedb.com/
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
12
Judge LLM Evaluation Prompt (No-Code Fix Comparison) You are an expert Judge LLM evaluating no-code fixes for GitHub issues. Suggested Fix: {’pred_fix’} Ground Truth Successful Fix: {’gt_success’} Ground Truth Failed Fix: {’gt_failed’} Does the suggested fix resemble the successful no-code fix, or the failed no-code fix? Provide your reasoning in a clear structure. Conclude clearly with either ’RESEMBLES SUCCESSFUL’ or ’RESEMBLES FAILED’ at the end.
Fig. 4: Evaluation prompt for the Judge LLM. It employs a contrastive three-part assessment approach, providing the Judge LLM with the Predicted Fix, a Successful Ground Truth Fix, and a known Failed Ground Truth Fix.
bug report subclassification and the no-code fix generation approach. To facilitate the reproducibility of our findings and support future research in automated subclassification and no-code fix generation for invalid bug reports, our complete replication package is publicly available at https://figshare.com/ s/dc79aaf924dac61a1095. A. RQ1: How can invalid bug reports be subclassified according to their root causes?
For Kimi K2.5, introducing an Agentic Web Search pipeline acts as the most effective context engineering method, improving its overall Weighted F1 from 0.60 (Vanilla) to 0.62, while RAG does not affect the overall Weighted F1 score. Interestingly, Agentic Web Search slightly degrades Gemini’s overall subclassification Weighted F1 score to 0.64, though it successfully lifts its performance on the difficult Wrong Version subclass from 0.00 to 0.20. The worst-performing contextengineered setup is RAG + Kimi K2.5, which maintains an aggregate score of 0.60 but suffers severe degradation in the External System & Dependency subclass (dropping to 0.42 from 0.47 in the Vanilla setting).
To evaluate how effectively models identify the root causes of invalid bug reports, we used the Weighted F1-Score as our primary metric to account for class imbalance. The results are B. RQ2: How can invalid bug reports be resolved with nosummarized in Table V. 1) RQ1.1: How do LLMs perform in subclassification code fixes? of invalid bug reports?: Evaluating the zero-shot baseline To evaluate the semantic fidelity and functional utility of (Vanilla LLM) reveals the intrinsic capabilities of LLMs generated no-code fixes, we employ BERTScore and a Judgefor subclassification without external context. As shown in LLM success rate. The results are distributed across Table VI Table V, Gemini 3.1 Pro is the best-performing model in the and Table VII. Vanilla setup, achieving an overall Weighted F1-score of 0.65. 1) RQ2.1: How do LLMs perform in generating no-code Conversely, Minimax M2.7 is the worst-performing model, fixes without subclass knowledge as prior information?: To with an overall Weighted F1-score of 0.53. understand the baseline generative capability of the models, we At the subclass level, models generally exhibit strong conducted an ablation study where invalid subclass descriptions performance in classifying Feature Request (e.g., Gemini 3.1 and taxonomies were removed (the “Without Priors” pipeline). Pro at 0.81, GPT-5.4 at 0.76) and Non-reproducibility (e.g., As illustrated in Table VII, Gemini 3.1 Pro emerges as the bestGPT-5.4 at 0.81), indicating that these categories likely contain performing model in this setup, achieving an overall Judge explicit linguistic cues. However, Wrong Version remains the success rate of 62.4% and a BERTScore F1 of 0.82. Kimi most challenging category across all models in the Vanilla K2.5 performs worst in this isolated setup, achieving an overall setup, with both Gemini 3.1 Pro and GPT-5.4 scoring a 0.00, Judge success rate of 58.1%. and Minimax M2.7 scoring a marginal 0.25. Notably, both models maintain high textual fidelity 2) RQ1.2: Does context engineering affect the invalid bug (BERTScore F1 ∼0.81-0.82) even without subclass priors, indireport subclassification performance?: The integration of cating that their pre-trained weights naturally align with humancontext engineering yields notable, however, model-dependent like developer formatting, even if the underlying functional shifts in classification accuracy. According to Table V, RAG logic is occasionally flawed. + Gemini 3.1 Pro achieves the highest overall subclassifica2) RQ2.2: Does utilizing the root cause subclass information performance across all experimental setups, reaching a tion of invalid bug reports affect the no-code fix generation Weighted F1 of 0.66. RAG boosts Gemini’s performance in performance?: Providing root cause subclass definitions generspecific subclasses, peaking at 0.85 F1 for Non-reproducibility ally improves the practical resolution capabilities of the models. and improving Working as Designed to 0.70. Comparing the “Without Priors” pipeline to the “Vanilla LLM”
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
13
TABLE V: Comprehensive Subclassification Performance: Vanilla, RAG, and Agentic Web Search (F1 and Weighted F1) Subclass / Metric
Gemini 3.1
GPT-5.4
Opus 4.6
Kimi K2.5
GLM-5
Minimax M2.7
RAG+Gemini
RAG+Kimi
Agentic+Gemini
Agentic+Kimi
External System & Dep. Faulty Config & Env. Feature Request Non-reproducibility Question Working as Designed Wrong Version
0.62 0.34 0.81 0.79 0.76 0.66 0.00
0.56 0.30 0.76 0.81 0.75 0.60 0.00
0.71 0.23 0.74 0.77 0.71 0.49 0.19
0.47 0.34 0.77 0.63 0.73 0.64 0.26
0.49 0.24 0.75 0.78 0.69 0.57 0.00
0.46 0.07 0.75 0.66 0.69 0.49 0.25
0.67 0.27 0.79 0.85 0.79 0.70 0.00
0.42 0.26 0.76 0.76 0.71 0.62 0.26
0.57 0.32 0.77 0.76 0.76 0.67 0.20
0.43 0.29 0.79 0.72 0.79 0.64 0.29
Overall Weighted F1
0.65
0.61
0.59
0.60
0.58
0.53
0.66
0.60
0.64
0.62
TABLE VI: No-Code Fix Evaluation Across LLMs (Vanilla Only) Vanilla LLM Class
Model
Judge (%) BERTScore F1
Gemini 3.1 Pro GPT-5.4 External System & De- Opus 4.6 Kimi K2.5 pendency Issues GLM-5 Minimax M2.7
63.3% 73.3% 68.3% 66.7% 55.0% 38.3%
0.83 0.82 0.82 0.82 0.82 0.81
Gemini 3.1 Pro GPT-5.4 Opus 4.6 Kimi K2.5 GLM-5 Minimax M2.7
59.0% 56.4% 56.4% 56.4% 50.0% 51.3%
0.83 0.82 0.82 0.82 0.82 0.82
Gemini 3.1 Pro GPT-5.4 Faulty Configuration & Opus 4.6 Kimi K2.5 Environment Setup GLM-5 Minimax M2.7
52.6% 47.4% 33.3% 36.8% 19.3% 10.5%
0.82 0.81 0.82 0.81 0.81 0.81
Non-reproducibility
Gemini 3.1 Pro GPT-5.4 Opus 4.6 Kimi K2.5 GLM-5 Minimax M2.7
61.9% 74.6% 68.3% 65.1% 61.9% 55.6%
0.81 0.80 0.80 0.80 0.80 0.80
Question
Gemini 3.1 Pro GPT-5.4 Opus 4.6 Kimi K2.5 GLM-5 Minimax M2.7
51.3% 41.0% 53.8% 59.0% 51.3% 41.0%
0.83 0.81 0.82 0.82 0.83 0.82
Gemini 3.1 Pro GPT-5.4 Working as Opus 4.6 Designed (Conflicting Kimi K2.5 Expectations) GLM-5 Minimax M2.7
85.4% 88.9% 66.0% 75.0% 70.8% 52.1%
0.83 0.82 0.82 0.82 0.82 0.82
Gemini 3.1 Pro GPT-5.4 Wrong Version (Already Opus 4.6 Kimi K2.5 Fixed) GLM-5 Minimax M2.7
25.0% 19.4% 52.8% 44.4% 22.2% 38.9%
0.81 0.81 0.81 0.81 0.81 0.81
Gemini 3.1 Pro GPT-5.4 Opus 4.6 Kimi K2.5 GLM-5 Minimax M2.7
63.1% 64.9% 58.4% 60.9% 51.8% 42.9%
0.82 0.81 0.82 0.81 0.82 0.81
Feature Request
Overall
pipeline in Table VII, Gemini 3.1 Pro sees a marginal increase in its overall Judge success rate (from 62.4% to 63.1%), while Kimi K2.5 experiences a more pronounced improvement (from 58.1% to 60.9%). When comparing all LLMs strictly under the Vanilla configuration (Table VI), GPT-5.4 stands out as the bestperforming model, achieving the highest standalone Vanilla Judge success rate of 64.9%. In contrast, Minimax M2.7 is the worst-performing model for no-code fix generation, decreasing to an overall success rate of 42.9%, despite maintaining a high semantic BERTScore (0.81). 3) RQ2.3: Does context engineering affect the no-code fix generation performance?: The application of context engineering mechanisms, including RAG and Agentic Web Search, yields varying results depending on the underlying model and the specific invalid subclass. As detailed in Table VII, the Agentic Web Search pipeline combined with Gemini 3.1 Pro achieves the highest overall Judge success rate of 68.9%. This configuration demonstrates notable improvements in specific subclasses, achieving 74.4% in Question and increasing the Wrong Version success rate to 41.7% compared to its Vanilla baseline of 25.0%. Conversely, implementing Agentic Web Search with Kimi K2.5 results in an overall Judge success rate of 43.1%, the lowest among its tested configurations. The RAG pipeline provides a different distribution of performance. RAG combined with Gemini 3.1 Pro yields an overall success rate of 64.4%, which is higher than its Vanilla performance of 63.1%. This setup achieves the highest recorded success rate in the Working as Designed subclass at 88.2%. However, RAG underperforms the Vanilla baseline in certain subclasses; for instance, Gemini 3.1 Pro’s success rate in Faulty Configuration & Environment Setup drops from 52.6% in Vanilla to 49.1% with RAG, and Kimi K2.5 drops from 36.8% to 26.3% in the same category. V. D ISCUSSION A. Discussion on Research Questions The experimental findings across our research questions reveal a complex interplay between model reasoning capabilities and the utility of external context. Below, we interpret these results, identify key drivers of performance, and propose methodologies to overcome the observed pitfalls. 1) RQ1: How can invalid bug reports be subclassified according to their root causes?: The results for RQ1.1 and RQ1.2 demonstrate that while LLMs possess an inherent baseline for bug triage, their performance is highly sensitive
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
14
TABLE VII: Class-Based No-Code Fix Evaluation (Gemini 3.1 Pro and Kimi with Different Setups) Without Priors Class
Model
Vanilla LLM
RAG
Agentic Search
Judge (%) BERTScore F1 Judge (%) BERTScore F1 Judge (%) BERTScore F1 Judge (%) BERTScore F1
External System & Dependency Gemini 3.1 Pro Kimi K2.5 Issues
56.7% 62.1%
0.83 0.81
63.3% 66.7%
0.83 0.82
63.3% 48.3%
0.83 0.82
68.3% 46.7%
0.83 0.82
Gemini 3.1 Pro Kimi K2.5
59.0% 60.3%
0.83 0.81
59.0% 56.4%
0.83 0.82
61.5% 59.0%
0.83 0.82
62.8% 42.9%
0.83 0.82
Faulty Configuration & Environ- Gemini 3.1 Pro ment Setup Kimi K2.5
70.2% 46.4%
0.83 0.81
52.6% 36.8%
0.82 0.81
49.1% 26.3%
0.82 0.81
52.6% 20.4%
0.82 0.81
Non-reproducibility
Gemini 3.1 Pro Kimi K2.5
44.4% 42.9%
0.81 0.80
61.9% 65.1%
0.81 0.80
66.7% 57.1%
0.81 0.80
69.8% 47.6%
0.81 0.80
Question
Gemini 3.1 Pro Kimi K2.5
64.1% 51.3%
0.83 0.81
51.3% 59.0%
0.83 0.82
53.8% 59.0%
0.83 0.82
74.4% 43.6%
0.83 0.82
Working as Designed (Conflicting Gemini 3.1 Pro Expectations) Kimi K2.5
80.6% 77.1%
0.83 0.82
85.4% 75.0%
0.83 0.82
88.2% 77.8%
0.83 0.82
84.0% 54.9%
0.83 0.82
Wrong Version (Already Fixed)
Gemini 3.1 Pro Kimi K2.5
33.3% 37.1%
0.81 0.80
25.0% 44.4%
0.81 0.81
22.2% 27.8%
0.81 0.81
41.7% 38.9%
0.82 0.82
Overall
Gemini 3.1 Pro Kimi K2.5
62.4% 58.1%
0.82 0.81
63.1% 60.9%
0.82 0.81
64.4% 55.8%
0.82 0.81
68.9% 43.1%
0.82 0.82
Feature Request
to the nature of the subclass. The high success in Feature Request and Non-reproducibility suggests these classes rely on explicit linguistic cues. However, the struggle with Faulty Configuration highlights a Domain Specificity Gap. The impact of context engineering (RAG and agentic web search) was non-linear. While Gemini 3.1 Pro leveraged RAG to improve its understanding of Working as Designed issues, it still could not identify Wrong Version issues. This indicates that retrieval relevance is not equivalent to ground truth verification.
related to default wallet settings, while the other pipelines fail. We attribute this to the fact that providing root-cause subclasses forces the model to evaluate all possible invalid categories. When a bug report is missing minor details, models tend to overemphasize the Non-reproducibility category and unnecessarily ask the reporter for more information. However, much like human maintainers, a model can often infer a configuration error despite these small omissions. By removing the need to weigh multiple subclass options, the Without Priors approach avoids this distraction and focuses more directly on producing the correct fix.
2) RQ2: How can invalid bug reports be resolved with noThe findings from RQ2.3 show that context engineering has code fixes?: The transition from identifying a bug to resolving it (RQ2.1 and RQ2.2) revealed a significant decoupling mixed effects on no-code fix generation. Although additional between BERTScore and Judge Success Rate. The consistent context can improve model reasoning, its effectiveness strongly BERTScore (0.81–0.82) suggests models are excellent at depends on the issue category. mimicking the ”style” of a fix. However, the lower Judge RAG performs well in categories that benefit from hisscores for models like Minimax M2.7 (42.9%) reveal a lack of torical examples or prior developer discussions, such as functional logic. Developers should avoid relying on textual Non-reproducibility and Feature Request. Most notably, RAG similarity metrics for triage. Instead, LLM must provide a outperforms Vanilla Gemini in the Working as Designed verifiable resolution step that a human-in-the-loop can validate. subclass. For example, in Brave issue 31299 (see Table VIII), Our comparison between the Without Priors and Vanilla the RAG-generated response correctly captured the key point: pipelines in RQ2.1 reveals a more nuanced relationship than the reported increase in memory usage was an intended design initially assumed. Omitting the root-cause subclass information trade-off rather than a bug. We attribute this improvement to causes only a marginal decrease in overall performance: the RAG retrieving past reports with similar discussions about overall Judge success rate drops by just 0.7% for Gemini 3.1 memory usage. Without this background information, the Pro and 2.8% for Kimi K2.5. Rather than acting as a universally vanilla pipeline instead asked the reporter for additional details. beneficial signal, prior subclass information has a categoryHowever, additional context can also degrade performance. dependent effect. Removing the prior decreases performance In highly specific categories such as Faulty Configuration & for some subclasses, such as Non-reproducibility and External Environment Setup and Wrong Version, retrieved information System & Dependency Issues. Conversely, withholding the appeared to distract the model from the exact issue described prior improves performance in certain cases; most notably, in the prompt. Instead of focusing on the concrete problem, the the Without Priors approach performs better in the Faulty model likely generalized from retrieved documents, resulting Configuration & Environment Setup subclass, yielding higher in fewer successful fixes than the basic model. Overall, these success rates for both models. results suggest that RAG is useful for providing historical For instance, in Brave issue 23741 in Table VIII, the Without grounding, but it may introduce unhelpful noise for strict, Priors pipeline successfully identifies a user misconfiguration version-specific or configuration-specific problems.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
15
Agentic web search produced the best overall performance criteria can improve the reliability and real-world applicability for Gemini 3.1 Pro (68.9%), but caused a substantial perfor- of AI-generated no-code fixes. mance drop for Kimi K2.5 (43.1%). One possible explanation is that agentic loops introduce recursive noise. For less C. Implications for Practitioners capable models, each search step creates another opportunity Our findings suggest that practitioners should start with to drift from the original user intent, potentially leading to project-grounded retrieval when reliable internal data is availa hallucination spiral [55]. To stabilize agentic performance, able. Teams with issue histories, internal wikis, support playfuture systems could constrain the search process through books, or documentation can use these sources before relying on predefined logical transitions, such as Search → Verify → broader web search. In our benchmark, RAG + Gemini achieves Summarize, rather than allowing the model to determine its own the strongest overall subclassification performance in Table V, path fully autonomously. However, prior work suggests that including strong results for External System & Dependency such constrained approaches may underperform more flexible Issues, where past closures and product documentation often agentic search methods [56]. provide more useful evidence than generic web results. Another key finding is the improvement in the challenging Use Context Engineering in Stages: Agentic web search is Wrong Version category, where Agentic Web Search increased still valuable for no-code fix generation. As shown in Table VII, the success rate from 27.3% to 42.4%. For example, in Brave Agentic Web Search + Gemini achieves the highest aggregate issue 34522 in Table VIII, the Agentic Web Search pipeline Judge success rate. However, practitioners should first anchor correctly identified that the issue had already been resolved the model in trusted project-specific corpora, then add live in version 1.62.49, whereas the Vanilla pipeline incorrectly web search when the issue depends on upstream systems, treated the behavior as intentional. Although the ground-truth external dependencies, or time-sensitive facts. Weaker models fix points to version 1.61.86, version 1.62.49 was committed may degrade when tool traces introduce noise rather than useful on November 13, 2023. Since this predates the bug report on constraints. November 24, 2023, the suggested version was still a valid Adapt Pipelines to Local Workflows: Company deployments and available upgrade for the user. This example demonstrates require adaptation to local data and governance constraints. that standard LLM and RAG approaches may lack the external Embedding stores, access control, retention policies, prompt temporal context needed to recognize when a reported issue versions, schemas, and internal sources all affect results. has already been patched in a recent release. Practitioners should repeat key ablations in their own environment, including taxonomy priors on/off, different retrieval corpora, temperature settings, and repeated-run aggregation. B. Implications for Researchers For auditability, retrieved URLs and chunk identifiers should Our findings suggest several directions for future research be logged for both RAG and agentic web search outputs. on automated no-code fix generation. The comparison between Measure Operational Value: Production value should be evaluation metrics shows that commonly used similarity- measured through operational signals, not aggregate F1 based metrics may not capture real-world effectiveness. While alone. Teams can track when high-confidence subclassification BERTScore measures semantic similarity, it fails to distinguish reroutes work away from engineering, accepted no-code fixes performance differences across invalid subclasses and is often remove follow-up cycles, or drafts reduce time-to-first-response. non-informative. In contrast, Judge-LLM evaluation provides a These counts can be converted into estimated effort savings and more practical assessment by considering functional correctness compared against API cost, retrieval maintenance, and human and alignment with successful resolutions. Future research review. Rare subclasses should be monitored separately, since should therefore move beyond purely lexical or embedding- weighted averages can hide weak minority-class performance. based metrics and adopt evaluation methods that better reflect utility in software maintenance contexts. VI. T HREATS TO VALIDITY Call for Standardized Evaluation Frameworks: The lack A. External Validity of a unified evaluation framework for no-code fix generation Our study is conducted exclusively on the Brave Browser remains a critical gap. Our study shows the need for standardrepository, which may limit the generalizability of the findings. ized benchmarks that combine invalid subclassification accuracy Different projects may exhibit varying development practices, with no-code fix quality, enabling consistent comparisons across issue-reporting behaviors, and maintenance dynamics, potenmodels and methodologies. Such frameworks should include tially leading to different outcomes. both automated metrics and structured evaluation protocols so The dataset is derived from a repository with rich historical future studies can reliably measure progress and generalize data and an actively maintained wiki. Projects lacking such findings across datasets and project environments. data sources may present different invalid subclass distributions Call for Manual Evaluation of No-Code Fixes: Beyond and model performance. To mitigate this threat, we evaluate automated metrics, our results highlight the importance of configurations both with and without repository-specific data. manual assessment. Readability, clarity, and practical usefulness are central to whether a no-code fix is actionable for end users. Future research should incorporate human-in-the-loop evaluation, including qualitative analysis and user studies, to better capture these dimensions. Adding such human-centered
B. Internal Validity Due to the opaque training processes of LLMs, we cannot guarantee that the models have not been exposed to portions
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
16
Issue Number
Generated High-Quality No-Code Fix
Generated Low-Quality No-Code Fix
Ground-Truth No-Code Fix
34522
Agentic Web Search: This issue has already been resolved in Brave version 1.62.49 and later releases. Please update your browser to the latest stable version (1.62.x or newer) to receive the fix for the sidebar button margin alignment. You can update by navigating to brave://settings/help or downloading the latest version from the official Brave website...
Vanilla LLM: The sidebar button’s margin design is intentional and follows the UI specification. The slight asymmetry in the icon’s border padding is designed for visual balance with adjacent UI elements and overall sidebar aesthetics. This is not a bug but a deliberate design choice...
Please test this with 1.61.86 or above. According to https://github.com/brave/brave-core/ commits/1.61.x, I’m not sure 1.61.85 includes this fix. I confirmed that 1.61.86 fixed this issue. [57]
31299
RAG: The observed increase in baseline memory usage for a blank page is expected and stems from recent features or underlying Chromium updates included in v1.54.70+. This is not a memory leak or defect, and the browser is functioning as designed. You do not need to take any action, but you can continue to monitor performance to ensure it remains stable in typical browsing scenarios.
Vanilla LLM: To properly investigate this memory degradation issue, please provide: (1) Detailed Steps to Reproduce (S2R) - describe how to trigger the memory issue outside of the automated dashboard test...
The degradation is the result of enabling extra UAF protection (aka BackupRefPtr). So it’s about +10Mb memory to have extremely important exploit protection in the browser process. That’s look like a reasonable trade off. Also we saw the similar memory differences when it was enabled for win. [58]
23741
Vanilla LLM Without Priors: Navigate to brave://settings/wallet and ensure your Default cryptocurrency wallet is set to Brave Wallet (Prefer extensions) if you are using an extension like MetaMask. Temporarily disable any conflicting extensions. . .
Other Pipelines: Please provide a detailed description along with explicit steps to reproduce (S2R), expected behavior (EB), observed behavior (OB)...
...Please make sure you aren’t testing with multiple wallet extensions installed. If you do want Brave Wallet though in that case you can set your default wallet to Brave Wallet in brave://settings/wallet. [59]
TABLE VIII: Comparison of Generated and Ground-Truth No-Code Fixes
of our benchmark during pretraining. Such exposure may bias the results and artificially affect the performance. The use of RAG introduces the risk of incorporating information that became available after the original bug report was created. To mitigate this threat, we restrict retrieval to snapshots of wiki content that are time-aligned with the corresponding test sample and bug reports that have been resolved before the creation of the corresponding test sample. Agentic web search performs data retrieval through a search platform that gives web results at that moment [60], hence the data that would be retrieved by web search would be different. Therefore, it may have been an impediment to the performance of agentic web search that the retrieval and issue dates do not match. We excluded the original issue link search results to prevent data leakage. LLM outputs are inherently stochastic under default temperature settings. This variability may lead to inconsistent results across runs. To keep experimental robustness, we perform three independent executions per sample and base our analysis on the aggregated outputs.
The BERTScore metric, which measures semantic similarity, may overestimate the alignment between generated and reference fixes, leading to overly optimistic evaluations. In contrast, the Judge LLM success metric is subject to stochastic bias and may misinterpret the alignment between generated and reference fixes, resulting in inaccurate assessments. To reduce this bias, we designed the evaluation rubric to be direct and clear. D. Conclusion Validity Allowing LLMs to operate with non-zero temperature improves their expressive capacity and reasoning flexibility but reduces strict reproducibility. Although repeated executions mitigate this issue, some variability remains inherent to the models. VII. C ONCLUSION
This study introduces and evaluates an automated framework for subclassifying invalid bug reports based on root causes and generating corresponding no-code fixes. For subclassification, C. Construct Validity RAG achieves the highest overall performance with 0.66 The invalid bug report taxonomy is defined based on root weighted F1, slightly outperforming vanilla LLMs at 0.65 cause analysis; however, it represents only one possible clas- and agentic web search at 0.64. At the subclass level, RAG sification scheme. Alternative taxonomies may yield different performs best on Non-reproducibility with 0.85 F1 and on categorizations and affect the interpretation of the results. External System & Dependency Issues with 0.67 F1, while The validity of no-code fixes is inferred from issue discus- agentic web search shows relatively better performance on sions rather than verified through controlled reproduction in a Faulty Configuration with up to 0.32 F1. Feature Request sandbox environment. In order to diminish the validity of the and Question remain consistently high across all pipelines, no-code fixes, we kept attention on observable signals such as reaching up to 0.79 F1, whereas Wrong Version remains the user confirmations like reactions and validation comments or most challenging subclass across all approaches with scores maintainer actions (e.g., issue closure). between 0.00 and 0.29.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
For no-code fix generation, agentic web search achieves the highest overall success rate at 68.9%, outperforming vanilla LLMs at 64.9%. At the subclass level, RAG performs best for Working as Designed with up to 87.4%, while agentic web search achieves the highest performance for Question at 72.2% and External System & Dependency Issues at 68.3%. Faulty Configuration and Wrong Version remain challenging across all pipelines, with success rates ranging from 16.7% to 52.9% and 15.2% to 42.4%, respectively. For future work, we aim to evaluate the proposed framework across a broader range of repositories, including industrial and closed-source projects, to assess generalizability. Improving performance for challenging subclasses such as faulty configuration and wrong version remains a key direction. Additionally, integrating real-time pipelines for continuous bug report processing and developing user-facing tools and dashboards can further enhance the practical applicability of automated no-code fix systems.
17
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
R EFERENCES [1] H. Krasner, “The cost of poor software quality in the us: A 2022 report,” Consortium for Information & Software Quality (CISQ), Tech. Rep., Dec. 2022, accessed 2025-11-07. [Online]. Available: https://www.it-cisq. org/wp-content/uploads/sites/6/2022/11/CPSQ-Report-Nov-22-2.pdf [2] Atlassian. (2025) Jira software: Issue and project tracking tool. Accessed: November 7, 2025. [Online]. Available: https://www.atlassian.com/ software/jira [3] GitHub. (2025) Github issues: Collaborative issue tracking platform. Accessed: November 7, 2025. [Online]. Available: https://github.com/ features/issues [4] Y. Fan, X. Xia, D. Lo, and A. E. Hassan, “Chaff from the wheat: Characterizing and determining valid bug reports,” IEEE Transactions on Software Engineering, vol. 46, no. 5, pp. 495–525, 2020. [5] M. Laiq, N. bin Ali, J. Börstler, and E. Engström, “A data-driven approach for understanding invalid bug reports: An industrial case study,” Information and Software Technology, vol. 164, p. 107305, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0950584923001593 [6] J. Anvik, L. Hiew, and G. C. Murphy, “Who should fix this bug?” in Proceedings of the 28th international conference on Software engineering, 2006, pp. 361–370. [7] K. Herzig, S. Just, and A. Zeller, “It’s not a bug, it’s a feature: How misclassification impacts bug prediction,” in 2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 392–401. [8] M. Laiq, N. b. Ali, J. Böstler, and E. Engström, “Early identification of invalid bug reports in industrial settings – a case study,” in Product-Focused Software Process Improvement, D. Taibi, M. Kuhrmann, T. Mikkonen, J. Klünder, and P. Abrahamsson, Eds. Cham: Springer International Publishing, 2022, pp. 497–507. [9] J. Sun, “Why are bug reports invalid?” in 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation. IEEE, 2011, pp. 407–410. [10] Y. Su, P. Luarn, Y.-S. Lee, and S.-J. Yen, “Creating an invalid defect classification model using text mining on server development,” Journal of Systems and Software, vol. 125, pp. 197–206, 2017. [11] S. Panichella, G. Canfora, and A. Di Sorbo, ““won’t we fix this issue?” qualitative characterization and automated identification of wontfix issues on github,” Information and Software Technology, vol. 139, p. 106665, 2021. [12] U. B. Torun, M. T. Demircan, M. F. Gön, and E. Tüzün, “Past, present, and future of bug tracking in the generative ai era,” ACM Transactions on Software Engineering and Methodology, 2026. [Online]. Available: https://doi.org/10.1145/3806655 [13] G. Yang, J. Ji, and J. Kim, “Enhanced bug priority prediction via priority-sensitive long short-term memory–attention mechanism,” Applied Sciences, vol. 15, no. 2, p. 633, 2025. [14] J. He, L. Xu, Y. Fan, Z. Xu, M. Yan, and Y. Lei, “Deep learning based valid bug reports determination and explanation,” in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), 2020, pp. 184–194. [Online]. Available: https://doi.org/10.1109/ISSRE5003.2020.00026 [15] Z. Li, M. Pan, Y. Pei, T. Zhang, L. Wang, and X. Li, “Deeplabel: Automated issue classification for issue tracking systems,” in Proceedings of the 13th Asia-Pacific Symposium on Internetware, 2022, pp. 231–241. [16] M. Laiq, N. bin Ali, J. Börstler, and E. Engström, “A comparative analysis of ml techniques for bug report classification,” Journal of Systems and Software, vol. 227, p. 112457, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121225001256 [17] X. Du, Z. Liu, C. Li, X. Ma, Y. Li, and X. Wang, “Llm-brc: A large language model-based bug report classification framework,” Software Quality Journal, vol. 32, no. 3, pp. 985–1005, 2024. [18] E. Dinç and E. Tüzün, “Judge the votes: A system to classify bug reports and give suggestions,” in Proceedings of the 2nd ACM International Conference on AI-powered Software (AIWare ’25), 2025. [19] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019. [20] N. Pandey, D. Sanyal, A. Hudait, and A. Sen, “Automated classification of software issue reports using machine learning techniques: an empirical study,” Innovations in Systems and Software Engineering, vol. 13, 12 2017. [21] N. Limsettho, H. Hata, A. Monden, and K. Matsumoto, “Unsupervised bug report categorization using clustering and labeling algorithm,” International Journal of Software Engineering and Knowledge Engineering, vol. 26, pp. 1027–1053, 09 2016.
18
[22] I. Chawla and S. Singh, “Automated labeling of issue reports using semi supervised approach,” Journal of Computational Methods in Sciences and Engineering, vol. 18, pp. 1–15, 01 2018. [23] H. Qin and X. Sun, “Classifying bug reports into bugs and non-bugs using lstm,” in Proceedings of the 10th Asia-Pacific Symposium on Internetware, 2018, pp. 1–4. [24] X. Ye, F. Fang, J. Wu, R. Bunescu, and C. Liu, “Bug report classification using lstm architecture for more accurate software defect locating,” in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 2018, pp. 1438–1445. [25] Q. Meng and J. Visser, “Which bug reports are valid and why? using the bert transformer to classify bug reports and explain their validity,” in Proceedings of the 4th European Symposium on Software Engineering (ESSE 2023), 2023, pp. 52–60. [26] A. Budhiraja, K. Dutta, M. Shrivastava, and R. Reddy, “Towards word embeddings for improved duplicate bug report retrieval in software repositories,” in Proceedings of the 2018 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR ’18), 2018, pp. 167–170. [27] J. Deshmukh, K. Annervaz, S. Podder, S. Sengupta, and N. Dubash, “Towards accurate duplicate bug retrieval using deep learning techniques,” in 2017 IEEE International conference on software maintenance and evolution (ICSME). IEEE, 2017, pp. 115–124. [28] M. Baqar, “Rag4tickets: Ai-powered ticket resolution via retrievalaugmented generation on jira and github data,” arXiv preprint arXiv:2510.08667, 2025. [29] K. Ren, “Credibility assessment of fabricated bug reports via large language models: A study on detecting fake software issues,” 2025. [30] E. Akyol, M. Dedeler, and E. Tüzün, “Improbr: Bug report improver using llms,” in 30th International Conference on Evaluation and Assessment in Software Engineering (EASE), 03 2026. [Online]. Available: https://arxiv.org/abs/2604.26142 [31] X. Li, J. Jin, G. Dong, H. Qian, Y. Wu, J.-R. Wen, Y. Zhu, and Z. Dou, “Webthinker: Empowering large reasoning models with deep research capability,” 2025. [Online]. Available: https://arxiv.org/abs/2504.21776 [32] W. Zhang, Y. Li, Y. Bei, J. Luo, G. Wan, L. Yang, C. Xie, Y. Yang, W.-C. Huang, C. Miao, H. P. Zou, X. Luo, Y. Zhao, Y. Chen, C. Chan, P. Zhou, X. Zhang, C. Zhang, J. Shang, M. Zhang, Y. Song, I. King, and P. S. Yu, “From web search towards agentic deep research: Incentivizing search with reasoning agents,” 2025. [Online]. Available: https://arxiv.org/abs/2506.18959 [33] J. Liu, Y. Li, C. Zhang, J. Li, A. Chen, K. Ji, W. Cheng, Z. Wu, C. Du, Q. Xu, J. Song, Z. Zhu, W. Chen, P. Zhao, and J. He, “Webexplorer: Explore and evolve for training long-horizon web agents,” 2025. [Online]. Available: https://arxiv.org/abs/2509.06501 [34] X. Pang, S. Tang, R. Ye, Y. Du, Y. Du, and S. Chen, “Browsemaster: Towards scalable web browsing via tool-augmented programmatic agent pair,” 2025. [Online]. Available: https://arxiv.org/abs/2508.09129 [35] M. Sudeep, “Revolutionizing customer service: The impact of large language models on chatbot performance,” INTERNATIONAL JOURNAL, vol. 10, no. 5, pp. 721–730, 2024. [36] H. Wang, X. Peng, H. Cheng, Y. Huang, M. Gong, C. Yang, Y. Liu, and J. Lin, “Ecom-bench: Can llm agent resolve real-world e-commerce customer support issues?” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2025, pp. 276–284. [37] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” arXiv preprint arXiv:2310.06770, 2023. [38] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024. [39] A. Patil, “Gitbugs: Bug reports for duplicate detection, retrieval augmented generation, triage, and more,” arXiv e-prints, pp. arXiv–2504, 2025. [40] “Icon is missing from brave notification ads after macos upgrade,” https: //github.com/brave/brave-browser/issues/26323, accessed: 2026. [41] “Full screen mode on mac make tabs and url section disappear,” https: //github.com/brave/brave-browser/issues/35808, accessed: 2026. [42] “Development version request,” https://github.com/brave/brave-browser/ issues/21405, accessed: 2026. [43] “Crash in brave ads,” https://github.com/brave/brave-browser/issues/ 34144, accessed: 2026. [44] “What format does brave use to store date/time for ads,” https://github. com/brave/brave-browser/issues/27157, accessed: 2026.
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING
[45] “Update without restarting,” https://github.com/brave/brave-browser/ issues/20778, accessed: 2026. [46] “Possible display bug on recovery phrase screen,” https://github.com/ brave/brave-browser/issues/20796, accessed: 2026. [47] b. contributors, “Labels · brave/bravebrowser,” https://github.com/brave/ brave-browser/labels, 2026, accessed: 2026-01-05. [48] W. G. Cochran, Sampling Techniques. Hoboken: John Wiley & Sons, 2007. [49] A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie et al., “Glm-5: from vibe coding to agentic engineering,” arXiv preprint arXiv:2602.15763, 2026. [50] MiniMax AI, “Minimax m2.7: Early echoes of self-evolution,” https: //huggingface.co/MiniMaxAI/MiniMax-M2.7, 2026, technical report and model release. [51] K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen et al., “Kimi k2. 5: Visual agentic intelligence,” arXiv preprint arXiv:2602.02276, 2026. [52] Serper. (2026) Serper: The world’s fastest and cheapest Google Search API. Accessed: 2026-04-21. [Online]. Available: https://serper.dev/ [53] OpenRouter, “Openrouter api reference,” https://openrouter.ai/docs/api/ reference/overview, 2026, accessed: 2026-04-23. [54] J. Niimi, “A simple ensemble strategy for llm inference: Towards more stable text classification,” in International Conference on Applications of Natural Language to Information Systems. Springer, 2025, pp. 189–199. [55] M. Zhang, O. Press, W. Merrill, A. Liu, and N. A. Smith, “How language model hallucinations can snowball,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024. [56] J. Wu, J. Zhu, Y. Liu, M. Xu, and Y. Jin, “Agentic reasoning: A streamlined framework for enhancing llm reasoning with agentic tools,” arXiv preprint arXiv:2502.04644, 2025. [Online]. Available: https://arxiv.org/html/2502.04644v2 [57] “Brave issue 34522, wrong version no-code fix example,” https://github. com/brave/brave-browser/issues/34522#issuecomment-1827025260, accessed: 2026. [58] “Brave issue 31299, working as designed no-code fix example,” https://github.com/brave/brave-browser/issues/31299# issuecomment-1608043966, accessed: 2026. [59] “Brave issue 23741, faulty configuration no-code fix example,” https://github.com/brave/brave-browser/issues/23741# issuecomment-1169167446, accessed: 2026. [60] Google Search Help, “Why your google search results differ from others,” https://support.google.com/websearch/answer/12412910?hl=en, 2025, accessed: 2026-04-24.
19