An Empirical Study on Improving LLM-Based Issue Classification through Domain-Informed Analysis

arXiv:2611.13470v1 [cs.SE] 10 Nov 2026

An Empirical Study on Improving LLM-Based Issue Classification through Domain-Informed Analysis Anubhav Gupta

Tanner Wright

Gema Rodrı́guez-Pérez

University of British Columbia Kelowna, BC, Canada anubhav.gupta@ubc.ca

University of British Columbia Kelowna, BC, Canada twrigh01@student.ubc.ca

University of British Columbia Kelowna, BC, Canada gema.rodriguezperez@ubc.ca

Abstract—Issue reports that describe Intrinsic Bugs, Extrinsic Bugs, and Not a Bug cases often use similar language, but they might point to different causes and require different maintenance actions. This makes three-way issue classification difficult for Large Language Models (LLMs), particularly when promptbased refinements begin to plateau. We study whether domaininformed analysis of LLM outputs can improve classification beyond prompt modification alone. Using 428 manually labeled issue reports, we evaluate six LLMs across 18 prompt configurations and analyze 7,704 model-generated rationales to identify recurring failure patterns in how models justify predictions, use evidence, and handle conflicting information. We translate these findings into a multi-stage pipeline that separates bug validity from bug origin. The pipeline improves macro F1 and Extrinsic Bug recall for five of the six models. The strongest pipeline configuration reaches 0.827 accuracy and 0.757 Extrinsic Bug recall, improving over the best-accuracy prompt-only baseline by +1.2 and +7.5 pp, respectively. The decomposition further reveals that bug origin classification is more reliable than validity, indicating that the bug-validity decision accounts for most of the remaining error. These results show that expert failure analysis can support more reliable LLM-based software engineering classifiers. Index Terms—GenAI, LLMs, Empirical Software Engineering, Software Maintenance and Evolution

I. I NTRODUCTION Issue reports are central artifacts in software maintenance, but their interpretation is often more difficult than it first appears. A report may mention crashes, failing tests, stack traces, unexpected behavior, or error messages and still not describe a software defect. Conversely, a report may describe a real defect whose root cause lies outside the focal project’s own source history, for example in a changed dependency, platform configuration, external service, or upstream API. This makes issue classification a compound problem: it requires deciding both bug validity [1], i.e., whether the report describes a real defect, and bug origin [2], i.e., whether the root cause of a valid defect is intrinsic or extrinsic to the focal project. In this paper, we study this problem as a three-way classification task: Intrinsic Bug, Extrinsic Bug, or Not a Bug. This combined validity-origin distinction matters for both maintenance practice and empirical software engineering research. Non-bug reports may require documentation updates, clarification, duplicate handling, or enhancement triage, while valid bug reports require root-cause analysis. Treating nonbugs or extrinsic bugs as intrinsic project defects can con-

taminate defect repositories, misdirect root-cause analysis, and distort conclusions drawn from software analytics studies [3]– [5]. This problem is not marginal: Herzig et al. manually inspected more than 7,000 issue reports and found that 33.8% of reports labeled as bugs did not describe corrective maintenance [3]. Prior work has also shown that intrinsic and extrinsic bugs can differ in ways that affect downstream defect prediction [2], [6]. As software engineering increasingly adopts automated classification and decision-support systems, the quality of these distinctions becomes a prerequisite for trustworthy automation. However, this three-way validity and origin classification task is not a simple text-classification problem. A report that mentions a dependency is not necessarily an Extrinsic Bug; the dependency may only appear in the execution context, or the focal project may be using the dependency incorrectly. Similarly, a report that uses failure-like language is not necessarily a bug; it may describe expected behavior, user error, an unsupported configuration, a documentation problem, or a feature request. Correct classification often depends on domain knowledge, expected behavior, project responsibility, and evidence distributed across the issue discussion. Large Language Models (LLMs) are a natural candidate for this task because they can classify issue text from instructions and examples without requiring a task-specific classifier to be trained from scratch [7], [8]. Prior studies show that GPTlike and transformer-like models can automate issue labeling [9], [10]. However, performance metrics alone do not show whether a model has learned the domain distinctions that define the classification problem. Two models may obtain similar F1 scores while failing for different reasons: one may confuse non-bugs with intrinsic bugs, another may overuse the extrinsic label whenever a dependency is mentioned, and another may rely on superficial failure terminology rather than root-cause evidence. In this paper, we investigate whether domain expertise can improve LLM-based issue classification beyond promptonly classification. We evaluate six LLMs under 18 promptonly configurations on 428 issue reports manually labeled as Intrinsic Bug, Extrinsic Bug, or Not a Bug. We then analyze both model predictions and model-generated rationales to understand how the models appear to interpret the task, which evidence they emphasize, and where their interpretations diverge

from expert judgment. Finally, we use this failure analysis to redesign the classification process as a domain-informed multistage pipeline. We evaluate the resulting pipeline using classlevel precision, recall, and F1, together with macro-averaged performance. Our results show that prompt-only changes reach an observed ceiling of 81.5% accuracy and 0.801 macro F1, motivating a deeper analysis of model-output failure patterns. Applying a structured rationale-inspection template to 7,704 representative model-generated rationales reveals two recurring patterns that distinguish correct from incorrect predictions across all three classes: contradictory evidence mishandling (L2-Q3) and conclusion-first reasoning (L2-Q5). Extrinsic Bug issues emerge as a distinct source of difficulty, exhibiting the highest misclassification rate (50.6%) and revealing that models frequently revise away from the correct label after initially identifying it. Translating these findings into a domaininformed multi-stage pipeline that separates bug validity from bug origin improves macro F1 for five of six models and raises the best configuration’s accuracy from 0.815 to 0.827 (+1.2 pp) while lifting Extrinsic Bug recall to 0.757 (+7.5 pp over the best-accuracy baseline). The pipeline also shows that each decision is independently reliable, with bug origin classification (up to 0.907 accuracy) more accurate than bug validity (up to 0.871), suggesting that the validity decision, rather than origin, accounts for much of the remaining error. This paper makes the following contributions: 1) An empirical evaluation: We evaluate six LLMs across three prompts (18 model-prompt configurations) for three-way issue classification and analyze performance variability and class-specific errors. 2) Rationale Analysis: We introduce a structured template for inspecting model rationales, focusing on evidence use, rule application, alternative labels, and contradictory information. 3) A domain-informed pipeline: We propose a multi-stage classification process that uses expert failure analysis to improve LLM-based issue classification. II. BACKGROUND AND R ELATED W ORK Bug Origins: Intrinsic and Extrinsic Bugs: Some software analytics techniques, especially defect prediction and bugmining approaches, assume that reported failures originate from changes in the same repository [11]–[13]. Rodrı́guezPérez et al. [2] challenged this assumption by distinguishing bugs according to their origin. An intrinsic bug is caused by a change within the analyzed project’s source code history, whereas an extrinsic bug is a genuine failure whose cause lies outside that history, for example in a dependency, external service, or upstream component. In a manual study of OpenStack Nova and Elasticsearch, they found that 9–21% of bugs were extrinsic [2]. A follow-up study showed that excluding Extrinsic bugs could change defect-prediction performance by as much as 16 AUC pp [6]. Recent work has begun to automate bug-origin classification: BuggIn is a TF–IDF-based classifier for identifying

Intrinsic bugs from issue reports [14], and Fan et al. [15] later proposed a TextCNN–KAN model for identifying Extrinsic bug reports. These studies show that textual signals can support origin classification, but they focus mainly on binary classification in a limited ecosystem setting. Wright et al. [16] extended bug-origin analysis to the npm ecosystem through the InEx-Bug dataset, which manually labels GitHub issues as Intrinsic Bug, Extrinsic Bug, Not a Bug, or Unknown. Dependency Bugs in Package Ecosystems: The npm ecosystem is especially relevant for studying Extrinsic Bug reports because packages depend on deep and evolving dependency networks. Prior ecosystem studies show that dependency networks grow and reorganize over time [17], [18], and that downstream packages can be affected by vulnerabilities, technical lag, deprecations, breaking changes, and abandoned dependencies [19]–[23]. These ecosystem phenomena are related but not equivalent to Extrinsic Bug. A vulnerability, deprecation, or breaking release is an upstream event or exposure mechanism; an Extrinsic Bug is the downstream failure attributed to such an event. Network studies can estimate how many packages are exposed to external risks, but they generally do not determine whether individual downstream reports describe a causally related failure. The two literatures operate at complementary levels: ecosystem studies explain how external risk can spread, while bug-origin studies identify how such failures are reported, interpreted, and handled in downstream projects. Automated Classification of Issue Reports: Issue-report classification has traditionally focused on validity or type, such as distinguishing bugs from feature requests, questions, documentation issues, or other maintenance tasks [24]–[27]. Herzig et al. [3] manually examined more than 7,000 issue reports and found that over 30% of reports labeled as bugs were misclassified. Subsequent work treated bug-versus-nonbug identification as a text-classification problem using bagof-words, n-grams, TF–IDF, topic models, and classifiers such as logistic regression, support-vector machines, decision trees, and random forests [24], [25], [28]–[31]. Recent work uses pre-trained language models and LLMs for issue classification [32]. Colavito et al. [9], [33] evaluated transformer-based models for issue classification and showed that data quality and label reliability materially affect evaluation. Koyuncu [34] studied how model and prompt choices affect fine-grained bug-report categorization. A recent mapping study identified 46 primary studies on automatic bug/non-bug classification and found that traditional machine learning, deep learning, BERT-family models, and zero- or few-shot methods are all represented in the literature [35]. LLM-BRC [36] further illustrates the use of language-model representations in hierarchical bug classification, although its goal is to classify behavioral properties of bug reports rather than bug origin. This literature provides useful methods and evaluation practices, but its target labels are not interchangeable with Intrinsic Bug and Extrinsic Bug labels. A classifier that separates bugs from non-bugs does not determine whether a valid bug was caused inside the focal repository or by an external

dependency, platform, or environment. Conversely, bug-origin classifiers such as BuggIn [14] and Fan et al.’s model [15] address origin classification, but do not fully address the preceding validity problem. Research Gap. Prior work supports parts of our problem but not their combination. Bug origin studies show that intrinsic and extrinsic bugs are empirically distinct and can affect software analytics results [2], [6]. Ecosystem studies show that dependency-intensive environments such as npm create many opportunities for externally caused failures [17], [18] . However, what remains underexplored is how LLMs fail when bug validity and bug origin must be decided together, and how expert analysis of those failures can be converted into a reproducible improvement process. Our study addresses this gap by evaluating prompt-only LLM classifiers, diagnosing their failure patterns through expert analysis of predictions and rationales, and translating those findings into a domaininformed pipeline for classifying issue reports as Intrinsic Bug, Extrinsic Bug, or Not a Bug. III. S TUDY D ESIGN Our goal is to investigate whether and how domain expertise can improve LLM-based classification of software issue reports. We focus on a three-way classification task in which each issue report must be labeled as Intrinsic Bug, Extrinsic Bug, or Not a Bug. In this study, domain experts are researchers or practitioners with experience in bug analysis and bug-origin classification. To address our goal, this paper formulates the following research questions (RQs): RQ1: How accurately do prompt-only LLM configurations classify issue reports as Intrinsic Bug, Extrinsic Bug, or Not a Bug? Motivation. Before introducing a domain-informed pipeline, we first establish the performance of prompt-only classification. This provides the baseline against which later pipeline changes are evaluated. It also shows whether prompt effects are consistent across models, prompts, and classes, and identifies which errors persist despite prompt refinement. RQ2: What recurring rationale-level failure patterns are associated with incorrect LLM classifications, and how do these patterns vary across Intrinsic Bug, Extrinsic Bug, and Not a Bug categories? Motivation. Aggregate metrics can show where models fail, but they do not reveal how models justify their predictions. RQ2 therefore analyzes model-generated rationales as diagnostic artifacts. The goal is to identify recurring rationale-level patterns that distinguish correct from incorrect predictions and to examine whether those patterns differ across the three ground-truth classes. These patterns provide the empirical basis for the pipeline evaluated in RQ3. RQ3: Does a domain-informed multi-stage classification pipeline improve performance beyond the strongest prompt-only configuration? Motivation. Prompt-only classification treats the task as a single three-way decision, but the labels combine two different judgments: bug validity and bug origin. This can lead to errors

when the model reasons about bug origin before establishing bug validity, or when it dismisses an extrinsic bug because the focal project is not responsible for the underlying cause. RQ3 tests whether these errors can be reduced by restructuring the classification pipeline itself. IV. M ETHODOLOGY Figure 1 illustrates the study’s five phases: (I) dataset preparation and LLM selection; (II) prompt refinement; (III) prompt evaluation; (IV) qualitative analysis of recurring failure patterns; and (V) construction and evaluation of the domaininformed classification pipeline. A. Dataset preparation and LLM selection Dataset: We use Version 3.0 of the InEx-Bug dataset [37], a manually annotated collection of closed GitHub issue reports from 103 npm repositories. Version 3.0 extends the original dataset [16] by excluding open and Unknown reports and adding 85 newly annotated reports. It contains 428 reports: 214 Not a Bug, 107 Intrinsic Bug, and 107 Extrinsic Bug. The dataset is therefore balanced between bug and non-bug reports, and also balanced between intrinsic and extrinsic bugs within the bug subset. Each record includes issue text (i.e., title, body, labels, and comments), repository and author metadata, timestamps, participants, maintainer activity, reopen events, assignees, milestones, closing pull requests/commits, and the final classification. Issue Report Representation: Each issue report is represented as a structured text document that is provided to the LLM as input. Each input includes: (i) the issue title; (ii) the repository name (e.g., axios/axios); (iii) the issue number and corresponding URL; (iv) the issue description, corresponding to the original report submitted by the user; and (v) the full sequence of comments associated with the issue, in chronological order. We also separately extract and append a maintainer comments section. Maintainer comments are defined as comments written by users whose GitHub role is marked as OWNER, MEMBER, or COLLABORATOR. We intentionally repeat them in a dedicated section to make maintainerprovided comments more noticeable to the model. This design choice is motivated by the fact that maintainer comments often contain the most reliable evidence for distinguishing between Intrinsic Bug, Extrinsic Bug, and Not a Bug categories, as maintainers are typically the ones who understand the project requirements and boundaries. LLM Selection and Configuration: To identify suitable models for our analysis, we did not want to restrict our study to a small, fixed set of LLMs. Instead, we began by selecting a diverse pool of recent and high-performing LLMs. Candidate models were selected based on availability, recency, reported performance on public benchmarks, and suitability for instruction-following and reasoning-oriented tasks. This initial pool included both proprietary and open-weight models accessed through the Gemini API and Google Cloud Vertex AI: Gemini 3.1 Pro, Gemini 3 Flash, Gemini 2.5 Pro, Gemini

(I) Dataset Preparation & LLM Selection

(II) Prompt Refinement

(III) RQ1: Prompt Evaluation

Dataset: InEx-Bug v3.0

Starting Point: Baseline Prompt (v0)

Experimental Setup

Closed Bug Reports

Constructing Variants

Issue Representation Title

Project

Issue No.

URL

Description

Comments

Maintainer comments

LLM Selection Gemini DeepSeek OpenAI OSS

(proprietary and open-weight

[3 prompts]

[6 LLMs]

Ablations

v0

Structural Linguistic

Pilot Study

12 prompt variants 10 different models Metrics Overall Accuracy Extrinsic Bug Recall

models)

(V) RQ3: Multi-Stage Pipeline

Representative Set

Task Decomposition into Multi-Stage Pipeline

Reduced to 7,704 responses using MV approach

428 GITHUB ISSUES (npm)

(IV) RQ2: Failure Analysis

54 Configs.

23,112 responses

Structured RationaleInspection Template Layer 1

Layer 2

Stage 1: Is it a Bug or Not a Bug? Not a Bug

Bug

Recover Evaluation Metrics

Prompt Interaction

Reasoning Patterns

Bug

Easy vs. Hard Issues Correct vs. Incorrect

Accuracy

F1 Per-class

Aggregation across Runs & Majority Vote (MV)

Recurring Failure Observations

Stage 2: Where did the bug originate?

Pipeline Evaluation

Selected Variants (v0, v1, v2)

Fig. 1. Integrated framework illustrating the domain-informed methodology for LLM-based issue classification.

2.5 Flash, Gemma 4 26B-A4B, GPT-OSS-120B, GPT-OSS20B, DeepSeek R1, GLM-5, and Qwen3.5 35B-A3B. The pilot study considered all candidate models. After the pilot study, we selected six model configurations for full evaluation. The final set includes three proprietary models and three open-weight models: Proprietary: Gemini 3.1 Pro (gemini-3.1-pro-preview), Gemini 3 Flash (gemini-3-flash-preview), Gemini 2.5 Pro (gemini-2.5-pro). Open-weight: DeepSeek R1 (deepseek-ai/deepseek-r1-0528-maas), GPTOSS-120B (openai/gpt-oss-120b-maas), GPT-OSS20B (openai/gpt-oss-20b-maas). These six models were chosen to satisfy three criteria. First, each model is capable of producing reasoning-style outputs and model-generated rationales. Second, the set spans three independent model providers (Google, OpenAI, and DeepSeek-AI) to mitigate single-vendor or single-family bias in our findings. Third, we include a balanced mix of proprietary and open-weight models to enable more reproducible experimentation. This design follows emerging guidance for empirical studies involving LLMs in software engineering, which emphasizes transparency about prompts and system design, reporting interaction traces, using appropriate baselines and metrics, and validating LLM outputs against human judgment [38]. All models are evaluated under identical experimental conditions: the same set of issue reports, the same prompt variants, and the same issue representation. We use a low fixed temperature of 0.2 to reduce variability while still allowing models to produce natural-language rationales, with top-p = 0.95. For models exposing top-k, we set it to 64. Each model is instructed to produce a Markdown response containing the following sections: a step-by-step rationale

(Reasoning), a three-way Probability Distribution over the bug classes, a categorical Confidence level (High/Medium/Low), and a single-label Final Answer. We do not enforce native structured output such as JSON mode. Instead, we preserve the model’s natural explanatory response because the generated rationale is part of the empirical material analyzed in RQ2. The final label, confidence level, and probability distribution are extracted using regular-expression parsing. The full response text is retained for qualitative analysis. B. Prompt Refinement Baseline Prompt: We adopt the classification prompt from the replication package accompanying Version 3.0 of the InExBug dataset [37] as our baseline (v0). The prompt instructs the model to classify each GitHub issue as Intrinsic Bug, Extrinsic Bug, or Not a Bug, using formal definitions, decision rules, and seven few-shot examples. Before use, we corrected only structural inconsistencies, without adding, removing, or semantically altering any definitions, rules, or examples. Prompt Variants: We create twelve variants of the baseline prompt to study how domain-specific rules and general prompt-engineering changes affect classification. The first set uses domain-informed ablations. We selectively remove rules that may overfit to specific issue patterns, such as upstreamlibrary attribution or temporal-signal heuristics. These rules were selected during a manual review of the baseline prompt. The second set uses general prompt-engineering variants, modifying prompt structure, wording, or ordering while preserving domain content, motivated by prior work on prompt sensitivity [39]–[41]. Our variants were constructed along four dimensions: •

Domain-informed ablation: Remove domain-specific rules to assess their impact on classification.

Structural ablation: Remove structural components (e.g., few-shot examples and probability outputs) to measure their impact on classification. • Structural sensitivity: Test whether formatting or ordering changes affect classification performance. • Linguistic sensitivity: Test whether tone, register, or persona changes affect classification performance. Pilot Study: We conduct a pilot study to select prompt variants for full-scale evaluation. The pilot uses a stratified sample of 30 labeled issue reports, consisting of 10 Intrinsic Bug, 10 Extrinsic Bug, and 10 Not a Bug reports, drawn with a fixed random seed for reproducibility. Each of the 13 prompt variants was evaluated on this sample using 24 configurations from 10 proprietary and open-weight models. Prompt performance is assessed using overall accuracy and Extrinsic Bug recall, aggregated across all 24 model configurations. We emphasize Extrinsic Bug recall because extrinsic bugs are the most domain-sensitive class. A prompt that improves overall accuracy while failing to identify extrinsic bugs would therefore not adequately address the task. All three authors jointly reviewed the accuracy and Extrinsic Bug recall results across the 13 variants and reached consensus on the three variants that performed most consistently well across models and were worth investigating further: the baseline prompt (v0), the no-upstream-library-rule variant (v10), and the maintainer-voice variant (v11), referred to as v0, v1, and v2 in the full evaluation study. •

C. RQ1: Prompt Evaluation We evaluate the three selected prompt variants on the full dataset of 428 issue reports using the six selected LLMs. This produces 18 (model × prompt) configurations in total. Mitigating LLM Non-Determinism: Since LLM outputs can vary across repeated runs [42], [43], we executed each model × prompt configuration three independent times at temperature=0.2, yielding 54 model × prompt × run configurations in total. We assess run-to-run stability using: (i) per-instance label agreement reported as the fraction of the 428 issues for which all three runs produced identical predictions, (ii) Fleiss’ κ with bootstrap 95% confidence intervals [44] treating the three runs as raters, (iii) mean, standard deviation, and min–max range of weighted F1, macro F1, and per-class recall across all runs, and (iv) Cochran’s Q tests on the per-instance binary correctness vector, where an instance was coded as correct if ground_truth == predicted_label and incorrect otherwise, to detect statistically significant differences in correctness rate across runs. Benjamini-Hochberg correction controlled the false discovery rate [45]. Aggregating Across Runs: We report performance in two ways: (i) per-run mean and standard deviation and (ii) majority-vote (aggregation inspired by self-consistency [46]), where for each issue, the final label is the one predicted by at least two of three runs. In rare cases where all three runs produce a different label, the tie is broken by selecting the label whose run reported the highest predicted class probability. If

probabilities are equal, we fall back to lexicographic order to ensure deterministic and reproducible results. Evaluation Metrics: We evaluate classification performance using accuracy, weighted F1, macro F1, and per-class precision, recall, and F1 for each of the three classes. To test prompt differences, we apply McNemar’s test to all three pairwise prompt comparisons within each model using majority-vote predictions. McNemar’s test is appropriate here because the predictions are paired at the issue level and the outcome is binary (correct/incorrect). Benjamini–Hochberg correction controls the false discovery rate at α = 0.05 [45]. D. RQ2: Qualitative Analysis of Recurring Failure Patterns To answer RQ2, we analyze the model outputs from RQ1. Each output includes the issue report, ground-truth label, model prediction, rationale, probability distribution, confidence level, prompt variant, model identifier, and run identifier. Since the full corpus contains 23,112 outputs across three repeated runs, we reduce it to one representative response for each (issue × model ×prompt). We retain the earliest response matching the majority prediction across the three runs; when all runs disagree, we use run 1 for deterministic selection. This produces 7,704 records. We analyze this reduced corpus in two complementary ways. First, we compute quantitative summaries of error distributions across models, prompts, classes, and issue reports to identify which classes and decision boundaries are most error-prone. We also categorize issues by classification difficulty to guide qualitative sampling. An issue is labeled model_hard if at most two of the six models classify it correctly, model_easy if at least four models classify it correctly, and model_mixed if exactly three models classify it correctly. This yields 93 hard, 316 easy, and 19 mixed issues. TABLE I R ATIONALE - INSPECTION TEMPLATE USED FOR RQ2. ID

Rationale Feature

Options

L1-Q1

Maintainer disagreement

L1-Q2

Rule grounding

L2-Q1

Rationale and response alignment Alternative-label consideration Contradictory evidence handling

Conflict acknowledged and reasoned through; Conflict present but not explicitly reasoned; No maintainer conflict present. Grounded; Asserted; Not applicable, No rule invoked. Aligned; Compressed; Reversed; Partial self-correction. Both alternatives; One alternative; None.

L2-Q2 L2-Q3 L2-Q4 L2-Q5

Correct-label abandonment Direction of reasoning

Addressed; Dismissed without supporting reasoning; Ignored entirely; No contradicting evidence present. Yes; No. Evidence-first; Conclusion-first; Recursive; Reversed.

Qualitative Inspection: We used an iterative manual qualitative inspection process to develop a structured template for identifying recurring patterns in model-generated rationales. The purpose of this process was not to conduct open-ended manual coding of the entire response corpus or a sample of it. Instead, our goal was to inspect a focused set of model-generated outputs, identify recurring rationale-level and

prompt-interaction patterns, and translate these observations into a structured template that could be applied consistently at scale. The inspection process proceeded in two rounds. In the first round, two authors inspected a randomly selected sample of 10 issue reports, consisting of five model_easy and five model_hard cases. The sample included issue reports from all three ground-truth categories. For each sampled issue, the authors individually inspected one representative model output selected from the reduced corpus, together with the issue evidence, the predicted label, and the ground-truth label. They noted recurring behaviors in how the model interpreted the issue, considered alternative labels, used or ignored contextual cues, and justified its prediction. After the independent inspection, the authors met to compare observations, discuss emerging patterns, resolve disagreements through discussion, and refine the candidate template questions. In the second round, the authors repeated this process on a different randomly selected sample of 10 issue reports, again consisting of five model_easy and five model_hard cases. Based on the observations and discussions from both rounds, the two authors formalized a preliminary inspection template consisting of structured questions and predefined answer choices. They then applied the preliminary template to the 20 inspected cases from Rounds 1 and 2 to assess whether the questions and answer choices captured the behaviors observed during manual inspection. The third author subsequently reviewed the template and helped refine ambiguous questions and response options. Across these iterations, the authors consolidated overlapping observations, clarified the wording of template questions, and revised the answer choices to better capture recurring patterns in the model-generated rationales. This process resulted in a finalized rationale-inspection template designed to identify recurring patterns associated with misclassifications and to guide the domain-informed pipeline design for RQ3. Table I summarizes the template questions and predefined answer options. LLM-Assisted Template Application: The finalized rationale-inspection template was converted into an LLMassisted inspection prompt. We used Gemini 2.5 Pro to apply the finalized template because it was the best-performing model in the RQ1 classification evaluation. Gemini 2.5 Pro was configured with a maximum thinking budget of 32,768 tokens. The model was instructed to produce one structured CSV row per response record, containing the seven template values in a fixed column order. We applied the inspection prompt to the full set of 7,704 representative responses. To assess the consistency of the LLM-assisted template application, we compared its outputs against human-coded labels on two disjoint ten-record samples corresponding to the two human-inspection rounds. The comparison showed cell-level agreement above 90%. E. RQ3: Domain-Informed Multi-Stage Pipeline To answer RQ3, we build a domain-informed multi-stage classification pipeline using the strongest RQ1 prompt-only

configuration, the no-upstream-library-rule variant (v1), as the baseline. Guided by the RQ2 failure analysis, the pipeline separates bug validity from bug origin and adds targeted checks for validity errors. Task Decomposition: The prompt-only baseline treats classification as a single three-way decision. In contrast, our pipeline decomposes the task into two sequential binary decisions. The first stage determines bug validity, while the second stage determines bug origin only for issues classified as valid bugs. 1) Stage 1: Bug Validity (Bug vs Not a Bug) determines whether the report describes a genuine software defect requiring a change to the package’s shipped code, configuration, or constraints, regardless of where the defect originates. 2) Stage 2: Bug Origin (Intrinsic Bug vs Extrinsic Bug) applies only to reports classified as bugs. This stage determines whether the defect originates from the package’s own logic or from an external dependency, environment, platform, or upstream change. The two-stage pipeline first asks whether a defect exists and then, only if a defect exists, asks where the defect originated. The Stage 1 prompt treats both internally and externally caused defects as Bug, does not assign the origin label, and instead defers the origin analysis to Stage 2. The Stage 2 prompt then guides the model to reason about where the defect originated and not where the fix may be applied. By decomposing the three-way classification task into two focused binary decisions, the pipeline avoids forcing a single prompt to arbitrate bug validity and bug origin at the same time. Domain-Informed Stage 1: The Stage 1 defect prompt is adopted from the v1 prompt selected in RQ1. We add two rules derived from the RQ2 analysis of how Extrinsic Bug reports are incorrectly dismissed. RQ2 showed that many Extrinsic Bug errors occur when a model recognizes an externally triggered failure but revises away from the bug label because the focal project is not directly responsible for the underlying cause. To address this failure mode, the Stage 1 prompt emphasizes that an issue can describe a genuine bug even when the root cause lies outside the focal repository, and that bug validity should be decided before assigning responsibility for the origin of the defect. Independent Re-check Stage (Recover): Between Stage 1 and Stage 2, we insert a re-check stage, called Recover. When Stage 1 predicts Not a Bug, Recover examines whether the issue nevertheless describes a genuine software defect that should instead be classified as a bug. Recover receives the Stage 1 verdict and rationale as a “prior pipeline assessment”, but is instructed to treat that assessment as a prior opinion that may be wrong rather than as a decision to ratify. This stage is conservative by design: it retains the Stage 1 verdict unless the issue contains clear evidence that the verdict is wrong. The re-check is performed by an independent judge model. In the preliminary pipeline iterations, using the same model for both Stage 1 and the re-check often led the model to re-derive and re-endorse its earlier decision. Recover is performed by gemini-3.1-pro-preview

for all pipelines except the one whose Stage 1 also uses gemini-3.1-pro-preview. There, to keep the judge independent, we substitute gemini-2.5-pro. Context Chaining between Stages: Each stage is executed as an independent LLM inference and therefore has no conversational memory. To preserve intermediate decisions, downstream stages receive the verdict and rationale generated by preceding stages. The Recover stage receives the Stage 1 prediction and rationale. Stage 2 receives the bug-validity verdict and the preceding rationale, and then assumes that bug validity has already been established. It then determines whether the defect originates from within the focal package (intrinsic) or from an external source (extrinsic). By carrying the accumulated reasoning forward in this pipeline, it allows for the avoidance of collapsing stages within a single run and allows for early mistakes to be caught later in the pipeline. Evaluation: For comparability with RQ1, we run the proposed pipeline end-to-end on the same 428-report dataset used in the prompt-only evaluation. Each issue receives a final three-way label: Intrinsic Bug, Extrinsic Bug, or Not a Bug. We then evaluate the proposed pipeline on an independent validation set of 30 manually annotated npm issue reports, with 10 reports per class. This validation set was not used during prompt selection, rationale-template development, failure analysis, or pipeline construction. The validation reports were manually annotated by the authors using the same three-way schema as InEx-Bug, with disagreements resolved through discussion. Lastly, we perform component ablations to isolate the contribution of each pipeline element. For the Recover stage, we record recovery flips and whether each flip improves or harms the final classification. Classifications that did not return a response or did not contain a parsable classification label were treated as invalid predictions and counted as incorrect. V. S TUDY R ESULTS Before addressing the research questions, we summarize the pilot study used to select prompt variants for full-scale evaluation. In the pilot study, accuracy ranged from 63.9% (v3) to 68.2% (v10) across the prompt variants. Extrinsic Bug recall varied from 17.1% (v5) to 35.4% (v10). Prompt v10 achieved the highest accuracy (+1.8 pp over v0) and the largest gain in Extrinsic Bug recall (+12.5 pp). Although v12, which combines v10 and v11, achieved similar gains, it did not outperform v10. We therefore selected the baseline prompt (v0), the no-upstream-library-rule variant (v10), and the maintainer-voice variant (v11) for full-scale evaluation. For readability, we refer to these three prompts in the full evaluation as v0, v1, and v2, respectively. A. RQ1: Prompt-Based Issue Classification Run Stability Analysis: Across all 18 (model × prompt) configurations, at least two of the three runs agreed on the predicted label for 98.8%–100% of issues (mean: 99.7%), confirming that majority vote labels were rarely determined by a single run. Full agreement ranged from 81.5% to 97.2% (mean: 89.5%). Treating runs as raters, Fleiss’ κ ranged from

TABLE II RQ1 CLASSIFICATION PERFORMANCE . M ACRO F1 IS REPORTED ON MAJORITY- VOTE PREDICTIONS v0 Model DeepSeek-R1 Gemini-2.5-Pro Gemini-3-Flash Gemini-3.1-Pro GPT-OSS-120B GPT-OSS-20B

F1 0.689 0.722 0.649 0.626 0.765 0.718

v1

v2

Acc

F1

Acc

F1

Acc

0.729 0.755 0.717 0.706 0.783 0.748

0.735†

0.752 0.815 0.787 0.762 0.792 0.764

0.704 0.743 0.669 0.631 0.765 0.755

0.738 0.776 0.727 0.708 0.783 0.773

0.801† 0.769† 0.733† 0.779 0.753

† Significant improvement over v0 under McNemar’s test with BH-FDR

correction (α = 0.05).

0.804 to 0.963 across all 18 configurations (mean: 0.883), with all configurations exceeding the conventional threshold for almost-perfect agreement κ = 0.80 [47]. Cochran’s Q tests on the per-instance binary correctness detected no significant differences across the runs after global BH-FDR correction. Class-conditional stability is the fraction of issues per groundtruth class for which all three runs agreed. Extrinsic Bug instances were least stable (mean: 80.1%), compared with Intrinsic Bug (94.1%) and Not a Bug (91.9%). Classification Results: Given the high inter-run agreement across all configurations (κ ≥ 0.804), we report RQ1 classification results using majority-vote predictions as point estimates, together with per-run mean ± SD. Table II shows macro F1 and accuracy for all 18 model × prompt configurations. Macro F1 on majority-vote (MV) predictions ranged from 0.626 (Gemini-3.1-Pro × v0) to 0.801 (Gemini-2.5-Pro × v1), with per-run mean ± SD confirming low sensitivity to any individual run across all configurations. The best-performing model was Gemini-2.5-Pro × v1 (macro F1 = 0.801, weighted F1 = 0.815, accuracy = 81.5%), followed by GPT-OSS120B × v1 (macro F1 = 0.779) and Gemini-3-Flash × v1 (macro F1 = 0.769). Prompt v1 achieved the highest macro F1 for five of six models, with gains over v0 for all six models ranging from +1.4 pp (GPT-OSS-120B) to +11.9 pp (Gemini-3-Flash). The only model for which v1 was not the top-performing prompt was GPT-OSS-20B, where v2 marginally but non-significantly outperformed v1 by 0.2 pp (macro F1 = 0.755 vs. 0.753). For four of the six models, Gemini-2.5-Pro, Gemini-3-Flash, Gemini-3.1-Pro, and DeepSeek-R1, the v0-to-v1 gains were statistically significant under McNemar’s test with BH-FDR correction (pFDR ≤ 0.022). Extrinsic Bug recall showed the largest and most consistent gains from v0 to v1 across all models, ranging from +9.3 pp (GPT-OSS-120B) to +35.5 pp (Gemini-3-Flash). Under v0, Extrinsic Bug recall was particularly low for the Gemini-3.1Pro (0.215) and Gemini-3-Flash (0.262) models. B. RQ2: Qualitative Analysis of Recurring Failure Patterns Across all 7,704 coded records, LLMs produced correct classifications in 5,829 cases (75.7%) and incorrect ones in 1,875 (24.3%). Error rates differed substantially across

TABLE III M ISCLASSIFICATION COUNTS BY DIRECTION ( ALL 7,704 RECORDS ). Direction

nwrong

ncorrect GT

% of all wrong

609 487 366 185 178 50

951 3,187 951 1,691 3,187 1,691

32.5% 26.0% 19.5% 9.9% 9.5% 2.7%

Extrinsic → Not a Bug Not a Bug → Intrinsic Extrinsic → Intrinsic Intrinsic → Not a Bug Not a Bug → Extrinsic Intrinsic → Extrinsic

L2_Q5: Conclusion-first reasoning

+35pp

L2_Q3: Contradicting evidence dismissed

+28pp

L2_Q3: Contradicting evidence ignored

+25pp

L2_Q4: Reached correct, then revised away

+18pp

L1_Q2: Rule applied without grounding

+17pp

L1_Q2: Rule applied with grounding

-17pp

L2_Q4: Never reached correct answer

-18pp

L2_Q5: Evidence-first reasoning L2_Q3: No contradicting evidence present

-50pp

-59pp

−60

Elevated in misclassified responses Elevated in correct responses −40

−20

0

20

40

Positive = more common in misclassified; Negative = more common in correctly classified

Fig. 2. Discrimination of coding questions between correct and incorrect predictions.

ground-truth categories: Extrinsic Bug issues were misclassified in 975 of 1,926 records (50.6%), compared with 235 of 1,926 (12.2%) for Intrinsic Bug and 665 of 3,852 (17.3%) for Not a Bug. Table III summarizes the six misclassification directions by volume. The two largest error directions are Extrinsic Bug predicted as Not a Bug (n = 609) and Not a Bug predicted as Intrinsic Bug (n = 487), together accounting for around 58% of all misclassifications. Cross-Class Failure Patterns. Two coding questions from the rationale-inspection template distinguish correct predictions from the incorrect ones with consistency across all three ground-truth classes: L2-Q3 (Contradictory evidence handling) and L2-Q5 (Direction of reasoning). Figure 2 visualizes the full discriminating profile. For L2-Q3, 63.4% of correct responses are coded as having no contradicting evidence present compared with only 4.3% of misclassified responses (∆ = −59 pp). In misclassified responses, contradictory evidence was much more often ignored or dismissed. Specifically, 31.0% ignore it entirely (∆ = +25 pp vs. 5.7% in correct responses), and 29.1% acknowledged but dismissed it without supporting reasoning (∆ = +28 pp vs. 1.1%). This pattern appeared across all three ground-truth classes: Intrinsic Bug (∆ = −66 pp), Not a Bug (∆ = −59 pp), Extrinsic Bug (∆ = −47 pp). For L2-Q5, evidence-first reasoning appears in 91.4% of correct responses but only 41.6% of misclassified ones (∆ = −49 pp). Conversely, conclusion-first reasoning appears in only 5.4% of correct responses but 40.8% of misclassified

responses (∆ = +35 pp). This pattern is consistent across all three categories: Extrinsic Bug (∆ = +33 pp), Intrinsic Bug (∆ = +42 pp), and Not a Bug (∆ = +35 pp). Extrinsic Bug Exhibits a Distinct Rationale Profile. Extrinsic Bug issues exhibit a different rationale profile from the other two classes. Conclusion-first (L2-Q5) reasoning appears in 22.4% Extrinsic Bug responses, compared with 12.0% for Intrinsic Bug and 10.8% for Not a Bug. Models also revise away (L2-Q4) from the correct label more often for Extrinsic Bug cases (10.3% vs. 2.3% for both other classes). Finally, contradicting evidence (L2-Q3) is more common in Extrinsic Bug records (71.9%) than in Intrinsic Bug (40.2%) or Not a Bug (45.9%), leaving fewer cases with no contradicting evidence (28.1% vs. 59.8% and 54.1%). Observed Patterns by Direction of Misclassifications. When Extrinsic Bug issues are misclassified as Intrinsic Bug (n = 366), the dominant profile combines conclusion-first reasoning with asserted rule application. Conclusion-first reasoning (L2Q5) appears in 51.6% of these wrong responses versus 5.3% of correct Extrinsic Bug responses (∆ = +46 pp). L1-Q2 is coded rule asserted without grounding in 35.5% of wrong responses versus 1.7% of correct ones (∆ = +33 pp). Contradicting evidence is dismissed in 35.2% of cases (∆ = +33 pp vs. 1.5%). When Extrinsic Bug issues are misclassified as Not a Bug (n = 609), 27.1% of responses first reach the correct label before revising away from it (L2-Q4), compared with 0.1% of correct Extrinsic Bug responses (∆ = +27 pp). Conclusion-first reasoning (∆ = +26 pp) and dismissed contradicting evidence (∆ = +23 pp) are also elevated. When Not a Bug cases are predicted as Intrinsic Bug (n = 487), one alternative considered (L2-Q2) appears in 33.5% of wrong responses versus 9.0% of correct Not a Bug responses (∆ = +24 pp). In contrast, no alternatives considered drops from 41.1% (correct) to 18.1% (wrong) (∆ = −23 pp). C. RQ3: Domain-Informed Multi-Stage Pipeline Table IV reports the proposed pipeline results across all models. The proposed pipeline improves macro F1 over the prompt-only baseline for five of the six models, with gains ranging from +0.8 pp (DeepSeek-R1) to +4.8 pp (Gemini-3Flash). The Extrinsic Bug recall also improves for five of the six models, most sharply for Gemini-3.1-Pro (0.505→0.738) and Gemini-3-Flash (0.617→0.757). The strongest promptonly configuration (baseline) was Gemini-2.5-Pro (accuracy 0.815, Extrinsic Bug recall 0.682). The strongest pipeline configuration, Gemini-3-Flash, reaches 0.827 accuracy, a +1.2 pp improvement, while simultaneously raising Extrinsic Bug recall to 0.757, +7.5 pp above the best-accuracy baseline. Each stage is independently reliable. Table V separates the two decisions. Stage 2 (Intrinsic Bug vs. Extrinsic Bug origin) is the more accurate stage, reaching 0.864–0.907 across all six models. Stage 1 (Bug vs. Not a Bug validity) reaches 0.785–0.871, with high Bug recall (0.813–0.930) but lower Not a Bug recall (0.720–0.836). The Recover pass produces a net-positive number of beneficial flips on five of six models, harming only DeepSeek-R1.

TABLE IV R ESULTS ACROSS MODELS AND PIPELINE CONFIGURATIONS . E ACH CELL REPORTS M ACRO -F1 / ACCURACY / E XTRINSIC R ECALL . Configuration

Gemini-2.5-Pro

Gemini-3.1-Pro

Gemini-3-Flash

GPT-OSS-120B

GPT-OSS-20B

DeepSeek-R1

Best prompt-only v1 (baseline) A: Stage 1 (v1) + Stage 2 (v1) B: A + context chaining C: B + Recover Proposed pipeline: C with domain-informed Stage 1

0.801 / 0.815 / 0.682 0.776 / 0.794 / 0.692 0.789 / 0.804 / 0.692 0.799 / 0.813 / 0.701

0.733 / 0.762 / 0.505 0.730 / 0.766 / 0.458 0.719 / 0.757 / 0.449 0.752 / 0.762 / 0.692

0.769 / 0.787 / 0.617 0.759 / 0.783 / 0.589 0.776 / 0.797 / 0.598 0.798 / 0.815 / 0.664

0.779 / 0.792 / 0.692 0.778 / 0.792 / 0.682 0.787 / 0.799 / 0.710 0.788 / 0.801 / 0.720

0.753 / 0.764 / 0.682 0.787 / 0.801 / 0.682 0.765 / 0.783 / 0.607 0.787 / 0.799 / 0.673

0.735 / 0.752 / 0.607 0.743 / 0.738 / 0.589 0.710 / 0.713 / 0.505 0.734 / 0.717 / 0.579

0.801 / 0.815 / 0.738

0.765 / 0.771 / 0.738

0.817 / 0.827 / 0.757

0.802 / 0.811 / 0.757

0.769 / 0.783 / 0.682

0.743 / 0.734 / 0.654

TABLE V S TAGE - LEVEL BREAKDOWN OF THE PROPOSED PIPELINE . Stage 1

Stage 2 Recovery

Model

Acc

Bug R NAB R

Acc

Flips

Gemini-2.5-Pro Gemini-3.1-Pro Gemini-3-Flash GPT-OSS-120B GPT-OSS-20B DeepSeek-R1

0.871 0.825 0.869 0.855 0.843 0.785

0.916 0.930 0.902 0.925 0.893 0.813

0.878 0.884 0.907 0.904 0.864 0.874

4 (4/0/+4) 65 (34/31/+3) 6 (4/2/+2) 6 (5/1/+4) 8 (5/3/+2) 7 (3/4/−1)

0.827 0.720 0.836 0.785 0.794 0.757

Bug R = Bug recall; NAB R = Not a Bug recall. Recovery flips are reported as total flips with good/bad/net in parentheses.

Validation Set: On the independent 30-issue validation set, Macro F1 ranges 0.829–0.935 and accuracy 0.800–0.933, with GPT-OSS-120B and GPT-OSS-20B achieving the highest macro F1 (0.935 and 0.933). Not a Bug recall stays high (0.900–1.000), while Extrinsic Bug recall (0.700–1.000) remains the most variable, consistent with the test-set results. VI. D ISCUSSION AND I MPLICATIONS Prompt-only refinement plateaus, and added domain rules can negatively impact performance. It is worth noting that the strongest prompt variant was not the one that encoded the most domain guidance. In fact, v1 was a domain-informed ablation that removed the upstream-library-attribution rule. It achieved the highest macro F1 for five of six models and improved over the baseline v0 for all six models. Our results align with prior evidence that, for LLMs, more context is not necessarily better. Prompt sensitivity shows that small, semantically minor edits to a prompt can swing the classification outcomes [39]–[41]. Prior work shows that added context can distract LLMs and reduce performance, whether through irrelevant details, excessive examples, or agent “cursor rules” [48]–[50]. Our results show that even valid expertauthored domain rules can have this effect when encoded directly in prompts. Implication 1: For domain-sensitive tasks, prompt-only refinement can face a diminishing-returns ceiling, and adding more domain rules or context is not necessarily beneficial. Domain guidance should therefore be evaluated empirically rather than assumed to improve performance. Rationales are useful diagnostic artifacts, but not explanations of internal reasoning. Our RQ2 analysis used modelgenerated rationales to identify recurring patterns in incor-

rect predictions. Correct predictions were much more likely to be evidence-first, whereas incorrect predictions frequently anchored on a conclusion. Similarly, misclassified responses often ignored or dismissed evidence that conflicted with the final label. These findings should not be interpreted as proof of the model’s internal reasoning process. Prior work has shown that chain-of-thought and generated explanations can be unfaithful to the actual decision process [51]. We therefore treat rationales as observable artifacts: they show how the model presents and justifies its interpretation, not necessarily how the model internally arrived at the label. Even under this cautious interpretation, rationales are valuable. They expose recurring failure signatures that aggregate metrics cannot show. Implication 2: Researchers should avoid claiming that a model “reasoned correctly” merely because its explanation is plausible. Instead, generated rationales can be treated as diagnostic artifacts that can help identify failure patterns and guide the design of more reliable classification pipelines. Extrinsic Bugs are a distinct challenge, not just a lowerperforming class. Prior bug origin research shows that extrinsic bugs are empirically important and can affect downstream software analytics [2], [6], while issue-report studies show that bug-labeled reports often contain non-bugs [3]. Our findings connect these concerns by showing that LLMs struggle when bug validity and bug origin must be judged together, even with explicit definitions, examples, and decision rules. The difficulty is not simply that Extrinsic Bug is a lower-performing class; it requires the model to recognize that a reported failure can be real even when the focal repository did not introduce the root cause. LLMs also revised away from the correct label more often for Extrinsic Bug reports. These results suggest that extrinsic bugs create a specific interpretive burden. This burden appears tied to how such reports are written and discussed, and it persisted despite explicit classification guidance in our evaluation. Implication 3: Extrinsic bug classification requires richer evidence than labels alone. Future datasets, issue templates, and reporting guidelines should capture dependency, environment, reproduction, maintainer, fix-location, and causal information needed to distinguish external causation from nonbug reports. For Extrinsic Bugs, the main bottleneck is bug validity, not bug origin. Our pipeline decomposes a complex task into smaller subtasks [52] and highlights where the difficulty lies when it comes to classifying Extrinsic Bugs. Extrinsic Bug was

the hardest class to classify, and its dominant error direction is revealing: 32.5% of all misclassifications were extrinsic bugs mistaken for Not a Bug. Once an issue is established as a genuine bug, the origin decision is reliable: Stage 2 achieved up to 90.7% accuracy on Intrinsic Bug vs. Extrinsic Bug. The end-to-end performance is instead bounded by Stage 1, where extrinsic bugs are discarded as Not a Bug before the origin decision is ever reached. The persistence of this error even under domain-informed prompting suggests the boundary may be genuinely ambiguous, potentially difficult for human annotators as well. This motivates an in-depth study of these cases to identify what evidence distinguishes an externallycaused defect from a non-bug. Implication 4: Datasets, tools, and classification pipelines should treat bug validity and bug origin as separable judgments. Future work should aim to improve extrinsic bug detection by targeting the Bug vs. Not a Bug boundary. Comparison with non-LLM classifiers. Prior automated origin classifiers target a different task, dataset, and label schema. BuggIn [14] performs binary intrinsic-vs-rest classification with classical ML to identify intrinsic bugs (best F1 0.78), and Fan et al.’s TextCNN–KAN [15] performs binary intrinsicvs-extrinsic classification (accuracy 0.75, F1 0.81 on 1,880 OpenStack samples). LLM-BRC [36] and Colavito et al. [9], [33], [53] classify behavioral or type-level properties rather than bug origin. The closest comparison to our work is at the binary-subtask level: our pipeline reaches 87.1% accuracy on the validity decision (Stage 1) and 90.7% on the origin decision (Stage 2), while jointly resolving both judgments that prior classifiers treat in isolation. VII. T HREATS TO VALIDITY Internal Validity: LLM outputs may vary across identical runs, affecting classifications, rationales, and confidence estimates in both the prompt-only evaluation and the downstream analysis. We mitigated this by running each model × prompt configuration three times at temperature=0.2, aggregating predictions by majority vote, and measuring stability with agreement, Fleiss’κ, metric variation, and Cochran’s Q tests. These results support using one representative response for RQ2 and single-run pipeline results for RQ3, though residual randomness remains. To code 7,704 rationales for RQ2, we applied the finalized template using Gemini 2.5 Pro, the strongest evaluated classifier. An automated coder may misread rationales, and using the model from the same family as some of the coded outputs may bias the coding. We mitigated this by developing the template through an iterative, human-driven inspection process and validating the automated coding against independent human coding on two disjoint ten-record samples, where cell-level agreement exceeded 90%. However, the validation set remains small. The template also encodes the authors’ judgments about which reasoning behaviors are salient, which may shape observed failures. To reduce this, two authors independently inspected samples across two rounds, reconciled

disagreements, and a third author refined ambiguous questions and answer options. External Validity: All 428 primary issues come from npm because InEx-Bug is the only public dataset with the required three-way schema. While npm’s dependency structure makes extrinsic bugs especially relevant, results may not transfer directly to ecosystems such as Maven or PyPI, industrial codebases, or projects with different triage practices. Construct Validity: Our failure analysis uses modelgenerated rationales, which may not faithfully reflect internal reasoning [51]. We therefore do not interpret rationales as explanations of why a model decided as it did. Instead, we treat them as observable artifacts of how a model presents its interpretation of the issue. We acknowledge that an inferential gap between stated and actual reasoning remains. The three classes reflect judgments about bug validity and origin, which can be ambiguous when evidence is incomplete. We treat InEx-Bug annotations as ground truth. Any residual label noise or borderline cases would affect both the performance metrics and the failure analysis. Finally, because LLMs are pre-trained on large web corpora, some evaluated models may have encountered GitHub issues during pre-training, raising the possibility of training-data leakage. We consider this risk limited, as the threat is not exposure to the issue text, which is the intended model input, but to the ground-truth labels. The manually annotated InEx-Bug dataset was released in 2026 and postdates the pre-training data of most evaluated models. Moreover, if leakage had materially influenced the results, we would expect uniformly high accuracy across all three categories. Instead, we observe a class-dependent error profile. Nevertheless, model training cutoffs and data sources are not fully transparent. We therefore cannot rule out exposure to some issue text and labels during pre-training. VIII. C ONCLUSION Overall, our findings show that the value of LLMs for issuereport classification should not be assessed only by asking whether they can assign labels accurately. Prompt-only refinement plateaued at 0.815 accuracy and 0.801 macro F1, but a domain-informed pipeline derived from expert failure analysis improved macro F1 and Extrinsic Bug recall for five of the six evaluated models, with its strongest configuration reaching 0.827 accuracy and 0.757 Extrinsic Bug recall. Decomposing the task further showed that origin classification is more reliable than validity, indicating that the bug-validity decision may account for much of the remaining error. More importantly, these results show that model failures can reveal domainspecific sources of confusion that, when analyzed by experts, can be translated into a more reliable classification process. This paper therefore demonstrates how expert diagnosis can help identify and address failure modes in LLM-based bug classification. R EFERENCES [1] Y. Fan, X. Xia, D. Lo, and A. E. Hassan, “Chaff from the wheat: Characterizing and determining valid bug reports,” IEEE transactions on software engineering, vol. 46, no. 5, pp. 495–525, 2018.

[2] G. Rodrı́guez-Pérez, G. Robles, A. Serebrenik, A. Zaidman, D. M. Germán, and J. M. Gonzalez-Barahona, “How bugs are born: a model to identify how bugs are introduced in software components,” Empirical Software Engineering, vol. 25, no. 2, pp. 1294–1340, 2020. [3] K. Herzig, S. Just, and A. Zeller, “It’s not a bug, it’s a feature: How misclassification impacts bug prediction,” in Proceedings of the 35th International Conference on Software Engineering, 2013, pp. 392–401. [4] X. Wu, W. Zheng, X. Xia, and D. Lo, “Data quality matters: A case study on data label correctness for security bug report prediction,” IEEE Transactions on Software Engineering, vol. 48, no. 7, pp. 2541–2556, 2021. [5] J. Zhang, X. Hu, S. Gao, X. Xia, D. Lo, and S. Li, “Less is more: On the importance of data quality for unit test generation,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 1293–1316, 2025. [6] G. Rodrı́guez-Pérez, M. Nagappan, and G. Robles, “Watch out for extrinsic bugs! a case study of their impact in just-in-time bug prediction models on the openstack project,” IEEE Transactions on Software Engineering, vol. 48, no. 4, pp. 1400–1416, 2020. [7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. [8] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024. [9] G. Colavito, F. Lanubile, N. Novielli, and L. Quaranta, “Leveraging gptlike llms to automate issue labeling,” in 21st International Conference on Mining Software Repositories, 2024, pp. 469–480. [10] G. Colavito, F. Lanubile, and N. Novielli, “Benchmarking large language models for automated labeling: The case of issue report classification,” Information and Software Technology, vol. 184, p. 107758, 2025. [11] R.-M. Karampatsis and C. Sutton, “How often do single-statement bugs occur? the manysstubs4j dataset,” in The 17th international conference on mining software repositories, 2020, pp. 573–577. [12] Z. Liu, S. Zhu, B. Qin, H. Chen, and L. Song, “Automatically detecting and fixing concurrency bugs in go software systems,” in Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2021, pp. 616–629. [13] T. Hirsch and B. Hofer, “Using textual bug reports to predict the fault category of software bugs,” Array, vol. 15, p. 100189, 2022. [14] P. Bhandari and G. Rodrı́guez-Pérez, “BuggIn: Automatic intrinsic bugs classification model using nlp and ml,” in Proceedings of the 19th International Conference on Predictive Models and Data Analytics in Software Engineering, 2023, pp. 2–11. [15] G. Fan, Y. Liang, L. Zu, H. Yu, Z. Huang, and W. Chen, “Automatic identification of extrinsic bug reports for just-in-time bug prediction,” Science of Computer Programming, vol. 249, p. 103410, 2026. [Online]. Available: https://doi.org/10.1016/j.scico.2025.103410 [16] T. Wright, A. Chen, and G. Rodrı́guez-Pérez, “InEx-Bug: A human annotated dataset of intrinsic and extrinsic bugs in the npm ecosystem,” in Proceedings of the 23rd International Conference on Mining Software Repositories. Rio de Janeiro, Brazil: ACM, 2026, preprint: https://arxiv.org/abs/2602.13400. [17] R. Kikas, G. Gousios, M. Dumas, and D. Pfahl, “Structure and evolution of package dependency networks,” in The 14th International Conference on Mining Software Repositories. IEEE, 2017, pp. 102–112. [18] A. Decan, T. Mens, and P. Grosjean, “An empirical comparison of dependency network evolution in seven software packaging ecosystems,” Empirical Software Engineering, vol. 24, no. 1, pp. 381–416, 2019. [19] A. Decan, T. Mens, and E. Constantinou, “On the impact of security vulnerabilities in the npm package dependency network,” in Proceedings of the 15th International Conference on Mining Software Repositories, 2018, pp. 181–191. [20] ——, “On the evolution of technical lag in the npm package dependency network,” in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2018, pp. 404–414. [21] F. R. Cogo, G. A. Oliva, and A. E. Hassan, “Deprecation of packages and releases in software ecosystems: A case study on npm,” Transactions on Software Engineering, vol. 48, no. 7, pp. 2208–2223, 2021. [22] D. Venturini, F. R. Cogo, I. Polato, M. A. Gerosa, and I. S. Wiese, “I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 4, pp. 1–26, 2023.

[23] C. Miller, M. Jahanshahi, A. Mockus, B. Vasilescu, and C. Kastner, “Understanding the response to open-source dependency abandonment in the npm ecosystem,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 2355–2367. [24] G. Antoniol, K. Ayari, M. Di Penta, F. Khomh, and Y.-G. Guéhéneuc, “Is it a bug or an enhancement? a text-based approach to classify change requests,” in The 2008 conference of the center for advanced studies on collaborative research: meeting of minds, 2008, pp. 304–318. [25] H. Qin and X. Sun, “Classifying bug reports into bugs and non-bugs using lstm,” in Proceedings of the 10th Asia-Pacific Symposium on Internetware, 2018, pp. 1–4. [26] R. Kallis, A. Di Sorbo, G. Canfora, and S. Panichella, “Ticket tagger: Machine learning driven issue classification,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2019, pp. 406–409. [27] ——, “Predicting issue types on github,” Science of Computer Programming, vol. 205, p. 102598, 2021. [28] P. Terdchanakul, H. Hata, P. Phannachitta, and K. Matsumoto, “Bug or not? bug report classification using n-gram idf,” in 2017 IEEE International Conference on Software Maintenance and Evolution, 2017, pp. 534–538. [29] E. Aktas, E. Cakmak, M. Inan, and C. Yilmaz, “Issue report validation in an industrial context,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE ’23. ACM, Nov. 2023. [30] Y. Zhou, Y. Tong, R. Gu, and H. Gall, “Combining text mining and data mining for bug report classification,” in 2014 IEEE International Conference on Software Maintenance and Evolution, 2014, pp. 311–320. [31] S. Herbold, A. Trautsch, and F. Trautsch, “On the feasibility of automated prediction of bug and non-bug issues,” Empirical Software Engineering, vol. 25, no. 6, pp. 5333–5369, 2020. [32] G. Aracena, K. Luster, F. Santos, I. Steinmacher, and M. A. Gerosa, “Applying large language models to issue classification,” in Proceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering, 2024, pp. 57–60. [33] G. Colavito, F. Lanubile, N. Novielli, and L. Quaranta, “Impact of data quality for automatic issue classification using pre-trained language models,” Journal of Systems and Software, vol. 210, p. 111838, 2024. [34] A. Koyuncu, “Exploring fine-grained bug report categorization with large language models and prompt engineering: An empirical study,” ACM Transactions on Software Engineering and Methodology, 2025. [35] M. Laiq and F. Dobslaw, “Automatic techniques for issue report classification: A systematic mapping study,” Automated Software Engineering, vol. 33, no. 3, p. 72, 2026. [36] X. Du, Z. Liu, C. Li, X. Ma, Y. Li, and X. Wang, “Llm-brc: A large language model-based bug report classification framework,” Software Quality Journal, vol. 32, no. 3, pp. 985–1005, 2024. [37] T. Wright, A. Chen, and G. Rodriguez-Perez, “InEx-Bug: A human annotated dataset of intrinsic and extrinsic bugs in the npm ecosystem,” https://github.com/STASER-Lab/InEx-Bug-A-Human-Annotated-Datas et-of-Intrinsic-and-Extrinsic-Bugs-in-the-npm-Ecosystem, University of British Columbia, Kelowna, British Columbia, Canada, 2025, version 3.0. [38] S. Baltes, F. Angermeir, C. Arora, M. M. Barón, C. Chen, L. Böhme, F. Calefato, N. Ernst, D. Falessi, B. Fitzgerald et al., “Guidelines for empirical studies in software engineering involving large language models,” arXiv preprint arXiv:2508.15503, 2025. [39] Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,” 2022. [Online]. Available: https: //arxiv.org/abs/2104.08786 [40] M. Mizrahi, G. Kaplan, D. Malkin, R. Dror, D. Shahaf, and G. Stanovsky, “State of what art? a call for multi-prompt llm evaluation,” 2024. [Online]. Available: https://arxiv.org/abs/2401.00595 [41] A. Salinas and F. Morstatter, “The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance,” in Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 4629–4651. [42] S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp. 1–28, 2025. [43] B. Atil, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Rajagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, and

B. Baldwin, “Non-determinism of ”deterministic” llm settings,” 2025. [Online]. Available: https://arxiv.org/abs/2408.04667 [44] J. L. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological Bulletin, vol. 76, no. 5, pp. 378–382, 1971. [45] Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,” Journal of the Royal statistical society: series B (Methodological), vol. 57, no. 1, pp. 289–300, 1995. [46] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” 2023. [Online]. Available: https://arxiv.org/abs/2203.11171 [47] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, pp. 159–174, 1977. [48] F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. Chi, N. Schärli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” 2023. [Online]. Available: https://arxiv.org/abs/2302.00093 [49] A. R. Oskooei, K. B. Cosdan, H. Isiktas, and M. S. Aktas, “When many-shot prompting fails: An empirical study of llm code translation,” 2025. [Online]. Available: https://arxiv.org/abs/2510.16809 [50] S. Jiang and D. Nam, “Beyond the prompt: An empirical study of cursor rules,” in Proceedings of the 23rd International Conference on Mining Software Repositories, 2026, to appear. [Online]. Available: https://arxiv.org/abs/2512.18925 [51] M. Turpin, J. Michael, E. Perez, and S. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chainof-thought prompting,” Advances in Neural Information Processing Systems, vol. 36, pp. 74 952–74 965, 2023. [52] T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal, “Decomposed prompting: A modular approach for solving complex tasks,” arXiv preprint arXiv:2210.02406, 2022. [53] G. Colavito, F. Lanubile, N. Novielli, C. Arreza, and Y. Shi, “Issue classification with llms: An empirical study of the nasa flight software systems,” Journal of Systems and Software, p. 112851, 2026.

Related documents

• Automatic Assistance to Reduce Information Overload during Software Issue Report Understanding#1614157
• Automatic Assistance to Reduce Information Overload during Software Issue Report Understanding#1571868
• Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs#987392
• Argumentation for Explainable and Globally Contestable Decision Support with LLMs#798800
• Argumentation for Explainable and Globally Contestable Decision Support with LLMs#797386