arXiv CSopen access

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

arXiv:2607.18550v1 [cs.SE] 20 Jul 2026

WENDKÛUNI C. OUÉDRAOGO, University of Luxembourg, Luxembourg YINGHUA LI∗ , Nanjing University of Science and Technology, China XUEQI DANG, University of Luxembourg, Luxembourg PAWEŁ BORSUKIEWICZ, University of Luxembourg, Luxembourg LIANG XIAO, Nanjing University of Science and Technology, China LINGFENG BAO, Zhejiang University, China ANIL KOYUNCU, Bilkent University, Turkey JACQUES KLEIN, University of Luxembourg, Luxembourg DAVID LO, Singapore Management University, Singapore TEGAWENDÉ F. BISSYANDÉ, University of Luxembourg, Luxembourg Bug resolution is a cross-artifact process: natural-language reports must expose actionable behavioral cues that can be reproduced by tests and corrected by patches. Yet the extent to which these signals are preserved across artifacts remains largely unquantified. We introduce Desc2Fix, a framework for measuring semantic alignment between bug reports, triggering tests, and developer-written fixes. Alignment is operationalized through structured behavioral anchors (e.g., reproduction steps, API/exception cues, expected vs. actual behavior), deterministic similarity metrics (ROUGE, SBERT, CodeBERT, OpenAI embeddings), and LLMbased judgments grounded in coverage, correctness, and specificity. We analyze 2,857 report–test–patch triplets from Defects4J and SWT-Bench using GPT-4o and DeepSeek-Chat. LLMs reliably extract structured signals (up to 90% completeness) and exhibit strong cross-model consistency, yielding a stable semantic input contract for downstream reasoning. However, alignment is highly representation-sensitive: lexical similarity alone is insufficient; full diffs provide the most stable basis for judging report–patch correspondence; and structured summaries trade surface similarity for predicate-level salience. Across more than 182,000 LLM alignment ratings, both models show systematic optimism relative to humans (1–2 points on 5-point scales) and only modest rank agreement, motivating bias-aware evaluation. Our results demonstrate that behavioral alignment is measurable but not reducible to similarity, and that structured anchors combined with embedding-based proxies provide reproducible signals for ranking and filtering tests and candidate patches. By transforming alignment into a controllable engineering signal, Desc2Fix enables more reliable test generation, semantics-aware fault localization, principled patch ranking, and improved bug report authoring. ∗ Corresponding author.

Authors’ addresses: Wendkûuni C. Ouédraogo, [email protected], University of Luxembourg, Luxembourg; Yinghua Li, [email protected], Nanjing University of Science and Technology, China; Xueqi Dang, [email protected], University of Luxembourg, Luxembourg; Paweł Borsukiewicz, [email protected], University of Luxembourg, Luxembourg; Liang Xiao, [email protected], Nanjing University of Science and Technology, China; Lingfeng Bao, [email protected], Zhejiang University, China; Anil Koyuncu, [email protected], Bilkent University, Turkey; Jacques Klein, [email protected], University of Luxembourg, Luxembourg; David Lo, [email protected], Singapore Management University, Singapore; Tegawendé F. Bissyandé, [email protected], University of Luxembourg, Luxembourg. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Association for Computing Machinery. XXXX-XXXX/2026/7-ART $15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn , Vol. 1, No. 1, Article . Publication date: July 2026.

2

Ouédraogo et al.

CCS Concepts: • Software and its engineering → Software testing and debugging; • Computer systems organization → Neural networks. Additional Key Words and Phrases: Bug Report, Semantic Alignment, Automated Test Generation, Program Repair, Fault Localization, LLM, Empirical Study ACM Reference Format: Wendkûuni C. Ouédraogo, Yinghua Li, Xueqi Dang, Paweł Borsukiewicz, Liang Xiao, Lingfeng Bao, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F. Bissyandé. 2026. Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches. 1, 1 (July 2026), 35 pages. https: //doi.org/10.1145/nnnnnnn.nnnnnnn

1

INTRODUCTION

Software systems are inherently prone to defects, making bug resolution a central task in software maintenance. In modern development workflows, bug reports are the primary interface for communicating software anomalies [4]. They typically describe reproduction steps, input conditions, stack traces, and expected versus actual behavior [20, 51], providing the semantic basis for constructing triggering tests and corrective patches. Yet, these behavioral signals are often diluted or partially lost as they propagate to executable artifacts. Tests may reproduce symptoms without encoding intended behavior, and patches may suppress failures without restoring semantic correctness. This semantic drift [37] weakens the correspondence between report, test, and fix, limiting precision, interpretability, and automation reliability. Bridging this gap between natural-language intent and code-level realization remains a fundamental challenge in automated software maintenance [11, 18]. Recent advances in automated testing, fault localization, and program repair have substantially improved maintenance efficiency [18, 25, 42, 45, 51]. However, these techniques typically treat bug reports as textual inputs rather than as structured semantic drivers, often overlooking contextual anchors such as input conditions, exception types, or expected outcomes. At the same time, large language models (LLMs) and code-aware transformers have enabled joint reasoning over natural language and code [7, 15, 38, 48]. Fault localization and repair systems have progressed from lexical retrieval to semantic and agentic reasoning [6, 8, 47, 51], and LLM-based test generation can now reproduce failures with increasing accuracy [9, 17, 36]. Yet these stages are still evaluated largely in isolation. Existing work rarely examines whether the semantics of the original bug description are preserved, transformed, or lost as they propagate from reports to tests and patches. A fundamental question thus remains largely unexplored: How are the behavioral signals expressed in bug reports preserved, transformed, or lost as they propagate to triggering tests and corrective patches? Answering this question is critical not only for understanding the interpretability of current LLM-based systems, but also for designing maintenance pipelines where bug reports act as controllable semantic drivers rather than passive textual inputs. Quantifying cross-artifact alignment provides an actionable engineering signal: reports that expose strong behavioral anchors can be transformed into reliable triggering tests and precise repair objectives, whereas weakly aligned tests may validate superficial fixes rather than semantic correctness, and patches that diverge from report-level intent may optimize for fail-to-pass transitions without restoring intended functionality. Measuring alignment therefore enables (i) early detection of non-actionable reports, (ii) alignment-guided test generation and filtering, (iii) semantics-aware fault localization, and (iv) principled ranking of candidate patches in automated repair workflows. To operationalize this perspective, we introduce Desc2Fix, a unified framework for quantifying cross-artifact semantic alignment. We first extract structured behavioral anchors (e.g., exception types, APIs, reproduction steps, expected vs. actual behavior) using LLM-assisted signal extraction, producing a controlled representation of developer intent. We then define alignment dimensions capturing how entity-level, scenario-level, and behavioral signals are reflected in tests and patches. , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

3

Alignment is assessed through complementary perspectives: deterministic similarity metrics (lexical and embedding-based), LLM-based semantic judgments grounded in coverage, correctness, and specificity, and human annotations used to validate model outputs. By triangulating these signals, Desc2Fix enables fine-grained analysis of semantic preservation and drift across the bug resolution pipeline. Our empirical evaluation spans two complementary datasets enabling cross-language and crossartifact analysis. Defects4J v3.0.1 [16] provides curated Java bugs with explicitly linked reports, triggering tests, and patches, supporting controlled alignment analysis. SWT-Bench [28] extends the study to Python and real-world GitHub issues paired with validated fail-to-pass tests, capturing more diverse and less curated development settings. Together, these corpora allow us to examine how behavioral signals propagate across artifacts and ecosystems, making this the first large-scale cross-language study of semantic alignment among bug reports, triggering tests, and corrective patches. This paper makes the following contributions: • A unified framework for cross-artifact behavioral alignment. We introduce Desc2Fix, a principled framework for quantifying how behavioral signals propagate from bug reports to triggering tests and corrective patches. The framework combines LLM-assisted structured extraction, a taxonomy of alignment dimensions (Entity, Scenario, Behavioral), and multi-perspective evaluation (lexical, embedding-based, and LLM-based), enabling fine-grained and human-grounded analysis. • A large-scale cross-language empirical study of semantic propagation. Across Defects4J (Java) and SWT-Bench (Python), we analyze 2,857 report–test–patch triplets to characterize how behavioral signals are preserved, transformed, or lost. Our study reveals representation-sensitive alignment effects, systematic LLM optimism, and the limitations of similarity-based proxies. • Desc2Fix: a reusable alignment benchmark. We release an annotated corpus linking structured bug reports, tests, and patches, enriched with embedding-based similarity scores and LLM-evaluated alignment judgments, supporting reproducible research on semantics-aware testing and repair. • Actionable guidance for semantics-aware maintenance. We identify which report attributes (e.g., explicit expected/actual contrast, salient steps) most strongly influence test adequacy and patch precision, and we derive alignment-guided engineering blueprints — operationalized as reusable algorithms — for test generation and repair workflows. • A fully reproducible research package. All datasets, prompts, evaluation scripts, and analysis code are publicly released1 to facilitate replication and extension in LLM-based maintenance research. The paper is organized as follows: Section 2 outlines key concepts. Section 3 details our study design. Section 4 presents findings for the three research questions. Section 5 discusses implications and limitations. Section 6 positions our study within existing literature, and Section 7 concludes with key insights and future directions. 2 2.1

BACKGROUND Bug Reports as Semantically Rich Artifacts

Bug reports are the primary entry point of the bug resolution process, describing failure context, reproduction scenarios, and expected versus actual behaviors [4, 20]. Beyond documenting symptoms, they encode linguistic and semantic signals that guide downstream activities such as test creation, fault localization, and patch generation. Report quality strongly influences resolution effectiveness: 1 https://anonymous.4open.science/r/Desc2Fix-EC04/

, Vol. 1, No. 1, Article . Publication date: July 2026.

4

Ouédraogo et al.

detailed reproduction steps and precise behavioral descriptions accelerate fixes [3], and structural and linguistic features correlate with fix success [30]. However, bug reports exhibit substantial variability in verbosity, terminology, and contextual explicitness, which can cause semantic drift when transitioning to tests and patches — key failure cues may be paraphrased, transformed, or omitted entirely.

2.2

From Fault Localization to Automated Program Repair

Fault localization aims to connect natural-language bug reports to faulty code elements. Early information-retrieval approaches (e.g., BugLocator [51], AmaLgam [42]) relied on lexical similarity between reports and source files, remaining sensitive to vocabulary variation and contextual ambiguity [21]. Later techniques incorporated embedding-based models and neural classifiers [8, 47], and more recently LLM-assisted methods further improved localization by reformulating reports or optimizing ranking objectives [6, 39]. Automated Program Repair (APR) extends this bridge from textual intent to executable fixes. Early text-driven systems demonstrated the feasibility of leveraging bug reports for patch generation [11, 25], while recent retrieval-augmented and agentic LLM systems synthesize validated patches directly from repository context [15, 18, 27, 48]. Despite this progress, most approaches optimize individual stages without explicitly examining semantic continuity across artifacts — the gap our study directly addresses.

2.3

Test Generation from Bug Reports

Test generation translates bug reports into executable specifications that reproduce observed failures. Early code-centric tools such as EvoSuite and Randoop largely ignored contextual information embedded in bug reports, limiting their ability to capture failure-specific inputs and behavioral constraints. More recent LLM-based approaches generate tests directly from issue descriptions, though a gap persists between syntactic executability and true behavioral relevance [36]. Extracting explicit and implicit inputs from reports has been shown to substantially improve test reproducibility [33, 34], and large-scale evaluations confirm the potential of LLM-driven test synthesis across diverse benchmarks [1, 2, 9, 17, 28]. However, existing studies primarily assess test correctness or executability rather than how behavioral cues propagate from reports into tests and subsequently into patches.

2.4

The Need for Cross-Artifact Semantic Alignment

Fault localization, test generation, and program repair have each advanced significantly, yet they are typically studied as independent stages. Three limitations persist across this body of work. First, most approaches rely on lexical similarity or task-specific optimization, inadequately capturing deeper correspondences between natural-language intent and executable behavior. Second, no unified framework systematically quantifies how behavioral signals propagate across artifacts: prior work combining report- and test-based signals improves repair performance [18, 27] but does not measure how cues are preserved, transformed, or lost from description to test to patch. Third, existing evaluations target generation performance within individual benchmarks [14, 16, 28] rather than cross-artifact semantic alignment. Desc2Fix directly addresses these gaps through a large-scale, cross-language examination of alignment among bug reports, tests, and patches. , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

3

5

STUDY DESIGN

3.1

Analysis Overview

Our methodology (Figure 1) is designed to analyze the textual and semantic alignment among three core artifacts involved in software bug resolution: the bug report, the triggering test, and the patch. We define a multi-layered analysis framework based on signal extraction, alignment taxonomy, and both metric- and LLM-based evaluation strategies. Datasets

Bug Reports & Tests (RQ2)

CoT DeepSeek-chat GPT-4o Defects4J

HE-1 Mode?

Raw Bug Reports

Raw

HE-2

Human Experts Verification

Ground Truth Bug Signal

Structured Bug Signal Extraction

Structured JSON Bug Reports

HE-1

HE-2

LLMs-Based vs. Metrics-Based Alignment Score Analysis

LLM & Human Alignment Evaluations

Cosine Similarities SBERT Bug Reports & Tests (RQ2)

CodeBERT

OpenAI Embeddings

Semantic Alignment ROUGE-L Scores

Triggering Tests

Bug Reports & Patches (RQ3)

Patch Perspectives Extraction Deletions Additions Patches

Alignment Score Analysis Human Alignment Scores

Human Experts Verification

LLMs-Based vs. Human-Based Signal Extraction Analysis (RQ1)

Structured

LLMs Alignment Scores

Large Language Models (LLMs)

Bug Reports & Patches (RQ3)

LLM-generated Bug Signal

Large Language Models (LLMs)

SWT-Bench

Triplets Extraction

DeepSeek-chat GPT-4o

Full

Jaccard Scores ROUGE-L

Lexical Alignment

Patch Views

Triplets of Core Artifacts

Alignment Score Analysis

Jaccard

Syntactic Alignment

POS Verbs Alignment Scores POS Nouns Alignment Scores

Metric-Based Alignment Evaluations

Fig. 1. Overview of the general workflow of Desc2Fix. 3.2

Textual and Semantic Signal Extraction

We extract high-level semantic signals from bug reports to support alignment analyses with triggering tests and corrective patches. In our setting, bug reports are treated as pre-fix problem descriptions—i.e., the developer-facing specification available before (or at the time) a failing test and a corrective patch are produced. This distinction matters because post-fix narratives (e.g., retrospective summaries) may directly incorporate solution details and would inflate apparent cross-artifact alignment. Signals are extracted at three abstraction levels: • Entities: exception types, API methods, constants, and error messages. • Scenarios: reproduction steps and input conditions required to trigger the bug. • Behaviors: expected vs. actual outcomes, including inferred fix intent. We prompt the LLM to produce a structured JSON representation aligned with these levels, improving consistency in the presence of paraphrases or implicit cues.2 An example output is shown below: Example: Structured Extraction Output (JSON) { " exception " : " NullPointerException " , " a p i _ i n v o l v e d " : [ " Form . s u b m i t " ] , " i n p u t _ c o n d i t i o n s " : [ " form ␣ i s ␣ empty " ] , " e x p e c t e d " : " V a l i d a t i o n ␣ e r r o r ␣ i s ␣ shown " , " a c t u a l " : " A p p l i c a t i o n ␣ c r a s h e s ␣ with ␣ N u l l P o i n t e r E x c e p t i o n " , " r e p r o d u c t i o n _ s t e p s " : [ " open ␣ t h e ␣ form " , " leave ␣ a l l ␣ f i e l d s ␣ blank " , " c l i c k ␣ submit " ] 2 https://anonymous.4open.science/r/Desc2Fix-EC04/

, Vol. 1, No. 1, Article . Publication date: July 2026.

6

Ouédraogo et al.

}

This representation enables systematic comparison across artifacts (e.g., whether reported exceptions, input conditions, and behavioral constraints are reflected in tests or patches), supporting our analysis of which signals are preserved, transformed, or lost throughout bug resolution. 3.3

Taxonomy of Alignment Dimensions and Metrics

To analyze how semantic signals propagate from bug reports to tests and patches, we define three alignment dimensions, each evaluated using lexical overlap and embedding-based semantic similarity. Entity Alignment measures whether report-level entities (e.g., exception types, APIs, literals) appear in corresponding tests or patches. Scenario Alignment assesses whether reproduction steps and input conditions are reflected in test setup or execution structure. Behavioral Alignment captures whether expected and actual outcomes are enforced in test assertions or addressed in patch logic. For lexical alignment, we use literal and fuzzy overlap metrics (Jaccard, ROUGE-L) and POS-based verb matching for scenarios. For semantic alignment, we compute similarity using SBERT, CodeBERT, BERTScore (behavioral dimension), and OpenAI embeddings. Each bug report is structured via LLM extraction (Section 3.2); alignment scores are then computed per dimension to analyze signal preservation, transformation, and loss across artifacts. 3.4

LLM-Assisted Semantic Analysis

To complement metric-based analyses, we use large language models (LLMs) as semantic evaluators of alignment between bug reports and corresponding tests or patches. We consider two input configurations: RAW, where the model receives the original bug report in free-form text, and STRUCTURED, where it is provided with the extracted JSON representation described in Section 3.2. This design allows us to assess the impact of structured inputs on semantic evaluation. For each report–artifact pair, the LLM produces a structured JSON assessment with three Likert-scale scores (1–5): coverage, correctness, and specificity, each accompanied by a brief justification. A final alignment score is computed as the rounded average of the three dimensions. This protocol enables consistent and interpretable comparison across models and input formats. We evaluate two widely adopted instruction-tuned LLMs representing distinct model families: GPT-4o [13], a state-of-theart commercial model, and DeepSeek-Chat [24], a competitive open-weight alternative. Both are accessed via API under consistent decoding settings (temperature= 1.0, top_p= 1.0), and all responses are serialized in JSON for downstream aggregation. Prompt templates and evaluation details are available in the replication package. 3.5

Research Questions

Our study investigates whether cross-artifact alignment can serve as a measurable and actionable signal for improving debugging and repair pipelines. RQ1: Can LLMs reliably extract structured semantic signals from bug reports to enable downstream alignment analysis? Before alignment can be quantified, bug reports must be transformed into structured representations capturing actionable anchors (e.g., APIs, input conditions, reproduction steps, expected/actual behavior). We evaluate GPT-4o and DeepSeek-Chat as independent extractors and assess whether their outputs are sufficiently complete, coherent, and human-aligned to serve as reliable inputs for repair-oriented analyses. RQ2: Can report–test alignment serve as an indicator of test adequacy and report testability? We quantify the semantic and structural alignment between bug reports and triggering tests, , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

7

and analyze which report characteristics (e.g., explicit expected/actual statements, API mentions, reproduction steps) are associated with stronger alignment. We examine whether alignment signals can identify tests that faithfully encode the reported failure and reports that are more test-ready. RQ3: Can report–patch alignment act as a semantic validation signal for program repair? We measure how closely patches address the behaviors and conditions described in bug reports across multiple patch views (full diff, additions, removals). We analyze whether alignment scores and structured anchors can help distinguish semantically grounded fixes from potentially overfitting or incomplete patches. 3.6

Prompting Techniques

All LLM-based evaluations use standardized prompts following the same structure: contextual grounding, explicit evaluation criteria, and JSON-formatted outputs. We adopt a lightweight Chainof-Thought strategy [43], instructing the model to briefly reason before producing structured scores. Each task is evaluated under two variants: RAW (natural-language report) and STRUCTURED (JSON-based representation from Section 3.2). Prompts are executed with GPT-4o and DeepSeekChat (temperature=1.0, top_p=1.0). Full templates are available in the replication package. Bug Report–Test Alignment. For report–test pairs, the model evaluates alignment along four dimensions: coverage, correctness, specificity, and overall alignment, providing a brief justification followed by a structured JSON assessment with scores (1–5): Bug Report–Test Alignment Output Schema { " a l i g n m e n t _ s c o r e " : <1 −5 > , " c o v e r a g e " : <1 −5 > , " c o r r e c t n e s s " : <1 −5 > , " s p e c i f i c i t y " : <1 −5 > , " justification " : " . . . " }

Bug Report–Patch Alignment. Patch evaluation follows the same scoring scheme. The model receives either the RAW or STRUCTURED report together with the patch (full diff, additions only, or deletions only) and produces scores using the same JSON schema above. On model and prompting choices. Our experimental design spans two datasets, six framing conditions (two report representations × three patch views), and four scoring dimensions, generating a large volume of alignment ratings that imposes practical constraints on model choice and prompting complexity. We rely on Chain-of-Thought prompting rather than native reasoning (“thinking”) modes or agentic orchestration, as our objective is to assess behavioral alignment as a measurable and reproducible signal rather than to maximize generation performance. The two selected models represent complementary paradigms — proprietary versus open-weight — ensuring that our findings are not artifacts of a single provider’s design choices. We further discuss the implications of model recency and prompting strategy in our threats to validity (Section 5.4). Discussion. Across tasks, prompts enforce uniform reasoning and structured outputs, enabling automated parsing and direct comparison with metric-based measures for large-scale alignment analysis. , Vol. 1, No. 1, Article . Publication date: July 2026.

8

3.7

Ouédraogo et al.

Datasets and Experimental Scope

We evaluate our framework on two complementary benchmarks covering Java and Python ecosystems: Defects4J and SWT-Bench. This dual-dataset design enables cross-language analysis of semantic alignment across bug reports, tests, and patches. We curate a structured corpus from Defects4J v3.0.1 [16], focusing on 16 projects with accessible and consistent issue metadata (excluding JFreeChart). For each bug, we reconstruct triplets linking the natural-language report, triggering/relevant tests, and the human-written patch. Bug reports are retrieved from JIRA, GitHub, or archived sources; patches are parsed into additions, deletions, and full diffs; and tests are linked using trigger_tests/ and relevant_tests/ metadata. This process yields 511 curated triplets spanning 16 Java projects. Table 1 summarizes corpus statistics. Table 1. Statistics of the curated Defects4J-v3 corpus. Metric

Value

Interpretation

Total instances (RAW) Unique projects Bug reports (retrieved) Avg. bug report length Avg. patch size Avg. source patch size Avg. test patch size Bugs with test patch Avg. trigger tests Avg. relevant tests Add/remove ratio Report sources

511 16 511 (100%) 142.7 words 52.4 lines 38.1 lines 14.3 lines 383 (74.9%) 1.2 per bug 2.5 per bug 1.31 JIRA (50%), GitHub (43.8%), Google Code (6%)

Triplets (bug report–test–patch) reconstructed from Defects4J v3. Java projects spanning multiple domains. Available via JIRA, GitHub, or Google Code. Textual richness of issue descriptions. Overall modification granularity (src + test). Code fix magnitude. Extent of test modification. Presence of test code in the correction. Tests failing on the buggy version. Tests explicitly linked to the bug. Slightly more additions than deletions. Provenance of natural-language reports.

To extend our analysis beyond curated Java benchmarks, we leverage SWT-Bench [28], a dataset of real-world Python issues paired with fail-to-pass tests and human-written patches. We merge the dev and test splits (2,519 instances) and apply a reproducible cleaning and structuring pipeline. We parse patches into additions, deletions, and full diffs, reconstruct executable test functions, and retain only instances containing all three artifacts (bug report, test, patch), resulting in 2,346 aligned triplets. Metadata such as repository name, patch size, and test type are aggregated for statistical profiling. Table 2 reports summary statistics. Table 2. Statistics of the curated SWT-Bench corpus. Metric

Value

Interpretation

Total instances (RAW) Triplets (bug–test–patch) Unique repositories Avg. bug report length Avg. test length Avg. patch size Add/remove ratio Executable tests Declarative tests

2,519 2,346 (93.1%) 18 119.6 words 33.7 lines 41.2 lines 4.48 2,346 (93.1%) 3 (0.1%)

All merged from Hugging Face (dev + test). Fully aligned artifacts across all repositories. Cross-project coverage across major Python ecosystems. Issue textual richness and contextual diversity. Behavioral scope of fail–pass validation tests. Typical granularity of human-written fixes. Balance between additive and corrective changes. Parsed Python test functions with assertions. YAML or DSL-based test specifications.

Combined, Defects4J and SWT-Bench yield 2,857 report–test–patch triplets, forming the empirical basis for all subsequent analyses (Sec. 4). 3.8

Metrics and Evaluation

All analyses combine three complementary perspectives: (i) LLM-based evaluation, (ii) metricbased similarity, and (iii) human evaluation for calibration. Evaluations are performed at the triplet level (report–test–patch) on both Defects4J and SWT-Bench to ensure cross-dataset comparability. LLM-based scores follow the four-dimension protocol described in Sec. 3.4 (coverage, , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

9

correctness, specificity, overall alignment), produced independently by GPT-4o and DeepSeek-Chat and averaged across models, with justifications retained for qualitative analysis. Metric-based evaluation covers lexical, semantic, and syntactic similarity between artifacts: lexical overlap via ROUGE-L and Jaccard; semantic similarity via cosine similarity over SBERT, CodeBERT, and OpenAI (text-embedding-3-small) embeddings; and structural similarity via POS-based noun and verb overlap using spaCy. We further analyze relationships between alignment scores and artifact-level features (e.g., report length, reproduction steps, exception mentions, test assertions) using non-parametric correlations (Spearman’s 𝜌, Kendall’s 𝜏, 𝑝<0.05), enabling cross-validation between LLM judgments and deterministic metrics. Table 3 summarizes all metrics and statistical procedures used throughout the study. Table 3. Summary of the metrics and evaluation protocol used across all analyses. Category

Metric / Dimension

Interpretation

Coverage, Correctness, Speci- Expert ratings from GPT-4o and DeepSeek-Chat assessing how ficity, Alignment (1–5) well artifacts (reports, tests, patches) align semantically and behaviorally. Lexical ROUGE-L, Jaccard Measures surface textual overlap between natural-language descriptions and code artifacts, reflecting literal term reuse. 3 Semantic SBERT, CodeBERT, OpenAI Captures conceptual relatedness beyond word overlap, linking + cosine similarity bug descriptions, tests, and patches at the embedding level. Syntactic POS overlap (verbs, nouns) via Evaluates linguistic and structural correspondence, measuring spaCy overlap in action verbs and entities across artifacts. Representation RAW vs. STRUCTURED re- Compares natural-language input with structured represenports tations (Entity, Scenario, Behavior) extracted through LLM prompting. Three complementary patch perspectives capturing the implePatch views Full / Additions / Deletions mentation, removal, or refactoring aspects of fixes. Statistical Spearman’s 𝜌, Kendall’s 𝜏 Non-parametric correlations between textual and semantic (two-sided, 𝑝<0.05) features of reports/tests and their alignment quality. LLM-based

3.9

Manual Verification and Quality Control

To anchor automated analyses in developer judgment, we conduct human evaluation on a shared sample of 400 bug reports (200 Defects4J, 200 SWT-Bench), covering the complete Desc2Fix pipeline across more than 29,000 individual field annotations and alignment ratings. Two annotators produced reference annotations for the six structured fields (exception, API, input conditions, reproduction steps, expected behavior, actual behavior), serving as ground truth for evaluating LLM-based extraction (precision, recall, F1). Inter-annotator agreement was high across fields, confirming that the schema supports reliable manual labeling. Annotators also qualitatively inspected representative report–test and report–patch triplets to validate LLM alignment scores, identifying recurrent scoring behaviors (e.g., optimistic ratings for minimal patches, sensitivity to sparse tests) that informed minor prompt clarifications while preserving evaluation independence. On the same sample, annotators independently rated report–test and report–patch behavioral alignment along coverage, correctness, specificity, and overall alignment. These ratings constitute the human reference for inter-human and Human–LLM agreement analyses (Spearman’s 𝜌, Kendall’s 𝜏, MAE, RMSE, Wilcoxon signed-rank tests with effect sizes). Report–test judgments were collected under RAW and STRUCTURED representations; report–patch judgments additionally covered Full, Add, and Remove patch views. 3.10

Implementation and Configuration

We conducted all experiments on a single workstation equipped with an Intel Core i9-14900K CPU (32 threads, 6.0 GHz), 64 GB RAM, and an NVIDIA RTX 5000 Ada GPU (32 GB VRAM). The entire pipeline was implemented in Python 3.10 and fully automated to enable scalable, , Vol. 1, No. 1, Article . Publication date: July 2026.

10

Ouédraogo et al.

reproducible execution across both Defects4J and SWT-Bench triplets. LLM-based evaluations were performed with GPT-4o and DeepSeek-Chat via their official APIs under consistent decoding settings (temperature=1.0, top_p=1.0), following the prompting scheme described in Section 3.6. Semantic similarities were computed with SBERT (sentence-transformers/paraphrase-mpnet-basev24 ) and CodeBERT (microsoft/codebert-base5 ) through the sentence-transformers framework, and with OpenAI’s text-embedding-3-small via the OpenAI API. Cosine similarity was used on all embedding spaces. Lexical metrics include ROUGE-L and Jaccard. Syntactic alignment features were obtained with spaCy6 (English model) for POS tagging and lemmatization of verbs and nouns in reports, tests, and patches. 4 4.1

RESULTS AND ANALYSIS RQ1: Reliability and Consistency of Structured Signal Extraction

[Experimental design]: RQ1 evaluates whether LLM-based structured extraction is reliable enough to serve as an enabling layer for alignment-driven debugging and repair analyses. Since RQ2 and RQ3 quantify cross-artifact alignment using structured report representations, extraction quality directly determines the validity of downstream signals. Using Defects4J (Java) and SWT-Bench (Python), we compare GPT-4o and DeepSeek-Chat as independent extractors. Each model generates a JSON summary with six fields (exception, api_involved, input_conditions, reproduction_steps, expected, actual), capturing actionable anchors across entity, scenario, and behavioral dimensions. We conduct three complementary analyses to assess extraction robustness. The first, structural coverage, measures completeness as the proportion of non-null fields to characterize schema population behavior. The second, extraction accuracy, evaluates agreement with 400 manually annotated reports (200 per dataset) using precision, recall, and F1 under both completeness-aware and content-only regimes. The third, cross-model consistency, measures structural agreement and semantic similarity (ROUGE-L, Jaccard, SBERT) to assess the stability of extracted signals across independent systems. Together, these analyses determine whether structured extraction provides a stable and semantically faithful representation suitable for alignment-based reasoning in test adequacy, fault localization, and patch validation tasks. [Results]: RQ1.1 – Structural Coverage and Descriptive Baselines. We first assess whether LLMs can populate our six-field schema at scale, as structural completeness directly determines how much report information becomes available for downstream alignment analyses (RQ2–RQ3). Table 4 shows that both GPT-4o and DeepSeek-Chat achieve consistently high completeness across corpora, with DeepSeek-Chat exhibiting a systematically denser extraction policy (Defects4J: 90.18% vs. 86.14%; SWT-Bench: 88.41% vs. 83.48%). Expected and Actual are near-saturated for both models across datasets (≈98–100%), indicating that behavioral outcome summaries are reliably populated regardless of extractor choice. Model differences instead concentrate on scenario specification fields (API, Input, Repro) and on Exception: DeepSeek-Chat consistently populates scenario anchors more frequently (e.g., Repro in SWT-Bench: 99.70% vs. 75.92%), suggesting a tendency to normalize implicit narrative descriptions into explicit procedural structure, whereas GPT-4o captures Exception more often (Defects4J: 48.92% vs. 44.03%; SWT-Bench: 39.41% vs. 32.48%), consistent with a salience-driven strategy that emphasizes explicit error cues. To contextualize these baselines against human behavior, we report the same presence statistics on matched 400-report subsets (Table 5). On both datasets, LLMs substantially exceed human 4 https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2 5 https://huggingface.co/microsoft/codebert-base 6 https://spacy.io/

, Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

11

Table 4. Presence rates (%) per field by LLM and dataset (full corpora). Dataset

Source

Compl.

Exception

API

Input

Repro

Expected

Actual

Defects4J

GPT-4o DeepSeek-Chat

86.14 90.18

48.92 44.03

92.37 98.83

91.19 99.22

85.91 99.22

99.41 99.80

99.02 100.00

SWT-Bench

GPT-4o DeepSeek-Chat

83.48 88.41

39.41 32.48

95.00 99.40

92.40 99.40

75.92 99.70

99.62 99.79

98.51 99.66

∗ Values are presence rates (%) for each field; Compl. is the mean percentage of the six fields that are non-null per instance.

Table 5. Structured signal extraction on a representative sample of 400 bug reports. Dataset

Source

Compl. (%)

Exception

API

Input

Repro

Expected

Actual

Defects4J

GPT-4o DeepSeek-Chat Human-1 Human-2

86.17 90.83 63.50 62.33

52.00 48.00 29.00 34.00

93.00 98.00 88.00 84.00

90.00 99.00 65.00 63.00

85.00 100.00 35.00 32.00

99.00 100.00 75.00 72.00

98.00 100.00 89.00 89.00

SWT-Bench

GPT-4o DeepSeek-Chat Human-1 Human-2

83.33 88.50 62.83 59.33

41.00 31.00 21.00 30.00

96.00 100.00 82.00 77.00

92.00 100.00 64.00 54.00

74.00 100.00 39.00 28.00

100.00 100.00 74.00 73.00

97.00 100.00 97.00 95.00

∗ Values are presence rates (%) for each field; Compl. is the average percentage of non-null fields per instance.

completeness (e.g., Defects4J: 90.83% DeepSeek-Chat / 86.17% GPT-4o vs. 62–64% for humans; SWT-Bench: 88.50% / 83.33% vs. 59–63%). This gap should not be read as humans being “worse”: manual annotations reflect principled abstention when a field is not explicitly supported by the report text, while LLMs more frequently infer and standardize implicit information (notably for Input and Repro). Structured extraction is therefore sufficiently dense to support downstream alignment measurements, but extractors differ in their fill-versus-abstain policies: DeepSeek-Chat is preferable when maximizing coverage of scenario anchors (useful to scaffold test intents and reproduction structure), whereas exception-centric workflows may benefit from cross-validating the Exception field with GPT-4o. Finding 1: LLMs extract dense structured bug-report signals at scale (83–90% completeness), but differ mainly by fill policy. DeepSeek-Chat more often reconstructs scenario anchors (API/Input/Repro), while GPT-4o more often surfaces Exception evidence; Expected/Actual is near ceiling for both (98–100%). Humans abstain more under ambiguity, suggesting LLM “extra completeness” reflects normalization rather than guaranteed evidence. Use LLM extraction as a reliable input contract, but treat scenario fields as coverage-oriented and exception fields as evidence-sensitive, with abstention rules or cross-model validation when needed. RQ1.2 – Extraction Accuracy. We next evaluate how faithfully extracted fields match human references (Table 6), distinguishing between two complementary regimes to disentangle semantic correctness from schema-completion policy. Completeness-aware accuracy (Table 6a) treats emptyversus-non-empty mismatches as errors, evaluating end-to-end extraction behavior including field population decisions. Under this regime, DeepSeek-Chat consistently outperforms GPT-4o (Defects4J: F1 = 0.595 vs. 0.514; SWT-Bench: 0.537 vs. 0.443), mirroring the denser population strategy observed in RQ1.1 rather than reflecting semantic disagreement. In contrast, content-only fidelity (Table 6b), which evaluates only fields where both model and human provide non-empty content, reveals near-ceiling semantic agreement for both systems (F1 ≥ 0.93 across datasets): DeepSeek-Chat leads on Defects4J (0.963 vs. 0.931), while GPT-4o shows a marginal advantage on SWT-Bench (0.981 vs. 0.970). The collapse of the performance gap under this regime shows that when models commit to extracting a field, their semantic representations closely match human annotations.

, Vol. 1, No. 1, Article . Publication date: July 2026.

12

Ouédraogo et al.

This contrast demonstrates that most discrepancies originate from fill-versus-abstain decisions rather than semantic distortion: extraction variability primarily reflects population policy, not content hallucination. This distinction is critical from an engineering standpoint, as alignment signals computed over populated fields are unlikely to be degraded by semantic extraction errors, and precision–coverage trade-offs can instead be explicitly controlled via abstention policies or similarity thresholds. Practically, DeepSeek-Chat offers stronger end-to-end completeness-aware performance, suiting coverage-oriented pipelines, while GPT-4o remains slightly favored on SWTBench for evidence-sensitive, conservative extraction. Table 6. Extraction accuracy against human references with semantic filtering. (a) Completeness-aware accuracy (all matched pairs) (b) Non-empty pairs only Dataset

Model

P

R

F1

Dataset

Model

P

R

F1

Defects4J

GPT-4o DeepSeek-Chat

0.500 0.583

0.542 0.618

0.514 0.595

Defects4J

GPT-4o DeepSeek-Chat

0.931 0.963

0.931 0.963

0.931 0.963

SWT-Bench

GPT-4o DeepSeek-Chat

0.438 0.531

0.454 0.549

0.443 0.537

SWT-Bench

GPT-4o DeepSeek-Chat

0.981 0.970

0.981 0.970

0.981 0.970

Finding 2: Extraction disagreements stem from field population policy, not semantic error. Under completeness-aware evaluation, DeepSeek-Chat achieves higher F1 due to more aggressive schema population. However, restricting evaluation to non-empty fields yields near-ceiling semantic fidelity for both models (F1 ≥ 0.93), indicating minimal semantic drift when content is produced. Structured extraction is therefore semantically stable, and downstream alignment analyses are unlikely to be biased by content-level hallucinations; coverage-precision trade-offs can be tuned via explicit abstention control. RQ1.3 – Model Consistency. Beyond accuracy against human annotations, we evaluate whether structured extraction is stable across independent LLM systems. High inter-model agreement would indicate that alignment signals derived from structured reports are not overly sensitive to extractor choice. As shown in Table 7, GPT-4o and DeepSeek-Chat make identical fill-versus-abstain decisions for the vast majority of fields (Defects4J: 93.61%; SWT-Bench: 92.16%), confirming that both systems converge on similar schema population decisions for most reports. When both models populate a field, semantic similarity is consistently high: mean ROUGE-L is approximately 0.65 across datasets, Jaccard ranges from 0.46 to 0.49, and SBERT cosine similarity reaches 0.80–0.81, indicating substantial semantic overlap despite lexical variation. DeepSeek-Chat maintains higher overall completeness (Defects4J: 90.25% vs. 86.14%, +4.11 pp; SWT-Bench: 88.43% vs. 83.67%, +4.76 pp), reflecting a more aggressive population strategy rather than divergent interpretation. Finding 3: Structured extraction is stable across independent LLM systems overall. GPT-4o and DeepSeek-Chat agree on 92–94% of fill-versus-abstain decisions and produce highly similar semantic content when both populate a field (SBERT ≈ 0.80). DeepSeek-Chat’s 4–5 pp completeness advantage reflects denser schema population rather than semantic divergence. Consequently, alignment signals derived from structured reports are robust to extractor choice, and tuning coverage policies yields larger gains than switching models. Breaking this down at the field level (Figure 2) reveals that consistency is not uniform. Entityoriented fields (API, Exception) exhibit very high lexical and embedding similarity (ROUGE-L ≥ 0.83; cosine ≥ 0.90), suggesting near-canonical representations across models. Scenario fields (Input, Repro) show moderate lexical overlap but strong embedding similarity (cosine ≈ 0.81– 0.83), consistent with paraphrased yet semantically equivalent procedural descriptions. Behavioral , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

13

Table 7. Cross-model consistency overview on full corpora. Structural

Semantic similarity (mean)

Completeness (%)

Dataset Agreement (%) Defects4J SWT-Bench

93.61 92.16

ROUGE-L

Jaccard

SBERT cosine

GPT-4o

DeepSeek-Chat

Δ (Deep − GPT)

0.647 0.653

0.459 0.494

0.804 0.811

86.14 83.67

90.25 88.43

4.11 4.76

Structural agreement (same non-null/empty decision per field), semantic similarity (mean ROUGE-L, Jaccard, SBERT cosine across fields), and average completeness by model (percentage of non-null fields).

Mean similarity

0.33

Mean similarity

0.50

uts

Inp

0.75

I AP

0.35

ion

0.51

0

0.74

0.41

0.2

ept Exc

Cosine

0.81

0.54

0.63

0.69

0.77

0.4

Jaccard

0.82

0.93

(a) Defects4J

0.91

ps ed ual Ste xpect Act E Field

0.6

0.84 0.71 0.76

0.30

uts

0.74

Inp

0.49

I

AP

0.32

tion

0.49

0.41

0

0.8

ROUGE-L

Metric

1

0.70

0.67 0.46

0.2

ep Exc

Cosine

0.83

0.68

0.68

0.4

Jaccard

0.81

0.90

0.85

0.6

0.90 0.74 0.83

0.8

ROUGE-L

Metric

1

ps ed ual ect Act Exp Field Ste

(b) SWT-Bench

Fig. 2. Semantic similarity between GPT-4o and DeepSeek-Chat by field and dataset. summaries (Expected, Actual) display the lowest lexical overlap (ROUGE-L ≈ 0.50) while maintaining moderate embedding similarity (cosine ≈ 0.70–0.75), reflecting natural summarization variability rather than semantic divergence. This gradient, from near-canonical entities to loosely paraphrased behaviors, confirms that inter-model differences primarily reflect field population policy rather than inconsistent semantic interpretation, and that extractor choice affects coverage density more than semantic content. Finding 4: Consistency is field-dependent: entity fields (API, Exception) converge almost canonically across models, whereas scenario (Input, Repro) and behavioral (Expected, Actual) fields show progressively stronger paraphrasing, with lower ROUGE-L but moderate-to-high embedding similarity. Literal-overlap metrics therefore suffice for entity anchors, while embedding-based similarity is necessary to reliably compare scenario and behavioral signals. Engineering implications for downstream debugging pipelines. RQ1 turns free-form bug reports into programmable artifacts: dense, semantically faithful structured representations that automated tools can consume directly. Three actionable levers follow. First, because most extraction discrepancies stem from fill-versus-abstain decisions rather than semantic errors, integration policies can be tuned rather than redesigned: conservative extraction (favoring precision) suits high-risk settings such as fault localization or patch validation, while coverage-oriented extraction maximizes scaffolding for generative tasks such as test synthesis or repair prompting. Second, the complementary extraction profiles of GPT-4o and DeepSeek-Chat suggest concrete ensemble strategies: pipelines prioritizing scenario reconstruction (API/Input/Repro) should favor DeepSeekChat, whereas exception-centric workflows benefit from cross-validating with GPT-4o. Third, since consistency is field-dependent (Finding 4), alignment tooling should apply literal-overlap checks for entity anchors but switch to embedding-based similarity for scenario and behavioral fields, avoiding a one-size-fits-all similarity threshold. In practice, structured extraction can serve as a normalization front-end that reduces narrative ambiguity, flags missing anchors, and stabilizes cross-artifact reasoning throughout debugging and repair pipelines, the foundation on which RQ2 and RQ3 build.

, Vol. 1, No. 1, Article . Publication date: July 2026.

14

Ouédraogo et al.

Summary of RQ1: LLM-based extraction reliably converts bug reports into dense, machineconsumable behavioral anchors (83–90% completeness), with DeepSeek-Chat and GPT-4o differing mainly in fill policy rather than semantic accuracy (F1 ≥ 0.93 on populated fields; 92–94% cross-model agreement). Structured anchors thus form a stable, model-robust input contract for RQ2–RQ3.

4.2

RQ2: Alignment Between Bug Descriptions and Triggering Tests

[Experimental design]: RQ2 studies report–test alignment as an actionable signal for debugging pipelines: can we quantify whether a triggering test truly encodes the failure described in the report, and which report properties make that translation easier? We use Defects4J as the main corpus and SWT-Bench for cross-validation, and we compare two ways of presenting the same report: its original narrative (RAW ) versus a structured view (STRUCTURED) obtained by extracting behavioral anchors (e.g., exception/API cues, input conditions, reproduction steps, and expected vs. actual behavior). We evaluate alignment with two complementary lenses. On one side, GPT-4o and DeepSeek-Chat act as LLM judges and score each report–test pair on a five-point scale along four axes (coverage, correctness, specificity, and overall alignment), producing short justifications. On the other side, we compute deterministic proxies that can scale to ranking and filtering: lexical overlap, embedding-based semantic similarity, and lightweight syntactic correspondence based on verb/noun overlap. Since alignment judgments may shift with the judge and with representation, we quantify inter-human agreement as a reliability ceiling and measure human–LLM agreement (Spearman’s 𝜌, Kendall’s 𝜏, and absolute error) to expose optimism bias and ranking inconsistency. We then connect alignment to report testability by correlating anchor-derived report attributes (e.g., presence of explicit expected/actual contrast, actionable steps, API/exception evidence, and concise content) with both judge scores and metric proxies. Together, these analyses characterize how natural-language bug descriptions translate into executable tests, how robust alignment is across representations and judges, and which behavioral anchors improve test adequacy beyond verbosity alone. [Results]: RQ2.1 – Bug Report–Test Semantic Alignment. We assess whether triggering tests reflect the semantics of their associated bug reports using two LLM judges (GPT-4o, DeepSeek-Chat) under two report views (RAW vs. STRUCTURED). Table 8 shows that alignment is non-trivial on both datasets (means typically > 3/5), but its magnitude is highly judge- and representation-dependent. Under RAW, DeepSeek-Chat is consistently more optimistic than GPT-4o, with the largest gaps on alignment and correctness (e.g., Defects4J alignment 3.81 vs. 3.25; correctness 3.98 vs. 3.26; SWT-Bench alignment 3.56 vs. 3.16; correctness 3.96 vs. 3.24), confirmed by Wilcoxon tests with medium-to-large paired effects (Defects4J: 𝑟 ≈ +0.46–+0.48; SWT-Bench: 𝑟 ≈ +0.43–+0.58; Table 23). STRUCTURED inputs reduce DeepSeek-Chat scores and narrow or reverse this judge effect: DeepSeek-Chat drops markedly and becomes more variable (e.g., Defects4J alignment 3.03 ± 1.62 vs. 3.81 ± 1.35 in RAW), while GPT-4o remains comparatively stable and tends to yield higher specificity (Defects4J: 3.51 vs. 3.28; SWT-Bench: 3.60 vs. 3.43); accordingly, Wilcoxon results reverse on Defects4J (GPT-4o > DeepSeek-Chat on all axes, |𝑟 | ≈ 0.10–0.14) and become axis-dependent on SWT-Bench. Overall, LLM-based alignment is not an absolute quantity: RAW tends to inflate scores (especially for DeepSeek-Chat), whereas STRUCTURED yields more stable, specificity-oriented assessments better suited for ranking or filtering tests. , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

15

Table 8. LLM-based alignment between bug reports and triggering tests (mean ± std). Dataset

Scenario RAW

Defects4J STRUCTURED RAW SWT-Bench STRUCTURED

Model

Alignment

Coverage

Correctness Specificity

GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat

3.25 ± 1.04 3.81 ± 1.35 3.28 ± 1.08 3.03 ± 1.62

3.33 ± 1.13 3.64 ± 1.49 3.32 ± 1.17 3.12 ± 1.68

3.26 ± 1.25 3.98 ± 1.43 3.27 ± 1.27 2.98 ± 1.74

3.42 ± 1.11 3.92 ± 1.28 3.51 ± 1.01 3.28 ± 1.58

GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat

3.16 ± 0.96 3.56 ± 1.09 3.21 ± 1.00 3.16 ± 1.26

3.31 ± 1.07 3.49 ± 1.19 3.22 ± 1.15 3.40 ± 1.34

3.24 ± 1.15 3.96 ± 1.19 3.23 ± 1.23 3.26 ± 1.48

3.40 ± 1.00 3.77 ± 1.10 3.60 ± 0.95 3.43 ± 1.21

Finding 5: Report–test alignment is non-trivial but judge- and representation-dependent. Triggering tests generally receive alignment scores > 3/5, yet absolute levels shift with both the evaluating LLM and the report view. Under RAW, DeepSeek-Chat scores are systematically higher than GPT-4o (medium-to-large paired effects), indicating a more optimistic judging style. Under STRUCTURED, DeepSeek-Chat scores drop and disperse, and the model gap narrows or can reverse (Defects4J: GPT-4o > DeepSeek-Chat across axes; SWT-Bench: axis-dependent). Thus, alignment should be treated as a controllable signal: RAW tends to inflate scores, whereas STRUCTURED yields more stable, specificity-oriented assessments that are better suited for ranking and filtering tests. Inter-human agreement is consistently strong across datasets and representations (Table 9), establishing a reliable reference for alignment evaluation. On Defects4J, RAW correlations exceed 𝜌 = 0.87 for alignment and coverage and approach perfect agreement for correctness and specificity (𝜌 ≥ 0.99); agreement decreases somewhat under STRUCTURED inputs (e.g., alignment 𝜌 from 0.87 to 0.71) but remains robust across axes. SWT-Bench follows a similar pattern (RAW: 𝜌 ∈ [0.82, 0.98]; STRUCTURED: 𝜌 ≥ 0.76), with error magnitudes uniformly small (𝑀𝐴𝐸 ≤ 0.21), an order of magnitude below model-level deviations. Human raters thus demonstrate stable ranking and consistent scoring behavior across representations. Table 9. Inter-human agreement between bug reports and triggering tests (b) SWT-Bench

(a) Defects4J RAW Axis Alignment Coverage Correctness Specificity

STRUCTURED

𝜌

𝜏

MAE

RMSE

𝜌

𝜏

MAE

0.87 0.91 1.00 1.00

0.85 0.88 1.00 1.00

0.14 0.11 0.00 0.02

0.37 0.33 0.00 0.14

0.71 0.88 0.99 0.97

0.69 0.85 0.99 0.95

0.21 0.13 0.02 0.05

RAW

RMSE Axis

𝜌

𝜏

MAE

0.46 Alignment 0.82 0.36 Coverage 0.90 0.14 Correctness 0.98 0.22 Specificity 0.96

0.79 0.87 0.96 0.93

0.18 0.12 0.03 0.06

STRUCTURED RMSE

𝜏

MAE

RMSE

0.41 0.76 0.73 0.35 0.85 0.82 0.17 0.95 0.92 0.24 0.92 0.89

0.20 0.15 0.06 0.09

0.44 0.38 0.25 0.29

𝜌

𝜌: Spearman correlation; 𝜏: Kendall correlation.

Against this stable human baseline, Human–LLM agreement remains limited (Table 10). Rank correlations are weak across most settings: on Defects4J/RAW, 𝜌¯ is near zero for GPT-4o (0.03) and negative for DeepSeek-Chat (−0.13), and while STRUCTURED modestly improves GPT-4o on Defects4J (𝜌¯ = 0.25) and SWT-Bench shows slightly higher correlations (up to 0.23), agreement remains far below inter-human levels throughout. Bias analysis reveals systematic score inflation: both models consistently over-score alignment relative to humans, with larger positive bias for DeepSeek-Chat (e.g., ≈ 1.99 on Defects4J/RAW vs. 1.53 for GPT-4o), and model–human MAE (1.35–2.38) exceeds inter-human MAE (≤ 0.21) by an order of magnitude. Wilcoxon effect sizes are moderate to large (¯𝑟 ≈ 0.58–0.77), confirming substantial distributional shifts. In summary, while humans provide a stable ranking of report–test alignment, LLM judges exhibit optimistic bias and weak rank consistency; STRUCTURED inputs partially reduce variance for GPT-4o but do not eliminate calibration gaps.

, Vol. 1, No. 1, Article . Publication date: July 2026.

16

Ouédraogo et al.

Table 10. Human–LLM agreement summary between bug reports and triggering tests. Dataset

Scenario

GPT-4o

DeepSeek-Chat

𝜌¯

Bias

MAE

𝑟¯

𝜌¯

𝑟¯

Bias

MAE

Defects4J

RAW STRUCTURED

0.03 0.25

1.53 1.59

1.75 0.73 1.71 0.77

-0.13 0.02

1.99 1.45

2.38 0.73 1.90 0.58

SWT-Bench

RAW STRUCTURED

0.22 0.09

1.00 1.51

1.35 0.58 1.67 0.75

0.23 0.08

1.44 1.63

1.67 0.74 1.88 0.73

Weighted averages computed using 𝑛𝑝𝑎𝑖𝑟𝑠 per axis. Spearman averages exclude undefined cases.

Finding 6: Human test-adequacy ratings are stable, but LLM judges are optimistic and weakly rank-aligned. Inter-human agreement is high (𝜌 > 0.8, 𝑀𝐴𝐸 ≤ 0.21), whereas Human–LLM rank agreement is modest (𝜌¯ mostly in [−0.13, 0.25]) with large positive bias and MAE up to 2.38. DeepSeek-Chat is more optimistic (especially in RAW), while GPT-4o improves slightly under STRUCTURED. Therefore, LLM judges should not be used out-of-the-box as decision signals; their scores require bias-aware calibration or should be complemented with deterministic proxies for ranking/filtering. Deterministic metrics provide a consistent and reproducible view of report–test relatedness across datasets and representations (Tables 11 and 12). Lexical overlap is uniformly minimal (ROUGEL: 0.03–0.07; Jaccard: 0.03–0.06), confirming that triggering tests rarely reuse report phrasing verbatim and that surface-level matching is a poor indicator of alignment, whereas embeddingbased similarity occupies a stable mid-range (SBERT: 0.47–0.54; OpenAI: 0.49–0.59), suggesting that tests capture report intent via semantic paraphrases rather than lexical copying. Across both corpora, RAW inputs yield higher embedding similarities than STRUCTURED, consistent with information loss when reports are condensed into extracted anchors. CodeBERT similarities are near-ceiling (≈0.92–0.95), useful as a coarse “recall” sanity check but limited in discriminative power for ranking or filtering. Syntactic overlap further indicates that tests preferentially reuse entities over actions: POS overlaps are modest overall (verbs: 0.03–0.09; nouns: 0.06–0.22), with noun overlap consistently exceeding verb overlap and higher overall on Defects4J than SWTBench, suggesting stronger explicit entity reuse in Java-centric reports/tests. Finally, on structured dimensions (STRUCTURED-only), DeepSeek-Chat yields higher similarity than GPT-4o across entity, scenario, and behavior components, most notably on SWT-Bench scenario (0.468 vs. 0.392), indicating that structured fields can amplify semantic matching in narrative-style corpora. Table 11. Lexical and embedding similarities between bug reports and triggering tests. (a) RAW Inputs (Summary + Description) Dataset

Model

ROUGE-L

Jaccard

Defects4J

GPT-4o DeepSeek-Chat

0.070 ± 0.087 0.069 ± 0.085

0.062 ± 0.071 0.062 ± 0.070

0.538 ± 0.130 0.948 ± 0.031 0.537 ± 0.130 0.949 ± 0.031

SBERT

CodeBERT

0.591 ± 0.096 0.589 ± 0.096

OpenAI

SWT-Bench

GPT-4o DeepSeek-Chat

0.037 ± 0.037 0.037 ± 0.039

0.033 ± 0.032 0.033 ± 0.032

0.523 ± 0.120 0.925 ± 0.042 0.524 ± 0.119 0.925 ± 0.041

0.524 ± 0.105 0.525 ± 0.105

(b) STRUCTURED Inputs (LLM-extracted signals) Dataset

Model

ROUGE-L

Jaccard

Defects4J

GPT-4o DeepSeek-Chat

0.032 ± 0.033 0.033 ± 0.031

0.032 ± 0.032 0.035 ± 0.029

0.473 ± 0.120 0.934 ± 0.025 0.503 ± 0.114 0.944 ± 0.021

SBERT

CodeBERT

0.555 ± 0.099 0.577 ± 0.092

OpenAI

SWT-Bench

GPT-4o DeepSeek-Chat

0.036 ± 0.038 0.038 ± 0.037

0.032 ± 0.032 0.034 ± 0.032

0.469 ± 0.127 0.919 ± 0.040 0.493 ± 0.126 0.926 ± 0.045

0.495 ± 0.112 0.518 ± 0.112

Split by input scenario (RAW vs. STRUCTURED). All values are mean ± std.

, Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

17

Table 12. POS overlaps and structured-dimension similarities between bug reports and triggering tests. POS (RAW) Dataset

POS (STRUCTURED)

Structured Similarities

Model Verb ov.

Noun ov.

Verb ov.

Noun ov.

Entity

Scenario

Behavior

Defects4J

GPT-4o DeepSeek-Chat

0.055 ± 0.112 0.124 ± 0.164 0.092 ± 0.127 0.217 ± 0.172

0.045 ± 0.144 0.077 ± 0.159

0.127 ± 0.194 0.213 ± 0.157

0.386 ± 0.146 0.370 ± 0.146 0.334 ± 0.114 0.418 ± 0.123 0.432 ± 0.116 0.361 ± 0.118

SWT-Bench

GPT-4o DeepSeek-Chat

0.026 ± 0.078 0.057 ± 0.100 0.027 ± 0.080 0.057 ± 0.102

0.035 ± 0.110 0.030 ± 0.104

0.084 ± 0.143 0.076 ± 0.130

0.383 ± 0.150 0.392 ± 0.163 0.394 ± 0.126 0.425 ± 0.129 0.468 ± 0.127 0.424 ± 0.125

POS overlaps are computed on both RAW and STRUCTURED; structured similarities are only available for STRUCTURED inputs.

Finding 7: Triggering tests align mainly semantically, not lexically. Report–test lexical overlap is near zero (ROUGE-L/Jaccard ≤ 0.07), while embedding similarity is moderate and consistent (SBERT/OpenAI ≈ 0.47–0.59), indicating paraphrased intent rather than textual copying (Table 11). POS overlap is low and noun-dominated (nouns > verbs), suggesting tests mirror entities (APIs/identifiers) more than actions (Table 12). Consequently, ranking/filtering should favor semantic embeddings and entity anchors over lexical matching; improving testability requires making steps and expected/actual behavior explicit, not increasing verbosity. OpenAI embedding cosine emerges as the most consistent deterministic predictor of LLM-based alignment judgments (Table 13). Across 15 out of 16 dataset–scenario–model combinations, OpenAI cosine achieves the highest absolute Spearman correlation with LLM scores, with moderate but stable and statistically significant correlations (GPT-4o: 𝜌 ≈ 0.32–0.45; DeepSeek-Chat: 0.25–0.42). Associations are generally stronger under STRUCTURED inputs, particularly for alignment and coverage, indicating that embedding similarity over distilled anchors captures the dimensions most emphasized by LLM judges. The only exception occurs for DeepSeek-Chat on Defects4J– STRUCTURED correctness, where jaccard_struct slightly exceeds cosine, suggesting that token-level overlap may retain marginal utility when reasoning over condensed summaries. Overall, OpenAI embedding cosine provides a stable and representation-robust approximation of LLM alignment scores across datasets: unlike LLM judges, it is deterministic, inexpensive to compute, and free of optimistic bias drift. Table 13. Best deterministic predictor per LLM axis (maximum |𝜌 |) by dataset, scenario, and model. Dataset

Scenario RAW

Defects4J STRUCTURED RAW SWT-Bench STRUCTURED

Model

Alignment (best)

Coverage (best)

Correctness (best)

Specificity (best)

GPT-4o openai_raw (0.382) openai_raw (0.387) openai_raw (0.375) DeepSeek-Chat openai_raw (0.305) openai_raw (0.329) openai_raw (0.264) GPT-4o openai_struct (0.452) openai_struct (0.425) openai_struct (0.353) DeepSeek-Chat openai_struct (0.249) openai_struct (0.249) jaccard_struct (0.245)

openai_raw (0.394) openai_raw (0.291) openai_struct (0.421) openai_struct (0.269)

GPT-4o openai_raw (0.388) openai_raw (0.394) DeepSeek-Chat openai_raw (0.418) openai_raw (0.410) GPT-4o openai_struct (0.393) openai_struct (0.408) DeepSeek-Chat openai_struct (0.421) openai_struct (0.419)

openai_raw (0.348) openai_raw (0.350) openai_struct (0.338) openai_struct (0.385)

openai_raw (0.326) openai_raw (0.375) openai_struct (0.318) openai_struct (0.356)

Finding 8: OpenAI embedding cosine is a stable and practical proxy for LLM alignment judgments. In 15/16 settings, OpenAI cosine is the strongest deterministic predictor of LLM scores (Table 13), with moderate but consistent correlations (up to 𝜌 ≈ 0.45). Associations are stronger under STRUCTURED inputs and for alignment/coverage axes. Embedding cosine can therefore replace LLM judges for ranking or filtering triggering tests, providing a reproducible, low-cost alignment signal without the optimistic bias observed in LLM evaluations. RQ2.2 – Bug Report Attributes and Test Quality. We next examine which report-level attributes make failures easier to translate into executable triggering tests. Although correlations are generally small in magnitude (Tables 14 and 21), consistent patterns emerge across datasets. Verbosity does not improve judged alignment. On Defects4J, report length (len_report_words) exhibits systematic negative correlations with all LLM judgment axes in both RAW and STRUCTURED views (𝜌 ≈ −0.12 to −0.21): longer descriptions increase narrative context but reduce judged , Vol. 1, No. 1, Article . Publication date: July 2026.

18

Ouédraogo et al.

alignment, suggesting a dilution effect where additional text does not translate into clearer, more testable behavioral cues. Notably, length-based features and avg_field_length correlate positively with deterministic similarity metrics (e.g., ROUGE-L, SBERT, OpenAI cosine), indicating that verbosity increases textual or embedding overlap without improving perceived behavioral faithfulness. Explicit behavioral anchors improve test alignment. On SWT-Bench, LLM judgments are more strongly associated with the presence of explicit anchors than with length: the presence of APIs (has_api) consistently improves alignment and coverage, and expected-outcome cues (has_expected) increase specificity, particularly for GPT-4o. In contrast, deterministic similarity metrics remain largely driven by structural richness (e.g., number of entities, length) rather than by behavioral precision. Taken together, LLM evaluators reward attributes that make report–test links behaviorally checkable (explicit APIs, steps, expected outcomes), whereas lexical or embedding similarity can increase mechanically with added text: tests align better with reports that expose operational anchors rather than narrative detail. Table 14. Top report attribute per LLM judgement axis (Spearman 𝜌), grouped by dataset and model. Dataset

Model

LLM axis

RAW Best attr.

STRUCTURED 𝝆

Best attr.

𝝆

DeepSeek-Chat

Alignment −0.197 Correctness −0.147 len_report_words Coverage −0.178 Specificity −0.200

−0.162 −0.151 len_report_words −0.145 −0.160

GPT-4o

Alignment −0.208 Correctness −0.160 len_report_words Coverage −0.160 Specificity −0.191

len_report_words

−0.166 −0.168 −0.123 −0.160

DeepSeek-Chat

Alignment Correctness avg_field_length Coverage Specificity

0.101 0.076 0.102 0.074

has_api avg_field_length has_api avg_field_length

0.080 0.079 0.093 0.041

GPT-4o

Alignment has_exception Correctness num_entities Coverage has_exception Specificity has_api

−0.051 −0.084 −0.043 0.041

has_api num_entities has_api has_expected

0.082 −0.081 0.125 0.074

Defects4J

SWT-Bench

len_report_words: word count of RAW report (summary+description). avg_field_length: mean word count over non-empty structured fields. has_exception: indicator that exception is present. num_entities: count of distinct entities (exception/API names). has_api: indicator that api_involved is present. has_expected: indicator that expected is present.

Finding 9: Testability depends more on explicit behavioral anchors than on verbosity. On Defects4J, longer reports increase lexical and embedding similarity but consistently reduce LLM alignment scores, indicating narrative dilution. Across datasets, the presence of explicit APIs, reproduction steps, and expected outcomes more reliably predicts well-aligned triggering tests than increasing descriptive length. To improve testability, bug reports should therefore prioritize concrete operational anchors (APIs, steps, expected behavior) over extended narrative context; tooling and templates should encourage structured behavioral cues rather than verbosity. Engineering implications for automated test generation. Findings 5–9 do more than characterize alignment descriptively: they translate directly into design decisions for test-from-report pipelines, which we operationalize as a concrete algorithm below. Three levers emerge. First, RQ1 shows that LLM-based extraction yields stable, semantically faithful structured representations that can serve as standardized input contracts (exception, APIs, inputs, steps, expected/actual), reducing generation variance compared to raw narratives. Second, RQ2 demonstrates that alignment is primarily semantic rather than lexical (Finding 7); selection objectives should therefore prioritize embedding-based similarity and explicit behavioral anchors over surface overlap. Third, because out-of-the-box LLM judges exhibit optimism and weak human rank alignment (Finding 6), deterministic proxies are preferable for automation: OpenAI embedding cosine offers the most consistent , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

19

approximation across datasets and representations (Finding 8). Together, these results enable an alignment-guided workflow that (i) normalizes reports into anchor-rich schemas, (ii) generates multiple candidate tests conditioned on these anchors, and (iii) ranks candidates using embeddingbased adequacy scores augmented with anchor coverage checks. Algorithm 1 operationalizes this workflow end-to-end. Given a bug report 𝑅, we first extract structured anchors 𝑆 (line 1, RQ1). If anchors are sparse, we optionally reinforce missing behavioral cues (e.g., explicit APIs, reproduction steps, expected/actual contrast), since anchor presence predicts stronger alignment than verbosity alone (line 3, Finding 9). Candidate tests are then generated explicitly conditioned on 𝑆: API and exception fields guide setup, input and step fields guide execution flow, and expected/actual fields guide oracle construction; this anchor-driven conditioning directly reflects our empirical finding that behavioral cues, not lexical overlap, determine alignment quality (Finding 7). Finally, candidates are ranked using a deterministic adequacy score combining embedding cosine between structured report and test representations with anchor-coverage checks (line 12, Finding 8), normalizing all components to [0, 1]. This blueprint requires no alignment-specific model training: it can be implemented with off-the-shelf LLM generators and embedding models, making it a readily reusable reference design for semantics-aware debugging pipelines. Algorithm 1 Alignment-guided anchor-conditioned test generation Require: Bug report 𝑅 (summary+description), generator 𝐺, embedding model 𝐸, 𝑁 candidates, top-𝑘 Ensure: Ranked tests T with adequacy scores 1: 𝑆 ← ExtractStructured(𝑅) ⊲ exception, API, inputs, steps, expected, actual — RQ1 2: 𝑎 ← AnchorScore(𝑆 ) 3: if 𝑎 < 𝜏𝑎 then 4: 𝑆 ← CompleteMissingAnchors(𝑅, 𝑆 ) ⊲ reinforce missing APIs/steps/oracles — Finding 9 5: end if 6: C ← ∅ 7: for 𝑖 ← 1 to 𝑁 do 8: 𝑐𝑖 ← BuildConstraints(𝑆 ) ⊲ API setup, reproduction steps, explicit oracle 9: 𝑡𝑖 ← 𝐺 (𝑆, 𝑐𝑖 ) ⊲ anchor-conditioned generation 10: 𝑟 𝑆 ← ConcatFields(𝑆 ) 11: 𝑟𝑡𝑖 ← ReprTest(𝑡𝑖 )
12: 𝑠𝑖embed ← cos 𝐸 (𝑟 𝑆 ), 𝐸 (𝑟𝑡𝑖 ) ⊲ deterministic proxy — Finding 8 anch 13: 𝑠𝑖 ← AnchorMatch(𝑆, 𝑡𝑖 ) 14: 𝑠𝑖oracle ← OracleCheck(𝑡𝑖 ) 15: 𝑠ˆ𝑖embed , 𝑠ˆ𝑖anch ← Normalize(·) 16: 𝑠𝑖 ← 𝛼 𝑠ˆ𝑖embed + 𝛽 𝑠ˆ𝑖anch − 𝛾 (1 − 𝑠𝑖oracle ) 17: C ← C ∪ { (𝑡𝑖 , 𝑠𝑖 ) } 18: end for 19: C ← Filter( C, 𝜏𝑠 ) 20: T ← TopK( C, 𝑘 ) 21: return T

, Vol. 1, No. 1, Article . Publication date: July 2026.

20

Ouédraogo et al.

Summary of RQ2: Report–test alignment is non-trivial but strongly judge- and representationdependent: DeepSeek-Chat is more optimistic under RAW, while GPT-4o is more stable and specificity-oriented under STRUCTURED. Human ratings form a reliable ceiling (𝜌 > 0.8), but both LLM judges show optimistic bias and weak rank agreement with humans, so their scores should not be used out-of-the-box. Alignment is predominantly semantic rather than lexical, with OpenAI embedding cosine the most consistent deterministic proxy for LLM judgments. Finally, explicit behavioral anchors, not verbosity, predict well-aligned tests, motivating the anchor-conditioned generation workflow of Algorithm 1.

4.3

RQ3: Alignment Between Bug Descriptions and Corrective Patches

[Experimental design]: RQ3 examines how faithfully human-written patches translate the entities, scenarios, and behavioral contrasts described in bug reports into source-level repair actions. We analyze report–patch pairs under a two-factor framing: report representation (RAW narratives vs. STRUCTURED anchors) and patch view (full diff, additions only, deletions only), yielding six conditions per dataset. Defects4J is our primary corpus and SWT-Bench provides cross-corpus validation. Alignment is measured from two complementary perspectives. First, GPT-4o and DeepSeek-Chat act as judges and assign four 1–5 scores (overall alignment, coverage, correctness, specificity) with short justifications. Second, we compute deterministic proxies—lexical overlap (ROUGE-L, Jaccard), embedding similarity (SBERT, CodeBERT, OpenAI cosine), and POS-based verb/noun overlap—to separate reproducible similarity signals from perception-based judgments. A human-rated subset provides a reliability ceiling and supports Human–LLM agreement analyses (rank agreement, bias, MAE, and Wilcoxon effect sizes) across framing conditions. Finally, we relate structured report attributes (e.g., length, reproduction steps, exception cues, expected/actual contrast) to patch alignment via Spearman’s 𝜌 and Kendall’s 𝜏, identifying which report semantics are associated with more behaviorally precise fixes. Together, this design characterizes patch directness as framing-dependent and quantifies how report semantics shape repair quality. [Results]: RQ3.1 – Directness of Patches to Behaviors. LLM judges generally rate human-written patches as well aligned with their bug reports, with the highest scores under RAW–full settings where mean alignment often exceeds 4/5 (Table 15). Yet this perceived directness is not an intrinsic property of a patch–report pair: it varies systematically with both the patch view and the report representation. Across datasets and models, full diffs provide the strongest and most stable signal: add-only views preserve part of the implementation intent but drop corrective context, while remove-only views are the most fragile and representation-sensitive. Wilcoxon tests confirm that full typically outperforms add and remove, often with large effects (|𝑟 | ≥ 0.5; Table 22), and the penalty is strongest under STRUCTURED inputs, where deletions lose semantic anchoring (e.g., SWT-Bench full–remove: GPT-4o 𝑟 = −0.81∗∗∗ ; DeepSeek-Chat 𝑟 = −0.70∗∗∗ ). Report framing amplifies these differences. RAW inputs generally yield higher and less dispersed scores than STRUCTURED summaries, with the largest drops for DeepSeek-Chat: its Defects4J remove alignment falls from 4.66 (RAW) to 3.05 (STRUCTURED), consistent with higher sensitivity to abstraction and cue loss, whereas GPT-4o remains more robust (full specificity: 4.58 RAW vs. 4.52 STRUCTURED). Overall, patch– report directness is a framing-dependent judgment: full diffs stabilize evaluation by preserving context, while isolated add/remove views and structured abstraction reduce anchoring and expose model-specific sensitivities. , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

21

Table 15. LLM-based patch–report alignment scores (mean ± sd, 1–5). Dataset

Model

Repr. RAW

GPT-4o STRUCTURED Defects4J RAW DeepSeek-Chat STRUCTURED

RAW GPT-4o STRUCTURED SWT-Bench RAW DeepSeek-Chat STRUCTURED

View

Alignment

Coverage

Correctness

Specificity

add full remove add full remove add full remove add full remove

3.16 ± 1.37 3.97 ± 1.24 3.82 ± 1.22 2.58 ± 1.05 3.69 ± 1.05 2.95 ± 1.08 4.09 ± 1.47 4.25 ± 1.37 4.66 ± 0.94 2.26 ± 1.24 3.35 ± 1.48 3.05 ± 1.65

3.38 ± 1.26 4.46 ± 0.91 4.10 ± 1.14 2.46 ± 1.16 3.67 ± 1.15 2.81 ± 1.18 4.11 ± 1.47 4.35 ± 1.34 4.68 ± 0.94 2.27 ± 1.30 3.41 ± 1.61 3.13 ± 1.73

3.07 ± 1.39 4.05 ± 1.21 3.91 ± 1.30 2.59 ± 1.27 3.71 ± 1.29 2.99 ± 1.30 4.04 ± 1.52 4.19 ± 1.45 4.64 ± 0.98 2.60 ± 1.51 3.47 ± 1.63 3.27 ± 1.77

3.50 ± 1.38 4.58 ± 0.79 4.24 ± 1.12 3.65 ± 1.13 4.52 ± 0.75 3.87 ± 1.10 4.27 ± 1.36 4.52 ± 1.13 4.71 ± 0.91 2.19 ± 1.39 3.57 ± 1.48 2.89 ± 1.69

add full remove add full remove add full remove add full remove

4.00 ± 1.13 4.17 ± 1.02 2.89 ± 1.33 3.73 ± 1.08 4.10 ± 0.99 2.45 ± 1.04 4.42 ± 1.01 4.63 ± 0.81 4.10 ± 1.47 3.50 ± 1.37 3.97 ± 1.21 2.40 ± 1.64

4.33 ± 0.90 4.54 ± 0.79 3.17 ± 1.34 3.71 ± 1.18 4.08 ± 1.08 2.25 ± 1.14 4.48 ± 0.99 4.68 ± 0.78 4.12 ± 1.48 3.59 ± 1.42 4.04 ± 1.25 2.44 ± 1.68

4.21 ± 1.02 4.46 ± 0.88 2.79 ± 1.42 3.88 ± 1.17 4.23 ± 1.03 2.39 ± 1.25 4.39 ± 1.08 4.61 ± 0.87 4.05 ± 1.51 3.81 ± 1.45 4.27 ± 1.20 2.56 ± 1.78

4.43 ± 0.91 4.64 ± 0.74 3.49 ± 1.37 4.29 ± 0.98 4.52 ± 0.82 3.35 ± 1.20 4.46 ± 1.06 4.66 ± 0.81 4.20 ± 1.46 3.31 ± 1.50 3.82 ± 1.37 2.28 ± 1.64

Finding 10: Behavioral directness is strongly framing-dependent. Full diffs yield the highest and most stable LLM alignment scores, whereas deletion-only views are systematically penalized, particularly under STRUCTURED representations. RAW inputs enhance robustness, especially for DeepSeek-Chat, while GPT-4o remains comparatively stable across formats. Patch view and representational abstraction therefore materially shape LLM-based assessments of patch–report alignment. Table 16 confirms that the human benchmark is highly reliable across patch views and representations. Inter-human agreement is near perfect for correctness and specificity (often 𝜌 ≥ 0.99 in RAW), consistently strong for coverage (𝜌 > 0.80), and moderate but stable for alignment, with MAE remaining small across settings (typically ≤ 0.44), indicating that representation and patch view introduce difficulty but not incoherence among human raters, and establishing a robust upper bound for model comparison. Against this stable baseline, Human–LLM rank agreement is generally modest (Table 17). Rank agreement. On Defects4J/RAW, average 𝜌¯ remains near zero for most views (e.g., GPT-4o: 0.03; DeepSeek-Chat: −0.08 on full), with similar patterns on SWT-Bench/RAW. The clearest improvement appears under STRUCTURED+add on SWT-Bench (𝜌¯ = 0.33 for GPT-4o; 0.26 for DeepSeek-Chat), suggesting that structured behavioral cues combined with addition-focused diffs partially recover human-consistent ranking, though even in this best case, agreement remains far below inter-human levels. Optimism bias. LLMs consistently assign higher scores than humans: mean bias is positive across all datasets, representations, and views, frequently exceeding +2 points and peaking at +3.03 (DeepSeek-Chat, Defects4J/RAW-remove), an order of magnitude larger than inter-human MAE, indicating systematic inflation rather than random disagreement. Improvements in 𝜌¯ do not imply closeness in absolute terms: MAE often remains above 2 in RAW settings and decreases only partially under STRUCTURED inputs. Wilcoxon effect sizes are consistently moderate to large (¯𝑟 typically 0.6–0.9 in RAW), confirming stable distributional divergence between human and LLM scores. Representation and patch view modulate agreement, but neither eliminates persistent

, Vol. 1, No. 1, Article . Publication date: July 2026.

22

Ouédraogo et al.

optimism bias; LLM judgments should therefore be interpreted as condition-sensitive alignment signals rather than human-equivalent evaluations. Table 16. Inter-human agreement between patches and bug reports (a) Defects4J RAW

(b) SWT-Bench STRUCTURED

RAW

View

Axis

𝜌

𝜏

MAE

𝜌

𝜏

MAE

Full

Align. Cov. Corr. Spec.

0.75 0.92 0.99 1.00

0.71 0.88 0.97 0.99

0.34 0.17 0.09 0.01

0.39 0.83 1.00 0.65

0.37 0.79 1.00 0.65

Add

Align. Cov. Corr. Spec.

0.58 0.87 1.00 1.00

0.56 0.86 1.00 0.99

0.41 0.10 0.01 0.01

0.48 0.78 1.00 0.64

Remove

Align. Cov. Corr. Spec.

0.53 0.84 1.00 0.99

0.52 0.82 1.00 0.98

0.37 0.14 0.01 0.01

0.48 0.78 1.00 0.51

STRUCTURED

View

Axis

𝜌

𝜏

MAE

𝜏

MAE

0.44 0.22 0.00 0.14

Full

Align. Cov. Corr. Spec.

0.92 – 1.00 –

0.90 – 0.99 –

0.05 0.55 0.52 0.00 0.89 0.83 0.05 1.00 1.00 0.00 0.85 0.84

0.40 0.26 0.00 0.10

0.47 0.75 1.00 0.64

0.41 0.19 0.00 0.18

Add

Align. Cov. Corr. Spec.

0.94 – 1.00 –

0.92 – 0.99 –

0.04 0.55 0.52 0.00 0.89 0.84 0.04 1.00 1.00 0.00 0.85 0.84

0.41 0.25 0.00 0.10

0.46 0.73 1.00 0.51

0.40 0.28 0.00 0.14

Remove

Align. Cov. Corr. Spec.

0.98 – 1.00 –

0.98 – 1.00 –

0.01 0.55 0.53 0.00 0.87 0.82 0.01 1.00 1.00 0.00 0.82 0.81

0.43 0.20 0.00 0.07

𝜌

𝜌: Spearman; 𝜏: Kendall. Dashes indicate undefined correlations due to constant inputs. Sample sizes vary by view (full: 200; add: 156/198; remove: 168/196)

Table 17. Human–LLM patch–report agreement summary. Dataset

Repr.

View

GPT-4o

DeepSeek-Chat

𝜌¯

Bias

MAE

𝑟¯

𝜌¯

Bias

MAE

𝑟¯

RAW

full add remove

0.03 0.16 0.04

2.19 1.36 2.24

2.30 1.64 2.40

0.83 0.67 0.81

-0.08 -0.07 0.05

2.22 2.28 3.03

2.48 2.48 3.05

0.80 0.80 0.88

STRUCT.

full add remove

0.15 0.11 0.13

1.87 1.09 1.43

2.06 1.34 1.61

0.77 0.62 0.71

0.10 0.12 0.13

1.35 0.66 1.35

1.80 1.06 1.73

0.62 0.38 0.58

RAW

full add remove

-0.00 -0.04 -0.06

2.76 2.49 1.57

2.78 2.56 1.80

0.89 0.86 0.71

0.01 -0.01 -0.10

2.88 2.70 2.30

2.93 2.78 2.58

0.90 0.88 0.80

STRUCT.

full add remove

0.19 0.33 0.09

2.32 2.04 1.09

2.36 2.12 1.43

0.86 0.84 0.58

0.21 0.26 0.12

2.16 1.67 0.88

2.26 1.86 1.47

0.83 0.76 0.38

Defects4J

SWT-Bench

Weighted averages computed using 𝑛𝑝𝑎𝑖𝑟𝑠 per axis. Spearman averages exclude undefined cases.

Finding 11. Human–LLM agreement is framing-dependent and systematically optimistic. Rank correlations are modest and improve mainly under STRUCTURED+add on SWT-Bench, yet remain far below inter-human levels. LLMs consistently over-score alignment by 1–3 points, with large and stable distributional shifts. Agreement and bias must therefore be reported jointly, and LLM scores interpreted as condition-sensitive signals rather than human substitutes. Across datasets and representations, full diffs consistently yield the strongest metric-based alignment, with add generally exceeding or matching remove (Table 18). This pattern holds across lexical, embedding, and POS-based measures, confirming that contextual completeness provides the richest semantic signal. Transitioning from RAW to STRUCTURED introduces a systematic surface–semantics trade-off: embedding similarity declines (SBERT drops by 11–14% on Defects4J full; OpenAI by 5–7%), while predicate-oriented POS overlap increases substantially (verbs +28– 54%, nouns +16–21%), and lexical dispersion contracts while embedding variance remains stable or slightly higher, indicating reduced surface redundancy but preserved semantic diversity. This RAW→STRUCTURED drop is more pronounced on Defects4J than SWT-Bench, suggesting stronger , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

23

reliance on lexical/contextual richness in Defects4J reports, whereas SWT-Bench maintains more stable embedding alignment under abstraction. CodeBERT, by contrast, remains consistently high (∼0.92–0.96) and largely insensitive to representation or view, evidencing stable code-level correspondence independent of surface framing. Overall, metric-based evidence corroborates the framing effects observed with LLM judges: full diffs maximize contextual signal, whereas structured abstraction shifts alignment from surface similarity toward predicate–argument salience. Table 18. Metric-based patch–report alignment scores (mean ± sd, 0–1). Dataset

Repr. RAW

Defects4J STRUCTURED

RAW SWT-Bench STRUCTURED

View

ROUGE-L

Jaccard

SBERT

CodeBERT

OpenAI

Verbs

Nouns

add full remove add full remove

0.02 ± 0.03 0.06 ± 0.05 0.03 ± 0.04 0.01 ± 0.03 0.04 ± 0.03 0.02 ± 0.03

0.02 ± 0.02 0.05 ± 0.04 0.02 ± 0.03 0.01 ± 0.02 0.04 ± 0.03 0.01 ± 0.02

0.37 ± 0.13 0.58 ± 0.11 0.41 ± 0.13 0.33 ± 0.14 0.50 ± 0.11 0.37 ± 0.14

0.91 ± 0.04 0.96 ± 0.02 0.92 ± 0.03 0.92 ± 0.05 0.95 ± 0.04 0.93 ± 0.04

0.39 ± 0.12 0.58 ± 0.11 0.42 ± 0.12 0.38 ± 0.14 0.55 ± 0.13 0.41 ± 0.13

0.02 ± 0.05 0.06 ± 0.08 0.03 ± 0.06 0.04 ± 0.13 0.10 ± 0.16 0.05 ± 0.13

0.03 ± 0.04 0.14 ± 0.11 0.03 ± 0.05 0.03 ± 0.06 0.17 ± 0.14 0.04 ± 0.07

add full remove add full remove

0.04 ± 0.04 0.04 ± 0.04 0.03 ± 0.04 0.04 ± 0.04 0.04 ± 0.04 0.03 ± 0.04

0.04 ± 0.04 0.04 ± 0.04 0.03 ± 0.03 0.04 ± 0.04 0.04 ± 0.03 0.02 ± 0.03

0.45 ± 0.14 0.47 ± 0.13 0.40 ± 0.14 0.42 ± 0.15 0.42 ± 0.13 0.37 ± 0.15

0.92 ± 0.04 0.93 ± 0.04 0.92 ± 0.04 0.92 ± 0.05 0.93 ± 0.05 0.93 ± 0.05

0.43 ± 0.13 0.44 ± 0.12 0.39 ± 0.12 0.42 ± 0.13 0.42 ± 0.12 0.37 ± 0.13

0.06 ± 0.11 0.06 ± 0.11 0.03 ± 0.08 0.07 ± 0.15 0.08 ± 0.15 0.03 ± 0.10

0.10 ± 0.12 0.13 ± 0.13 0.06 ± 0.09 0.13 ± 0.16 0.15 ± 0.16 0.07 ± 0.11

Finding 12: Metric-based evidence confirms strong framing effects. Full diffs yield the highest alignment across lexical and embedding measures. STRUCTURED abstraction reduces surface and embedding similarity but increases predicate-oriented POS overlap, indicating a shift from surface fidelity to argument-level salience. This trade-off is stronger on Defects4J than SWTBench. CodeBERT remains near-ceiling and representation-invariant, providing a stable code-level anchor. RQ3.2 – Report Semantics and Patch Precision. Associations between report attributes and LLM-judged patch quality display corpus-dependent patterns (Table 19). On Defects4J, GPT-4o shows virtually no stable associations, whereas DeepSeek-Chat exhibits coherent but small negative correlations across all axes (median 𝜌 in the −0.05 to −0.09 range): increased procedural or lexical density (more steps, entities, longer fields) slightly reduces perceived alignment, coverage, and specificity. Effect sizes remain modest (|𝜌 | ≤ 0.08) but directionally consistent. On SWT-Bench, both judges predominantly reward behavioral contrast instead: explicit expected/actual statements and concise, content-bearing fields positively associate with alignment, coverage, and specificity (median 𝜌 ≈ 0.03–0.05), and DeepSeek-Chat mirrors GPT-4o’s positive trends while additionally penalizing excessive input enumeration (negative correlations for num_inputs). Overall, clarity and contrast outweigh sheer descriptive volume: verbosity is weakly detrimental on Defects4J (DeepSeek-Chat), whereas explicit behavioral contrast improves precision on SWT-Bench, small but internally consistent, corpus-specific sensitivities. Finding 13: Report–patch precision is corpus- and judge-dependent. On SWT-Bench, explicit behavioral contrast (expected/actual cues, concise salient steps) consistently improves LLMjudged alignment. On Defects4J, increased procedural or lexical density weakly reduces DeepSeekChat scores, while GPT-4o remains largely insensitive. Clarity and contrast matter more than volume, but their impact depends on corpus characteristics. Correlations between report attributes and metric-based report–patch similarity also exhibit corpus-dependent structure (Table 20). On Defects4J, associations are modest and mixed: codeaware embeddings (SBERT, CodeBERT) account for most positive effects, whereas surface metrics , Vol. 1, No. 1, Article . Publication date: July 2026.

24

Ouédraogo et al.

Table 19. Direction and magnitude of LLM-based correlations between report attributes and patch-quality scores.

(a) Defects4J Score

Model

Sig +

Sig –

Median 𝜌 (sig)

0 0 0 0 0 0 0 0

4 0 4 0 4 1 3 0

–0.072 – –0.085 – –0.079 –0.044 –0.054 –

DeepSeek-Chat Alignment GPT-4o DeepSeek-Chat Correctness GPT-4o DeepSeek-Chat Coverage GPT-4o DeepSeek-Chat Specificity GPT-4o

Median 𝜏 (sig)

Max |𝜌 | (sig)

–0.056 – –0.068 – –0.063 –0.033 –0.047 –

0.044 – 0.052 – 0.042 0.033 0.031 –

Median 𝜏 (sig)

Max |𝜌 | (sig)

0.027 0.036 0.030 0.039 0.022 0.041 0.026 0.039

0.079 0.074 0.055 0.052 0.082 0.068 0.073 0.055

(b) SWT-Bench Score

Model

Sig +

Sig –

Median 𝜌 (sig)

5 6 6 3 5 7 3 7

1 0 1 0 1 0 1 0

0.031 0.042 0.032 0.048 0.026 0.045 0.032 0.046

DeepSeek-Chat GPT-4o DeepSeek-Chat Correctness GPT-4o DeepSeek-Chat Coverage GPT-4o DeepSeek-Chat Specificity GPT-4o Alignment

Sig + / Sig –: count of report attributes with significant positive / negative correlation (significance if Spearman 𝑝 < .05 or Kendall 𝑝 < .05). Median 𝜌 (sig) / Median 𝜏 (sig): median Spearman / Kendall among significant effects within the block. Max |𝜌 | : largest absolute Spearman coefficient among significant effects in the block.

(ROUGE-L, Jaccard) show sparse or unstable associations, and POS-based overlaps frequently tilt negative, indicating that simple lexical or predicate cues can invert under this corpus’ reporting style. On SWT-Bench, the pattern is substantially more uniform: lexical, embedding, and POS metrics predominantly show positive correlations, especially for verbs and code-aware embeddings, and although effect sizes remain small (median |𝜌 | ≈ 0.03–0.08), their directional consistency suggests stable semantic reinforcement when reports are clearer and more content-bearing. Across both datasets, SBERT and CodeBERT provide the most consistent positive associations, while surface and POS metrics are corpus-sensitive, reinforcing the need to report multiple metric families per dataset rather than relying on a single similarity signal. Finding 14: Metric-based correlations are small but structurally consistent. Code-aware embeddings (SBERT, CodeBERT) show stable positive associations between informative reports and patch similarity across datasets. Surface and POS metrics are corpus-sensitive—mixed or negative on Defects4J, but uniformly positive on SWT-Bench. Embeddings therefore provide reliable primary signals, while lexical/POS cues offer complementary, dataset-dependent evidence. Engineering implications for automated test generation and repair. Findings 10–14 show that patch–report alignment is representation-sensitive and signal dependent rather than intrinsic, and translate directly into three design decisions for repair pipelines, which we operationalize as a concrete algorithm below. First, evaluate patches as full diffs: both LLM- and metric-based analyses indicate that contextual completeness provides the most stable alignment signal, while add-only or remove-only views lose semantic anchoring, especially under structured abstraction (Findings 10, 12). Repair validation should therefore operate on full diffs rather than isolated edits. Second, prioritize embedding-based semantic alignment over surface overlap: code-aware embeddings (SBERT/CodeBERT/OpenAI) provide the most stable signals across corpora, whereas lexical and POS cues are dataset-sensitive (Finding 14). Embedding proximity between report anchors and candidate patches offers a lightweight proxy for behavioral directness during ranking or filtering. Third, condition generation and validation on report clarity: explicit expected/actual contrasts and a few salient steps consistently strengthen alignment signals, whereas procedural verbosity without contrast can dilute repair precision (Finding 13). Together, alignment can be operationalized as a , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

25

Table 20. Direction and magnitude of metric-based correlations between report attributes and patch alignment metrics.

(a) Defects4J Metric CodeBERT Jaccard Noun overlap OpenAI (emb) ROUGE-L SBERT Verb overlap

Model GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat

Sig +

Sig –

Median 𝜌 (sig)

7 2 2 1 1 0 6 2 2 1 10 4 1 0

0 0 0 1 1 2 3 3 0 2 0 0 2 1

0.066 0.058 0.068 0.018 –0.0065 –0.0575 0.057 –0.054 0.065 –0.072 0.075 0.113 –0.054 –0.066

Median 𝜏 (sig)

Max |𝜌 |

0.054 0.043 0.052 0.012 –0.0025 –0.043 0.045 –0.041 0.051 –0.058 0.055 0.081 –0.043 –0.056

0.125 0.068 0.082 0.089 0.071 0.060 0.155 0.091 0.072 0.101 0.227 0.169 0.080 0.066

Median 𝜏 (sig)

Max |𝜌 |

0.060 0.048 0.053 0.048 0.026 0.036 0.039 0.057 0.035 0.051 0.056 0.053 0.037 0.032

0.160 0.060 0.131 0.187 0.112 0.139 0.180 0.225 0.123 0.165 0.146 0.207 0.123 0.158

(b) SWT-Bench Metric CodeBERT Jaccard Noun overlap OpenAI (emb) ROUGE-L SBERT Verb overlap

Model GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat GPT-4o DeepSeek-Chat

Sig +

Sig –

Median 𝜌 (sig)

11 2 9 10 6 6 9 9 14 11 11 12 5 8

0 0 0 0 5 5 3 2 0 0 0 0 3 1

0.079 0.059 0.064 0.057 0.030 0.042 0.050 0.074 0.043 0.062 0.068 0.065 0.040 0.037

Sig + / Sig –: count of report attributes with significant positive / negative correlation (significance if Spearman 𝑝 < .05 or Kendall 𝑝 < .05). Median 𝜌 (sig) / Median 𝜏 (sig): median Spearman / Kendall among significant effects within the block. Max |𝜌 | : largest absolute Spearman coefficient among significant effects in the block.

controllable repair signal: (i) normalize reports into structured behavioral schemas, (ii) generate and evaluate patches as full diffs, and (iii) rank candidates using embedding-based proximity augmented with anchor checks. Algorithm 2 operationalizes this workflow end-to-end. Given a bug report 𝑅, we extract structured anchors 𝑆 and emphasize the expected/actual contrast (line 2, Finding 13), since explicit behavioral contrast is the strongest report-level predictor of patch precision. Candidate patches are generated as full diffs by default (line 5), reflecting the framing effect established in Finding 10. Each candidate is then scored via a deterministic adequacy function combining embedding cosine (line 6, Finding 14), anchor coverage, and a minimality prior that penalizes broad or unrelated edits, avoiding reliance on LLM judges, whose systematic optimism and weak human-rank agreement (Finding 11) make them unsuitable for automated decisionmaking. Final validation applies the standard fail→pass criterion, optionally followed by regression tests. Like Algorithm 1, this blueprint requires no alignment-specific model training and can be implemented with off-the-shelf LLM generators and embedding models, making it a directly reusable reference design for semantics-aware repair validation.

, Vol. 1, No. 1, Article . Publication date: July 2026.

26

Ouédraogo et al.

Algorithm 2 Full-diff patch generation and validation guided by report anchors Require: Bug report 𝑅, codebase 𝐶, failing test suite T𝑓 𝑎𝑖𝑙 (available), patch generator 𝐺 𝑃 , embedding model 𝐸, 𝑁 candidates, top-𝑘 Ensure: Validated patch(es) P and ranked candidates C 1: 𝑆 ← ExtractStructured(𝑅) ⊲ expected/actual, exception, API, steps – RQ1 2: 𝑆 ← EmphasizeContrast(𝑆 ) ⊲ make expected vs actual explicit – Finding 13 3: C ← ∅ 4: for 𝑖 ← 1 to 𝑁 do 5: 𝑝𝑖 ← 𝐺 𝑃 (𝐶, 𝑆 ) ⊲ generate full diff candidate by default – Finding 10
6: 𝑢𝑖𝑒𝑚𝑏𝑒𝑑 ← cos 𝐸 (repr(𝑆 ) ), 𝐸 (repr(𝑝𝑖 ) ) ⊲ deterministic proxy – Finding 14 7: 𝑢𝑖𝑎𝑛𝑐ℎ ← AnchorMatch(𝑆, 𝑝𝑖 ) ⊲ API/exception/entities/behavioral cues present 8: 𝑢𝑖𝑚𝑖𝑛 ← Minimality(𝑝𝑖 ) ⊲ penalize broad edits / unrelated files 9: 𝑢𝑖 ← 𝛼𝑢𝑖𝑒𝑚𝑏𝑒𝑑 + 𝛽𝑢𝑖𝑎𝑛𝑐ℎ + 𝜆𝑢𝑖𝑚𝑖𝑛 10: C ← C ∪ { (𝑝𝑖 , 𝑢𝑖 ) } 11: end for 12: C ← TopK( C, 𝑘 ) 13: P ← ∅ 14: for all (𝑝𝑖 , 𝑢𝑖 ) ∈ C do 15: if Validate(𝐶 ⊕ 𝑝𝑖 , T𝑓 𝑎𝑖𝑙 ) = PASS then 16: P ← P ∪ {𝑝𝑖 } 17: end if 18: end for 19: return ( P, C)

Summary of RQ3: Patch–report alignment is highly framing-dependent: full diffs provide the richest and most stable signal, RAW representations inflate scores, and STRUCTURED abstractions sharpen behavioral focus at the cost of surface similarity. Human judgments remain stable across views, but LLM judges show systematic optimism (+1–2 points) and only modest rank agreement with humans, making them unsuitable as out-of-the-box decision signals. Embeddingbased similarity (SBERT, CodeBERT, OpenAI) offers the most stable cross-dataset proxy, while explicit behavioral contrast, not verbosity, best predicts patch precision, motivating the anchorconditioned validation workflow of Algorithm 2.

5 5.1

DISCUSSION Engineering Implications

Beyond the RQ-specific engineering implications and reference algorithms presented in Section 4, our findings translate into concrete guidance for four classes of debugging automation pipelines. Test Generation. Make bug reports programmable before synthesis: use a STRUCTURED view (API/exception, inputs, steps, expected/actual) as an input contract to reduce generation variance (RQ1). Select among generated candidates using deterministic adequacy signals, such as embedding cosine (the most reliable proxy), anchor coverage, and oracle checks, rather than lexical overlap or out-of-the-box LLM judging (RQ2). Anchor richness, not verbosity, drives test quality. Fault Localization. Use behavioral anchors to constrain the search space: structured cues (APIs, exceptions, scenario steps) provide stable semantic queries for ranking suspicious components (RQ1). Prioritize embedding-based or code-aware matching over surface similarity, which can inflate with report length rather than with relevance (RQ2). Program Repair. Treat patch evaluation as representation-dependent: full diffs yield the most stable alignment signal, while add/remove-only views can distort interpretation (RQ3). Rank patches using code-aware semantic signals and treat LLM scores as heuristics given their systematic optimism , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

27

and limited human agreement (RQ3); explicit expected/actual contrasts and salient steps remain the most predictive cues for precise fixes. Bug Report Quality. Actionability depends on explicit anchors, not length. Naming APIs, specifying reproduction steps, and clearly stating expected vs. actual behavior improve downstream automation, whereas verbosity alone can dilute alignment despite increasing similarity scores (RQ2–RQ3). These four levers share a common principle: across every stage of the debugging pipeline, explicit behavioral anchors consistently outperform verbosity as a driver of automation quality, a pattern we revisit from the individual developer’s perspective in Section 5.3. 5.2

Implications for Researchers

Our findings expose methodological blind spots in how empirical software engineering evaluates bug resolution, and suggest directions for more diagnostic, semantics-aware benchmarks. Behavioral alignment is not reducible to similarity. Across RQ2–RQ3, lexical overlap is minimal and embedding similarity only moderately reflects behavioral correspondence: similarity can inflate with verbosity, saturate with code-aware embeddings, or undervalue concise yet correct fixes. Researchers should therefore separate surface resemblance from behavioral correspondence and explicitly measure the anchors linking reports to tests and patches. Structured representations materially change evaluation outcomes. An anchor-rich schema (API/exception, inputs, steps, expected vs. actual) reduces ambiguity and distinguishes extraction policy effects from genuine semantic errors. Benchmarks relying solely on raw narratives risk conflating failure modes; publishing and evaluating both RAW and STRUCTURED views improves comparability and diagnostic power. Patch view is a methodological variable. Full diffs provide the most stable alignment signal, while add-only and remove-only views shift both LLM judgments and metric behavior. Conclusions about report–patch alignment therefore depend on representation and should be stratified accordingly. LLM-based scoring is condition-sensitive rather than authoritative. Despite strong inter-human agreement, Human–LLM agreement remains modest and models exhibit systematic optimism that varies by representation and view. LLM scores should thus be reported with agreement and bias analyses rather than treated as ground truth. Taken together, these results support a behavior-centric framing of defect resolution as a propagation of semantic cues across artifacts, rather than a collection of isolated IR or APR tasks. Crucially, quantifying alignment provides a practical optimization signal for debugging automation: alignment scores can be used to (i) filter weakly grounded tests before they become repair oracles, (ii) rank candidate patches beyond fail-to-pass criteria, (iii) prioritize fault localization targets that preserve reported behavioral anchors, and (iv) detect under-specified bug reports that lack sufficient cues for reliable automation. By turning alignment into a measurable objective, researchers can design controllable, semantics-aware pipelines that better reflect developer intent. 5.3

Implications for Developers

Our findings also translate into concrete, low-effort practices that make day-to-day LLM-assisted maintenance workflows more reliable, complementing the pipeline-level guidance of Section 5.1 with habits any developer can adopt individually. Write behavioral anchors into reports, not just narrative. LLM tools behave more consistently when a report exposes a small set of checkable fields (API/exception, inputs, reproduction steps, and expected vs. actual behavior) rather than a long free-form description. Even jotting these down manually, without any tooling, reduces ambiguity for whoever (or whatever) picks up the bug next. Don’t trust a passing test until it encodes the behavior. A triggering test that merely fails-then-passes is weak evidence: what matters is whether it explicitly encodes the reported behavior (anchors + oracle), not just that a test exists. , Vol. 1, No. 1, Article . Publication date: July 2026.

28

Ouédraogo et al.

Before closing a bug, check that the test would actually fail for the right reason. Review full diffs by default; use add/remove views only to probe intent. Full diffs are the most reliable basis for judging whether a patch restores intended behavior. When reviewing or triaging with an LLM assistant, reserve add-only or remove-only views for a specific question, such as whether a change merely suppresses the symptom or actually restores behavior, rather than using them as the primary review artifact. Treat an LLM’s alignment or correctness score as a hint, not a verdict. LLM judges are systematically optimistic and only weakly rank-aligned with human assessment. If a tool surfaces an alignment or confidence score during triage or review, cross-check it against something concrete: does the test cover the reported input condition? Does the patch address the stated expected/actual contrast? Only then act on the score. Overall, stating behavioral anchors explicitly, insisting on tests with real oracles, and reviewing complete diffs are small habits with outsized effect: they are exactly the properties our results show LLM-assisted tools depend on to reason reliably about a bug. 5.4

Threats to Validity

Construct validity. Behavioral alignment between bug reports, tests, and patches is an abstract concept that we operationalize through structured anchors, LLM-based judgments, and semantic similarity metrics. Each proxy captures only part of the phenomenon: lexical or embedding similarity may reflect surface overlap rather than true behavioral correspondence, while LLM judgments can be sensitive to framing, prompt design, or representation format. We mitigate this risk by triangulating across heterogeneous signals (structured anchors, deterministic metrics, and LLM scores) and by validating key analyses against human references. The convergence of qualitative trends across these complementary measures reduces the likelihood that our findings are artifacts of any single operationalization. Internal validity. Our results may be influenced by experimental design choices, including prompt wording, model configuration, and artifact representation. To assess robustness, we evaluate two distinct LLM families (GPT-4o and DeepSeek-Chat), two report representations (RAW and STRUCTURED), and three patch views (full, add, remove), and further analyze trends across two independent datasets. While residual model-specific or prompt-specific effects cannot be entirely excluded, the stability of qualitative patterns across configurations and corpora supports the internal consistency of our conclusions. A further internal validity concern relates to model and prompting recency. Our study evaluates GPT-4o and DeepSeek-Chat under a lightweight Chain-of-Thought strategy (Sec. 3.6), rather than native multi-step reasoning (“thinking”) modes or agentic orchestration frameworks that have since become more prevalent. This choice reflects the practical constraints of our experimental scope: two datasets, six framing conditions, and four scoring dimensions, yielding a very large volume of alignment ratings that would be difficult to reproduce reliably and at comparable cost with reasoning-heavy or multi-agent pipelines. We note, however, that Desc2Fix evaluates alignment as a measurable signal over existing artifacts rather than as a generation or repair capability to be maximized; the framework itself (structured anchors, deterministic proxies, and bias-aware LLM judging) is agnostic to which underlying model or prompting strategy produces the report–test–patch triplets or the judgments, and can be directly re-applied to newer reasoning-oriented or agentic models as they become available. We leave such replication to future work. External validity. We conduct our study on Defects4J and SWT-Bench, which span multiple projects and programming languages (Java and Python) and represent diverse bug categories. However, they do not cover all ecosystems (e.g., C/C++, mobile, industrial proprietary systems). Our experiments also involve two contemporary LLMs and widely used embedding models; future model generations or domain-specific systems may exhibit different behaviors. Nonetheless, the , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

29

consistency of observed patterns across datasets and artifact types suggests that the identified representation and alignment effects are not dataset-specific. Our structured schema and alignment framework are model-agnostic and designed to facilitate replication on additional corpora. Reliability and annotation bias. Human evaluations were conducted by two annotators with software engineering expertise following task-specific guidelines and iterative calibration. We explicitly measured inter-annotator agreement using Spearman’s 𝜌, Kendall’s 𝜏, and MAE across evaluation dimensions. Agreement was consistently strong, particularly for correctness and specificity in RQ3, indicating stable and reproducible judgment criteria. All annotations were performed on the same fixed report sample, ensuring comparability across RQs and artifact types. Although manual assessment inherently involves some subjectivity, the high observed agreement supports the reliability of the human reference used throughout our analyses. 6 6.1

RELATED WORK Bug Reports and Structured Information Extraction

Bug reports are inherently noisy and heterogeneous, which has motivated extensive research on improving their structure and machine interpretability. Prior work has explored template-based report improvement [3], information retrieval and classification models [19], and more recently fine-grained entity and relation extraction using NER and knowledge-aware representations [49, 50]. These approaches enhance report understanding by identifying entities, semantic relations, or enriched embeddings that support downstream tasks such as bug triaging or categorization. However, most prior studies treat bug reports as isolated textual artifacts: they focus on improving representation quality but do not examine whether extracted signals are preserved, transformed, or lost in downstream executable artifacts such as triggering tests and corrective patches. In contrast, our work models structured behavioral anchors (e.g., exceptions, APIs, reproduction steps, expected vs. actual behavior) and evaluates their propagation across artifacts. This cross-artifact perspective moves beyond report enrichment toward semantic validation of the entire bug resolution pipeline. 6.2

Bug Localization, Test Generation, and Report–Code Linking

A substantial body of work investigates automated links between bug reports and code artifacts, particularly for fault localization and test generation. Traditional fault localization techniques combine spectrum-based and mutation-based signals [10], while more recent approaches integrate heterogeneous features (including textual similarity, complexity, and learned representations) into deep learning models such as DeepFL [22]. IR-based methods remain common baselines for aligning bug reports to source components [31], but predominantly rely on surface-level similarity between report text and code identifiers. Test generation research has increasingly incorporated textual cues from bug reports to synthesize fault-revealing tests, particularly in LLM-based settings. However, these approaches typically evaluate generated tests based on execution outcomes (e.g., fail-to-pass transitions) rather than on explicit semantic correspondence with the reported behavior, so alignment with developer intent is often assumed rather than measured. Similarly, recent LLM-driven systems support patch generation and report-aware repair, ranging from single-pass generation-and-validation pipelines [12, 15, 48] to autonomous agentic frameworks that iteratively plan, edit, and validate repairs [5]. On the test-generation side, recent work similarly moves toward reproducing failures directly from issue text [1, 29] rather than assuming report–test correspondence. Yet across both single-pass and agentic paradigms, few studies examine whether generated or developer-written patches semantically reflect the behavioral signals expressed in the original bug report. In contrast, our work treats report–test–patch relationships as a unified semantic propagation problem. Rather than optimizing localization, test generation, or repair independently, , Vol. 1, No. 1, Article . Publication date: July 2026.

30

Ouédraogo et al.

we quantify how behavioral anchors flow across artifacts and provide reproducible alignment signals that can inform and control each stage of the debugging pipeline, irrespective of whether that stage is implemented as a single-pass generator or an agentic loop. 6.3

Limitations of Embedding-Based Metrics for Semantic Alignment

Vector-based similarity metrics (e.g., TF-IDF, SBERT, CodeBERT) are widely used to approximate semantic relatedness between bug reports and code artifacts, supporting tasks such as duplicate detection, classification, and bug triage [35]. While contextual embeddings provide richer representations than lexical overlap alone, prior studies have noted that such metrics remain imperfect proxies for developer intent and behavioral nuance [32, 50]. Our results empirically confirm these limitations in a cross-artifact setting. Embedding similarity can remain high for artifacts that share entities or surface vocabulary while diverging in behavioral intent, and conversely may appear low for concise yet behaviorally correct tests or patches. We further observe that metric behavior is representation-sensitive: structured abstractions reduce surface similarity while increasing predicate-level correspondence, and code-aware embeddings (e.g., CodeBERT) can approach ceiling values without guaranteeing behavioral restoration. These findings suggest that embedding similarity should be treated as a useful but insufficient proxy. We therefore advocate for behavior-aware evaluation frameworks that integrate structured signal extraction and cross-artifact validation rather than relying solely on vector similarity. 6.4

Large Language Models and Semantic Validation

Large language models (LLMs) have rapidly become central to software engineering research, particularly in automated program repair (APR), code generation, and patch suggestion. A recent systematic review covering 189 studies (2020–2025) highlights the breadth of LLM-based repair paradigms and evaluation strategies [46]. These systems increasingly move beyond lexical heuristics toward semantic reasoning and agentic repair workflows. The broader shift toward agentic repair pipelines further motivates our focus on alignment as a foundational signal: a recent systematic study of six agentic and non-agentic repair systems on SWE-bench Verified [26] shows that even iterative, tool-using agents exhibit substantial performance variation and opaque failure modes, underscoring the need for artifact-level, model agnostic alignment signals, such as those Desc2Fix provides, that remain applicable regardless of whether the underlying repair strategy is single-pass or agentic. More recently, LLMs have also been studied as evaluators of code and patch quality. For example, Li et al. [23] demonstrate that LLM-based naturalness modeling can distinguish buggy, overfitting, and correctly repaired patches at scale. Other approaches incorporate self-evaluation mechanisms such as round-trip validation, where models reassess their own outputs for syntactic and semantic plausibility [40]. However, most existing evaluation protocols still rely on indirect proxies (compilation success, test-suite passing, token-level similarity, or distributional naturalness [41, 44]) which do not explicitly verify whether a patch restores the behavioral intent expressed in the original bug report. Our work complements and extends this line of research by treating LLMs as behavioral alignment evaluators grounded in developer intent. Rather than assessing patch plausibility in isolation, we measure semantic correspondence across bug reports, triggering tests, and fixes. By integrating structured behavioral anchors, LLM-based scoring, deterministic metrics, and human-grounded validation, Desc2Fix provides a reproducible framework for evaluating cross-artifact semantic consistency at scale, applicable to both single-pass and agentic repair paradigms. To our knowledge, no prior work systematically quantifies behavioral signal propagation across reports, tests, and patches using both structured representations and human-calibrated LLM judgments. , Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

7

31

CONCLUSION AND FUTURE WORK

Bug resolution is not a single artifact task but a semantic propagation process: behavioral signals originate in natural-language reports and are expected to materialize in triggering tests and corrective patches, yet the extent of that propagation has remained largely unmeasured. In this work, we introduced Desc2Fix, a unified framework for quantifying how these signals are preserved, transformed, or lost across artifacts. Our study demonstrates three core findings. First, structured behavioral anchors can be reliably extracted from bug reports, turning natural-language descriptions into stable and reproducible semantic input contracts. Second, cross-artifact alignment is measurable but highly representation-sensitive: lexical similarity alone is insufficient, patch view materially affects evaluation outcomes, and structured abstractions trade surface resemblance for predicate-level salience. Third, LLM-based alignment judgments exhibit systematic optimism and only moderate agreement with humans, underscoring the need for bias-aware and multi-perspective evaluation. Beyond empirical characterization, Desc2Fix reframes alignment as an actionable engineering signal rather than an observational property. Quantifying alignment enables early detection of weak or non-actionable reports, alignment-guided test generation, semantics-aware fault localization, and principled ranking of candidate patches in automated repair workflows. By grounding crossartifact reasoning in structured anchors and reproducible metrics, our framework strengthens the controllability and interpretability of LLM-driven maintenance pipelines, whether those pipelines are built from single-pass generators or agentic, tool-using systems. Future work will extend this paradigm in four directions. First, we plan to incorporate additional artifact sources (e.g., stack traces, developer discussions) to model richer intent signals. Second, we will close the loop from evaluation to generation by using alignment signals as optimization objectives for test synthesis and patch generation. Third, we envision interactive debugging workflows in which alignment feedback supports human–LLM collaboration, helping developers diagnose semantic drift and validate repair intent. Fourth, as reasoning-oriented and agentic models become more prevalent (Sec. 5.4), we plan to re-apply Desc2Fix to such systems to test whether our representation- and framing-sensitivity findings generalize beyond single-pass instruction-tuned LLMs. Ultimately, treating bug reports as controllable semantic drivers, rather than passive textual inputs, turns behavioral alignment from an afterthought of debugging automation into one of its central design variables. ACKNOWLEDGEMENTS This research was funded in whole, or in part, by the Luxembourg National Research Fund (FNR), grant reference AFR PhD bilateral, project reference 17185670. This work was also supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No. 949014) and the Fundamental Research Funds for the Central Universities (AE89991/478). For the purpose of open access, and in fulfilment of the obligations arising from the grant agreement, the author has applied a Creative Commons Attribution 4.0 International (CC BY 4.0) license to any Author Accepted Manuscript version arising from this submission. REFERENCES [1] Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Validate SWE Patches. arXiv preprint arXiv:2502.05368 (2025). [2] Alif Al Hasan, Subarna Saha, Mia Mohammad Imran, and Tarannum Shaila Zaman. 2025. LLPut: Investigating Large Language Models for Bug Report-Based Input Generation. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1652–1659.

, Vol. 1, No. 1, Article . Publication date: July 2026.

32

Ouédraogo et al.

[3] Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report?. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering. 308–318. [4] Tegawendé F Bissyandé, David Lo, Lingxiao Jiang, Laurent Réveillere, Jacques Klein, and Yves Le Traon. 2013. Got issues? who cares about it? a large scale investigation of issue trackers from github. In 2013 IEEE 24th international symposium on software reliability engineering (ISSRE). IEEE, 188–197. [5] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200. [6] Partha Chakraborty, Mahmoud Alfadel, and Meiyappan Nagappan. 2024. RLocator: Reinforcement learning for bug localization. IEEE Transactions on Software Engineering (2024). [7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [8] Fan Fang, John Wu, Yanyan Li, Xin Ye, Wajdi Aljedaani, and Mohamed Wiem Mkaouer. 2021. On the classification of bug reports to improve bug localization. Soft Computing 25, 11 (2021), 7307–7323. [9] Sidong Feng and Chunyang Chen. 2024. Prompting is all you need: Automated android bug replay with large language models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–13. [10] Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419. [11] Qing Gao, Hansheng Zhang, Jie Wang, Yingfei Xiong, Lu Zhang, and Hong Mei. 2015. Fixing recurring crash bugs via analyzing q&a sites (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 307–318. [12] Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. 2024. A deep dive into large language models for automated bug localization and repair. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1471–1493. [13] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). [14] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023). [15] Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. Inferfix: End-to-end program repair with llms. In Proceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering. 1646–1656. [16] René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis. 437–440. [17] Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2312–2323. [18] Anil Koyuncu, Kui Liu, Tegawendé F Bissyandé, Dongsun Kim, Martin Monperrus, Jacques Klein, and Yves Le Traon. 2019. iFixR: Bug report driven program repair. In Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 314–325. [19] Ahmed Lamkanfi, Serge Demeyer, Emanuel Giger, and Bart Goethals. 2010. Predicting the severity of a reported bug. 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010) (2010), 1–10. https://api.semanticscholar. org/CorpusID:9312951 [20] Ahmed Lamkanfi, Serge Demeyer, Quinten David Soetens, and Tim Verdonck. 2011. Comparing mining algorithms for predicting the severity of a reported bug. In 2011 15th European Conference on Software Maintenance and Reengineering. IEEE, 249–258. [21] Jaekwon Lee, Dongsun Kim, Tegawendé F Bissyandé, Woosung Jung, and Yves Le Traon. 2018. Bench4bl: reproducibility study on the performance of ir-based bug localization. In Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis. 61–72. [22] Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. 2019. Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis. 169–180. [23] Yuning Li, Wenkang Zhong, Zongwen Shen, Chuanyi Li, Xiang Chen, Jidong Ge, and Bin Luo. 2025. An empirical study on the code naturalness modeling capability for LLMs in automated patch correctness assessment. Automated Software Engineering 32, 2 (2025), 35.

, Vol. 1, No. 1, Article . Publication date: July 2026.

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

33

[24] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [25] Chen Liu, Jinqiu Yang, Lin Tan, and Munawar Hafiz. 2013. R2Fix: Automatically generating bug fixes from bug reports. In 2013 IEEE Sixth international conference on software testing, verification and validation. IEEE, 282–291. [26] Xiangxin Meng, Zexiong Ma, Pengfei Gao, and Chao Peng. 2024. An Empirical Study on LLM-based Agents for Automated Bug Fixing. arXiv preprint arXiv:2411.10213 (2024). [27] Manish Motwani and Yuriy Brun. 2023. Better automatic program repair by using bug reports and tests together. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1225–1237. [28] Niels Mündler, Mark Müller, Jingxuan He, and Martin Vechev. 2024. SWT-bench: Testing and validating real-world bug-fixes with code agents. Advances in Neural Information Processing Systems 37 (2024), 81857–81887. [29] Noor Nashid, Islem Bouzenia, Michael Pradel, and Ali Mesbah. 2025. Issue2test: Generating reproducing test cases from issue reports. arXiv preprint arXiv:2503.16320 (2025). [30] Anh Tuan Nguyen, Tung Thanh Nguyen, Hoan Anh Nguyen, and Tien N Nguyen. 2012. Multi-layered approach for recovering links between bug reports and fixes. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. 1–11. [31] Feifei Niu, Chuanyi Li, Kui Liu, Xin Xia, and David Lo. 2025. When Deep Learning Meets Information Retrieval-based Bug Localization: A Survey. Comput. Surveys 57, 11 (2025), 1–41. [32] Yuki Noyori, Hironori Washizaki, Yoshiaki Fukazawa, Keishi Ooshima, Hideyuki Kanuka, and Shuhei Nojiri. 2023. Deep learning and gradient-based extraction of bug report features related to bug fixing time. Frontiers in Computer Science 5 (2023), 1032440. [33] Wendkûuni C Ouédraogo, Laura Plein, Kader Kaboré, Andrew Habib, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2024. Extracting Relevant Test Inputs from Bug Reports for Automatic Test Case Generation. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 406–407. [34] Wendkûuni C Ouédraogo, Laura Plein, Kader Kabore, Andrew Habib, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2025. Enriching automatic test case generation by extracting relevant test inputs from bug reports. Empirical Software Engineering 30, 3 (2025), 85. [35] Avinash Patil, Kihwan Han, and Aryan Jadon. 2023. A comparative study of text embedding models for semantic text similarity in bug reports. arXiv preprint arXiv:2308.09193 (2023). [36] Laura Plein, Wendkûuni C Ouédraogo, Jacques Klein, and Tegawendé F Bissyandé. 2024. Automatic generation of test cases based on bug reports: a feasibility study with large language models. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 360–361. [37] Sarah Rastkar, Gail C Murphy, and Gabriel Murray. 2010. Summarizing software artifacts: a case study of bug reports. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1. 505–514. [38] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). [39] Shuai Shao and Tingting Yu. 2024. Enhancing IR-based Fault Localization using Large Language Models. arXiv preprint arXiv:2412.03754 (2024). [40] Asankhaya Sharma. 2024. Patched RTC: evaluating LLMs for diverse software development tasks. arXiv preprint arXiv:2407.16557 (2024). [41] Jiajun Sun, Fengjie Li, Xinzhu Qi, Hongyu Zhang, and Jiajun Jiang. 2025. Empirical Evaluation of Large Language Models in Automated Program Repair. arXiv preprint arXiv:2506.13186 (2025). [42] Shaowei Wang and David Lo. 2014. Version history, similar report, and structure: Putting them together for improved bug localization. In Proceedings of the 22nd international conference on program comprehension. 53–63. [43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837. [44] Aidan ZH Yang, Sophia Kolak, Vincent J Hellendoorn, Ruben Martins, and Claire Le Goues. 2024. Revisiting unnaturalness for automated program repair in the era of large language models. arXiv preprint arXiv:2404.15236 (2024). [45] Klaus Changsun Youm, June Ahn, Jeongho Kim, and Eunseok Lee. 2015. Bug localization based on code change histories and bug reports. In 2015 Asia-Pacific Software Engineering Conference (APSEC). IEEE, 190–197. [46] Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2024. A systematic literature review on large language models for automated program repair. arXiv preprint arXiv:2405.01466 (2024). [47] Wen Zhang, Ziqiang Li, Qing Wang, and Juan Li. 2019. FineLocator: A novel approach to method-level fine-grained bug localization by query expansion. Information and Software Technology 110 (2019), 121–135.

, Vol. 1, No. 1, Article . Publication date: July 2026.

34

Ouédraogo et al.

[48] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement, 2024. arXiv preprint arXiv:2404.05427 (2024). [49] Cheng Zhou, Bin Li, Xiaobing Sun, and Hongjing Guo. 2018. Recognizing software bug-specific named entity in software bug repository. In Proceedings of the 26th Conference on Program Comprehension. 108–119. [50] Cheng Zhou, Bin Li, Xiaobing Sun, and Sheng Yu. 2023. Leveraging multi-level embeddings for knowledge-aware bug report reformulation. Journal of Systems and Software 198 (2023), 111617. [51] Jian Zhou, Hongyu Zhang, and David Lo. 2012. Where should the bugs be fixed? more accurate information retrievalbased bug localization based on bug reports. In 2012 34th International conference on software engineering (ICSE). IEEE, 14–24.

A ADDITIONAL TABLES Table 21. Top bug-report attribute per deterministic metric (Spearman 𝜌), by dataset, scenario, and model. (a) Defects4J Scen.

(b) SWT-Bench 𝝆

Model

Metric

Best attribute

DeepSeek-Chat

behavior_sim codebert_raw entity_sim rougeL_raw sbert_raw scenario_sim verb_overlap_raw

num_entities len_report_words num_entities avg_field_length num_entities has_api len_report_words

0.160 0.339 0.231 0.188 0.194 0.111 0.106

GPT-4o

behavior_sim codebert_raw entity_sim openai_raw rougeL_raw sbert_raw scenario_sim verb_overlap_raw

has_api has_steps has_api has_api has_steps num_entities has_steps len_report_words

0.193 0.367 0.356 RAW 0.239 0.245 0.275 0.434 0.124

DeepSeek-Chat

codebert_struct num_entities rougeL_struct avg_field_length sbert_struct num_entities verb_overlap_struct avg_field_length

GPT-4o

codebert_struct has_steps 0.364 openai_struct has_api 0.246 rougeL_struct avg_field_length 0.287 sbert_struct num_entities 0.283 verb_overlap_struct len_report_words 0.175

RAW

STR.

Scen. Model

Best attribute

DeepSeek-Chat

behavior_sim codebert_raw entity_sim jaccard_raw noun_overlap_raw openai_raw rougeL_raw sbert_raw scenario_sim verb_overlap_raw

has_api 0.078 len_report_words 0.223 num_entities 0.197 len_report_words 0.206 has_exception -0.074 len_report_words 0.207 len_report_words 0.217 len_report_words 0.287 num_entities 0.113 len_report_words 0.130

GPT-4o

behavior_sim codebert_raw entity_sim openai_raw rougeL_raw sbert_raw scenario_sim verb_overlap_raw

has_api num_entities has_api len_report_words len_report_words len_report_words has_steps len_report_words

DeepSeek-Chat

codebert_struct num_entities 0.119 jaccard_struct len_report_words 0.139 noun_overlap_struct has_exception -0.082 openai_struct avg_field_length 0.176 rougeL_struct len_report_words 0.126 sbert_struct len_report_words 0.155 verb_overlap_struct avg_field_length 0.090

GPT-4o

codebert_struct openai_struct rougeL_struct sbert_struct verb_overlap_struct

0.164 0.190 0.162 0.159

STR.

Scen.: scenario (RAW vs. STRUCTURED). STR.: structured.

, Vol. 1, No. 1, Article . Publication date: July 2026.

𝝆

Metric

num_entities has_api num_entities num_entities len_report_words

0.072 0.262 0.330 0.215 0.217 0.292 0.388 0.139

0.215 0.184 0.157 0.211 0.078

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

35

Table 22. Patch-view effect on LLM scores (Wilcoxon signed-rank). Dataset

Representation

Model

Axis

full–add

full–remove

add–remove

GPT-4o

Alignment Coverage Correctness Specificity

−0.68∗∗∗ (70) −0.67∗∗∗ (369) −0.58∗∗∗ (369) −0.63∗∗∗ (369)

−0.09 (100) −0.28∗∗∗ (492) −0.12∗∗ (492) −0.25∗∗∗ (492)

+0.43∗∗∗ (93) +0.45∗∗∗ (352) +0.44∗∗∗ (352) +0.41∗∗∗ (352)

DeepSeek-Chat

Alignment Coverage Correctness Specificity

−0.17∗∗ (369) −0.25∗∗∗ (370) −0.13∗ (367) −0.18∗∗∗ (370)

+0.23∗∗∗ (492) +0.17∗∗∗ (493) +0.23∗∗∗ (487) +0.15∗∗∗ (493)

+0.37∗∗∗ (356) +0.38∗∗∗ (356) +0.38∗∗∗ (351) +0.32∗∗∗ (356)

GPT-4o

Alignment Coverage Correctness Specificity

−0.74∗∗∗ (370) −0.73∗∗∗ (370) −0.67∗∗∗ (370) −0.66∗∗∗ (370)

−0.51∗∗∗ (494) −0.52∗∗∗ (494) −0.43∗∗∗ (494) −0.52∗∗∗ (494)

+0.24∗∗∗ (355) +0.19∗∗∗ (355) +0.17∗∗ (355) +0.12∗ (355)

DeepSeek-Chat

Alignment Coverage Correctness Specificity

−0.70∗∗∗ (371) −0.68∗∗∗ (371) −0.58∗∗∗ (367) −0.71∗∗∗ (371)

−0.17∗∗∗ (494) −0.15∗∗∗ (494) −0.10∗ (494) −0.32∗∗∗ (494)

+0.39∗∗∗ (356) +0.40∗∗∗ (356) +0.30∗∗∗ (352) +0.31∗∗∗ (356)

GPT-4o

Alignment −0.18∗∗∗ (764) −0.63∗∗∗ (620) −0.55∗∗∗ (659) Coverage −0.28∗∗∗ (2321) −0.73∗∗∗ (1971) −0.62∗∗∗ (1956) Correctness −0.30∗∗∗ (2321) −0.75∗∗∗ (1971) −0.66∗∗∗ (1956) Specificity −0.23∗∗∗ (2321) −0.64∗∗∗ (1971) −0.52∗∗∗ (1956)

DeepSeek-Chat

Alignment −0.26∗∗∗ (2326) −0.32∗∗∗ (1979) −0.11∗∗∗ (1963) Coverage −0.26∗∗∗ (2326) −0.33∗∗∗ (1979) −0.13∗∗∗ (1963) Correctness −0.26∗∗∗ (2302) −0.33∗∗∗ (1963) −0.12∗∗∗ (1954) Specificity −0.21∗∗∗ (2326) −0.24∗∗∗ (1979) −0.07∗∗ (1963)

GPT-4o

Alignment −0.44∗∗∗ (2321) −0.81∗∗∗ (1973) Coverage −0.39∗∗∗ (2321) −0.81∗∗∗ (1973) Correctness −0.38∗∗∗ (2321) −0.79∗∗∗ (1973) Specificity −0.27∗∗∗ (2321) −0.71∗∗∗ (1973)

DeepSeek-Chat

Alignment −0.43∗∗∗ (2326) −0.70∗∗∗ (1979) −0.50∗∗∗ (1963) Coverage −0.40∗∗∗ (2326) −0.69∗∗∗ (1979) −0.50∗∗∗ (1963) Correctness −0.37∗∗∗ (2308) −0.69∗∗∗ (1957) −0.52∗∗∗ (1945) Specificity −0.36∗∗∗ (2326) −0.66∗∗∗ (1978) −0.45∗∗∗ (1962)

RAW

Defects4J

STRUCTURED

RAW

SWT-Bench

STRUCTURED

−0.70∗∗∗ (1958) −0.72∗∗∗ (1958) −0.70∗∗∗ (1958) −0.57∗∗∗ (1958)

Significance: ∗∗∗ 𝑝 < 0.001, ∗∗ 𝑝 < 0.01, ∗ 𝑝 < 0.05; otherwise not significant. Cells report 𝑟 with significance stars and paired sample size 𝑛 in parentheses.

Table 23. Model effect on LLM scores (Wilcoxon signed-rank). Dataset

Scenario

Alignment

Coverage

Correctness

Specificity

Defects4J

RAW STRUCTURED

+0.46∗∗∗ (501) −0.13∗∗ (502)

+0.23∗∗∗ (501) −0.10∗ (502)

+0.48∗∗∗ (501) −0.14∗∗ (502)

+0.41∗∗∗ (501) −0.10∗ (502)

SWT-Bench

RAW STRUCTURED

+0.43∗∗∗ (2344) −0.04∗ (2340)

+0.20∗∗∗ (2344) +0.17∗∗∗ (2340)

+0.58∗∗∗ (2342) +0.04 (2339)

+0.37∗∗∗ (2344) −0.12∗∗∗ (2340)

Significance: ∗∗∗ 𝑝 < 0.001, ∗∗ 𝑝 < 0.01, ∗ 𝑝 < 0.05; otherwise not significant. Cells report effect size 𝑟 with significance stars and paired sample size 𝑛 in parentheses.

, Vol. 1, No. 1, Article . Publication date: July 2026.