A Taxonomy of Failing Bug Reproduction Tests on SWT-Bench

arXiv:2611.13395v1 [cs.SE] 10 Nov 2026

A Taxonomy of Failing Bug Reproduction Tests on SWT-Bench Luca Filippelli

Hamed Taherkhani

Hung Viet Pham

Hadi Hemmati

York University Toronto, Canada lucafili@my.yorku.ca

York University Toronto, Canada hamedth@yorku.ca

York University Toronto, Canada hvpham@yorku.ca

York University Toronto, Canada hemmati@yorku.ca

Abstract—Automatic Program Repair (APR) tools require high-quality test generation models to validate candidate patches. Despite the advancements of automated bug reproduction testing tools, they continue to struggle with issues of higher complexity. These struggles may be caused by a framework’s lack of problemspecific context. We address the unclassified nature of these framework failures by providing a taxonomy of bug reproduction tests. This study uses the AssertFlip and OpenHands frameworks, the two highest-scoring open-source tools on SWT-Bench Verified. The significance of model selection on the causes of failure is also explored by utilizing GPT-5-mini and GPT-4o-mini on both tools. The error traces of these frameworks are categorized to determine granular modes of failure. Through this process, mechanical and implementation failures were the most common failure classes. Overall, a tool’s generation approach has a significant impact on precision and coverage, while the selection of the model alters the quality and quantity of generated tests. Specifically, more powerful models with higher reasoning capability tended to present fewer failures due to a lack of technical or problemspecific context. Our experiment also shows evidence that our taxonomy can be used as concrete guidance to augment the issue description to improve the generation quality. Index Terms—Taxonomy, Bug Reproduction Testing, SWTBench, LLM4SE, Test Generation

I. Introduction The costly nature of program repair spurred the development of the Automated Program Repair (APR) subfield. APR tools automatically generate patches for software bugs when provided with an issue description, ground-truth tests, and a buggy codebase [1], [2], [3], [4], [5], [6]. The effectiveness of these tools depends on the availability and quality of comprehensive contextual information, particularly high-quality bugreproduction tests that validate candidate patches. To address this dependency, automated bug reproduction test generation has emerged as a complementary research area, providing the necessary test suites that APR systems require. Recent advances in Large Language Models for Software Engineering (LLM4SE) have catalyzed progress in this domain, leveraging the capacity of LLMs to both generate and evaluate test cases. Open-source frameworks such as AssertFlip, (a hybrid LLMand-rule-based bug reproduction test tool) [7] and OpenHands (a general-purpose agentic platform) [8] exemplify this trend, This work was supported by the NSERC Discovery Grant and Vector Institute Scholarship in AI.

enabling researchers to generate bug-reproduction tests at scale through locally executable, LLM-driven approaches. Despite these advancements, current state-of-the-art LLMdriven bug reproduction frameworks exhibit substantial limitations when faced with complex software issues, particularly those requiring deep comprehension of external library semantics and intricate API interactions. Empirical evidence from the SWT-Bench Verified benchmark reveals that even top-performing open-source frameworks, AssertFlip (45.5% resolution rate) and OpenHands (62.4% resolution rate) with GPT-5-mini, ranking fourth and seventh respectively on the leaderboard [9], fail to reproduce more than one-third of the test cases. More importantly, the underlying causes of these failures remain uncharacterized in existing literature. Without a systematic understanding of failure patterns, practitioners cannot effectively diagnose weaknesses in their approaches, nor can researchers identify targeted interventions to advance the state of the art. A comprehensive taxonomy of bug reproduction test failures would address this gap by providing: (1) a structured framework for analyzing tool limitations, (2) insights into how tool choices and model selection influence failure patterns, and (3) an evidence base for developing targeted mitigation strategies. Such a taxonomy can transition bug reproduction research from incremental performance improvements to principled, failure-informed design. This study addresses these gaps through a systematic empirical analysis of bug reproduction test failures across multiple dimensions. We examine two leading open-source frameworks, AssertFlip and OpenHands. To isolate the effects of model selection on failure patterns, we evaluate the frameworks using GPT-4o-mini and GPT-5-mini. This controlled experimental design enables us to disentangle tool-specific limitations from model-dependent weaknesses. We analyze 435 failing test instances across all tool-model configurations and manually categorize each failure. Our granular analysis reveals not only which failure patterns dominate, but also how the distribution of these patterns shifts as a function of tool architecture and model reasoning capacity. In addition, we investigate whether simple template-based augmentation guided by our taxonomy can remediate the identified failure patterns. Specifically, we apply a targeted method to a representative sample of failing

instances, augmenting issue descriptions with failure-classspecific guidance derived from our taxonomy. This paper seeks to provide insights on the common causes of failure within open source bug reproduction frameworks. We particularly seek to analyze the effect of tool selection and model selection in determining the prevalence of specific bug classes. Given these two objectives, the following research questions can be formulated: • RQ1: What are the primary failure modes of LLM-based bug reproduction tools? • RQ2: How does the tool’s test generation approach affect failure class distributions? • RQ3: How does the chosen model impact failure distributions? Contributions.This work makes the following contributions to automated bug reproduction research: • A comprehensive taxonomy of bug reproduction test failures: We develop a new failure taxonomy to capture the specific failure mode of LLM-based bug reproduction, introducing fine-grained subclasses to enable deeper analysis. Our taxonomy provides a standardized vocabulary for characterizing 435 real-world failures across state-of-the-art frameworks. • Empirical analysis of failure pattern distributions: We quantify how tool architecture and model selection influence failure distributions, revealing that tool choice can affect the precision and coverage of test generation and model reasoning capability significantly impacts the quantity and quality of generated tests. • Evidence for taxonomy-driven remediation: We demonstrate that targeted remediation strategies are plausible with the help of our taxonomy. Applying to previously failing instances, 7 out of 11 augmentation attempts successfully improved the generated test. This indicates that with our taxonomy guidance, practitioners would have a concrete remediation method without much effort. The remainder of the paper will provide some background on bug reproduction research (Section II), discuss related work (Section III), describe the study methodology (Section IV), list experimental setup (Section V), present the results and findings (VI), discuss the case study of a possible mitigation method (Section VII), elaborate the threats to validity (Section VIII), and conclude (Section IX). Data Availability We make our labelling results available 1 II. Background This section reviews essential background on bug reproduction testing and benchmarking practices. LLM-Driven Bug Reproduction Testing Bug reproduction testing aims to automatically generate test cases that expose reported software defects, and it is a key prerequisite for automated program repair and regression testing [10]. A bugreproducing test must satisfy two essential properties: it fails on the buggy version and passes after the fix, thereby serving 1 https://zenodo.org/records/19075731

as an executable oracle for both defect presence and patch correctness [10]. In practice, the main source of problemspecific context is the bug report, yet producing reliable reproductions manually is labor-intensive and often deferred or ignored by developers [11], [12]. This motivates automated reproduction pipelines that translate issue context into runnable tests. Earlier work explored automated crash reproduction strategies that leverage partial signals such as crash traces to recover triggering executions [12]. More recently, LLM-driven frameworks generalize beyond crash-only settings by synthesizing reproduction tests directly from natural language issue descriptions and iterative execution feedback, enabling reproduction at scale [13], [7]. A central challenge in this setting is not only generating test code, but also constructing correct assertions (oracles) that align with the issue’s expected behavior. CodeT addresses this by validating candidates through cross-sample agreement using a RANSAC-inspired consensus mechanism to filter inconsistent tests [14]. Complementarily, semantic enrichment methods such as TELPA [15] leverage counterexamples to steer the model toward deeper systemunder-test understanding. AssertFlip [7] operationalizes oracle construction in the reproduction setting by generating a passing test from an LLM-produced plan. Benchmarking: Benchmarking provides standardized evaluation protocols for systematic comparison of competing systems under controlled conditions [16], [17], [18], [19], [20]. Historical benchmarks have enabled quantitative assessment across diverse domains, including LAN networks, database management systems, and information extraction [21]. The advances in machine learning has motivated domain-specific benchmarks tailored to particular evaluation objectives. In the security domain, TDDBench [22] assesses methods for detecting training data exposure. For software development, BaxBench [23] evaluates automated backend generation systems on functionality and security criteria. In the LLM4SE domain, SWE-Bench [1] establishes evaluation baselines using real-world software engineering issues. Reproduction Test Benchmarks: Automated bug reproduction test generation requires standardized benchmarks for evaluation. SWT-Bench [9] addresses this need by adapting problems from SWE-Bench to create a bug reproduction benchmark. GitBugJava [24] provides benchmarking functionality for Java-based bug reproduction, comprising 199 bugs with patches from 55 open-source repositories. To broaden coverage beyond Python/Java issue-to-test settings, CrashJS [25] contributes a benchmark of 453 Node.js crashes for automated crash reproduction, while BugsInDLLs [26] curates 112 reproducible environments across JAX, TensorFlow, and PyTorch to enable systematic evaluation of testing techniques against deep-learning-library bugs. For lower-level crash scenarios, Live-kBench [27] further provides an evolving Linux kernel crash-resolution benchmark. Thematic Analysis: Thematic Analysis (TA) is a qualitative analysis method for reporting patterns in data. There are several approaches to TA. One of the most prominent is the six-step method proposed by Braun and Clarke [28], which

entails familiarization, code generation, theme generation, theme review, theme definition, and report production. Braun and Clark also identified a subgrouping of TA known as Coding Reliability. This grouping focuses on using procedures to minimize researcher bias [29]. Template Analysis, a Coding Reliability TA approach proposed by Brooks et al., [30] follows a similar approach to Braun and Clarke with the use of a priori themes, the generation of new themes, and the modification of applied labels. The Framework TA method entails the same initial process of familiarization and coding followed by the development of an analytical framework to label subsequent data, after which it is charted and interpreted [31]. Our taxonomy employs a similar structure to Template Analysis, and the details will be discussed further in section IV. III. Related Work This section reviews existing taxonomies for test failure classification and state-of-the-art frameworks that underpin our study design and taxonomy construction. Taxonomy in Software Testing: Taxonomies establish standardized classification schemes that enable systematic analysis and comparison across studies in software testing [32], [33], [34], [35]. Beizer’s seminal defect taxonomy introduced a foundational classification framework comprising nine distinct fault categories [36]. Subsequent taxonomies have adapted and extended this foundation to address evolving testing paradigms. Felderer and Beer refined Beizer’s categories to better capture failures in requirements-based testing contexts [37]. Beyond defect classification, taxonomies have been developed to categorize testing methodologies themselves, as exemplified by Shaukat et al.’s systematic classification of testing techniques [38]. Taherkhani et al. [39] present a taxonomy of LLM-generated test failures comprising seven invalidity categories. We take inspiration from this taxonomy to create one expressive enough to describe LLM-driven bug-reproduction test-tool failures. Automated Reproduction Test Generation: Automated bug reproduction test generation approaches fall into two categories. Agentic approaches prioritize planning capabilities and task autonomy at higher computational cost. Agentless approaches offer lower-cost alternatives that, as demonstrated by Xia et al. [40], achieve competitive effectiveness. Several agentic approaches have been developed for bug reproduction test generation. SWE-Agent [41] introduced a software-engineering-specific agentic system with an enriched user interface. OpenHands [8] ranks among the most effective agentic systems on SWT-Bench, extending SWEAgent through multi-capability agent deployment to provide a more generalized framework. AEGIS [42] represents the first domain-specific agent for bug reproduction, incorporating a text generation module followed by an FSM-based feedback module. BRT Agent [43] augments the established, LIBRO [13] framework by integrating a reasoning module into its reproduction pipeline. LIBRO [13] represents an early agentless framework that combines few-shot prompting with

post-generation ranking. Otter [44] employs LLM-driven planning instead of ranking, sequentially localizing files, generating a resolution plan, revising it, and executing. e-Otter++ [45]mu extends this approach by generating multiple candidate tests using Otter and critiquing and repairing them via LLM feedback. Issue2Test [46] advances SWT-Bench performance over SWE-Agent, and LIBRO by incorporating an error categorization step to refine failing tests. AssertFlip [7] adopts a similar refinement strategy through assertion inversion by generating a passing test from an LLM-produced test plan, flipping the assertion, and validating through LLM-based criticism. IV. Study Design This study focuses on investigating the common pitfalls of automatic reproduction test generation and how our findings can help practitioners utilize state-of-the-art LLM-based generation tools. This study was conducted with a selection of two top open-source generation frameworks on the SWTBench Verified leaderboard [9]. We focus the study on the SWT-Bench Verified benchmark, as, unlike GitBugJava or CrashJs, it provides a containerized execution environment [24][25]. Developing this environment from scratch for these benchmarks would be a significant undertaking in and of itself. The environment, like the one SWT-Bench provides, ensures all evaluation attempts are consistent, thus reducing a potential source of error within a paper’s evaluations. We also perform our study on two general-purpose and lightweight LLMs, GPT4o-mini and GPT-5-mini, which represent typical usage in the real world. Furthermore, selecting the GPT family of models allows this study to closely align with the results of OpenHands and AssertFlip on the SWT-Bench Verified leaderboard [9]. Finally, based on the existing taxonomy [39], we expand and label the invalidity of generated reproduction tests. The SWT Bench evaluation harness is used as a uniform evaluation method for each tool-model pair. There may be issues in each pair that cannot be assessed, as they do not generate a test for the harness to evaluate. For instance, AssertFlip with GPT-5-mini may have more labelled failures than AssertFlip on GPT-4o-mini, as the latter was not able to generate the same quantity of inferences for the SWT Bench harness to evaluate. Finally, to test the practicality of our findings, we apply some concrete insights from our taxonomy to improve the test generation from the practitioner’s point of view and observe significant improvement with simple issue description augmentation. A. Study Subjects Several reproduction test generation frameworks are reported on the SWT-Bench leaderboard; however, we focus on open-source frameworks to aid with taxonomy construction and reasoning about failure modes, as closed-source frameworks would hinder our ability to closely investigate the root causes of generation failures. AssertFlip and OpenHands represent the top two open-source frameworks on the SWTBench Verified leaderboard. Even if AssertFlip and OpenHands reflect the leading open-source frameworks at the time

TABLE I: Definitions and Examples of Every Taxonomy Class and Subclass Class

Subclass

Definition

Example

Not Implemented

A test is not provided by the tool

matplotlib-22719: # Task List #1. Explore repository to find appropriate test location +Inspected repository structure and located top-level project directory...

Output Format Inconsistency

A test, despite being logically correct, expects an output to be formatted differently from its actual format

xarray-6599: assert (’axis’, ’azimuth_time’) == (’azimuth_time’, ’axis’)

Environment Error

An error within the test environment occurs prior to execution

astropy-7166: pytest.py: error: unrecognized arguments: –no-header

Incorrect File Reference

The generated test references an incorrect file path

matplotlib-23299: reproduction.py not found at /testbed/reproduction.py

Wrong API Call

Incorrect interaction with an external library

django-13807: loaddata crashes when table names are SQL keywords

Incorrect Input/Mock

The test fails to provide correct input or mock input values to reproduce the buggy state

pytest-5262: Mock fails to mimic the behaviour of standard Python I/O interfaces

Incorrect Assertion

The oracle checks the wrong variable or state

astropy-14309: assertion only checks for an empty list

Logical Failure

The test case executes a sequence of operations that is syntactically correct but semantically disconnected from the bug’s trigger conditions

matplotlib-23314: Monkeypatching Axes3D.draw to detect invisibility logic rather than verifying the rendered output layers.

Misunderstanding Edge Case Logic

A test does not sufficiently acknowledge or address an edge case within its logic

matplotlib-25960: Does not account for a shared reference being a shallow copy

Misunderstanding Function Logic From Natural Language

A test fails to sufficiently interpret natural language logic

django-15629: migration library requirement is overlooked in the test

Mechanical Failure

Misimplementations

Requirement Misunderstanding

of this study rather than absolute state-of-the-art performance overall, this does not limit our findings. Our primary focus is identifying recurring modes of failure and actionable avenues for improvement from a practitioner’s perspective. In this study, GPT-4o-mini and GPT-5-mini are chosen for their lightweight reasoning capability. The dichotomy between these two models enables the taxonomy results to be analyzed on the basis of model as well as the generation frameworks. This further analysis will provide insights on the model’s influence in terms of generated reproduction test failure. B. Taxonomy Construction In general, taxonomization aids in categorizing failing tests by using a standard language. Individual test failure taxonomies were decided upon after reviewing the available existing taxonomies in the background and related work. As existing taxonomy categories do not provide specific categories necessary for classifying bug reproduction test failures, we decided to develop our own. This synthesis enables a detailed analysis of test generation failures and provides more concrete insights so that practitioners can improve the generated test with targeted issue description augmentation. The thematic construction of this taxonomy required three of the authors to participate in the labelling process. Two authors began the process by agreeing on an initial set of categories and independently labelling all failing instances from our selected frameworks and LLMs. In the event of a labelling disagreement, or an ambiguous case, a third author acted as a tie-breaker to ensure every label received a majority consensus. Our initial set included Output Format Inconsistency, Wrong API Call, Incorrect Input/Mock, Logical Failure, Misunderstanding Edge Case Logic, and Misunderstanding

Function Logic From Natural Language. Instances that did not adhere to this initial set were skipped. Once the initial labelling step was complete, the authors agreed on new labels to independently label the remaining instances. Once the entire labelling process was complete, the authors grouped the labels into overarching themes, creating the class-subclass dynamic presented in this paper. This process can be exemplified through the issues django13807 and astropy-14309 on AssertFlip GPT-5-mini as well as django-15629 on Openhands GPT-5-mini (as listed in table I). django-13807 was classified as Wrong API Call due to the attempt to create a table not including a command to generate a table. This failure pattern is confirmed from the test’s execution trace, which crashes due to the table not existing. Therefore, it fit the original definition of Wrong API Call as a result of improperly generating an SQL table. Conversely, astropy-14309 failed due to the generated test’s assertion being formatted to only search for empty lists. This pattern did not fit the original labels, as, except for the final assertion, the test adhered to the correct implementation and issue logic. This failure mode was confirmed by reviewing the assertion error generated within the execution trace and by reviewing the issue text. As such, in the second round of labelling, it was assigned Incorrect Assertion. While the generated test for django-15629 was formatted correctly, it failed as a result of a misinterpretation of the original prompt. Specifically, the generated test ignored the requirement of using the migrations library in Django, thus resulting in an invalid test environment. Since the test failed to follow a clear requirement within the problem prompt, it was classified as Misunderstanding Function Logic From Natural Language.

To validate the reliability of our manual annotation process, we calculated the inter-annotator agreement using Cohen’s Kappa coefficient (𝜅). This metric is particularly suited for our study as it accounts for the agreement occurring by chance. The coefficient is defined as: 𝑝𝑜 − 𝑝𝑒 𝜅= 1 − 𝑝𝑒

The SWT-Bench evaluation harness [9] was used to assess the generated tests from each tool-model pair. The failing tests, along with their associated error traces (the trace of a test’s execution) from the evaluation harness, were then extracted to label and construct the taxonomy.

where 𝑝 𝑜 represents the relative observed agreement among raters, and 𝑝 𝑒 is the hypothetical probability of chance agreement. In our evaluation, the annotators achieved a 𝜅 of 0.953. Following the widely accepted scale by Landis and Koch [47], this value indicates “almost perfect“ agreement, confirming the high degree of consistency and reliability in our dataset labeling. Table I shows our full taxonomy with subcategories along with their definitions and examples. We present the Misimplementations, Mechanical Failure, and Requirement Misunderstanding classes to respectively represent instances of improper test state, external failure, and an incapability to understand the requirements of an issue. Misimplementations contains the Logical Failure, Incorrect Assertion, and Incorrect Input / Mock subclasses to distinguish the specific manner by which a test fails to generate a reflective bug reproduction test. Additionally, Mechanical Failure is expanded upon by including the Wrong API Call, Output Format Inconsistency, Environment Error, Not Implemented, and Incorrect File Reference subclasses to represent instances of incorrect library interaction, failed environment setup, and invalid test generation. Lastly, Requirement Misunderstanding includes the Misunderstanding Edge Case Logic and Misunderstanding Function Logic From Natural Language subclasses to articulate the source of a tool’s inability to internalize a problem’s requirements. Returning to the example issues, astropy- 14309, django- 13807, and django-15629, the former’s error pattern was grouped with Misimplementations as the generation resulted in an improper test state. Conversely, the failure pattern in django- 13807 was consistent with the Mechanical Failure grouping as an instance of external failure. The latter issue’s failure pattern, being due to a misunderstanding of the issue’s requirements, was thematically grouped within Requirement Misunderstandings. These categories, being based on software engineering phenomena, ensure the taxonomy’s generalizability outside of SWT-Bench as a tool, but must still resolve these same challenges of reproducing a bug from an issue.

In this section, we provide the investigation and answer for each research question. Specifically, we present the overall distribution of failure classes and discuss the common failure patterns to answer RQ1. Next, we analyze the differences between frameworks and LLMs and discuss the specific failure modes when different frameworks and LLMs are used. And finally, we present some concrete experiments showing how the finding from our taxonomy can help practitioners make concrete arguments to test generation frameworks and improve their performance.

V. Experiment Setup The test generation frameworks are executed on a 2019 MacBookPro with 32 GB of memory running macOS 15.6.1. Following the replication instructions on the respective GitHub repositories for AssertFlip [7] and OpenHands [8], Docker with Anaconda is configured. To minimize resource wastage, the experiment on OpenHands with GPT-5-mini was reused from prior work [9]. The hyperparameters, such as the number of generation iterations, for both AssertFlip and OpenHands are kept at the default values to ensure consistent results.

VI. Results

A. RQ1: What Are the Primary Failure Modes of LLM-Based Bug Reproduction Tools? To answer this research question, three authors manually reviewed the generated reproduction test failures and labeled each instance with one of the categories from our taxonomy presented in table I. This taxonomy was constructed concurrently with labeling to ensure consistency and completeness. Figure 1.a shows the distributions of the reproduction test failures across all three framework-LLM combinations. Each bar represents a failure class with texturing to represent failure subclasses. When reading the graph from left to right, the bars respectively represent Mechanical Failure, Misimplementations and Requirement Misunderstanding. The distribution is presented as the percentage of the particular class with respect to the total number of generated test cases. This will show the relative distribution between different tool-model combinations. However, since each configuration would have a different number of generated test cases, table II also shows the raw number of failed tests in each failure class, as well as the number of valid tests and the total number of generated tests. Note that unless otherwise specified, all percentages are reported as relative to total generated tests for a specific toolmodel pairing. Overall, the most prevalent failure classes across the 435 test failures are Mechanical Failure and Misimplementations. These two classes were observed in 30.0% of all generated tests. The three most prevalent subclasses of Mechanical Failure include Environment Error (12.1%), Wrong API Call (10.2%) and Output Format Inconsistency (2.6%). The first subclass presents an interesting mode of failure by which a test cannot be executed due to an error in establishing its runtime environment. Furthermore, the prevalence of Wrong API Call highlights the detriment of a lack of library-specific context. For instance, sympy-15875 failed as the model did not assume that the method under test could return None. Therefore, a relatively superficial assumption in the library’s contract led to a failed bug reproduction test.

a) Overall Failure Rates

30%

Failure Rate (%)

26.0%

Failure Rate (%)

20%

10%

1.9%

Failure Rate (%)

15.9%

0%

70% 60% 50% 40% 30% 20% 10% 0% 70% 60% 50% 40% 30% 20% 10% 0%

b) AssertFlip GPT 4o mini

c) AssertFlip GPT 5 mini

25.4%

14.1%

64.8%

8.0%

7.6%

0.0% d) OpenHands GPT 4o mini

1.5%

e) OpenHands GPT 5 mini

25.0%

21.8%

15.1% 2.4%

2.0%

FAILURE HIERARCHY MISIMPLEMENTATIONS REQUIREMENT MISUNDERSTANDING Incorrect Input/Mock Misunderstanding Function Logic From Natural Language Incorrect Assertion Misunderstanding Edge Case Logic Logical Failure

MECHANICAL FAILURE Not Implemented Output Format Inconsistency Environment Error Incorrect File Reference Wrong API Call

Fig. 1: A Global Distribution of Test Failures TABLE II: Raw Failure Distributions in Each Tool-Model Pairing and Overall Class

Subclass

Overall

AssertFlip GPT-4o-mini

AssertFlip GPT-5-Mini

OpenHands GPT-4o-mini

OpenHands GPT-5-mini

Not Implemented

7

0

0

2

5

Output Format Inconsistency

26

2

4

7

13 34

Environment Error

120

0

5

81

Incorrect File Reference

4

0

0

1

3

Wrong API Call

101

8

11

36

46

Class Total

258

10

20

127

101

Incorrect Input/Mock

35

2

7

2

24

Incorrect Assertion

58

13

8

12

25

Logical Failure

65

3

6

35

21

Class Total

158

18

21

49

70

Misunderstanding Function Logic

6

0

2

0

4

Misunderstanding Edge Case Logic

13

0

2

4

7

Class Total

19

0

4

4

11

Failing Tests

435

28

45

180

182

Generated Tests

994

71

263

196

464

Mechanical Failure

Misimplementation

Requirement Misunderstanding

The three subclasses in Misimplementation, including Logical Failure (6.5%), Incorrect Assertion (5.8%), and Incorrect Input/Mock (3.5%), represent the main challenges of test generation: how to generate (1) a valid test oracle with (2) appropriate input, with (3) the correct testing logic. For instance, in the issue astropy-14309, the model assumed that resolving the given IndexError implied asserting an empty file registry despite there being other file types within the issue,

thus resulting in an Incorrect Assertion. This demonstrates that when the tool fails to generate an effective test, it typically does so for superficial contextual or logical reasons. The subclasses of Requirement Misunderstanding were observed at a significantly lower rate than their peers. Misunderstanding Edge Case Logic and Misunderstanding Function Logic From Natural Language were respectively observed at

a rate of 1.3% and 0.6%. These failures, unlike the most common subclasses, typically include added logical complexity within their class-specific domain. For instance, a non-Latin character translation failure demonstrates the added complexity of Misunderstanding Function Logic From Natural Language as it requires domain knowledge on non-Latin alphabets and Domain Name Service (DNS) translation. Overall, the majority of overall failing issues tend to fail as a result of poor test implementation or a lack of library context. Conversely, the least observed failure classes tend to suffer from a higher logical complexity. This finding suggests that most issues fail as a result of a lack of logical or contextual clarity rather than a high logical complexity, and further clarification in the issue text would help the model generate more correct test cases. We will explore applying this finding in Section VII. Finding 1: Mechanical Failure and Misimplementations are the most prevalent failures. These classes typically fail due to superficial logical or contextual assumptions, whereas the least common failure classes tend to fail on account of their more complex logic. B. RQ2: How Does the Tool’s Test Generation Approach Affect Failure Class Distributions? In this RQ, we seek to determine the influence of a tool’s test generation strategy on the distribution of observed failure classes. To do so, we investigate the differences in frequency and proportion of observed failures between AssertFlip’s 263 generated tests (figure 1.c) and OpenHands’ 464 generated tests (figure 1.e) on the same LLM (GPT-5-mini). The same analysis is done on GPT-4o-mini (figures 1.b and 1.d). The overall per-tool distributions differ significantly in their top two observed classes. For instance, Misimplementations (8.0% vs. 15.1%) and Mechanical Failure (7.6% vs. 21.8%) go from being relatively equal on AssertFlip GPT-5-mini to significantly imbalanced on OpenHands GPT-5-mini, with Requirement Misunderstanding remaining relatively marginal. This trend is primarily driven by the lower relative quantity of Environment Error on AssertFlip-GPT-5-mini relative to OpenHands (1.9% vs. 7.3%). For instance, although pytest5262 fails on both tool-model pairs, AssertFlip produces an Incorrect Input/Mock whereas OpenHands produces an Environment Error. This failure class imbalance is also observed on both tools when using GPT-4o-mini. The higher proportion of Mechanical Failure (14.1% vs. 64.8%) compared to Misimplementations (25.4% v 25.0%) on Openhands is driven by the significantly higher proportion of Environment Error (0.0% vs. 41.3%). Due to its agentic approach, OpenHands might misconfigure the test, leading to missing files and incorrect environmental setup errors. Intriguingly, the relative proportion in Misimplementations remains stable across both tools on GPT-5-mini. For instance, Incorrect Assertion (3.0%) on AssertFlip GPT-5-mini and OpenHands GPT-5-mini (5.4%) comprise approximately a third of their respective Misimplementations class. While

the subclass proportion changes on GPT-4o-mini, the same pattern of relative Misimplementation stability across both tools remains. This proportion suggests a failure parity; despite different operational paradigms, AssertFlip’s specialized inversion versus OpenHands’ agentic interaction, both tools reach a similar bottleneck where implementation failure presents itself at nearly identical rates. While both tools reach similar bottlenecks, OpenHands produces more tests while AssertFlip has a higher precision in test generation. On one hand, AssertFlip, with its verify and flip approach, already pre-verifies many test implementation details prior to reproduction test generation. This method means that AssertFlip is less likely to generate tests that have Mechanical Failure or Misimplementations leading to an overall more precise set of generated tests at the cost of reduced generation volume. On the other hand, OpenHands, with its agentic approach, generates the tests directly using LLM without implementation details verification which led to a higher rate of Mechanical Failure due largely to incorrect environmental setup. Overall, different approaches can have different effects on how precisely the tests are generated. Specifically, AssertFlip’s implementation details verification reduce the amount of generated tests but improve precision, whereas OpenHands’s agentic approach favors coverage with more tests generated at the cost of lower precision. Finding 2: Different tools using different verification approaches can alter the precision and coverage tradeoff. By pre-verifying the implementation details, AssertFlip generates fewer tests than OpenHands, 263 vs 464 on GPT5-mini and 71 vs. 196 on GPT-4o-mini, but avoids greater rates of error; 17.1% vs. 39.2% on GPT-5-mini and 39.4% vs. 91.8% on GPT-4o-mini. C. RQ3: How Does the Chosen Model Impact Failure Distribution? Given that both tools heavily rely on LLMs to generate test cases, we seek to investigate the effect of model selection on the distribution of failure classes. To this end, RQ3 will compare the frequency and proportion of observed failures between GPT-4o-mini (figure 1.b and figure 1.d) and GPT-5mini (figure 1.c and figure 1.e) on AssertFlip and OpenHands, respectively. Overall, the distribution changes significantly between GPT4o-mini and GPT-5-mini. Using AssertFlip, with GPT-4omini incurs a larger portion of Mechanical Failures (14.1% vs. 7.6%) and Misimplementations (25.4% vs. 8.0%). This pattern is also observable on OpenHands with GPT-4o-mini having more Mechanical Failure (64.8% vs. 21.8%) and Misimplementations (25.0% vs. 15.1%). This difference might indicate that GPT-4o-mini makes more mistakes due to its smaller knowledge base [48] and weaker reasoning ability [49]. GPT-5-mini can generate more precise tests without Mechanical Failure and Misimplementations. Notably, on AssertFlip, Wrong API Call and Incorrect Assertion are the most prevalent

failure modes in GPT-4o-mini generated tests. For example, in the case of django-15499, GPT-4o-mini generates a test that fails due to a misunderstanding that optimization logic in Django’s reduce method is a multi-step sequence. But the tool attempted to verify it using a single comparison call while asserting that the result should contain the cumulative changes of two separate operations. Since the API only handles one peer-to-peer merger at a time, the second set of changes was never actually processed. GPT-5-mini rectifies this test by utilizing the MigrationOptimizer class, which provides the correct high-level abstraction. By passing the entire list of operations into the optimizer, the test allows Django to handle the iterative reduction logic internally, accurately reflecting the desired outcome where multiple operations collapse into a single, updated CreateModel. Therefore, GPT-5-mini, unlike GPT-4o-mini, leverages its stronger knowledge base to interact with the correct API method to generate an accurate bug reproduction test. While failures generally decrease from GPT-4o-mini to GPT-5-mini, the rate of Requirement Misunderstanding increases both on AssertFlip (0.0% vs. 1.5%) and OpenHands ( 2.0% vs. 2.4%). This slight increase may account for a subset of issues which, despite overcoming Misimplementation and Mechanical Failure, fail to satisfy a problem’s requirements. For example, the issue django-15629, previously discussed in section IV, failed on OpenHands GPT-5-Mini due to a Requirement Misunderstanding. On OpenHands GPT-4-Mini, it could not even proceed to execution as a result of an Environment Error due to a Module Not Found error. Therefore, while GPT-5-mini decreases failures overall, Requirement Misunderstandings increase due to the conversion of Mechanical Failure and Misimplementations Furthermore, GPT-5-mini was able to generate significantly more tests compared to GPT-4o-mini; 263 vs. 71 on AssertFlip and 464 vs. 196 on OpenHands. This difference indicates that GPT-5-mini is much more capable at creating syntactically correct tests, which leads to both better precision and coverage. Finding 3: Models with a larger knowledge base and higher reasoning capability generate more tests with fewer failures. Specifically, GPT-5-mini significantly outperforms GPT4o-mini with more generated tests; 263 vs. 71 on AssertFlip and 464 vs. 196 on OpenHands and GPT-5-mini has a much lower failure rate relative to GPT-4o-mini; 17.1% vs. 39.4% on AssertFlip and 39.2% vs. 91.8% on OpenHands. VII. Discussion: Taxonomy provides insight to improve test generation with context enrichment To determine if our taxonomy can give concrete insights to practitioners, we conduct a small experiment where the insights from our findings can guide augmentations that are applied directly to an issue’s natural language description. This method can help improve the outcome of the generated test. Table III presents our attempts with specific issues (Col Issue), in different subclasses (Col Failure Subclass). The Issue

Summary column describes an issue’s problem statement, and finally, we provide our concrete addition to the issue description that can help improve the generated test quality. Specifically, for each failure mode, we craft an augmentation template that focuses the LLMs on avoiding the mistake in the first place. For this experiment, we utilize AssertFlip and GPT-5-mini. Wrong API Call: For failures of this type, we focus the LLMs with the following template: Ensure to fully understand and properly use the X libraries to the fullest extent necessary to correctly replicate the bug, where X is the specific API(s) that the LLM needs to pay attention to. Table III shows the issues to which we apply this augmentation strategy. Of the six issues, two generate correct test cases after the augmentation. Specifically django-15382, django-13195, sympy-19495 and sympy-15976 fail to produce valid bug reproduction tests. Unlike the following failure categories, these samples rely upon a considerable understanding of the library they will be using to effectively replicate a bug. In each failing test, the mode of failure remained the same. This concentration of failures demonstrates how library understanding requires a significant amount of context which the tool may not be able to accommodate. For example, in django-13195, the failure persists due to the model’s inability to comprehend the complexities of the web-framework along with the necessary configuration for a web cookie. This failure can be contrasted by the success of django-13807. Whereas django-13195 failed due to insufficient API context, the subject matter of django13807, generating a simple database table, was of reasonable complexity for the model to properly replicate. Therefore, failures within the Wrong API Call type occur due to a need for a deeper contextual understanding of the libraries. Incorrect Input/Mock: To focus the LLMs on making sure the input are generated or mocked correctly, we use the following template: Ensure to fully, to the extent needed, understand and emulate the behaviour, input requirements and state of X, where X is the specific input object or class that needs to be correctly initialized or mocked. We apply this augmentation strategy to pytest-5262 and xarray6721, which are presented in Table III. Of the two issues, both generate correct test cases after the augmentation. By providing a simple command to sufficiently mimic the behavior of their target mock, the tools replicated their respective issue’s bug. This success demonstrates that, when given a minimal amount of context, the tool can create an effective mock, thus replicating the bug. Incorrect Assertion: To ensure the tool generates accurate assertions, we use the following template: Review your target output and the expected behaviour, make sure your assertions fully reflect the X, where X is specific target which the assertion should focus on. We apply this augmentation strategy to astropy-14309, django-13212 and sphinx-10466 which are presented in Table III. Of the three issues, all generate correct test cases after the augmentation. Therefore, the augmentation to review each issue’s expected behaviour and target output successfully resolved the failing instances.

TABLE III: Examples of Context-Enrichment Mitigation Issue

Failure Type

Issue Summary

Problem Statement Addition

Resolved PostAugmentation?

django-13807

loaddata crashes when table names ++Ensure to fully understand and properly use the yes are SQL keywords django.db.connections library to the fullest extent necessary to correctly replicate the bug

sympy-15875 Wrong API Call

A complex integer expression ++Ensure to fully understand and properly use the yes which evaluates to zero does not sympy I and sympy simplify libraries to the fullest return true when is_zero() is extent necessary to correctly replicate the bug called

sympy-15976

A symbol ending with a number ++Ensure to fully understand and properly use the no is invisible as the API wraps the sympy.printing.mathml and xml.dom libraries to expression in a <mi> tag the fullest extent necessary to correctly replicate the bug

sympy-19495

the .sub method corrupts a bound ++Ensure to fully understand and properly use the no variable ImageSet, ConditionSet and FiniteSet libraries to the fullest extent necessary to correctly replicate the bug

django-13195

Deleting a cookie should preserve ++Ensure to fully understand and properly use the no its samesite django.test.override_settings library to the fullest extent necessary to correctly replicate the bug

django-15382

Django filter command removes ++Ensure to fully understand and properly use the no WHERE command django.models.exists and base python libraries to the fullest extent necessary to correctly replicate the bug

pytest-5262

Incorrect Input/Mock

xarray-6721

EncodedFile should not accept bi- ++Ensure to fully, to the extent needed, understand yes nary and emulate the behaviour, input requirements and state of RawIOBase or BufferedIOBase interface. Accessing array chunks loads the ++Ensure to fully, to the extent needed, understand yes whole array and emulate the behaviour, input requirements and state of xarray.Variable

astropy-14309

Resolve IndexError Incorrect Assertion

++Review your target output and the expected be- yes haviour, make sure your assertions fully reflect all possible file formats that .ecsv should return

django-13212

ValidationErrors should include the ++Review your target output and the expected be- yes provided value in a custom error haviour, make sure your assertions fully reflect the message population of the params dictionary of ValidationError

sphinx-10466

make.gettext returns duplicated list ++Review your target output and the expected be- yes locations haviour, make sure your assertions fully reflect the Catalog as the sole source of truth

For instance, astropy-14309 resolves its index error by either checking for an empty list or a list of valid file formats. This is in contrast to the failing test which only asserted an empty list. Ultimately, this augmentation encourages the LLM to undertake a deeper review of the issue’s requirements and methods of assessment. Overall, our quick experiment shows a general success of context enrichment guided by our failure taxonomy with 7 out of 11 augmentations successfully improving the generated tests. This result shows that our taxonomy can be a concrete guide to users in improving reproduction test generation using LLMs, and future research can utilize such a taxonomy to develop techniques that automatically improve the overall performance of LLM-generated reproduction tests.

Furthermore, language-specific failure modes are therefore not captured within the presented taxonomy

VIII. Threats to Validity

Human Subjectivity: While the labelling process followed a majority consensus model, the potential for Researcher Bias remains a threat. Individual experience with the Python ecosystem or specific libraries may have influenced the classification of failure modes. The extension of an existing taxonomy, blind labelling, and consensus methodology mitigate this threat by limiting interpretation and encouraging evidence-based justification, especially in the event of disagreement.

Scope: This study analyzes OpenHands and AssertFlip using the Python language. Both tools capture the diversity of methodologies seen in LLM-driven bug reproduction tools (agentic loop and rule-based hybrid frameworks). Despite this capture, hyper-specific, tool-based failure modes from novel methodologies may be excluded from this taxonomy.

Data Leakage: The risk of data leakage increases as models continually undergo training, since it is possible for the models to be trained on the issues within the SWT-Bench datasets. This form of leakage could introduce false positive results, as passing bug reproduction tests would be achieved via memorization rather than learning. LLM Stochasticity: Given the probabilistic nature of LLMs, the exact results, even when using the same input parameters, may differ. Naturally, this threat may diminish the independent ability to verify this taxonomy through direct replication. Despite this threat, the taxonomy can be independently assessed via extension to other tools and models.

IX. Conclusion Given the importance of automated bug reproduction frameworks in improving program repair, understanding the failures of these tools is integral. This study addresses the existing literature gap by generating a taxonomy of automated bug reproduction test failures of state-of-the-art open-source frameworks using two modern GPT models. Our findings indicate that different framework approaches can lead to different precision and coverage tradeoff and more powerful LLMs are more likely to increase the quality and quantity of generated tests. We also demonstrate that our taxonomy can be used as guidance to targeted augmentation which can lead to improvement of generated reproduction tests. References [1] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve realworld github issues?” in ICLR, 2024. [2] F. Li, J. Jiang, J. Sun, and H. Zhang, “Hybrid automated program repair by combining large language models and program analysis,” TOSEM, 2025. [3] J. Xu, Y. Fu, S. H. Tan, and P. He, “Aligning the objective of LLM-based program repair,” arXiv, 2024. [4] Y. Ouyang, J. Yang, and L. Zhang, “Benchmarking automated program repair: An extensive study on both real-world and artificial bugs.” ACM, 2024. [5] B. Yang, Y. Xiong, and C. Zhang, “A survey of LLM-based automated program repair: Taxonomies, design paradigms, and applications,” arXiv, 2025. [6] Q. Zhang, C. Fang, Y. Xie, Y. Zhang, W. Sun, and Z. Chen, “A systematic literature review on large language models for automated program repair,” arXiv, 2024. [7] L. Khatib, N. S. Mathews, and M. Nagappan, “AssertFlip: Reproducing bugs via inversion of LLM-generated passing tests,” in ICSE, 2026. [8] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “OpenHands: An open platform for AI software developers as generalist agents,” arXiv, 2024. [9] N. Mündler, M. N. Müller, J. He, and M. Vechev, “SWT-Bench: Testing and validating real-world bug-fixes with code agents,” arXiv, 2024. [10] T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schroter, and C. Weiss, “What makes a good bug report?” IEEE TSE, 2010. [11] O. Chaparro, C. Bernal-Cárdenas, J. Lu, K. Moran, A. Marcus, M. Di Penta, D. Poshyvanyk, and V. Ng, “Assessing the quality of the steps to reproduce in bug reports,” in ESEC/FSE, 2019. [12] M. Soltani, A. Panichella, and A. van Deursen, “Evolutionary testing for crash reproduction,” in SBST, 2016. [13] S. Kang, J. Yoon, and S. Yoo, “Large language models are few-shot testers: Exploring LLM-based general bug reproduction,” in ICSE, 2023. [14] B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “CodeT: Code generation with generated tests,” in ICLR, 2023. [15] C. Yang, J. Chen, B. Lin, Z. Wang, and J. Zhou, “Advancing code coverage: Incorporating program analysis with large language models,” ACM Trans. Softw. Eng. Methodol., 2026. [16] G. Bai, J. Liu, X. Bu, Y. He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zheng, and W. Ouyang, “MT-Bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues,” in ACL, 2024. [17] S. Ahuja, D. Aggarwal, V. Gumma, I. Watts, A. Sathe, M. Ochieng, R. Hada, P. Jain, M. Ahmed, K. Bali, and S. Sitaram, “MEGAVERSE: Benchmarking large language models across languages, modalities, models and tasks,” in NAACL, 2024. [18] S. Ni, G. Chen, S. Li, X. Chen, S. Li, B. Wang, Q. Wang, X. Wang, Y. Zhang, L. Fan, C. Li, R. Xu, L. Sun, and M. Yang, “A survey on large language model benchmarks,” arXiv, 2025. [19] P.-H. Q. Le, M. Nauta, V. B. Nguyen, S. Pathak, J. Schlötterer, and C. Seifert, “Benchmarking eXplainable AI: A survey on available toolkits and open challenges,” in IJCAI, 2023.

[20] E. Davis, “Benchmarks for automated commonsense reasoning: A survey,” ACM Comput. Surv., 2024. [21] S. E. Sim, S. Easterbrook, and R. C. Holt, “Using benchmarking to advance research: A challenge to software engineering,” in ICSE, 2003. [22] Z. Zhu, Y. Yang, and D. Lian, “TDDBench: A benchmark for training data detection,” arXiv, 2024. [23] M. Vero, N. Mündler, V. Chibotaru, V. Raychev, M. Baader, N. Jovanović, J. He, and M. Vechev, “BaxBench: Can LLMs generate correct and secure backends?” arXiv, 2025. [24] A. Silva, N. Saavedra, and M. Monperrus, “GitBug-Java: A reproducible benchmark of recent java bugs,” in MSR, 2024. [25] P. Oliver, J. Dietrich, C. Anslow, and M. Homer, “CrashJS: A Node.js benchmark for automated crash reproduction,” in MSR, 2024. [26] M. M. A. Naziri, A. K. Singh, B. Wu, F. Qin, S. Dutta, and M. d’Amorim, “BugsInDLLs: A database of reproducible bugs in deep learning libraries to enable systematic evaluation of testing techniques,” in ISSTA, 2025. [27] C. Huang, A. Mathai, F. Yu, A. Nogikh, P. Maniatis, F. Ivančić, E. Wu, K. Kaffes, J. Yang, and B. Ray, “Outrunning LLM cutoffs: A live kernel crash resolution benchmark for all,” in ICML, 2026. [28] V. Braun and V. Clarke, “Using thematic analysis in psychology,” Qual. Res. Psychol., 2006. [29] ——, “Toward good practice in thematic analysis: Avoiding common problems and be(com)ing a knowing researcher,” IJTH, 2023. [30] J. Brooks, S. McCluskey, E. Turley, and N. King, “The utility of template analysis in qualitative psychology research,” Qual. Res. Psychol., 2015. [31] N. K. Gale, G. Heath, E. Cameron, S. Rashid, and S. Redwood, “Using the framework method for the analysis of qualitative data in multidisciplinary health research,” BMC Med. Res. Methodol., 2013. [32] C. Winston and R. Just, “A taxonomy of failures in tool-augmented LLMs,” in AST, 2025. [33] F. Dobslaw, R. Feldt, J. Yoon, and S. Yoo, “Challenges in testing large language model based software: A faceted taxonomy,” arXiv, 2025. [34] H. Tian, C. Wang, B. Yang, L. Zhang, and Y. Liu, “A taxonomy of prompt defects in LLM systems,” arXiv, 2025. [35] V. Vinay, “Failure modes in LLM systems: A system-level taxonomy for reliable AI applications,” arXiv, 2025. [36] B. Beizer, Software Testing Techniques, 2nd ed. International Thomson Computer Press, 1990. [37] M. Felderer and A. Beer, “Using defect taxonomies for testing requirements,” IEEE Softw., 2015. [38] K. Shaukat, U. Shaukat, F. Feroz, S. Kayani, and A. Akbar, “Taxonomy of automated software testing tools,” Int. J. Comput. Sci. Innov., 2015. [39] H. Taherkhani, A. DaghighFarsoodeh, M. Chowdhury, H. V. Pham, and H. Hemmati, “Consistency meets verification: Enhancing test generation quality in large language models without ground-truth solutions,” arXiv, 2026. [40] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying LLM-based software engineering agents,” arXiv, 2024. [41] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” arXiv, 2024. [42] X. Wang, P. Gao, X. Meng, C. Peng, R. Hu, Y. Lin, and C. Gao, “AEGIS: An agent-based framework for general bug reproduction from issue descriptions,” arXiv, 2024. [43] R. Cheng, M. Tufano, J. Cito, J. Cambronero, P. Rondon, R. Wei, A. Sun, and S. Chandra, “Agentic bug reproduction for effective automated program repair at Google,” arXiv, 2025. [44] T. Ahmed, J. Ganhotra, R. Pan, A. Shinnar, S. Sinha, and M. Hirzel, “Otter: Generating tests from issues to validate SWE patches,” arXiv, 2025. [45] T. Ahmed, J. Ganhotra, A. Shinnar, and M. Hirzel, “Heterogeneous prompting and execution feedback for SWE issue test generation and selection,” in ICSE, 2026. [46] N. Nashid, I. Bouzenia, M. Pradel, and A. Mesbah, “Issue2Test: Generating reproducing test cases from issue reports,” arXiv, 2025. [47] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, 1977. [48] OpenAI. (2025, aug) Introducing GPT-5 for developers. OpenAI. [49] J. Qiu, Z. Liu, Z. Liu, R. Murthy, J. Zhang, H. Chen, S. Wang, M. Zhu, L. Yang, J. Tan, Z. Cen, C. Qian, S. Heinecke, W. Yao, S. Savarese, C. Xiong, and H. Wang, “LoCoBench: A benchmark for long-context large language models in complex software engineering,” arXiv, 2025.

Related documents

• AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems#1534283
• If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs#997036
• Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports#200557
• TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble#1234609
• TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble#1227388