PatchRecall: Patch-Driven Retrieval for Automated Program Repair Mahir Labib Dihan, Faria Binta Awal, Md. Ishrak Ahsan Department of Computer Science and Engineering Bangladesh University of Engineering and Technology (BUET) {mahirlabibdihan, faria12mou, ishrak26}@gmail.com
arXiv:2604.10481v1 [cs.SE] 12 Apr 2026
Abstract
MBPP (Austin et al., 2021; Chen et al., 2021) evaluate isolated and simplistic tasks that fail to capture the complexity of real-world repositories. Empirical studies demonstrate that Large Language Models (LLMs) struggle with repository-level tasks requiring class- and project-wide reasoning (Du et al., 2023; Liu et al., 2023). To bridge this gap, SWE-bench (Jimenez et al., 2023) was introduced, providing a benchmark derived directly from real GitHub issues and pull requests across 12 widely used Python projects. Its tasks demand realistic, multi-file, multi-function modifications, offering a rigorous testbed for evaluating automated program repair (APR) systems. Despite these advances, state-of-the-art models still fall short. Even sophisticated LLMs like Claude-2 have solved less than 2% of SWE-bench tasks (Jimenez et al., 2023). More recent frameworks such as MAGIS (Tao et al., 2024) introduced multi-agent collaboration and achieved up to 13.94% resolution rates, an order-of-magnitude improvement over GPT-4 baselines. Nonetheless, challenges remain: performance declines sharply as issue complexity grows, precise localization of edits remains unreliable, and generalization across repositories is inconsistent. These limitations underscore the necessity for new retrieval and localization strategies that can scale to repository-level complexity. To address this gap, we propose PatchRecall, a hybrid retrieval approach designed to balance recall with conciseness in APR workflows. Our method integrates two complementary strategies: (1) codebase retrieval, where issue descriptions are matched against repository contents to identify potentially relevant files, and (2) history-based retrieval, where past issues and their associated edits guide candidate file selection. By merging and reranking candidates from both strategies, PatchRecall produces a concise yet comprehensive set of target files. Experiments on SWE-bench show that PatchRecall
Retrieving the correct set of files from a large codebase is a crucial step in Automated Program Repair (APR). High recall is necessary to ensure that the relevant files are included, but simply increasing the number of retrieved files introduces noise and degrades efficiency. To address this tradeoff, we propose PatchRecall, a hybrid retrieval approach that balances recall with conciseness. Our method combines two complementary strategies: (1) codebase retrieval, where the current issue description is matched against the codebase to surface potentially relevant files, and (2) history-based retrieval, where similar past issues are leveraged to identify edited files as candidate targets. Candidate files from both strategies are merged and reranked to produce the final retrieval set. Experiments on SWE-Bench demonstrate that PatchRecall achieves higher recall without significantly increasing retrieved file count, enabling more effective APR.
1
Introduction
Modern software development relies heavily on collaborative platforms such as GitHub, where issues serve as central artifacts for tracking bugs, feature requests, and maintenance tasks. Large, popular repositories—for example, Django with more than 34K reported issues (Foundation, 2024) —undergo rapid and extensive evolution. Addressing these issues often requires edits across multiple files and functions, making precise localization within a codebase both challenging and crucial (Bissyandé et al., 2013). Effective resolution of such issues depends on identifying the correct set of candidate files that need modification. However, navigating vast repositories exacerbates the tradeoff between recall and efficiency: retrieving too few files risks omitting relevant ones, while retrieving too many introduces noise and reduces efficiency. Existing literature highlights several limitations in this space. Benchmarks such as HumanEval and 1
• 1-4 hours tasks: 42.9% failure rate, showing moderate difficulty
achieves higher recall without significantly increasing the number of retrieved files, thereby enabling more effective downstream APR. In summary, this work contributes a novel retrieval framework tailored for large-scale repositories, motivated by the shortcomings of existing approaches and validated on the SWE-bench benchmark. By focusing on retrieval precision and efficiency, our framework moves one step closer to making repository-level automated program repair more practical and scalable. Method TRAE Refact.ai Agent Moatless Tools OpenHands SWE-agent SWE-Fixer
Model Claude 4 Sonnet Claude 4 Sonnet Claude 4 Sonnet Claude 4 Sonnet Claude 4 Sonnet Qwen-2.5-7b
• 15 min - 1 hour tasks: 12.3% failure rate • <15 min fix tasks: Only 3.1% failure rate This distribution demonstrates that current APR systems struggle disproportionately with tasks requiring deeper understanding and more extensive code changes.
2.1.3 Temporal Patterns % Resolved The year distribution (Figure 1a) shows that failure instances peaked during 2020-2021 (approximately 75.20 13 instances each year), with notable occurrences 74.40 throughout 2019-2023. The 2018 baseline of 3 70.80 instances was relatively low. This temporal pattern 70.40 may reflect the evolution of issue complexity in 66.60 mature codebases or changes in how issues are 32.80 reported and categorized.
Table 1: Performance comparison on SWE-bench Verified leaderboard.
2
2.1.4 Error Type Patterns The error frequency analysis (Figure 1b) across repositories revealed:
Empirical Study
• TypeError dominated in sphinx-doc (nearly 100 occurrences), suggesting typerelated issues are particularly prevalent in documentation-focused codebases
The empirical study was conducted on the SWEBench benchmark, which consists of 268 realworld software engineering issues from popular Python repositories. The evaluation focused on analyzing instances where TRAE (Zhang et al., 2024), an Automated Program Repair (APR) agent, failed to resolve issues, providing insights into the challenges and limitations of current approaches. 2.1
• AssertionError was common across multiple repositories (django, sympy, pydata, sphinxdoc), indicating test failures and expectation mismatches
Key Findings
• AttributeError appeared frequently in django and sphinx-doc
2.1.1 Failure Analysis Across Repositories The study identified django/django as the most challenging repository, with 23 instances where all agents failed. This was followed by sympy/sympy (9 instances) and astroid/astroid (8 instances). The concentration of failures in specific repositories suggests that certain codebases present systematic challenges for automated repair approaches, possibly due to their complexity, size, or architectural patterns.
The error distribution suggests that different repositories exhibit distinct failure patterns, likely reflecting their domain-specific characteristics and coding patterns.
2.1.2 Difficulty Level Distribution The analysis revealed a strong correlation between task difficulty and agent failure rates (Figure 1a):
2.1.5 Failure Count Distribution A critical finding shown in the distribution analysis (Figure 1c) reveals:
• KeyError was notably present in django • ValueError showed modest frequency across astropy and django
• >4 hours tasks: 66.7% failure rate across all agents, indicating that complex, timeintensive issues remain largely unsolvable
• 41 instances had only 1 agent failure • 30 instances had 2 agent failures 2
• A long tail of instances with varying failure counts
3
Methodology
3.1
Traditional File Retrieval Approaches
• 58 instances had 18 agent failures (highlighted in red), representing cases where all evaluated agents completely failed
When an issue is reported in a large software repository, the first step in automated resolution is to identify the relevant files where code modifications are required. A widely adopted approach is to treat the issue description as a natural language query and retrieve candidate files from the entire codebase. Classical information retrieval techniques such as BM25 (Robertson et al., 2009) are employed to rank files by their textual similarity to the issue description. The system then selects the top-k files as the context for subsequent code generation or patch suggestion. This retrieval paradigm is also reflected in the SWE-bench benchmark (Jimenez et al., 2023), where the baseline relies on BM25 to map issue descriptions to potentially relevant files. The retrieved files are then fed into large language models (LLMs) as context, with the expectation that the true locus of change is among them. While this method scales to large repositories with thousands of files, it introduces significant noise: models are frequently distracted by irrelevant context and fail to localize the exact site of modification. To better understand the distribution of code edits, we analyzed the SWE-bench-verified dataset, which contains 500 human-validated issue–patch pairs. Our analysis revealed that in more than 400 cases (over 80% of the dataset), the resolution required changes in only a single file, as illustrated in Figure 2. This finding underscores a key limitation of traditional retrieval approaches: although they retrieve a set of candidate files, in the vast majority of cases, only one file actually requires modification. Therefore, the retrieval stage must not only recall the correct file but also avoid overwhelming the downstream LLM with extraneous context. This motivates our investigation into hybrid retrieval strategies.
The mean failure count was 9.3, with a median of 8.0 and maximum of 18, indicating substantial variability in task difficulty across the benchmark. 2.2
Implications
These findings underscore several challenges for the proposed hybrid retrieval approach: 1. Repository-specific adaptation: The concentration of failures in specific repositories suggests that retrieval strategies may need to be tailored to different codebase characteristics 2. Scalability to complex tasks: The high failure rate for >4 hour tasks indicates that improved file retrieval alone may be insufficient; deeper semantic understanding and multi-file reasoning capabilities are needed 3. Error-type awareness: The diverse error patterns suggest that incorporating error-type information into the retrieval strategy could improve the identification of relevant files 4. Historical learning opportunity: The 58 universal failure cases represent particularly valuable learning opportunities for improving both retrieval and repair strategies, as they likely contain common patterns that current approaches systematically miss Taken together, these findings point to file localization as a persistent bottleneck in automated program repair. Failures often arise not only from complex or long-duration tasks but also from the inability of existing systems to consistently identify the correct files for modification. Improving the retrieval stage becomes critical: retrieving too many files overwhelms the model with noise, while missing the true file leads to guaranteed failure. Motivated by this, our methodology focuses on enhancing file retrieval strategies, combining both repository-level and history-based signals to better balance recall and conciseness.
3.2
Proposed Hybrid Retrieval Strategy
While traditional retrieval methods rely solely on matching the issue text to the codebase, we propose a hybrid retrieval framework that integrates information from both verified and unverified instances of the SWE-bench dataset. The goal is to more accurately localize the files that require modification while minimizing the inclusion of irrelevant context. 3
Figure 1: (a) Overview of failure patterns across repositories, difficulty levels, time periods, and failure counts. Top-left shows the top 10 repositories with the most all-failed instances. Top-right displays the percentage of each difficulty level that failed across all agents. Bottom-left presents the temporal distribution of all-failed instances from 2018 to 2023. Bottom-right shows the distribution of failure counts, with the red bar highlighting 58 instances where all 18 agents failed. (b) Error frequencies by repository across different error types. The chart reveals repository-specific error patterns, with TypeError dominating in sphinx-doc, AssertionError prevalent across multiple repositories, and varying frequencies of AttributeError, KeyError, and ValueError across different codebases. (c) Distribution of instance failure counts across agents. The histogram shows the number of instances for each failure count (1-18 agents). The red-highlighted bar at failure count 18 represents 58 instances where all agents failed, indicating particularly challenging cases in the benchmark.
3.2.1
Distribution of Number of Changed Files per Patch 400
We consider the SWE-bench-verified subset (500 human-validated issue–patch pairs) as the gold standard for evaluation, while treating the remainder of SWE-bench as an unverified pool of auxiliary data. When a new issue is presented, we first retrieve semantically similar issues from the unverified pool using a sentence transformer (Reimers and Gurevych, 2019) model. From this step, we select the top-10 most relevant issues and extract their corresponding patch files. We consider 10 as a safe number, because Figure 2 provides us with the evidence that all the patches in the verified dataset required less than 10 files to modify. The files modified in these patches are then aggregated
Number of Patches
300
200
100
0
0
1
2
3
4
5
6
7
8
Leveraging Verified and Unverified Datasets
9 10 11 12 13 14 15 16 17 18 19 20 21 22 Number of Changed Files
Figure 2: Frequency distribution plot showing how many files were modified in the patches included in the SWE-bench-verified dataset. Our empirical analysis reveals that in more than 80% of cases, only a single file actually requires modification. And in all of the patches, less than 10 files were modified.
4
Detailed Retrieval Workflow
and scored according to their frequency and patch relevance, producing a candidate set of files likely to be relevant.
(A) History-Based Retrieval
(B) Direct Codebase Retrieval
Current Issue
Current Issue
"Fix pagination bug in admin panel"
"Fix pagination bug in admin panel"
SWE-Bench Unverified (1,794 issues)
Target Repository (3,247 files) ӈ src/ Ӌ admin/views.py Ӌ pagination.py
Issue #1234: Pagination error...
ӈ utils/
Issue #5678: Admin display bug...
Ӌ display.py
⋮ (Top-10 similar issues via BM25)
3.2.2
Parallel BM25 Retrieval from the Codebase
1. admin/views.py (score: 0.87)
pagination.py (freq: 5)
2. core/admin.py (score: 0.72)
utils/display.py (freq: 3)
3. pagination.py (score: 0.68)
Patch-Based Scores (s_patch)
BM25 Scores (s_bm25)
admin/views.py: 0.92
admin/views.py: 0.87
pagination.py: 0.78
core/admin.py: 0.72
utils/display.py: 0.65
pagination.py: 0.68
Hybrid Scoring Function H(f) = α · s_patch(f) + (1-α) · s_bm25(f) (α = 0.4 for optimal balance)
Final Ranked Files (Top-K) 1. admin/views.py 2. pagination.py 3. core/admin.py → Fed to LLM for patch generation
Figure 3: Overview of our proposed hybrid retrieval workflow. (A) History-Based Retrieval: The current issue is compared with the SWE-bench-unverified dataset using sentence transformer to retrieve the top-10 most similar issues. From their associated patches, the modified files are extracted, and scored. (B) Direct Codebase Retrieval: In parallel, the current issue is matched against the entire target repository using BM25, producing a ranked list of candidate files by textual relevance. (C) Hybrid Score Fusion & Re-ranking: Scores from both retrieval streams are min–max normalized and combined via the hybrid scoring function H(f ). The fused list is re-ranked to produce the top-k files, which are supplied as input context to the LLM for patch generation.
Re-ranking with Hybrid Scoring
After obtaining candidate files from both retrieval streams, we re-rank the union of their results to select the most relevant top-k files. Each stream produces a ranked list of (docid, score) tuples with method-specific scoring. To make scores comparable, we apply min–max normalization per method, per instance: s′m (f ) =
BM25 Ranked Files
admin/views.py (freq: 7)
(C) Hybrid Score Fusion & Re-ranking
In parallel, we apply the traditional BM25-based retrieval directly on the target repository. This process computes the similarity between the input issue text and all files in the repository, selecting the top-k candidate files. Thus, we obtain two ranked lists of candidate files: one derived from issue–patch similarity in the unverified dataset, and another derived from direct issue–file similarity within the target codebase. 3.2.3
⋮
Extract Modified Files from Patches
sm (f ) − min(sm ) , max(sm ) − min(sm ) + ϵ
where m ∈ {SentenceTransformer, BM25}, sm (f ) is the raw score for file f , and ϵ is a small constant for stability. We then compute a hybrid score
4
Experiments and Analysis
4.1
Experimental Setup
To evaluate the effectiveness of our retrieval strategies, we conduct experiments on the SWE-bench dataset (Jimenez et al., 2023). We partition the dataset into two subsets: SWE-bench-verified, which consists of 500 human-validated issue–patch pairs, and SWE-bench-unverified, which consists of the remaining 1,794 instances. Our retrieval experiments are primarily benchmarked on the verified subset, as it serves as the gold standard for correctness. We implement three retrieval baselines:
H(f ) = α · s′ST (f ) + (1 − α) · s′BM25 (f ), where s′ST (f ) comes from a Sentence Transformer retriever (we use all-mpnet-base-v2 (Song et al., 2020; Reimers, 2021) indexed with FAISS over the unverified pool) and s′BM25 (f ) from sparse retrieval over the target codebase. Files missing from one stream receive a normalized score of 0 for that stream. We sort candidates by H(f ) and keep the top-k. To study sensitivity, we sweep α ∈ [0, 1] and k ∈ {1, . . . , 10}, computing recall@k on SWEbench-verified. This exposes the trade-off between dense semantic similarity and sparse lexical matching; intermediate α values typically yield the most robust recall. The re-ranked top-k files are then provided as context to the downstream LLM for patch generation. The full workflow of our pipeline has been illustrated in Figure 3.
1. BM25: a probabilistic sparse retrieval method widely used in SWE-bench baselines. 2. TF–IDF (Salton and Buckley, 1988): a classical term-weighting scheme for ranking files by cosine similarity with the issue description. 3. Sentence Transformer: a dense retrieval model (we use the all-mpnet-base-v2 variant) that encodes both issue text and file contents into embeddings, and ranks files by semantic similarity. 5
65 60
TFIDF BM25 Sentence-Transformer
80 70
50 45
Recall (%)
Recall (%)
55
40 35
60
alpha=0.0 alpha=0.2 alpha=0.4 alpha=0.6 alpha=0.8 alpha=1.0
50
30 25
2
4
6
File Number (Top-K)
8
40
10
30
Figure 4: Comparison of recall scores for BM25, TF– IDF, and Sentence Transformer retrievers on SWEbench-verified. The x-axis denotes the number of retrieved files (k), while the y-axis shows recall. The sentence transformer demonstrates the highest recall across all retrieval depths, highlighting its advantage in bridging semantic gaps between issue descriptions and code.
6
File Number (Top-K)
8
10
Figure 5 shows recall results for varying α. We find that hybrid retrieval improves robustness across different k values, with intermediate α values (e.g., α = 0.4 to 0.6) yielding the best overall performance. This confirms that combining historical patch-level information with direct code retrieval allows for more precise localization.
Baseline Retrieval Performance
Figure 4 presents recall performance across different values of k. We observe that the sentence transformer consistently outperforms BM25 and TF–IDF across all settings, demonstrating the benefit of semantic embeddings in capturing relationships between issue descriptions and source code. BM25 performs competitively at low k, but its recall saturates more quickly. TF–IDF trails both methods, indicating limitations in handling natural language and code vocabulary mismatch. 4.3
4
Figure 5: Recall scores of the proposed hybrid retrieval strategy on SWE-bench-verified for different values of α. When α = 0, the system reduces to pure BM25 retrieval; when α = 1, it relies solely on issue–patch similarity. Hybrid configurations with α between 0.4 and 0.6 achieve the highest recall, demonstrating the benefit of balancing patch-derived evidence with direct textual similarity.
Each method takes as input the issue description and retrieves the top-k files from the codebase. We evaluate retrieval quality using recall—the percentage of gold (ground-truth) edited files contained within the retrieved set. This metric directly captures how effectively a retrieval strategy localizes the true locus of change. 4.2
2
5
Related Work
MAGIS (Tao et al., 2024) introduces a multi-agent framework for GitHub issue resolution, where specialized agents collaborate to identify relevant files, generate candidate patches, and validate solutions. A key contribution of MAGIS lies in its emphasis on repository-scale context handling and multistep coordination, addressing some of the limitations of single-agent retrieval and patching systems. Despite achieving substantial improvements over baseline methods (e.g., 13.94% resolution rate compared to 1.96% for Claude 2 on SWE-bench), MAGIS still struggles with precise file localization, particularly in repositories with large and complex dependency structures. Their reported recall curves (Figure 6) highlight that while BM25 provides a reasonable sparse retrieval baseline, performance saturates quickly and leaves significant room for enhancement. In contrast, our work explores hybrid retrieval strategies that fuse semantic retrieval models with patch-derived signals from the unverified portion of SWE-bench. As shown in Figure 6, our sentence transformer model achieves consistently higher re-
Hybrid Retrieval with Score Fusion
Building on these baselines, we evaluate our proposed hybrid retrieval approach, which integrates issue–patch similarity from the unverified subset with direct BM25 retrieval on the codebase. We combine the two sources of evidence by normalizing scores and computing a hybrid relevance score: H(f ) = α · s′ST (f ) + (1 − α) · s′BM25 (f ), where s′ST (f ) comes from a Sentence Transformer retriever (we use all-mpnet-base-v2 indexed with FAISS over the unverified pool) and s′BM25 (f ) from sparse retrieval over the target codebase. 6
80
across different retrieval depths. Compared with existing multi-agent frameworks such as MAGIS, our approach advances the retrieval stage itself, achieving consistently higher recall with a lightweight, modular design that can complement more complex agent-based systems. Looking forward, we envision extending our framework in several directions. First, adaptive weighting strategies for score fusion could dynamically tune α based on issue complexity. Second, integrating structural signals from abstract syntax trees (ASTs) or dependency graphs may provide more fine-grained localization. Finally, combining our retrieval improvements with generationand validation-focused systems has the potential to substantially raise end-to-end resolution rates on SWE-bench and beyond. By strengthening the retrieval stage, our work takes a step towards making automated program repair more practical, scalable, and reliable in real-world software engineering contexts.
Ours (alpha = 0.4) SWE-Bench (BM25)
Recall (%)
70 60 50 40 2
4
6
File Number (Top-K)
8
10
Figure 6: Recall@K comparison between our retrieval methods and the BM25 baseline used in MAGIS. The x-axis denotes the number of retrieved files (k), while the y-axis shows recall percentage. Our sentence transformer retriever, combined with hybrid scoring (α = 0.4), achieves consistently higher recall, highlighting the effectiveness of semantic and patch-augmented retrieval compared to sparse BM25 retrieval alone.
call than BM25 across all retrieval depths. Moreover, by incorporating hybrid score fusion (with α = 0.4), we surpass the BM25 baseline used in MAGIS, demonstrating that lightweight hybrid retrieval can yield complementary benefits to multiagent systems. This comparison underscores that while MAGIS focuses on agent collaboration, our method advances the retrieval stage itself, which is a critical bottleneck for effective LLM-based issue resolution.
6
References Jacob Austin et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Tegawendé F. Bissyandé, David Lo, Lingxiao Jiang, Laurent Réveillère, Jacques Klein, and Yves Le Traon. 2013. Got issues? who cares about it? A large scale investigation of issue trackers from github. In IEEE 24th International Symposium on Software Reliability Engineering, ISSRE 2013, Pasadena, CA, USA, November 4-7, 2013, pages 188–197. IEEE Computer Society.
Conclusion
In this work, we addressed the challenge of file retrieval for automated program repair in large software repositories. Through an empirical study on the SWE-bench dataset, we confirmed that traditional retrieval approaches such as BM25, while widely used, introduce significant noise by retrieving multiple irrelevant files despite the fact that over 80% of SWE-bench-verified tasks require edits in only a single file. To mitigate this limitation, we proposed PatchRecall , a hybrid retrieval strategy that integrates direct codebase retrieval with history-based retrieval from past issue–patch pairs. By normalizing and fusing scores from both sources, our method balances precision and recall, ensuring that downstream LLMs are provided with more concise yet comprehensive contexts. Our experiments demonstrated that dense retrieval with sentence transformers outperforms sparse methods like BM25 and TF–IDF, and that the hybrid score fusion further improves recall
Mark Chen et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Xueying Du et al. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861. Django Software Foundation. 2024. #30255 (docutils reports an error rendering view docstring when the first line is not empty). https://code. djangoproject.com/ticket/30255. Carlos E. Jimenez et al. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770. Nelson F. Liu et al. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. Nils Reimers. 2021. all-mpnet-base-v2. https: //huggingface.co/sentence-transformers/ all-mpnet-base-v2.
7
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. Gerard Salton and Chris Buckley. 1988. Termweighting approaches in automatic text retrieval. Information Processing & Management, 24(5):513– 523. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2020. Mpnet: Masked and permuted pretraining for language understanding. In Advances in Neural Information Processing Systems. Wei Tao et al. 2024. Magis: Llm-based multi-agent framework for github issue resolution. In Advances in Neural Information Processing Systems, volume 37, pages 51963–51993. Yunlong Zhang, Junqiang Chen, Xiang Liu, Zhengzi Wu, and Qing Wang. 2024. Trae: Trajectory-based automated program repair. In Proceedings of the 46th International Conference on Software Engineering, pages 1–13. ACM.
8