An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code Jelena Ilić Vulićević
arXiv:2604.23361v1 [cs.SE] 25 Apr 2026
Independent Researcher Master of Electrical and Computer Engineering Email: [email protected]
to run capable LLMs on consumer grade hardware. Models such as LLaMA 3.2 (8B) [3] and Mistral 7B [4] demonstrate competitive performance despite their relatively smaller size, making them suitable candidates for local deployment in practical settings.
Abstract—Large language models (LLMs) have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysis. However, most prior work relies on cloud-based models or specialized hardware, limiting practical applicability in privacy-sensitive or resource-constrained environments. In this paper, we present a systematic empirical evaluation of two locally deployed LLMs, LLaMA 3.2 and Mistral, for realworld Python bug detection using the BugsInPy benchmark. We evaluate 349 bugs across 17 projects using a zero-shot prompting approach at the function level and an automated keyword-based evaluation framework. Our results show that locally executed models achieve accuracy between 43% and 45%, while producing a large proportion of partially correct responses that identify problematic code regions without pinpointing the exact fix. Performance varies significantly across projects, highlighting the importance of codebase characteristics. The results demonstrate that local models can identify a meaningful share of bugs, though precise localization remains difficult for locally executed LLMs, particularly when handling complex and context dependent bugs in realistic development scenarios.
Despite these advances, there is limited work that systematically evaluates the effectiveness of locally deployed LLMs for bug detection in real world software projects. Most existing studies focus on large cloud based models, rely on simplified benchmarks, or target specific categories such as security vulnerabilities. As a result, the ability of smaller locally executable models to handle realistic debugging scenarios remains insufficiently understood.
The Python programming language provides a suitable context for this evaluation. Python is widely used across domains such as web development, data science, and machine learning [5]. At the same time, its dynamic typing and flexible syntax can make certain types of bugs difficult to detect using traditional static analysis techniques. In contrast, LLMs can reason about program behavior based on semantic patterns I. I NTRODUCTION without relying on explicit type information. The availability Software bugs remain a major challenge in modern software of the BugsInPy dataset [6], which contains real bugs collected development. Identifying and fixing bugs consumes a substanfrom well known Python projects, further enables a rigorous tial portion of development time, while undetected errors can empirical evaluation. lead to serious failures in production systems. A software bug To fill this gap, we evaluate two locally executed LLMs, can be defined as an unintended error in source code that results in incorrect or unexpected behavior, ranging from minor issues LLaMA 3.2 and Mistral, on the BugsInPy dataset. The dataset to critical system failures. Consequently, there is a strong need includes real bugs from 17 Python projects such as pandas, for automated techniques that can assist developers in detecting keras, matplotlib, and scrapy. We adopt a zero-shot prompting approach, where models are queried without task specific and resolving such problems. Recent advances in large language models (LLMs), which examples, and evaluate their responses at the function level by are neural networks trained on large-scale text and code corpora, analyzing individual functions instead of complete files. have demonstrated promising capabilities in code understanding The main contributions of this paper are as follows: and generation. Models such as GPT-4 [1] and Claude [2] have • We present a systematic empirical evaluation of two locally shown strong performance on tasks related to bug detection. executed LLMs on a dataset of real world Python bugs. However, these models are typically accessed through cloud • We design and implement an automated keyword based based services, requiring internet connectivity and incurring evaluation method for assessing free text bug detection usage costs. In addition, sending proprietary code to external responses. servers raises important concerns related to privacy and data • We analyze performance variation across multiple Python security. Locally executed models offer an alternative approach. projects and identify factors that influence detection Unlike cloud based solutions, these models run entirely on accuracy. local hardware, ensuring data privacy and eliminating usage • We provide all code and data required to reproduce our costs. Recent progress in model efficiency has made it feasible experiments.
1
II. R ELATED W ORK
TABLE I N UMBER OF PROCESSED BUGS PER PROJECT FROM THE B UGS I N P Y
Large Language Models (LLMs) have recently emerged as a prominent research direction in software engineering, with applications in code generation, bug detection, and automated debugging. Chen et al. [7] introduced Codex, a GPT-based model finetuned on publicly available code, evaluating its ability to generate functional Python programs from natural language descriptions using the HumanEval benchmark. Feng et al. [8] proposed CodeBERT, trained jointly on natural language and programming language data, enabling tasks such as code search and documentation generation. Both approaches demonstrate strong performance on code understanding but focus on generation rather than detection of bugs in realistic settings. Bug detection and fault localization have received growing attention in subsequent work. Kang et al. [9] proposed AutoFL, which uses LLMs for fault localization while generating natural language explanations alongside predictions, improving interpretability in large codebases. A related effort by Mhatre et al. [10] evaluated cloud-based models on a benchmark spanning both Python and C++, finding that defect complexity is the primary factor governing detection accuracy. These studies converge on the observation that performance degrades when bugs require cross-function reasoning. Test quality and structured bug reporting represent a different dimension of the same problem. Santana et al. [11] measured how well LLMs detect and refactor test smells, reporting variation across model families and task complexity. Acharya and Ginde [12] applied instruction-tuned models to convert unstructured bug reports into structured form, finding that openweight models can approach proprietary system performance on this task, though results vary across report types. Limitations in handling long contexts and complex program logic have been studied directly. Lee et al. [13] examined large Python codebases and documented consistent performance degradation as code complexity grows. Tambon et al. [14] analyzed bugs produced by LLMs themselves, proposing a taxonomy of recurring failure patterns in generated code. Both studies highlight that current models remain sensitive to context length and structural complexity. Benchmark construction has also been a productive research direction. Widyasari et al. [6] released BugsInPy, a curated set of real Python bugs with reproducible test cases drawn from well-known open-source projects. Aguilar et al. [15] later identified reproducibility issues in this dataset and proposed revisions to support more reliable evaluation. Pushkar et al. [16] extended evaluation to multi-vulnerability settings, showing that model performance drops systematically as the number of co-occurring defects increases. Zhang et al. [17] contributed a systematic review of LLM-based automated program repair, categorizing methods and identifying open challenges. The availability of open-weight models such as LLaMA [18] and Mistral [4] has made local deployment a practical option, eliminating cloud dependencies and usage costs. Despite this, prior work has largely evaluated large cloud-hosted models,
DATASET
Project
Bugs Used
pandas scrapy thefuck luigi youtube-dl keras black matplotlib ansible fastapi tornado tqdm spacy httpie sanic cookiecutter PySnooper
96 32 29 28 34 31 19 19 13 13 10 7 5 4 4 3 2
Total
349
leaving the capabilities of consumer-grade offline deployments inadequately characterized. This study evaluates LLaMA 3.2 and Mistral in a fully offline setting on the BugsInPy benchmark, providing an empirical basis for assessing their utility in realistic, resource-constrained debugging scenarios. III. M ETHODOLOGY A. Dataset We use the BugsInPy dataset [6], which contains 501 real bugs collected from 17 well known open source Python projects, including pandas, keras, matplotlib, scrapy, ansible, and others. These projects cover a diverse set of domains such as data science, web development, machine learning, and developer tools, providing a representative sample of real world Python code. Each bug instance includes a patch file in unified diff format indicating the modified lines, metadata with buggy and fixed commit identifiers, a failing test case that exposes the bug, and scripts for reproducing the execution environment. We attempt to process all 501 bugs in the dataset. A bug is excluded if one of the following conditions is met: (1) the original source file cannot be retrieved due to repository changes or unavailable commits, (2) the buggy line cannot be located in the retrieved file, or (3) no enclosing def or class statement can be identified to extract a complete function. After applying these criteria, 349 bugs are successfully processed and used in the evaluation, corresponding to 69.7% of the dataset. The distribution of processed bugs across projects is shown in Table I. B. Code Extraction For each bug, we extract three elements from the patch file: the file path, the buggy lines, and the corresponding fixed lines. The original source file is then retrieved using the buggy commit identifier through the GitHub raw content API, ensuring that the exact version of the code is obtained.
2
From the retrieved file, we identify and extract the function that contains the buggy line. This is done by scanning upward to locate the nearest def or class statement that marks the beginning of the function, and scanning downward to identify the next such statement, which marks its end. This procedure captures the full function, including its signature, documentation, and implementation. Function level extraction is performed based on preliminary experiments with alternative approaches. Providing the entire file often leads models to produce general summaries instead of focusing on the bug, as the relevant lines are diluted within a larger context. In contrast, providing only the buggy line with limited surrounding context does not offer sufficient information for understanding the intended behavior. Extracting the full function provides a balance between context and focus. In cases where multiple buggy lines are present, the first occurrence is used as an anchor for function extraction.
not include keywords but indicates the presence of a problem using terms such as bug, issue, or error. Otherwise, the response is labeled wrong. This evaluation approach is conservative. Correct responses that use different terminology may not match the extracted keywords and can therefore be assigned a lower score. As a result, the reported accuracy should be interpreted as a lower bound of the true performance. In addition to fully correct predictions, some responses are labeled as "partial" when they identify the general location or nature of the bug but do not provide a precise or complete explanation. To assess the reliability of the automated evaluation, we manually inspected a random sample of 50 model responses. The manual evaluation showed a high level of agreement with the keyword-based labels, indicating that the automated method provides a reasonable approximation of model performance despite its limitations. For the purpose of statistical evaluation, we map model outputs to binary outcomes. Responses labeled as "correct" are treated as correct predictions, while "partial" and other responses are treated as incorrect.
C. Prompt Design Each model is queried using the same zero-shot prompt format: Here is a Python function. It contains a bug. Find the bug and explain how to fix it.
E. Models and Hardware
A zero-shot setting is used for two reasons. First, it reflects We evaluate two locally executed LLMs using the Ollama a realistic usage scenario in which developers interact with framework [19], which enables efficient model execution models without task specific configuration. Second, it allows on consumer hardware. The first model is LLaMA 3.2 (8B evaluation of the model’s inherent capability for bug detection parameters) [3], and the second is Mistral (7B parameters) [4]. without influence from example based prompting. Both are referred to by their short names throughout the The prompt explicitly states that the function contains a bug. remainder of this paper. This design focuses the evaluation on the model’s ability to All experiments are conducted on a MacBook Pro 14 identify and explain the bug, rather than on binary classification inch (2021) equipped with an Apple M1 Pro processor and of correctness, and avoids trivial responses that always predict 16GB of unified memory, running macOS Sequoia 15.7.3. the absence of errors. The function code is appended directly The software environment includes Python 3.13 and Ollama after the prompt. version 0.18.0. All experiments are executed fully offline, with no communication with external services. D. Response Evaluation The average response time per query was approximately 7 Model outputs are evaluated automatically using a keyword seconds for LLaMA 3.2 and 13 seconds for Mistral, resulting based approach. The evaluation consists of three steps. in a total experiment runtime of approximately 40 minutes First, for each bug, we extract keywords from the fixed lines for LLaMA 3.2 and 75 minutes for Mistral. All models that do not appear in the buggy version. These keywords were executed using 4-bit quantization (Q4_K_M) through the typically correspond to new identifiers, function calls, or Ollama framework to ensure efficient inference on consumerconcepts introduced by the fix. grade hardware. Second, we remove common English words and tokens shorter than four characters to reduce noise and avoid acF. Reproducibility cidental matches. We note that the evaluation approach is heuristic and may not fully capture semantic correctness. In To ensure reproducibility, all code and experimental results particular, keyword matching may produce false positives when are publicly available.1 This includes scripts for dataset prokeywords appear as substrings, and false negatives when correct cessing, source code retrieval, model interaction, and response answers use alternative terminology. Therefore, the reported evaluation. All outputs are stored in JSON format and can be results should be interpreted as an approximation of model regenerated using a single script. The BugsInPy dataset is also performance rather than an exact measure. publicly available.2 Third, each response is assigned one of three labels. A 1 https://github.com/insajder/llm-bug-detection response is considered correct if it includes at least one keyword 2 https://github.com/soarsmu/BugsInPy associated with the fix. A response is labeled partial if it does
3
TABLE II OVERALL R ESULTS BY M ODEL Model LLaMA 3.2 Mistral
TABLE III ACCURACY BY P ROJECT (%)
Correct
Partial
Wrong
Accuracy
Project
151 155
171 161
27 33
43.3% 44.4%
PySnooper black fastapi ansible keras httpie tornado matplotlib scrapy spacy pandas youtube-dl thefuck cookiecutter luigi sanic tqdm
LLaMA 3.2
Mistral
100.0 73.7 69.2 61.5 54.8 50.0 50.0 47.4 40.6 40.0 38.5 38.2 34.5 33.3 28.6 25.0 0.0
100.0 68.4 61.5 69.2 48.4 50.0 60.0 47.4 43.8 20.0 44.8 35.3 37.9 33.3 28.6 0.0 14.3
B. Results by Project Table III presents the accuracy achieved by each model across individual projects. The results show substantial variation, ranging from 0% on tqdm to 100% on PySnooper. LLaMA 3.2 consistently outperforms Mistral on several Fig. 1. Overall bug detection results by model. projects, including fastapi (by 7.7 percentage points) and spacy (by 20 percentage points). In contrast, Mistral achieves better performance on projects such as ansible (by 7.7 percentage G. Bug Type Classification points), pandas (by 6.3 percentage points), and tornado (by 10 To enable fine-grained analysis, each of the 349 bugs was percentage points). manually assigned to one of nine categories based on the As a simple baseline, random keyword matching would yield nature of the change between the buggy and fixed versions: negligible accuracy (below 5%), indicating that both models Null/None Check, Return Value, Conditional Logic, Indexing, perform substantially above chance. Error Handling, Loop Logic, Type Conversion, Comparison To evaluate whether the observed difference between the Operator, and Other/Complex. Classification was performed by two models is statistically significant, we applied McNemar’s inspecting the modified tokens in each patch without reference test [20] on paired predictions. Model outputs were aligned to model outputs. by bug identifier and mapped to binary outcomes (correct vs. incorrect). Of 349 paired predictions, both models agreed on IV. R ESULTS 126 correct and 169 incorrect cases. LLaMA 3.2 was correct This section presents the empirical evaluation of LLaMA where Mistral was not in 25 cases, while Mistral was correct 3.2 and Mistral on the BugsInPy benchmark. We first report where LLaMA 3.2 was not in 29 cases. The exact binomial overall performance, followed by a detailed analysis across McNemar’s test yields p = 0.68, indicating no statistically significant difference between the two models. individual projects and score distributions. Figure 2 shows the accuracy of both models across all projects, highlighting substantial variation in performance. A. Overall Results All reported metrics are computed directly from the raw Table II presents the overall results. experimental outputs to ensure consistency and reproducibility. As shown in Table II, Mistral achieves 44.4% accuracy, slightly outperforming LLaMA 3.2 by 1.1 percentage points C. Score Distribution by Project (43.3%). Both models produce a substantial proportion of Table IV presents the distribution of correct, partial, and partial responses, indicating that they are often able to recognize that a problem exists in the code, but struggle to precisely wrong responses for both models across all projects. identify the underlying issue and its fix. These results indicate The results reveal several consistent patterns across projects. that, in practice, locally deployed LLMs tend to be more reliable Both models achieve perfect performance on PySnooper, with at detecting the presence of bugs than at accurately localizing 100% correct responses and no errors. In contrast, projects such and explaining them. Figure 1 illustrates the distribution of as tqdm show very low correctness, with LLaMA 3.2 achieving correct, partial, and wrong responses for both models. 0% and Mistral 14.3%, while the majority of responses are
4
TABLE V ACCURACY BY B UG T YPE (%) Bug Type Null/None Check Return Value Conditional Logic Indexing Error Handling Loop Logic Comparison Operator Type Conversion Other/Complex
TABLE IV S CORE D ISTRIBUTION BY P ROJECT (%). ROWS MAY NOT SUM TO EXACTLY 100 DUE TO ROUNDING .
Correct
Partial
Wrong
Correct
Partial
Wrong
PySnooper black fastapi ansible keras tornado httpie matplotlib scrapy spacy pandas youtube-dl thefuck cookiecutter luigi sanic tqdm
100.0 73.7 69.2 61.5 54.8 50.0 50.0 47.4 40.6 40.0 38.5 38.2 34.5 33.3 28.6 25.0 0.0
0.0 26.3 30.8 38.5 38.7 50.0 50.0 15.8 59.4 60.0 45.8 58.8 65.5 66.7 71.4 50.0 85.7
0.0 0.0 0.0 0.0 6.5 0.0 0.0 36.8 0.0 0.0 15.6 2.9 0.0 0.0 0.0 25.0 14.3
100.0 68.4 61.5 69.2 48.4 60.0 50.0 47.4 43.8 20.0 44.8 35.3 37.9 33.3 28.6 0.0 14.3
0.0 31.6 38.5 30.8 48.4 20.0 50.0 15.8 56.2 80.0 35.4 61.8 62.1 66.7 67.9 75.0 71.4
0.0 0.0 0.0 0.0 3.2 20.0 0.0 36.8 0.0 0.0 19.8 2.9 0.0 0.0 3.6 25.0 14.3
LLaMA 3.2
Mistral
59.5 51.3 40.0 36.4 50.0 36.4 50.0 0.0 21.7
60.8 51.3 36.4 47.7 62.5 45.5 50.0 0.0 16.7
D. Results by Bug Type To better understand model behavior, we categorize the 349 evaluated bugs into nine types based on the nature of the change between the buggy and fixed versions. Categories are derived by inspecting the modified tokens in each patch: Null/None Check, Return Value, Conditional Logic, Indexing, Error Handling, Loop Logic, Type Conversion, Comparison Operator, and Other/Complex for cases involving multiple interacting components. Table V presents the accuracy achieved by each model across bug categories, ordered by frequency. Across bug categories, consistent differences emerge. Both models perform best on Null/None Check bugs, achieving around 60% accuracy, which reflects the syntactic regularity of such errors, missing guard clauses or incorrect None comparisons tend to follow recognizable patterns that LLMs can match against training data. Return Value bugs also yield competitive accuracy near 51% for both models, as incorrect
Mistral
Project
79 78 55 44 16 11 2 4 60
classified as partial, indicating that models often recognize the presence of an issue but fail to identify the exact fix. Projects such as matplotlib exhibit a relatively high proportion of wrong responses for both models (36.8%), suggesting that bugs in these codebases are more difficult to diagnose accurately. Similarly, pandas and sanic show a non-negligible share of wrong predictions, particularly for Mistral. On the other hand, projects such as ansible, black, and fastapi are dominated by correct and partial responses, with very few or no wrong predictions, indicating that bugs in these projects are easier to localize and interpret. Figures 3 and 4 provide a visual comparison of score distributions across all projects for each model. LLaMA 3.2 achieves strong performance on projects such as black (73.7%) and fastapi (69.2%), while also exhibiting a high proportion of partial responses on projects like luigi (71.4%) and tqdm (85.7%). Mistral shows comparable performance on several projects, outperforming LLaMA 3.2 on ansible (69.2% vs 61.5%) and tornado (60.0% vs 50.0%), but also produces more partial and wrong responses on projects such as spacy (80.0% partial) and sanic (25.0% wrong). All reported metrics are computed directly from the raw experimental outputs to ensure consistency and reproducibility.
Fig. 2. Bug detection accuracy by project.
LLaMA 3.2
Count
5
Fig. 3. Score distribution by project for LLaMA 3.2.
Fig. 4. Score distribution by project for Mistral.
return expressions are often localized within a single line and differences suggest complementary strengths, and motivate future work on ensemble or model selection strategies. do not require cross-function reasoning. Figure 5 summarizes these results visually. Performance drops considerably for Conditional Logic and All reported metrics are computed directly from the raw Indexing bugs, where correct detection requires understanding experimental outputs to ensure consistency and reproducibility. of program state and data shape. Type Conversion bugs prove the most difficult, with both models scoring 0%, likely because V. D ISCUSSION such errors manifest only at runtime and depend on implicit type contracts that are not apparent from the function body A. Interpretation of Results alone. The results indicate that locally deployed LLMs are capable The Other/Complex category, which captures bugs involving of identifying a substantial portion of real-world Python bugs, interactions between multiple components, yields the lowest achieving accuracy between 43% and 45%. While this level of accuracy after type conversion (21.7% for LLaMA 3.2, 16.7% performance is not sufficient for fully automated debugging, for Mistral). This aligns with the function-level analysis it is significant given that both models operate entirely on limitation noted in Section III: when a bug’s root cause spans consumer-grade hardware without GPU acceleration, external multiple functions or relies on shared state, a single-function APIs, or internet connectivity. Removing both the cost and prompt does not provide sufficient context for reliable detection. connectivity requirements makes such systems accessible to Notably, Mistral outperforms LLaMA 3.2 on Indexing (+11.3 individual developers and resource-constrained teams. pp) and Error Handling (+12.5 pp) bugs, while LLaMA 3.2 Importantly, the BugsInPy dataset consists of real defects shows a slight advantage on Conditional Logic (+3.6 pp). These extracted from production-level open-source projects, which are
6
In such cases, understanding the bug requires broader contextual information beyond a single function, including interfunction dependencies and framework-specific behavior. This highlights a fundamental limitation of function-level analysis and suggests that incorporating additional context may be necessary for more accurate bug detection. D. Threats to Validity The primary threat to validity lies in the keyword-based evaluation approach. While efficient and fully automated, this method provides only an approximate measure of correctness and may both underestimate and overestimate true model performance. In particular, semantically correct responses that use different terminology may be misclassified as partial or incorrect, while coincidental keyword matches may lead to false positives. As a result, the reported accuracy should be interpreted as a conservative lower bound rather than an exact measure of model capability. Incorporating manual evaluation or semantic similarity metrics would provide a more reliable assessment. Additional threats include the limited scope of the evaluation. Although BugsInPy is widely used, it does not cover all programming styles or domains, and only two models are evaluated in this study. Expanding the analysis to additional datasets and models would improve the generalizability of the findings. Another limitation is that the prompt explicitly states that the function contains a bug. This may inflate model performance compared to real-world scenarios where the presence of a bug is not known in advance.
Fig. 5. Bug detection accuracy by bug type for both models. Numbers in parentheses indicate the count of bugs per category.
considerably more challenging than synthetic benchmarks. The fact that local models correctly identify over 40% of such bugs suggests that they can already provide meaningful assistance in practical debugging workflows. A key observation is the high proportion of partial responses, which account for nearly half of all outputs. In these cases, models successfully identify that a function contains problematic logic but fail to precisely locate the root cause or propose an exact fix. Although not fully correct, such responses remain practically useful, as they narrow the search space and highlight suspicious regions of code.
E. Practical Implications Even so, the results show that locally deployed LLMs can serve as effective first-pass debugging assistants. Rather than replacing developers, they are most useful for identifying suspicious code regions and providing initial diagnostic insights. The offline nature of local deployment is particularly valuable in privacy-sensitive environments, where source code cannot be shared externally. In addition, the absence of API costs makes such systems accessible to a wider range of users, including students, independent developers, and small teams.
B. Comparative Performance Although the overall accuracy difference between LLaMA 3.2 and Mistral is relatively small (approximately 1 percentage point), the results reveal substantial variation across projects. This indicates that model effectiveness is highly dependent on codebase characteristics rather than global model superiority. LLaMA 3.2 tends to perform better on projects with more structured and conventional code patterns, while Mistral shows competitive or superior performance on projects involving less predictable logic. These results suggest that the models exhibit complementary strengths, and that combining multiple models may be a promising direction for improving robustness.
VI. C ONCLUSION A. Summary of Findings In this paper, we presented a systematic empirical evaluation of locally deployed LLMs for real-world Python bug detection using the BugsInPy dataset. We evaluated two models, LLaMA 3.2 and Mistral, on 349 real bugs across 17 projects using a zero-shot prompting approach and automated evaluation. The results show that locally executed models achieve accuracy between 43% and 45%, while a large proportion of responses remain partially correct, indicating strong detection capability but limited precision in localization and repair. Performance varies significantly across projects, confirming that codebase characteristics play a critical role in model effectiveness. While both models perform well on structured code, they struggle in scenarios requiring broader context or complex reasoning.
C. Failure Cases The lowest performance is observed on projects such as tqdm, sanic, luigi, and spacy, where accuracy drops significantly. These projects typically involve complex abstractions, implicit dependencies, or interactions across multiple components.
7
B. Future Work
[16] C. Pushkar, S. Kabra, D. Kumar, and J. Challa, “Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detection,” arXiv preprint arXiv:2512.22306, 2025. [17] Q. Zhang, C. Fang, Y. Xie, et al., “A Systematic Literature Review on Large Language Models for Automated Program Repair,” arXiv preprint arXiv:2405.01466, 2025. [18] H. Touvron, T. Lavril, G. Izacard, et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302.13971, 2023. [19] Ollama, “Ollama,” 2023. Available: https://ollama.com [20] Q. McNemar, “Note on the sampling error of the difference between correlated proportions,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947, doi:10.1007/BF02295996.
Future work should focus on improving both evaluation methodology and model performance. Incorporating manual or semantically aware evaluation would provide more accurate estimates of true capability. Integrating richer context, such as multiple functions or project-level information, stands as a promising direction from a modeling perspective. Furthermore, the development of retrieval-augmented approaches and hybrid systems could significantly enhance the overall robustness of the bug detection process. Additional improvements may also be achieved through fewshot prompting or alternative inference strategies. Expanding the evaluation to additional models, datasets, and programming languages would further strengthen the generality of the findings. Overall, these results demonstrate that meaningful AIassisted debugging is already feasible on widely available consumer hardware. R EFERENCES [1] J. Achiam et al., “GPT-4 Technical Report,” arXiv preprint arXiv:2303.08774, 2023. Available: https://arxiv.org/abs/2303.08774. “Claude 3 Model Card,” Technical Re[2] Anthropic, port, 2024. Available: https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3. pdf. [3] Meta AI, “LLaMA 3.2 Model Card,” Technical Report, 2024. Available: https://www.llama.com/docs/model-cards-and-prompt-formats/ llama3_2/. [4] A. Jiang et al., “Mistral 7B,” arXiv preprint arXiv:2310.06825, 2023. Available: https://arxiv.org/abs/2310.06825. [5] TIOBE Index, “TIOBE Programming Community Index,” 2024. Available: https://www.tiobe.com/tiobe-index/. [6] R. Widyasari et al., “BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies,” in Proc. of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2020, pp. 1556–1560, doi: 10.1145/3368089.3417943. [7] M. Chen, J. Tworek, H. Jun, et al., “Evaluating Large Language Models Trained on Code,” arXiv preprint arXiv:2107.03374, 2021. [8] Z. Feng, D. Guo, D. Tang, et al., “CodeBERT: A Pre-Trained Model for Programming and Natural Languages,” in Proc. EMNLP, 2020, pp. 1536–1547. [9] S. Kang, G. An, and S. Yoo, “A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization,” arXiv preprint arXiv:2308.05487, 2023. [10] A. Mhatre, N. Nader, P. Diehl, and D. Gupta, “LLM-GUARD: Large Language Model-Based Detection and Repair of Bugs and Security Vulnerabilities in C++ and Python,” arXiv preprint arXiv:2508.16419, 2025. [11] E. G. Santana, M. Rehman, and F. Perez, “Evaluating LLMs Effectiveness in Detecting and Correcting Test Smells,” arXiv preprint arXiv:2506.07594, 2025. [12] J. Acharya and G. Ginde, “Enhancing Bug Report Quality Using LLMs,” arXiv preprint arXiv:2504.18804, 2025. [13] H. Lee, S. Sharma, and B. Hu, “Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks,” arXiv preprint arXiv:2406.15325, 2024. [14] F. Tambon, L. Smith, and J. Kim, “Bugs in Large Language Models Generated Code: An Empirical Study,” arXiv preprint arXiv:2403.08937, 2024. [15] F. Aguilar, S. Grayson, and D. Marinov, “Reproducing and Improving the BugsInPy Dataset,” in Proc. IEEE SCAM, Bogotá, Colombia, 2023, pp. 260–264, doi:10.1109/SCAM59687.2023.00036.
8