HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair Fazle Rabbi
Jinqiu Yang
Concordia University Montreal, Canada [email protected]
arXiv:2605.02215v2 [cs.SE] 5 May 2026
Abstract Recent Large Language Models (LLMs) have shown strong performance on automated program repair across standard benchmarks. However, these benchmarks evaluate models on a single canonical form of buggy code and do not reflect the syntactic variations commonly observed in real-world software, leaving robustness largely unexamined. In this work, we construct HEJ-Robust, a robustness benchmark built from HumanEval-Java-Bug using eight semanticspreserving code transformations, resulting in 1,450 transformed instances. We evaluate five fine-tuned LLMs on this benchmark and show that model performance drops by over 50% under several transformations, indicating that current LLM-based repair models lack robustness to minor syntactic variations.
CCS Concepts • Software and its engineering → Software testing and debugging; Source code generation; • Information systems → Language models.
Keywords Large Language Models, Automated Program Repair, Benchmark, Robustness Testing ACM Reference Format: Fazle Rabbi and Jinqiu Yang. 2026. HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair. In Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware ’26), July 6–7, 2026, Montreal, QC, Canada. ACM, New York, NY, USA, 5 pages. https://doi.org/ 10.1145/3805760.3814929
1
Concordia University Montreal, Canada [email protected] problem, where buggy code is translated into its fixed version [11, 39]. Pre-trained LLMs, such as PLBART [1], and CodeT5 [34], have shown strong repair capability when fine-tuned on bug-fix data [4, 10, 37]. More recent studies further explore instruction-tuned and agent-based LLMs for automated program repair [2, 8, 38]. To evaluate these approaches, existing benchmarks commonly rely on Defects4J [12] or HumanEval-Java-Bug [10], which assume a fixed syntactic representation of buggy programs. Existing APR benchmarks evaluate repair accuracy on a single canonical buggy program, ignoring syntactic diversity among semantically equivalent code. Prior studies show neural code models are sensitive to semantics-preserving transformations [24, 33]. While robustness testing via transformations, fuzzing, and adversarial examples has been studied in other SE tasks [22, 36], robustness evaluation for LLM-based APR on function-level, human-crafted benchmarks remains largely unexplored. A related effort, Defects4JTRANS [15], applies transformations to project-level real bugs in Defects4J; our benchmark is complementary, focusing on functionlevel, human-crafted bugs from HumanEval-Java-Bug. We address this gap by introducing a transformation-based robustness benchmark for automated program repair. Constructed by applying eight semantics-preserving transformations to HumanEvalJava-Bug [10], our benchmark enables controlled evaluation of repair consistency. We use it to assess the robustness of five fine-tuned LLM repair models against code perturbations. The contributions of this paper are as follows: (1) We introduce a transformation-based robustness benchmark built on HumanEval-Java-Bug, covering eight semanticspreserving transformations and providing function-level, test-executable robustness evaluation for APR. (2) We provide a systematic evaluation of LLM-based repair models under semantics-preserving transformations. (3) We release the benchmark to facilitate future research on robust and reliable automated program repair.
Introduction
Automated program repair (APR) aims to automatically generate patches that fix buggy programs. Early APR approaches primarily followed the generate-and-validate paradigm, where candidate patches are synthesized using predefined or learned repair operators and validated against test suites. Representative systems include GenProg [14], PAR [13], and systematic mutation-based repair techniques [23]. While these approaches have demonstrated effectiveness on specific bug classes, they often suffer from scalability limitations and test-suite overfitting. Recent advances in deep learning have significantly reshaped APR research by formulating program repair as a code translation
This work is licensed under a Creative Commons Attribution 4.0 International License. AIware ’26, Montreal, QC, Canada © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2601-9/2026/07 https://doi.org/10.1145/3805760.3814929
Our Code, dataset, and Artifacts are publicly available 1
2
Related Work
Automated program repair has been extensively studied over the past two decades. Early work primarily follows the generate-andvalidate paradigm, where candidate patches are generated and validated against test suites [14, 23]. While effective on curated benchmarks such as Defects4J [12], these approaches suffer from overfitting and scalability issues [35]. More recently, deep learning-based APR approaches reformulate bug fixing as a neural machine translation problem, translating 1 https://github.com/frabbisw/hej-robust
AIware ’26, July 6–7, 2026, Montreal, QC, Canada
Fazle Rabbi and Jinqiu Yang
Table 1: Fine-tuned models against different semantics-preserving code transformations on HumanEval-Java-Bug. (a) Local Variable Renaming (100 bugs)
plbart_base plbart_large codet5_small codet5_base codet5_large
(b) Method Renaming (149 bugs)
orig.
Pass@10 trans. change
CodeBLEU orig. trans.
14.53 21.88 19.35 24.81 23.66
6.54 9.91 8.26 12.28 11.5
82.11 82.75 82.17 82.02 80.74
54.99↓ 54.71↓ 57.31↓ 50.5↓ 51.39↓
81.91 82.15 81.43 81.58 80.92
plbart_base plbart_large codet5_small codet5_base codet5_large
orig.
Pass@10 trans. change
CodeBLEU orig. trans.
17.86 22.6 20.3 24.77 24.41
18.69 23.33 19.1 24.41 23.7
83.63 84.11 83.92 83.86 82.67
4.65↑ 3.23↑ 5.91↓ 1.45↓ 2.91↓
83.84 83.86 84.02 83.94 82.55
plbart_base plbart_large codet5_small codet5_base codet5_large
(e) Loop Exchange (142 bugs)
plbart_base plbart_large codet5_small codet5_base codet5_large
Pass@10 trans. change
CodeBLEU orig. trans.
19.32 25.26 21.55 26.04 28.28
18.39 23.66 17.92 23.66 26.8
84.66 84.77 84.08 84.58 83.29
84.76 85.5 84.61 85.12 84.36
plbart_base plbart_large codet5_small codet5_base codet5_large
(g) Insert Log Statement (173 bugs)
plbart_base plbart_large codet5_small codet5_base codet5_large
19.46 23.98 21.16 25.87 24.75
18.13 22.8 19.46 24.37 23.98
82.96 83.34 83.1 83.06 81.73
6.83↓ 4.92↓ 8.03↓ 5.8↓ 3.11↓
82.78 83.04 82.96 82.71 81.69
orig.
Pass@10 trans. change
CodeBLEU orig. trans.
12.5 22.22 22.22 22.22 12.5
22.22 30.0 22.22 12.5 12.5
86.03 86.14 83.68 86.02 85.26
77.76↑ 35.01↑ 0% 43.74↓ 0%
85.8 85.69 83.3 85.47 84.67
orig.
Pass@10 trans. change
CodeBLEU orig. trans.
16.88 21.41 19.7 23.25 23.45
15.69 18.48 17.92 20.99 21.62
83.83 84.17 84.01 83.94 82.56
7.05↓ 13.69↓ 9.04↓ 9.72↓ 7.8↓
85.64 86.1 85.87 85.75 85.04
(h) Insert Try catch (114 bugs)
orig.
Pass@10 trans. change
CodeBLEU orig. trans.
17.22 22.07 19.53 24.45 24.78
16.43 22.42 18.4 22.07 24.45
83.64 84.1 83.86 83.8 82.61
4.59↓ 1.59↑ 5.79↓ 9.73↓ 1.33↓
CodeBLEU orig. trans.
(f) Reorder Condition (603 bugs)
orig.
4.81↓ 6.33↓ 16.84↓ 9.14↓ 5.23↓
Pass@10 trans. change
(d) Boolean Exchange (7 bugs) †
(c) Parameter Renaming (162 bugs)
plbart_base plbart_large codet5_small codet5_base codet5_large
orig.
83.6 83.85 83.69 83.57 82.91
plbart_base plbart_large codet5_small codet5_base codet5_large
orig.
Pass@10 trans. change
CodeBLEU orig. trans.
16.91 21.53 19.29 25.17 26.14
13.74 19.29 11.02 18.12 18.12
83.86 84.17 84.52 84.29 83.16
18.75↓ 10.4↓ 42.87↓ 28.01↓ 30.68↓
83.65 83.94 84.49 84.3 83.52
† Results for Boolean Exchange are based on only 7 instances and are statistically underpowered; they are excluded from headline robustness claims.
buggy code into fixed code [11, 20, 32, 39]. Pre-trained LLMs further improve repair performance by leveraging large-scale code corpora before fine-tuning on repair data [1, 4, 5, 34, 37]. Most of these approaches evaluate on bug-fix pairs (BFPs) [5, 32], which largely consist of abstract or canonicalized code. More recent benchmarks derived from HumanEval [6] enable functional validation using test cases [10]. Complementary studies explore LLM-based repair in competitive programming and agent-based settings [2, 8, 38]. Parallel to APR research, robustness testing of neural models for code has gained attention. Prior work demonstrates that neural code
models are vulnerable to small, semantics-preserving transformations [28]. Transformation-based testing, fuzzing, and adversarial example generation have been applied to code models [22, 28], with later work emphasizing natural and context-aware transformations [36]. Works [24, 33] evaluate the robustness of code generation models under semantics-preserving perturbations. More recently, Defects4J-TRANS [15] evaluates the generalizability of LLMs in APR by applying transformations to project-level bugs in Defects4J. Defects4J-TRANS applies five transformations: variable renaming, loop transformation, switch transformation, dead code
HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair
injection, and boolean transformation. Our benchmark shares three of these (variable renaming, loop exchange, and boolean exchange), and adds five transformations not present in Defects4J-TRANS: method renaming, parameter renaming, condition reordering, log statement insertion, and try–catch insertion. Switch transformation was not applicable to HumanEval-Java-Bug as the dataset contains no switch statements. The two benchmarks are therefore complementary: Defects4J-TRANS targets project-level real bugs, while HEJ-Robust targets function-level, human-crafted bugs. While robustness has been studied for tasks such as code summarization and code representation learning, function-level robustness evaluation for automated program repair on human-crafted benchmarks remains largely unexplored. In particular, existing APR benchmarks do not systematically evaluate the robustness of repair models under semantics-preserving code transformations. Beyond program repair, LLMs have been applied to related code tasks, including code translation [26, 27, 31] and secure code generation [7, 16–18], with studies also revealing reliability concerns such as social bias in LLM-generated code [19, 25]. These findings collectively highlight that robustness and trustworthiness of LLM outputs remain open challenges across code tasks, motivating systematic evaluation frameworks such as the benchmark we propose.
3 Robustness Benchmark Design 3.1 Base Dataset We adopt the HumanEval-Java-Bug dataset introduced by Jiang et al. [10], which is derived from HumanEval [6]. The dataset contains 164 Java programs with manually injected bugs and annotated buggy-line locations. Each instance is accompanied by executable test cases and human-written patches. We select this dataset because it is manually curated, recent, and less likely to suffer from data leakage issues common in earlier APR benchmarks.
3.2
Semantics-preserving Code Transformations
We apply eight semantics-preserving code transformations that reflect common syntactic variations observed in real-world software. These transformations correspond to common coding practices such as identifier renaming during refactoring, loop restructuring for style conventions, and defensive programming patterns such as try-catch insertion and logging, making them representative of syntactic variation encountered in real-world Java development. The eight transformations are as follows: (1) Local variable renaming renames all the occurrences of a local variable. An LLM generates the new identifier of the variable. (2) Method renaming renames a method name using the same strategy as local variable renaming. (3) Parameter renaming renames a parameter using the same strategy as local variable renaming and method renaming. (4) Log statement insertion adds System.out.println("log") as the first code statement in a method. Since HumanEvalJava-Bug is a function-level benchmark, test cases validate return values rather than standard output; this transformation therefore does not affect test outcomes.
AIware ’26, July 6–7, 2026, Montreal, QC, Canada
Table 2: Examples of the eight semantics-preserving transformations. Transformation Before
After
Local var. ren.
int temp = 0;
int count = 0;
Method ren.
int compute()
int calculate()
Parameter ren.
void foo(int x)
void foo(int val)
Insert log
int foo(int x) {
int foo(int x) { System.out.println("log");
Insert try-catch
int r = f(s);
try { int r = f(s); } catch (Exception e) {}
Boolean exc.
boolean res = true; return res;
boolean res = false; return !(res);
Loop exc.
for (int i=0;i<n;i++)
int i=0; while (i<n) {...; i++;}
Condition re.
if (a == b)
if (b == a)
(5) Try-catch insertion adds a try-catch block at a random applicable code location. (6) Boolean exchange changes the initialization value of a boolean variable and wraps its subsequent uses in !(...) to ensure semantic equivalence; for example, boolean res = true; becomes boolean res = false; and any subsequent return res; becomes return !(res);. (7) Loop exchange replaces a for statement with an equivalent while statement and vice versa. (8) Condition reordering swaps the two operands of == and != expressions. Table 2 illustrates a before-and-after example for each of the eight transformations. Renaming transformations (1–3). For identifier renaming, we adopt the naturalness-aware substitution strategy proposed by Yang et al. [36]. Unlike prior approaches that use random strings or fixed patterns [22, 28], this method generates context-aware and developer-natural identifiers, ensuring that performance degradation reflects robustness issues rather than unnatural code artifacts. We use masked language prediction with CodeBERT and GraphCodeBERT to generate candidate identifiers and select substitutions based on cosine similarity in embedding space. Java code is parsed using tree-sitter [3] to ensure consistent replacement across all occurrences. To control transformation strength, only one identifier is renamed per program. Structural and syntactic transformations (4–8). The remaining transformations are implemented using JavaTransformer [29], which applies AST-based modifications via JavaParser. Transformations are applied only when syntactically valid. This yields 1,450 transformed instances in total.
3.3
Benchmark Construction and Task Formulation
After applying transformations, the locations of buggy lines may change. We manually re-annotate the buggy-line locations for all transformed programs by inspecting each transformed instance and mapping the original buggy statement to its updated position. Instances where a transformation directly modifies the buggy line
AIware ’26, July 6–7, 2026, Montreal, QC, Canada
itself are removed from the dataset to avoid ambiguity. To reduce annotation errors, a second author independently verified more than 10% of the re-annotated instances, and disagreements were resolved by discussion. Combined with the original human-written patches and test cases, this yields a fully executable benchmark suitable for robustness evaluation. Model outputs are evaluated using both code-similarity metrics, such as CodeBLEU [21, 30], and functional correctness via test-based metrics (e.g., pass@10) provided by HumanEval-Java-Bug [10]. The benchmark is extensible: new transformations can be added via JavaTransformer and re-annotated following the same protocol.
4
Experimental Setup
To evaluate the proposed benchmark, we consider five LLMs: two PLBART variants (base and large) and three CodeT5 variants (small, base, and large), all fine-tuned and released by Jiang et al. [10]. We select these models because they are the only publicly available fine-tuned APR models evaluated on HumanEval-Java-Bug, which allows us to study robustness of APR-specific models under our transformations without introducing confounds from retraining. We directly evaluate these models without any modification to their original training or decoding configuration. Following the evaluation protocol of Jiang et al. [10], we generate 10 candidate patches per bug and evaluate using Pass@10, where a bug is considered fixed if at least one generated patch passes all developer-written test cases. We do not modify seed settings, and we use the same generation procedure as provided in the original released models. Pass@10 serves as the primary metric for all robustness conclusions. CodeBLEU is reported as a reference control metric only; as our results confirm, it does not reliably capture functional robustness degradation.
5
Results
The evaluation results are summarized in Table 1, which reports the performance of five fine-tuned models across eight transformed datasets. Each transformation is presented in a separate subtable, showing Pass@10 and CodeBLEU scores for both the original and transformed datasets. We also report the relative change from the original to the transformed dataset, indicated by ↑ for improvements and ↓ for degradations. Across all eight transformations, we observe drops in Pass@10 for most models, with the largest degradation occurring under the Local Variable Renaming transformation, where performance decreases by 50.5% to 57.31%. This is likely because fine-tuned models rely heavily on identifier patterns learned during training; renaming local variables introduces distribution shifts in token sequences that disrupt the model’s ability to identify the buggy location and generate a correct patch. Similar behavior has been reported in code generation robustness studies [33]. In contrast, transformations such as Parameter Renaming and Insert Log Statement cause smaller drops, suggesting that method-level context or appended logging code is less disruptive to the repair process. Structural transformations such as Loop Exchange cause moderate drops, as they change control-flow structure while preserving variable names. Notably, robustness does not correlate with model size: larger models often degrade more than their smaller counterparts (e.g., CodeT5_large vs. CodeT5_base, and PLBART_large vs.
Fazle Rabbi and Jinqiu Yang
PLBART_base) across multiple transformations. Results for Boolean Exchange are excluded from headline claims due to only 7 applicable instances; a single prediction shift changes the percentage by over 12 points, making any reported change uninterpretable. The apparent improvement for plbart_base (+77.76%) should be interpreted as noise rather than a meaningful robustness signal. Two transformations show marginal improvements for some models. Under parameter renaming, plbart_base and plbart_la rge improve by 4.65% and 3.23% respectively. Although parameter renaming uses the same naturalness-aware substitution strategy as local variable renaming, the two transformations affect different parts of the code and may interact differently with the model’s learned repair patterns. We do not draw strong conclusions from these marginal improvements, as they may reflect noise or datasetspecific characteristics rather than a systematic robustness effect. In contrast, CodeBLEU scores remain largely stable across transformations, with only minor increases or decreases. This confirms that CodeBLEU is an unreliable indicator of robustness for APR: models can produce syntactically similar but functionally incorrect patches without any detectable drop in CodeBLEU. We recommend that future APR robustness benchmarks rely on execution-based metrics such as Pass@k.
6
Threats to Validity
Our benchmark is constructed entirely from HumanEval-Java-Bug, which consists of small, isolated function-level programs; findings may not generalize to project-level bugs involving inter-file dependencies. The eight transformations do not cover all possible syntactic variations, and not all transformations are applicable to every program, explaining the varying instance counts per transformation. For renaming, we rename only one identifier per program, which may introduce bias across instances with different numbers of variables. Finally, we evaluate models from only two fine-tuned LLM families, PLBART and CodeT5; results may not generalize to instruction-tuned or agent-based LLMs.
7
Conclusion
We present HEJ-Robust, a robustness benchmark of 1,450 bug instances constructed from HumanEval-Java-Bug using 8 semanticspreserving transformations. Evaluating five fine-tuned LLMs, we show that even minor syntactic variations cause consistent drops in Pass@10, revealing substantial robustness gaps in current repair models. Future work will study richer transformation spaces, project-level benchmarks with inter-file dependencies, evaluation of instruction-tuned and agent-based LLMs in zero-shot settings, and improved robustness-aware training and evaluation metrics.
Acknowledgement This research was supported by the Fonds de recherche du Québec (Grant No.2024-NOVA346499)[9], Natural Sciences and Engineering Research Council of Canada (NSERC) through the Alliance, Grant (Grant No.586838-23), the NSERC Discovery Grant (Grant No. RGPIN-2019-07007 and Grant No. DGECR-2019-00464), and NSERC CREATE Grant (Grant No.555406-2021). We gratefully acknowledge the support of all funding agencies.
HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair
References [1] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. arXiv:2103.06333 [cs.SE] https://arxiv.org/abs/2103.06333 [2] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. arXiv:2403.17134 [cs.SE] https://arxiv.org/abs/2403.17134 [3] Max Brunsfeld and contributors. 2024. tree-sitter. https://github.com/tree-sitter/ tree-sitter. Accessed: 2024-05-23. [4] Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, and Baishakhi Ray. 2022. NatGen: Generative pre-training by "Naturalizing" source code. arXiv:2206.07585 [cs.SE] https://arxiv.org/abs/2206.07585 [5] Saikat Chakraborty and Baishakhi Ray. 2021. On Multi-Modal Learning of Editing Source Code. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, Piscataway, NJ, USA, 443–455. [6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374 [cs.LG] https://arxiv.org/abs/2107.03374 [7] Cheng Cheng and Jinqiu Yang. 2025. CFCEval: Evaluating Security Aspects in Code Generated by Large Language Models. In 2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware). IEEE, Piscataway, NJ, USA, 01–10. [8] Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, Piscataway, NJ, USA, 1469–1481. [9] Fonds de recherche du Québec. 2024. FRQNT-NSERC NOVA Program, Grant No. 2024-NOVA-346499. https://doi.org/10.69777/346499 [10] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, Piscataway, NJ, USA, 1430–1442. [11] Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. CURE: Code-aware neural machine translation for automatic program repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, Piscataway, NJ, USA, 1161–1173. [12] René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis. ACM, New York, NY, USA, 437–440. [13] Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. 2013. Automatic patch generation learned from human-written patches. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, Piscataway, NJ, USA, 802–811. [14] Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. GenProg: A Generic Method for Automatic Software Repair. IEEE Transactions on Software Engineering 38, 1 (2012), 54–72. https://doi.org/10.1109/TSE.2011.104 [15] Fengjie Li, Jiajun Jiang, Jiajun Sun, and Hongyu Zhang. 2025. Evaluating the generalizability of llms in automated program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE, Piscataway, NJ, USA, 91–95. [16] Junjie Li, Fazle Rabbi, Cheng Cheng, Aseem Sangalay, Yuan Tian, and Jinqiu Yang. 2026. An exploratory study on fine-tuning large language models for secure code generation. Empirical Software Engineering 31, 4 (2026), 81. [17] Junjie Li, Fazle Rabbi, Bo Yang, Song Wang, and Jinqiu Yang. 2025. Prompt, Synthesize, Fine-Tune: A Secure Code Generation Recipe. arXiv:2510.07189 [cs.SE] https://arxiv.org/abs/2510.07189 [18] Junjie Li, Fazle Rabbi, Bo Yang, Song Wang, and Jinqiu Yang. 2025. Secure-Instruct: An Automated Pipeline for Synthesizing Instruction-Tuning Datasets Using LLMs for Secure Code Generation. arXiv:2510.07189 [cs.SE] https://arxiv.org/abs/2510. 07189 [19] Lin Ling, Fazle Rabbi, Song Wang, and Jinqiu Yang. 2025. Bias unveiled: Investigating social bias in LLM-generated code. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. AAAI Press, Washington, DC, USA, 27491–27499. [20] Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020. Coconut: combining context-aware neural translation models using ensemble for program repair. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, New York, NY, USA, 101–114. [21] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the
AIware ’26, July 6–7, 2026, Montreal, QC, Canada
40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Stroudsburg, PA, USA, 311–318. [22] Maryam Vahdat Pour, Zhuo Li, Lei Ma, and Hadi Hemmati. 2021. A search-based testing framework for deep neural networks of source code embedding. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, Piscataway, NJ, USA, 36–46. [23] Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. 2015. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In Proceedings of the 2015 International Symposium on Software Testing and Analysis. ACM, New York, NY, USA, 24–36. [24] Fazle Rabbi, Zishuo Ding, and Jinqiu Yang. 2025. A Multi-Language Perspective on the Robustness of LLM Code Generation. arXiv:2504.19108 [cs.SE] https: //arxiv.org/abs/2504.19108 [25] Fazle Rabbi, Lin Ling, Song Wang, and Jinqiu Yang. 2026. Social Bias in LLMGenerated Code: Benchmark and Mitigation. arXiv preprint (2026). https: //arxiv.org/abs/2605.00382 arXiv:2605.00382. [26] Fazle Rabbi, Soumit Kanti Saha, Tri Minh Triet Pham, Song Wang, and Jinqiu Yang. 2025. BabelCoder: Agentic Code Translation with Specification Alignment. arXiv:2512.06902 [cs.SE] https://arxiv.org/abs/2512.06902 [27] Fazle Rabbi, Soumit Kanti Saha, and Jinqiu Yang. 2026. Beyond Translation Accuracy: Addressing False Failures in LLM-Based Code Translation. arXiv preprint (2026). https://arxiv.org/abs/2605.02195 arXiv:2605.02195. [28] Md Rafiqul Islam Rabin, Nghi DQ Bui, Ke Wang, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour. 2021. On the generalizability of Neural Program Models with respect to semantic-preserving program transformations. Information and Software Technology 135 (2021), 106552. [29] Md Rafiqul Islam Rabin, Ke Wang, and Mohammad Amin Alipour. 2019. Testing neural program analyzers. arXiv:1908.10711 [cs.SE] https://arxiv.org/abs/1908. 10711 [30] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv:2009.10297 [cs.SE] https://arxiv.org/abs/2009.10297 [31] Soumit Kanti Saha, Fazle Rabbi, Song Wang, and Jinqiu Yang. 2024. SpecificationDriven Code Translation Powered by Large Language Models: How Far Are We? arXiv:2412.04590 [cs.SE] https://arxiv.org/abs/2412.04590 [32] Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM) 28, 4 (2019), 1–29. [33] Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. 2023. ReCode: Robustness evaluation of code generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Stroudsburg, PA, USA, 13818–13843. [34] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifieraware unified pre-trained encoder-decoder models for code understanding and generation. arXiv:2109.00859 [cs.SE] https://arxiv.org/abs/2109.00859 [35] Bo Yang and Jinqiu Yang. 2020. Exploring the differences between plausible and correct patches at fine-grained level. In 2020 IEEE 2nd International Workshop on Intelligent Bug Fixing (IBF). IEEE, Piscataway, NJ, USA, 1–8. [36] Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022. Natural Attack for Pre-trained Models of Code. arXiv:2201.08698 [cs.SE] https://arxiv.org/abs/2201.08698 [37] Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2022. CoditT5: Pretraining for Source Code and Natural Language Editing. arXiv:2208.05446 [cs.SE] https://arxiv.org/abs/2208.05446 [38] Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, and Zhenyu Chen. 2023. A critical review of large language model on software engineering: An example from chatgpt and automated program repair. arXiv:2310.08879 [cs.SE] https://arxiv.org/abs/2310.08879 [39] Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. 2021. A syntax-guided edit decoder for neural program repair. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, New York, NY, USA, 341–353.
Received 2026-02-15; accepted 2026-03-28