An Extensive Empirical Study on Code Translation Technique
Analyze & explore related research in Conceptio
Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
1
An Extensive Empirical Study on Code Translation Techniques
arXiv:2608.20776v1 [cs.SE] 21 Aug 2026
Ruihang Fan, Jiajun Jiang, Xinpeng Wang, Jiateng Fu, Fengjie Li and Jiasi Shen
Abstract—Automated code translation is increasingly important for software evolution, yet the relative strengths and limitations of learning-based and large language model (LLM)based techniques remain insufficiently understood. To address this gap, we conduct a large-scale empirical study comparing representative code translation techniques across methodological paradigms and translation granularities. We evaluate learningbased methods, LLM-based methods, and general-purpose LLMs on multilingual method-level and class-level benchmarks involving multiple programming languages. Our analysis considers executable correctness, code similarity, translation direction, translation granularity, and failure patterns. The results show that LLMs and LLM-based methods generally outperform learningbased methods in method-level correctness, although similarity metrics alone do not reliably reflect functional correctness. Translation direction substantially affects performance, particularly when translating between languages with different type-system characteristics. Class-level translation remains considerably more difficult than method-level translation because it requires preserving global semantics, interfaces, member relationships, and cross-method dependencies. Our error analysis further shows that static semantic errors and logical errors are the primary challenges in existing code translation systems. These findings provide empirical evidence and practical guidance for developing more robust, type-aware, structure-aware, and context-aware code translation techniques. Index Terms—Code translation, Program migration, Empirical study
I. I NTRODUCTION ODE translation, the process of automatically converting source code into functionally equivalent versions in other programming languages, has become a cornerstone of modern software evolution. It underpins critical tasks such as cross-platform program migration [1]–[3], legacy system modernization [4], [5], and library upgrades [6], [7]. As software systems evolve rapidly, the demand for such automated translation has grown substantially, since manual translation remains resource-intensive and time-consuming, while automation techniques can significantly reduce developer effort and maintenance costs [8], [9]. Consequently, automated techniques have emerged to address this need. Over the past few decades, a plethora of approaches have been proposed, ranging from early rigid, rule-based methods
C
Jiajun Jiang is the corresponding author. Ruihang Fan, Jiajun Jiang, Xinpeng Wang, Jiateng Fu and Fengjie Li are with School of Computer Software, College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. Jiasi Shen is with the Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong 999077, China. (E-mail: {frexvite, jiangjiajun, wxp3023244137, world 2004, fengjie}@tju.edu.cn; sjs@cse.ust.hk).
grounded in abstract syntax trees and pattern matching [10]– [15], to sophisticated deep learning models [16]–[24], and, most recently, techniques based on large language models (LLMs) [25]–[28]. In recent years, methods from different paradigms have achieved impressive results according to their respective metrics, yet the rapid pace of development has outpaced our understanding of their relative strengths and weaknesses. Although several empirical studies on code translation techniques have been conducted [29]–[34], they have typically adopted tools from only a single paradigm, i.e., either exclusively LLM-based methods or purely neural translation approaches. Consequently, there remains a lack of systematic, large-scale empirical research that performs genuine crossparadigm, cross-task level, and cross-dataset comparisons of existing code translation techniques under unified settings, in order to assess the current state of the art. To fill this gap, we conduct an extensive empirical study to systematically evaluate representative advanced techniques. Specifically, we first survey existing automated code translation approaches, primarily published between 2020 and 2025, and apply rigorous selection criteria, ultimately identifying 11 state-of-the-art representative techniques. These include four deep learning based methods (VIM-PT, StructCoder, TransCoder-ST, and a general-purpose code model, CodeT5+), four LLM-based methods (two variants of InterTrans, UniTrans, and ExeCoder), along with three recent general-purpose LLMs (GPT-3.5, GPT-4o-mini, and DeepSeek-V4-Flash). Finally, we conduct experiments on two method-level multilingual benchmarks and one class-level translation benchmark (G-TransEval, TransCoder-Uni, and ClassEval-T) to enable comprehensive and fair comparisons across code translation tasks involving five programming languages (C++, C#, Java, JavaScript, and Python). Through in-depth analysis of the experimental results, we derive a set of findings, among which we highlight the following findings: LLM-based methods generally achieve better correctness on method-level translation. LLMs and LLM-based methods generally outperform learning-based methods in CA and CSR. However, improvements over the corresponding base LLM are not uniform and depend on the method design, the capability of the base model, and the target task. The advantage of LLM-based methods is also less pronounced when measured by CodeBLEU. • Translation direction affects correctness. The direction of code translation has a clear impact on translation performance. Translating from statically typed languages to dynamically typed languages generally achieves better re•
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
sults than translating in the opposite direction, whereas the reverse direction tends to perform worse. This effect is also influenced to some extent by the inherent characteristics of the benchmark dataset. • Class-level translation remains substantially more difficult than isolated method-level translation. On ClassEvalT, methods obtain relatively high CSR but much lower CA, indicating that producing compilable class-level code is considerably easier than preserving class semantics, member relationships, and behavioral equivalence. • Language effects are pronounced in class-level translation. Within the class-level setting, translations targeting Python generally achieve higher CA than translations targeting Java or C++, while translation into C++ is particularly difficult in several directions. Larger contexts and crossmember dependencies amplify the challenges introduced by language-specific type and structural constraints. • Static semantic and logical errors are the dominant failure categories. Among the sampled method-level failed cases, static semantic errors account for 49.37% and logical errors account for 27.46%, together comprising 76.83% of all failures. At the class level, static semantic errors account for 70.67% of the sampled failures. Function signature violations are the most common static semantic error, highlighting the difficulty of type mapping, interface preservation, symbol binding, and logical equivalence. These findings lead to several implications for future research. In particular, code translation systems should move beyond isolated method-level translation, combine correctnessoriented and similarity-oriented evaluation, explicitly address direction-specific type inference and type mapping, and strengthen semantic verification for larger translation units. In summary, this paper makes the following contributions: To the best of our knowledge, we conduct the first largescale empirical study that evaluates code translation techniques across multiple methodological paradigms and translation granularities, including 11 representative techniques spanning learning-based methods, LLM-based methods, and general-purpose LLMs under unified experimental settings. • We perform a cross-paradigm, cross-dataset, and crossgranularity evaluation using two method-level multilingual benchmarks and one class-level translation benchmark. This enables controlled comparisons across translation methods, programming languages, translation directions, and translation units. • We provide a multi-dimensional analysis of code translation performance based on correctness and similarity metrics, including an investigation of language-direction effects, benchmark characteristics, and the differences between method-level and class-level translation. • We conduct a systematic manual analysis of translation failures. We annotate 570 method-level failed outputs and 150 class-level failed outputs using a structured error taxonomy, with high inter-rater agreement measured by Fleiss’ kappa. The analysis identifies static semantic errors, logical errors as major challenges. • We summarize a set of findings and implications for •
2
developing more robust, type-aware, and context-aware code translation techniques, and open-source the experimental results and associated implementations to facilitate replication and future comparison. II. R ELATED W ORK A. Code Translation Methods Numerous code translation methods have been proposed to automatically convert source code written in one programming language into another functionally equivalent code in another language. Existing approaches can be broadly categorized into rule-based, learning-based, and LLM-based methods. We also provide a brief summary of the recent representative methods in Table I, showing the publishing year, supported languages, the benchmarks used for evaluation, and whether the source code is open-source. Rule-based methods perform translation through handcrafted rules that explicitly map source-language constructs to their counterparts in the target-language. Represent works include MetaLift [35], DuoGlot [12], CxGo [15], Sharpen [10] and C2Rust [11]. These methods typically operate by parsing source code into structured intermediate forms and then applying predefined transformation rules to produce targetlanguage code. However, most of them are language-specific and require substantial manual effort to construct and maintain the translation rules. As a result, they are difficult to adapt to newly introduced features, evolving language standards, and the translated code may still require manual refinement [36]. Learning-based methods learn translation patterns from data instead of relying on hand-crafted rules. Early machine learning–based approaches to code translation were largely inspired by statistical machine translation (SMT). Nguyen et al. [8] investigated whether SMT techniques developed for natural languages could be adapted to programming languages. Karaivanov et al. [37] explored phrase-based methods and demonstrated the feasibility of SMT-based code translation. Nguyen et al. [2] further proposed mppSMT, a divide-andconquer method that achieved strong performance at that time. Later, deep neural networks were introduced for code translation. Chen et al. [38] proposed a tree-based neural model for translating between JavaScript and CoffeeScript. However, such supervised approaches are often constrained by the availability and quality of parallel corpora. To address this limitation, Roziere et al. [9] introduced TransCoder, a fully unsupervised neural transcompiler. And follow-up works further improved it with unit test filtering and intermediate representations [23], [39]. LLM-based methods have recently emerged as a promising direction for code translation. Unlike earlier learningbased approaches that mainly depend on task-specific training, these methods typically leverage the reasoning and iterative refinement abilities of LLMs to improve translation quality. For example, Jana et al. [26] presented CoTran, which finetunes an LLM with reinforcement learning guided by compiler and symbolic execution feedback, directly optimizing for compilation correctness and functional equivalence. Zheng et al. [42] introduced CodeGeeX, a large multilingual pretrained model capable of both code generation and translation
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
3
TABLE I: A brief summary of code translation techniques. Name TransCoder [9]
Rule-based
LLM-based
Learning-based
PLBART [40] TransCoderST [23]⋆
Year↑ 2020 2021 2021
Supported Languages C++, Java, Python C#, Java C++, Java, Python C++, Go, Java, Rust C#, Java C#, Java C++, Java, Python Java, Python C#, Java C/C++, C#, Java, JavaScript, Python, PHP
TransCoder-IR [39]
2022
NatGen [22] Contrabert [24]
2022 2023
SDA-Trans [1]
2023
SumGenToBT [41] StructCoder [19]⋆
2023 2024
VIM-PT [20]⋆
2024
CodeT5 [17] CodeT5+ [18]⋆ CodeGeeX [42]
2021 2023 2023
FSCTrans [43]
2024
Mftcoder [44] CodeFuse-13B [45]
2024 2024
multiple Multiple Multiple C#, Java, Python Multiple Multiple
TransAGENT [28]
2024
Multiple
CoTran [26] UniTrans [27]⋆ InterTrans [25]⋆
2024 2024 2025
ExeCoder [46]⋆
2025
Multiple Multiple Multiple C++, Java, Python
Sharpen [10] C2Rust [11] CxGo [15]
2008 2018 2021
DuoGlot [12]
2023
MetaLift [35]
2023
C#, Java C++, Rust C++, Go Python, JavaScript C++, Java
Benchmark D1 D4 D1
OSS ✓ ✓ ✓
TABLE II: Benchmarks for code translation evaluation. Name
Year↑
Languages
Probs.
Tests
D1. TransCoder-test [9]
2020
C++, Java, Python
948
✓
D2. CodeNet [47]
2021
C/C++, Go, Java, Python
200
✗
D3. AVATAR [48]
2021
Java, Python
250
✗
D4. CodeXGLUE [49]
2021
C#, Java
1000
✗
D5. CoST [50]
2022
C/C++, C#, Java, JavaScript, Python, PHP
1625
✗
D1
✓
D4 D4
✓ ✓
D6. XLCoST [51]
2022
C/C++, C#, Java, JavaScript, Python, PHP
11265
✓
D1
✗
D7. G-TransEval [31]
2023
C++, C#, Java, JavaScript, Python
400
✓
D1 D4
✓ ✓
2023
C++, Go, Java, JavaScript, Python
164
✓
D5
✓
D9. TransCoder-Uni [27]
2024
C++, Java, Python
568
✓
D10. AVATAR-TC [52]
2024
Java, Python
1746
✓
D4 D4 D8, D6
✓ ✓ ✓
D11. ClassEval-T [30]
2025
C++, Java, Python
94
✓
D1, D4
✓
CodeFuseEval CodeFuseEval TransAGENTtest D10 D9 D8, D1, D2 TransCodertest-X
✓ ✓
-
✓ ✓ ✓
D1
✓
Spark, Domino
✓
✗ ✓ ✓ ✓ ✓
“⋆”: studied methods in this paper. “-”: the tool did not explicitly evaluate on any benchmarks, and “Dk” denotes dataset shown in Table II.
across 23 programming languages, with its evaluation benchmark HumanEval-X extending beyond Python to multiple languages. Yang et al. [27] proposed UniTrans, which enhances translation through automated test-driven feedback. Yuan et al. [28] introduced TranAgent, a multi-agent framework that uses multiple LLMs to correct syntactic and semantic errors during translation. Macedo et al. [25] developed InterTrans, which leverages intermediate languages to bridge syntactic gaps between source and target languages. Additionally, a wide range of LLMs for code generation can also be applied to code translation tasks, such as CodeBERT [16], CodeT5 [17], CodeT5+ [18], and MftCoder [44]. Several of these models have achieved strong performance on code translation benchmarks and are even regarded as competitive baselines—CodeT5+ being a notable example. B. Code Translation Benchmarks Over the past few years, several benchmarks have been proposed for evaluating code translation methods. CodeNet [47] provides large-scale parallel programs collected from online coding platforms, covering 55 different programming languages. Ahmad et al. introduce Avatar [48], which focuses on more challenging Python–Java translation tasks. CodeXGLUE [49] provides a code-to-code translation benchmark based on parallel Java–C# programs. Although they
D8. HumanEval-X [42]
*“Probs” denotes the number of programming problems covered in the dataset.
cover a wide range of programming languages, many of them lack executable test cases, which limits the reliable evaluation of the functional correctness of translated code. More recent benchmarks address this issue by providing executable test suites. Roziere et al. [23] proposed a benchmark (referred to as TransCoder-Uni) from GeeksforGeeks programs in Python, C++, and Java, and 568 out of 948 cases include test suites. Jiao et al. [31] introduced G-TransEval, which contains 400 parallel translation instances across five languages, along with difficulty labels and complete test suites for every language pair. Xue et al. [30] proposed ClassEvalT, a class-level code translation benchmark covering Python, C++, and Java, featuring longer code snippets, complex dependencies, and comprehensive test suites. We list several recognized benchmarks in the field of code translation in Table II. After our verification, all benchmarks in the table have been used by at least two works in the code translation domain.
C. Empirical Studies on Code Translation To investigate the effectiveness of existing code translation techniques, several studies have been conducted. Jiao et al. [31] proposed a four-type taxonomy for code translation and systematically evaluated state-of-the-art neural translation models across these translation types. Gong et al. [53] studied the execution efficiency of LLM-based code translation, and Pan et al. [32] studied the effectiveness of LLMs for automated code translation and analyzed LLM-introduced translation bugs. UniTrans [27] further enhanced code translations through test generation, execution-based checking, and iterative repair. Tao et al. [34] further introduced PolyHumanEval and used it to conduct a large-scale empirical study of LLMbased code translation. However, these studies either focus on neural or LLM-based (especially focus on end-to-end LLM methods) methods or are limited to a small set of programming languages and evaluation perspectives. In contrast, our work aims to compare both recent learning-based and LLM-based methods under a unified setting and provides a more comprehensive evaluation from multiple perspectives, including quantitative metrics and error type analysis.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
III. M ETHODOLOGY A. Code Translation Method Selection As mentioned in the introduction, existing studies predominantly focus on individual paradigms (e.g., LLM-based methods, especially focus on end-to-end LLMs), while a systematic comparison of different approaches under a unified setting remains underexplored. As discussed in Section II-A, numerous code translation methods have been proposed in the past decade. This study aims to comprehensively compare the performance of the latest state-of-the-art techniques. Accordingly, we focus on methods published between 2020 and 2025. We further limit our scope to open-source approaches, primarily for two reasons: (1) reproducing a large number of methods would require significant time and effort, and (2) reimplementations often fail to achieve the performance reported in the original papers, potentially confounding the evaluation of each method’s intrinsic capabilities. Finally, we selected the representative methods from two categories, i.e., learning-based and LLM-based approaches (listed in Table I), to ensure the feasibility of our study and the reliability of our experimental results. To identify the most representative approaches within these categories, we applied the following three selection criteria. 1) Performance (outstanding performance): Among methods developed under comparable timelines, we prioritize those achieving best performance on at least one benchmark, ensuring that selected systems genuinely represent the current best performance within their respective technical paradigms. 2) Recency (latest techniques): Higher preference is given to models and methods released or updated after 2024, so that our experiments can reflect the latest advancements in the field of code translation. 3) Language diversity (applicable to diverse languages): We favor methods supporting the largest number of programming languages, aiming to obtain more comprehensive and generalizable experimental results. Based on the above criteria, we ultimately selected eight state-of-the-art code translation methods from existing studies. These methods include four learning-based approaches (three translation-specific methods: VIM-PT, StructCoder, TransCoder-ST, and one general-purpose code model: CodeT5+), as well as four LLM-based methods (two variants of InterTrans, UniTrans, ExeCoder). The rationale is as follows: VIM-PT, StructCoder, UniTrans, ExeCoder and InterTrans are the best-performing open-source methods in recent years in the fields of learning-based code translation and LLM-based code translation, respectively (TransAGENT surpasses UniTrans but is not open-source). TransCoder-ST and CodeT5+ have been employed as classic competitive baselines in prior code translation studies, and we include them in our experiments to obtain more representative results [19], [26], [27], [43]. In addition, to conduct a comprehensive evaluation, we also incorporate three recent general-purpose LLMs into our study, which have demonstrated outstanding performance across various software engineering tasks [54]–[56], including two commercial LLMs (GPT-3.5-turbo and GPT-4o-mini) and
4
one open-source LLM (DeepSeek-V4-Flash). Consequently, we investigate 11 representative approaches in our evaluation, which (except for LLMs) are marked with ⋆ in Table I and described in detail below. VIM-PT [20]: A unified multilingual program translation model based on variational inference and information disentanglement. It jointly learns language-shared semantics and language-specific features, effectively leveraging nonparallel and partially missing data to significantly improve accuracy and generalization in cross-language code translation. StructCoder [19]: A structure-aware Transformer model that explicitly incorporates abstract syntax trees (ASTs) and data flow graphs (DFGs) into both encoder and decoder. It further enhances performance on code translation and textto-code generation by introducing two auxiliary tasks in the decoder: AST path prediction and data flow prediction. TransCoder-ST [23]: An unsupervised code translation approach based on self-training. It automatically generates high-coverage unit tests for source code and uses these tests to filter semantically equivalent translations, thereby constructing high-quality pseudo-parallel corpora for iterative model refinement. CodeT5+ [18]: A model featuring that supports encoderonly, decoder-only, and encoder-decoder modes within a unified architecture. It is trained with multi-stage objectives, including span denoising, causal language modeling, contrastive learning, and text-code matching, and achieves strong performance across code understanding and generation tasks. InterTrans [25]: An LLM-based code translation method that constructs an “intermediate-language translation path” and employs tree search combined with test verification to substantially improve functional correctness in crosslanguage translation. In our experiments, we use InterTrans implementations based on Magicoder-SDS-7B [57] and StarCoder2-15B [58], referred to hereafter as IT(Magi) and IT(Star), respectively. UniTrans [27]: A general-purpose LLM-based code translation framework that validates and repairs translation outputs by generating and executing test cases. It further iteratively refines prompts using error feedback to enhance both functional correctness and robustness in cross-language code translation. ExeCoder [46]: An LLM specifically designed for code translation, aimed at utilizing executability representations such as functional semantics, syntax structures, and variable dependencies to enhance the capabilities of LLMs in code translation. GPT-3.5-turbo [59]: An efficient and cost-balanced LLM from OpenAI. It was widely used as a baseline for code understanding and generation tasks in previous studies [27]. Specifically, we adopted the version of GPT-3.5-turbo-1106 in our experiment. GPT-4o-mini [60]: A lightweight OpenAI model offering low latency and strong reasoning capabilities, ideal for real-time applications and code generation. Specifically, we
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
employed the version of GPT-4o-mini-2024-07-18 in our experiment. DeepSeek-V4-Flash [61]: A high-performance model from DeepSeek featuring a Mixture-of-Experts architecture, delivering strong results in code and reasoning tasks with high cost-effectiveness. B. Benchmark Selection As introduced in Section II-B, a number of benchmarks for code translation have been proposed, as shown in Table II. Similarly, we also applied several selection criteria analogous to those used for the studied method selection: 1) Multilingual (covering diverse languages): To enable a comprehensive comparison among diverse methods and investigate their translation performance when applied to different programming languages of the same task (A task denotes a distinct function regardless of implementation languages), we prefer the benchmarks that have more tasks and involve more parallel implementations of different languages. 2) Verifiability (including test suite): Since the same functionality can be implemented in different ways, directly comparing the equivalence of the translated code to the ground-truth implementation is insufficient. To enable a more in-depth analysis and make the evaluation results verifiable in an automatic manner, we require the benchmark including associated test suites, which play as the (incomplete) semantic specification of the target code, and thus can automatically and reliably examine the functional correctness of the translated code. 3) Compatibility (applicable to studied methods): We require that the benchmarks should accommodate as many of the selected methods introduced in Section III-A, facilitating more generalizable conclusions rather than findings limited to a single dataset. According to these selection criteria, we ultimately selected two representative method-level code translation benchmarks in this study, G-TransEval and TransCoder-Uni (the cleaned version of the TransCoder-ST dataset prepared by Yang et al. [27]), as well as one class-level code translation benchmark, ClassEval-T [30], involving 1,062 distinct translation tasks in total. All three benchmarks feature comprehensive test suites and multilingual parallel implementations. For the methodlevel benchmarks, they are applicable to nearly all of our studied methods, while the class-level benchmark is also compatible with most large language models and LLM-based approaches (learning-based methods are difficult to adapt due to context length limits). We also emphasized these three datasets in bold in Table II. We exclude XLCoST from testtime evaluation because it is used for fine-tuning the learningbased methods in our study (Section IV-C). IV. E XPERIMENTAL S ETUP A. Research Questions In this study, we aim to answer the following research questions:
5
RQ1: How do existing methods perform overall on method-level code translation tasks? To answer this question, we conduct extensive experiments on two largescale benchmarks and compare the overall correctness of all selected methods. Specifically, we evaluate the translated code using compilation and functional correctness metrics, and further analyze the results with CodeBLEU for a multidimensional view of translation quality. RQ2: How do existing methods perform on different translation tasks and language directions? Since the same method may behave differently when translating between different language pairs or in opposite directions, we further investigate task-level performance variations across sourcetarget language combinations. This RQ aims to identify language-direction effects and examine how programming language characteristics influence translation performance. RQ3: How does code translation perform at the class level? Compared with method-level translation, class-level translation involves longer contexts, richer inter-method dependencies, and more complex type structures. In this RQ, we extend the evaluation to class-level code translation and study whether existing methods can still preserve correctness under more complex code contexts. RQ4: What kinds of translation errors do current methods commonly make? Although existing methods achieve strong performance overall, they still produce incorrect outputs. To better understand these failures, we manually analyze translation errors produced by the selected methods, categorize them into major error types, and examine their distributions across datasets and methods. B. Evaluation Metrics We use three evaluation metrics: Computational Accuracy (CA), Compilation Success Rate (CSR), and CodeBLEU (CB) [62]. Following prior work [9], [23], [27], CA measures the proportion of generated programs that pass all official test cases, while CSR measures the proportion that compile successfully. CodeBLEU measures the similarity between generated code and reference code by combining n-gram matching, keyword matching, abstract syntax tree matching, and data flow matching. We use the standard setting in which all four components are equally weighted. We collectively refer to CA and CSR as correctness metrics, since they measure the functional and syntactic correctness of generated code. We refer to CodeBLEU as a similarity metric, since it evaluates how closely the generated code matches the reference implementation in textual and structural terms. C. Implementation and Configuration Learning-based Methods. For TransCoder-ST [23], we used the best checkpoint provided by the original authors. For VIM-PT [20], StructCoder [19], and CodeT5+ [18], we followed prior work and trained/fine-tuned the models on the XLCoST [51] dataset. In addition, because TransCoderUni and XLCoST are derived from the same data source, we removed from the training set any examples that overlap
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
6
TABLE III: Configurations of hyperparameters for model training. Hyperparameter Parameter size Learning rate Batch size Gradient accumulation steps Epoch *
StructCoder
VIM-PT
CodeT5+
∼ 224M 1e-5 25 / 25
∼ 222M 1e-4 8 2 25*
∼ 220M 5e-5 8 4 10
VIM-PT includes 10 auto-encoding epochs and 15 formal epochs.
with our evaluation benchmarks. The training hyperparameters are listed in Table III, and all settings follow the default configurations of the corresponding open-source repositories. For each method, we evaluated using the checkpoint that achieved the best performance on the validation set. This choice was motivated by two factors: 1) learning-based methods exhibit less randomness compared to LLMs; 2) repeated training would incur substantial computational costs (e.g., VIM-PT training took approximately 1,266 GPU hours). LLM-based Methods. The officially released UniTrans [27] cannot be directly executed on G-TransEval [31] due to the tight coupling of part of its code with the source dataset. We therefore adapted the modules for general translation enhancement and translation repair to evaluate and rectify translation outputs generated by LLMs. For each method, we follow prior works [27], [30] and set the hyperparameters to temperature = 0.8, top p = 0.9, and n = 1, i.e., only the first generated output was used. We then conducted three independent runs and reported the average results. Experimental environment Our experiments were conducted on a local machine equipped with dual Intel Xeon 6388 CPUs, 512GB RAM, and two NVIDIA A800-SXM4-80GB GPUs, running Ubuntu 20.04.6LTS. V. R ESULTS A NALYSIS A. RQ1: Method-Level Code Translation Performance In this research question, we evaluate the overall performance of different approaches on two large-scale benchmarks. Specifically, we consider the first output of each method by default, namely CA@1 and CSR@1, and further use CodeBLEU as a similarity metric for multi-dimensional evaluation. Table IV and Table V report the correctness results of different tools across all language pairs on G-TransEval [31] and TransCoder-Uni [27], respectively. Figure 1 and Figure 2 show the distributions of different metrics for all methods. Detailed CodeBLEU results are available on the project website. In terms of correctness metrics, LLM-based methods, including general-purpose LLMs and methods enhanced for code translation, generally outperform state-of-the-art learning-based methods across metrics and datasets. Specifically, except for IT(Star), most LLM-based methods maintain high average CA and CSR scores on both datasets, clearly outperforming traditional learning-based methods. In contrast, the best-performing learning-based method, TransCoder-ST, achieves CA and CSR scores of only 56.96% and 70.50%,
respectively, which are much lower than those of most LLMbased methods. Overall, among all translation tasks, UniTrans achieves the best performance, with average CSR (CA) scores of 96.01% (94.36%) on G-TransEval and 96.59% (91.66%) on TransCoder-Uni, significantly outperforming not only learning-based methods but also other LLM-based approaches. Among all methods that enhance performance using large language models, UniTrans improves the CSR (CA) over its base model (GPT-3.5-turbo) by 1.18 percentage points (1.13 percentage points) and 1.44 percentage points (1.57 percentage points) on the two datasets, respectively, while ExeCoder and IT (Magi) achieve performance close to that of GPT-4o-mini. This suggests, to some extent, that current LLM methods designed for enhancing code translation tasks generally achieve stable and relatively high correctness, but the extent of improvement over the base LLM still depends on the specific method design, the capability of the base model, and the particular task at hand. Finding 1: In method-level code translation tasks, LLMs and LLM-based methods significantly outperform learning-based methods in correctness metrics. Methods enhanced for code translation usually achieve strong performance, but their improvements over base LLMs depend on the specific method design, the capability of the base model, and the target tasks. In terms of similarity metrics, almost all methods achieve average CodeBLEU scores above 70.00% on both datasets. However, the advantage of LLM-based methods on this metric is less pronounced than on correctness metrics. Specifically, although learning-based methods usually obtain lower CA scores, their similarity scores are close to those of LLM-based methods. For example, StructCoder and VIM-PT perform poorly in correctness metrics, but their similarity scores are close to those of UniTrans. This indicates that learning-based methods often generate outputs that are superficially similar to the reference translations but fail to compile or pass tests. To further investigate this phenomenon, we group all passed and failed outputs generated by different methods and compute their CodeBLEU scores. We exclude InterTrans to avoid the impact of empty parser outputs on the calculation. As shown in Figure 3, the average CodeBLEU gap between passed and failed samples is 15.96% to 28.32% for learning-based methods, while it is only 9.51% to 13.25% for LLM-based methods, which is substantially smaller. This suggests that LLM-based methods tend to generate functionally correct outputs even when their surface similarity to the reference is relatively low, indicating that they focus more on understanding the semantic functionality of source programs rather than performing line-by-line translation. Combining correctness and similarity analyses, we conclude that recent advances in LLMs have dominant effectiveness in code translation: they consistently outperform learning-based methods, provide more robust translation results, and emphasize functional equivalence rather than surface similarity.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
7
TABLE IV: Translation performance on G-TransEval across methods. The table shows results for both translation directions (e.g., Java to C# and C# to Java) between each pair of languages (e.g., Java⇌C#). Empty cells indicate that the corresponding method is not applicable. A color gradient is used, darker shades highlight larger values and thus better performance (%). Name
DS-V4-Flash GPT-3.5-turbo GPT-4o-mini UniTrans
Java⇌C#
Java⇌C++
Java⇌Py.
Java⇌JS
C++⇌Py.
C++⇌JS
C++⇌C#
Py.⇌C#
CSR
CA
CSR
CA
CSR
CA
CSR
CA
CSR
CA
CSR
CA
CSR
CA
CSR
CA
CSR
CA
CSR
CA
99.17 99.92 98.92 99.92 96.75 99.00 99.25 100.00
98.92 99.92 98.58 99.67 96.33 98.33 96.08 99.83
92.67 99.75 94.42 99.17 90.00 97.50 99.42 100.00 81.50 92.92 94.33 93.00 70.08 81.17
92.67 99.75 94.25 99.17 88.58 96.67 99.25 99.67 77.58 90.83 91.17 90.83 66.67 78.83
99.75 93.25 99.42 92.83 99.08 85.67 99.50 95.50 98.83 82.92 92.75 92.75 75.67 78.67
99.58 91.75 98.92 90.33 96.17 81.58 99.17 93.17 94.08 74.92 87.25 85.58 66.83 70.92
99.00 92.17 99.50 91.92 98.58 88.00 99.75 93.83
97.42 88.92 97.50 89.58 93.17 84.17 93.17 91.17
96.92 74.17 96.67 77.42 92.25 71.67 92.92 90.42
98.33 93.17 98.83 96.08 91.75 95.08 99.33 99.33
98.08 92.83 98.17 95.67 91.00 92.67 96.67 99.00
88.17 99.25 89.42 99.33 68.17 98.67 88.08 99.42
86.92 99.08 86.42 98.75 64.42 95.75 83.58 98.83
96.08 99.50 96.08 99.58 95.50 97.92 97.58 99.75
93.33 99.50 91.17 99.25 86.67 96.08 93.92 99.33
97.67 80.00 98.83 87.58 98.33 69.00 98.83 86.75
96.50 75.42 96.42 84.50 92.33 65.92 97.00 82.42
90.50 89.17 58.17 77.08
99.25 76.33 96.17 76.08 94.75 62.25 95.75 89.67 92.25 59.83 92.25 68.67 54.92 45.17
98.67 77.58 99.25 79.92 98.58 74.17 99.33 87.58
97.50 93.75 69.50 81.17
99.42 77.67 96.75 78.92 97.33 67.00 97.58 92.25 96.33 67.75 97.67 77.00 61.50 51.67
97.25 75.75 59.58 51.75
90.42 68.67 48.08 47.00
94.00 84.75 59.75 48.33
91.58 81.50 58.08 46.00
83.08 98.50 55.08 68.58
74.83 93.75 51.08 60.67
94.25 93.33 66.83 70.83
80.75 89.00 53.42 67.42
97.33 81.00 61.58 52.42
90.58 91.48 85.90 73.92 48.83 65.17 59.30 50.00
51.25 31.00 84.00 70.00 84.25 55.00
21.75 24.75 70.25 52.50 63.75 45.50
66.50 43.75 47.25 40.00 35.50 29.25 71.00 53.25 81.75 63.00 83.25 55.50 48.50 50.75
17.75 19.25 60.50 35.75 66.50 39.50
61.75 42.25 49.00 44.25 33.50 24.00 50.50 26.75 63.50 43.25 48.10 34.30 46.00 40.75 53.25 48.00 50.75 20.00 50.00 13.50 43.00 40.00
ExeCoder
95.42 IT(Magi) 92.17 65.17 IT(Star) 71.67 20.50 StructCoder 18.00 52.00 52.25 48.00 VIM-PT 40.75 44.75 39.75 76.00 64.75 TransCoder-ST 68.75 58.00 75.00 67.25 62.75 54.00 CodeT5+ 57.75 53.25 60.50 55.25
C#⇌JS
Average CSR
CA
94.06 92.86 94.83 93.23 90.30 87.04 96.01 94.36 86.71 81.58
22.88 19.25
70.50 56.96 79.50 59.75 71.50 64.25 60.25 51.25 80.25 56.50 79.00 59.25 68.80 56.06 52.25 44.00 58.00 49.00 86.25 65.00 81.00 63.00 61.50 52.75
& $
& 6 5
' 6
' 6
' 6
* 3 Y ) 7 O D V K * 3 W 7 X U E 8 Q R P L R L 7 Q ( [ U D Q V L H & , 7 R G H U 0 , 7 D J L 6 W U 6 W D U X F W 9 & R G 7 U , 0 H U D Q V & 3 7 & R R G H U G H 7 6 7
* 3 Y ) 7 O D V K * 3 W 7 X U E 8 Q R P L R L 7 Q ( [ U D Q V L H & , 7 R G H U 0 , 7 D J L 6 W U 6 W D U X F W 9 & R G 7 U , 0 H U D Q V & 3 7 & R R G H U G H 7 6 7
* 3 Y ) 7 O D V K * 3 W 7 X U E 8 Q R P L R L 7 Q ( [ U D Q V L H & , 7 R G H U 0 , 7 D J L 6 W U 6 W D U X F W 9 & R G 7 U , 0 H U D Q V & 3 7 & R R G H U G H 7 6 7
& R G H % / ( 8
97.08 94.50 66.25 73.08 24.25 21.50 56.25 42.25
Py.⇌JS
& $
& 6 5
' 6 9 * 3 ) O D 7 V K * 3 W X U 7 E R R 8 Q P L Q L 7 U L , 7 D Q V 0 D , 7 J L 6 W D 9 , U 0 7 U 3 7 D Q V & R G & R H U 6 G H 7 7
' 6 9 * 3 ) O D 7 V K * 3 W X U 7 E R R 8 Q P L Q L 7 U L , 7 D Q V 0 D , 7 J L 6 W D 9 , U 0 7 U D Q 3 7 V & R G & R H U 6 G H 7 7
' 6 9 * 3 ) O D 7 V K * 3 W X U 7 E R R 8 Q P L Q L 7 U L , 7 D Q V 0 D , 7 J L 6 W D 9 , U 0 7 U D Q 3 7 V & R G & R H U 6 G H 7 7
& R G H % / ( 8
Fig. 1: Result distribution regarding all metrics for each method on G-TransEval.
Fig. 2: Result distribution regarding all metrics for each method on TransCoder-Uni.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
8
TABLE V: Translation performance on TransCoder-Uni. Name
Java⇌C++
Java⇌Py.
C++⇌Py.
CSR
CSR
CSR
97.57 94.95 97.29 GPT-3.5-turbo 94.33 97.29 GPT-4o-mini 89.14 97.86 UniTrans 95.30 91.94 ExeCoder 99.59 96.93 IT(Magi) 95.71 62.38 IT(Star) 52.56 76.02 VIM-PT 71.16 92.72 Transcoder-ST 75.10 85.65 CodeT5+ 81.33 DS-V4-Flash
CA 94.93 92.05 94.00 91.70 92.43 86.72 94.86 92.74 87.22 96.61 92.29 92.12 59.39 50.69 68.95 66.18 85.01 70.95 75.16 78.63
CA
97.49 92.60 97.13 91.77 94.68 84.99 97.84 94.61 94.68 97.93 91.09 93.98 57.26 66.74 59.05 52.07 78.88 72.61 83.62 75.31
91.31 88.38 90.80 87.34 87.72 80.15 91.52 90.46 87.14 88.73 83.05 87.00 50.14 61.27 29.31 42.95 69.18 62.24 71.77 70.95
Average
CA
96.34 88.51 95.11 95.43 93.89 91.36 96.55 97.50 94.75 88.08 92.24 95.29 53.66 76.95 55.17 55.25 72.41 76.66 82.97 82.44
CSR
89.30 82.73 86.42 90.29 85.42 85.58 88.29 92.08 86.78 77.94 81.68 87.01 46.34 66.45 25.43 38.33 62.93 62.10 72.20 69.59
CA
94.56
89.79
95.15
90.09
91.84
86.31
96.59
91.66
94.49
87.41
94.22
87.23
61.58
55.73
61.46
45.33
78.03 81.85
73.07
3 D V V ) D L O
& R G H % / ( 8
G-TransEval
Model DS-V4-Flash GPT-3.5-turbo GPT-4o-mini UniTrans ExeCoder IT(Magi) IT(Star) StructCoder VIM-PT TransCoder-ST CodeT5+
TransCoder-Uni
Java
C++
Python
C#
JS
Java
C++
Python
97.15 97.31 93.56 98.00 85.83 91.09 64.21 20.50 41.38 67.50 62.00
98.50 97.55 93.67 97.77 91.54 91.27 59.98 / 36.00 59.25 61.44
87.08 86.00 73.73 87.23 67.38 77.46 55.15 / 23.69 44.12 48.19
97.08 97.63 94.77 98.66 / 89.50 56.79 18.00 38.00 / 56.62
84.50 87.69 79.46 90.12 / 80.19 60.38 / 32.44 / 52.06
93.13 92.41 90.08 91.60 87.18 87.68 54.78 / 49.19 77.12 73.47
90.70 89.11 86.08 93.47 91.70 87.00 48.56 / 46.19 67.02 75.48
85.60 88.80 82.82 89.91 83.34 87.00 63.82 / 40.67 62.28 70.28
TABLE VII: Average CA for translation from all other languages to each target language (presented in the table). G-TransEval
Model
68.75
TABLE VI: Average CA for translation from each source language (presented in the table) to all other target languages.
DS-V4-Flash GPT-3.5-turbo GPT-4o-mini UniTrans ExeCoder IT(Magi) IT(Star) StructCoder VIM-PT TransCoder-ST CodeT5+
TransCoder-Uni
Java
C++
Python
C#
JS
Java
C++
Python
95.09 94.69 90.19 95.38 82.88 89.44 74.62 18.00 35.19 55.25 50.62
84.00 85.86 78.79 90.66 68.71 77.50 51.21 / 39.00 50.25 46.62
99.35 98.27 95.69 98.31 93.17 90.56 62.46 / 18.25 65.38 64.56
89.84 91.92 79.42 90.94 / 83.94 56.08 20.50 40.06 / 58.88
96.04 95.44 91.11 96.50 / 88.06 52.12 / 39.00 / 59.62
90.21 89.52 83.44 93.20 92.67 89.56 55.98 / 54.56 66.60 74.79
88.83 92.15 89.01 90.56 82.58 89.65 62.92 / 53.64 73.66 72.38
90.30 88.61 86.57 91.25 86.96 82.36 48.24 / 27.37 66.06 71.98
is used as the source or target language on the two datasets. The results are shown in Table VI and Table VII.
& R G H 7
6 7 R G H U
3 7
V &
9 ,
0
7 U D Q
G H U & R X F W
D Q V
6 W U
L 7 U 8 Q
U E R * 3 7 R P L Q L
W X * 3 7
' 6
9
) O
D V K
Fig. 3: CodeBLEU gaps between passed and failed samples. Finding 2: Learning-based methods often generate outputs that are similar to correct translations but cannot compile or pass tests. In contrast, LLM-based methods achieve CodeBLEU scores comparable to learning-based methods and show a smaller CodeBLEU gap between correct and incorrect outputs, indicating that they focus more on semantic functionality rather than line-by-line translation. B. RQ2: Performance on Different Translation Tasks According to the results shown in Table IV and Table V, the same method may achieve substantially different results when translating between the same pair of programming languages in opposite directions. For example, when using DeepSeek-V4-Flash on the G-TransEval dataset, the CA score for C++ to Python reaches 99.25%, whereas the score in the reverse direction drops sharply to 76.33%. Similar patterns can also be observed for other large language models and LLMbased methods. To better understand this issue, we conduct a deeper analysis of the results for each programming language. Specifically, we compute the average CA when each language
The results confirm that the choice of source and target languages has a clear impact. On the G-TransEval dataset, we observe a phenomenon consistent with prior studies [30], [34]: translation performs better when Python is used as the target language. We further extend and summarize this observation. Specifically, when Python or JavaScript is used as the target language, CA scores are significantly higher than when Java or C++ is used as the target language. Conversely, when Java or C++ is used as the source language, CA scores are significantly higher than when Python or JavaScript is used as the source language. We attribute this to the difference in the amount of explicit information encoded in different programming languages. For example, when translating from Python, where type information is relatively implicit, to Java or C++, where type information is more explicit, the model needs to infer variable types, parameter types, and return types, which may introduce errors. However, on the TransCoder-Uni dataset, the performance decrease or improvement caused by using Python as the source or target language is much less pronounced. This phenomenon motivates us to investigate the differences between the two datasets. Based on our study of failure cases, we speculate that this is because TransCoder-Uni contains fewer type annotations and more complex logic than G-TransEval, as shown in Table X and further discussed in RQ4. This characteristic weakens the disadvantage or advantage usually associated with using Python as the source or target language. Overall, programming languages do affect translation correctness, but the extent of this effect also depends on the type complexity and logic complexity of the dataset.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Incomplete Structure
A1
Ø Using illegal characters or keywords. Exceptional Runtime Conditions
A2 C1
Ø Triggering exceptions by illegal operations or invalid states during program runtime
Logic Errors Ø Program produces incorrect output.
A
B1
D1
Type Mismatch
Ø Type mismatch in expression or assignment.
Syntax Errors
Ø Lacking essential grammatical elements. Invalid Token / Keyword Misuse
9
C
B
Runtime Errors
Static Semantic Errors
Bug Type
D Logic Errors
B2
Undefined / Undeclared Identifier
Ø Using undeclared variables, functions, or types.
B3
Function Signature Violation
Ø Incorrect argument or return types. Declaration / Binding Violations
B4Ø Operations that violate variable declaration, scope, or mutability rules.
B5
API / Library Usage Error
Ø Using incorrect APIs or library dependencies.
Fig. 4: Categories of failures caused by the code translation methods studied in our experiments. Finding 3: Translation generally tends to perform better when translating from statically typed languages to dynamically typed languages, whereas the opposite direction results in notably lower performance. This suggests that, in practical code translation, using a statically (dynamically) typed language as the source (target) language is usually a more favorable setting.
C. RQ3: Class-Level Code Translation Performance To further evaluate the practicality of existing code translation techniques in real-world software engineering scenarios, we extend our study from method-level translation to classlevel translation. Compared with method-level translation, class-level translation involves longer contexts, richer intermethod dependencies, and more complex type structures. These characteristics make class-level translation not merely a scaled-up version of method-level translation, but a qualitatively more difficult task that requires models and tools to preserve global class semantics, member relationships, and cross-language type consistency. Table VIII summarizes the class-level translation results in terms of CA and CSR. Overall, all methods show a clear performance decline on class-level translation, especially in CA. Although some methods still achieve relatively high CSR, their CA remains low, indicating that generating compilable class-level code is much easier than generating semantically correct class-level translations. From the overall averages, the best-performing UniTrans achieves an average CA of 24.82% and an average CSR of 89.39%, while the three general-purpose LLMs obtain average CA values between 18.82% and 22.87% and average CSR values between 72.15% and 83.51%. This result indicates that even the best-performing method can correctly translate only a limited portion of class-level code. In other words, existing techniques still cannot reliably solve class-level code translation in practical settings.
Finding 4: Compared with method-level code translation, both LLMs and LLM-based methods show decreases in CA and CSR on class-level code translation. However, the decrease in CSR is much smaller than that in CA, indicating that the main challenge in current class-level code translation is not compilation, but how to generate semantically correct code under more complex code contexts.
A more detailed analysis shows that, compared with method-level translation, the gap between CA and CSR becomes much larger for almost all methods and translation directions. Specifically, in Java→C++ and Python→C++, all methods produce extremely low CA values, with several settings close to or equal to zero, while CSR remains moderate or even high. Similarly, for C++→Java and Python→Java, CSR is generally above 73.40%, but CA remains limited (≤ 25.53%). This discrepancy indicates that many translated classes can compile, but still fail to preserve the original program semantics, class structure, or behavioral equivalence. This further suggests that existing methods can handle syntactic issues in translation relatively well, but when source code becomes longer and more complex, involving more dependencies and types, they struggle to achieve the high correctness observed in method-level translation. The results also reveal a clear language effect in class-level translation. Translations targeting Python achieve relatively higher CA than those targeting Java or C++. In contrast, translating into C++ is consistently the most challenging direction. For example, all methods obtain significantly low CA (≤ 12.77%) on Java→C++, and only marginally better results on Python→C++. This trend indicates that when class structures become larger and more interdependent, translating language-specific type systems, object models, and declaration conventions becomes more difficult. Compared with methodlevel tasks, class-level tasks are more sensitive to language differences, especially when the target language is statically typed and imposes stricter structural and type constraints.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
10
GPT-4o-mini CA
70.21 62.77 86.17 90.43 58.51 80.85
43.62 35.11 11.70 22.34 1.06 2.13
Average
60.46
12.06
89.39
24.82
83.51
22.87
72.15
18.82
74.82
19.33
Finding 5: Language effects become stronger in classlevel code translation. As the translation unit becomes larger and type interactions become more complex, accurately understanding and translating cross-language type systems becomes a major bottleneck. Therefore, practical class-level code translation should place greater emphasis on type-aware translation, especially for tasks targeting statically typed languages, where strict type and structural constraints make translation more difficult.
D. RQ4: Case Study of Translation Errors Although existing methods achieve strong performance on the studied benchmarks, they still produce incorrect translations. To better understand these errors and identify the main challenges, we manually analyze the failed outputs generated by the studied methods. We adopt a two-step classification process. First, we randomly sample 30 failed cases to establish an initial set of error categories. Then, three evaluators, each with at least three years of software development experience, independently annotate the remaining cases in batches of 30. They discussed ambiguous cases and iteratively refined the coding categories. Under the final category criteria, the Fleiss’kappa [63] coefficient was 0.9446, indicating substantial agreement among the three raters. Ultimately, all remaining disagreements were resolved through discussion. The final error taxonomy, shown in Figure 4, divides errors into four major categories: syntax errors, static semantic errors, runtime errors, and logical errors. Each major category contains mutually exclusive subcategories with brief descriptions. Each failed case is assigned to exactly one category. When a failure involves multiple error types, we classify it according to the following severity order: syntax errors have higher priority than static semantic errors, static semantic errors have higher priority than runtime errors, and runtime errors have higher priority than logical errors. 1) RQ4.1 Method-Level Error Analysis: For the methodlevel setting, we annotate 570 failed outputs generated by the studied methods. Specifically, we collect 30 failed cases for each method on each dataset, except that StructCoder produces only 30 failed cases in total. The final distribution of error types is shown in Figure 5. Static semantic errors (49.37%) and logical errors (27.46%) account for the vast majority of all failed cases. This indicates that generating semantically and functionally correct programs remains the main challenge for code translation methods, including both learning-based and large language model (LLM)-based approaches. In contrast, syntax errors (18.25%) and runtime errors (4.92%) have a relatively smaller impact on the overall results.
Fig. 5: Distribution of error types across datasets. The left side of each method represents the results on G-TransEval, and the right side represents the results on TransCoder-Uni. , Q F R P S O H W H 6 W U X F W X U H $ , Q Y D O L G 7 R N H Q . H \ Z R U G 0 L V X V H $ 7 \ S H 0 L V P D W F K %
8 Q G H I L Q H G 8 Q G H F O D U H G , G H Q W L I L H U % ) X Q F W L R Q 6 L J Q D W X U H 9 L R O D W L R Q % ' H F O D U D W L R Q % L Q G L Q J 9 L R O D W L R Q V %
$ 3 , / L E U D U \ 8 V D J H ( U U R U % ( [ F H S W L R Q D O 5 X Q W L P H & R Q G L W L R Q V & / R J L F ( U U R U V '
D Q V
CSR
42.55 32.98 13.98 18.09 1.06 4.26
8 Q L 7 U
CA
69.15 60.64 92.47 81.91 59.57 69.15
L Q L
CSR
42.55 30.85 18.09 25.53 12.77 7.45
$ 3 , / L E U D U \ 8 V D J H ( U U R U % ( [ F H S W L R Q D O 5 X Q W L P H & R Q G L W L R Q V & / R J L F ( U U R U V '
P
CA
68.09 69.15 93.62 94.68 88.30 87.23
* 3 7 R
CSR
54.26 41.49 15.96 20.21 7.45 9.57
W X U E R
CA
79.79 72.34 94.19 95.35 96.05 98.61
* 3 7
CSR
23.40 24.47 11.70 8.51 3.19 1.06
R G H U
CA
46.81 55.32 80.85 73.40 60.64 45.74
D V K
CSR C++ → Py. Java → Py. C++ → Java Py. → Java Java → C++ Py. → C++
) O
GPT-3.5-turbo
3 H U F H Q W D J H R I ( U U R U V
DS-V4-Flash
8 Q G H I L Q H G 8 Q G H F O D U H G , G H Q W L I L H U % ) X Q F W L R Q 6 L J Q D W X U H 9 L R O D W L R Q % ' H F O D U D W L R Q % L Q G L Q J 9 L R O D W L R Q V %
' 6 9 * 3 ) O D V 7 K * 3 W X U E R 7 R P L Q 8 Q L L 7 U D Q ( [ V H & R G H , 7 U 0 D J L , 7 6 W D U 9 7 U D , 0 3 Q V & 7 R G H U 6 & R 7 G H 6 W U 7 X F W & R G H U
UniTrans
3 H U F H Q W D J H R I ( U U R U V
ExeCoder
' 6 9
Task
, Q F R P S O H W H 6 W U X F W X U H $ , Q Y D O L G 7 R N H Q . H \ Z R U G 0 L V X V H $ 7 \ S H 0 L V P D W F K %
( [ H &
TABLE VIII: CSR and CA results across different translation directions on ClassEval-T.
Fig. 6: Error case distribution on ClassEval-T. Finding 6: Among method-level failed cases, static semantic errors and logical errors are the most common, indicating that generating semantically and functionally correct programs remains a major challenge for existing code translation methods. Among static semantic errors, function signature violations are the most critical subcategory, accounting for 70.18% of all static semantic errors. This problem arises because translation methods must implicitly infer type mappings between the source and target languages. It is particularly challenging when translating from an information-sparse source language to an information-rich target language. Even for state-of-the-art LLMs, accurately determining the appropriate target-language types requires a deep understanding of the semantics of the translated code. For example, Figure 7 shows an incorrect translation from JavaScript to C#, in which the generated method signature is incompatible with the expected interface. According to the reference answer, the return type and the first two parameter types of ConcatLargeNumbers should be List<int>. However, the translated code uses int[] for the return type and both list parameters, and correspondingly returns the result through ToArray() instead of ToList(). As a result, the
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Source function concatLargeNumbers(numbersA, numbersB, flag) { const filteredA = numbersA.filter(number => number > flag); const filteredB = numbersB.filter(number => number > flag); const allNumbers = [...filteredA, ...filteredB]; return allNumbers; }
Translation int[] ConcatLargeNumbers( int[] numbersA, int[] numbersB, int flag) { return numbersA .Where(n => n > flag) .Concat(numbersB.Where(n => n > flag)) .ToArray(); }
The translated code wrongly uses int[] instead of List<int> for the return type and parameters, and returns with ToArray() rather than ToList()
Ground truth List<int> ConcatLargeNumbers( List<int> numbersA, List<int> numbersB, int flag) { List<int> allNumbers = numbersA .Where(n => n > flag) .Concat(numbersB.Where(n => n > flag)) .ToList(); return allNumbers; }
Fig. 7: An example of function signature violation Source def findMinimumAngle(arr, n): l = 0 _sum = 0 ans = 360 for i in range(n): _sum += arr[i] while _sum >= 180: ans = min(ans, 2 * abs(180 - _sum)) _sum -= arr[l] l += 1 ans = min(ans, 2 * abs(180 - _sum)) return ans
Translation
The incorrect translation updates ans outside the loop, leading to incorrect results.
Ground Truth
int findMinimumAngle(int arr[], int n) { int findMinimumAngle(int arr[], int n) { int l = 0, sum = 0, ans = 360; int l = 0, sum = 0, ans = 360; for (int i = 0; i < n; i++) { for (int i = 0; i < n; i++) { _sum += arr[i]; _sum += arr[i]; while (_sum >= 180) { while (_sum >= 180) { ans = min(ans, 2 * abs(180 - _sum)); ans = min(ans, 2 * abs(180 - _sum)); _sum -= arr[l]; _sum -= arr[l]; l += 1; l += 1; } } } ans = min(ans, 2 * abs(180 - _sum)); ans = min(ans, 2 * abs(180 - _sum)); } return ans; return ans; } }
Fig. 8: An example of a logic error generated method does not match the required interface and is incompatible with the benchmark tests. This example shows that, when translating from an information-sparse language such as JavaScript to a statically typed language such as C#, accurately inferring collection types and mapping them to the expected target-language abstractions remains a key bottleneck. For logical errors, we observe that many failures involving complex code logic are caused by misplaced statements. For example, Figure 8 shows a logical error made by IT(Magi) when translating Python to C++. The assignment to the variable ans is incorrectly placed outside the loop, resulting in incorrect program behavior. Although the translated code can be compiled, it produces incorrect outputs. Such errors are difficult to detect, especially when the code involves complex algorithms, unless developers have a thorough understanding of the underlying logic. Furthermore, we analyze the differences in error distributions between the two benchmarks. As shown in Figure 5, the distributions differ substantially, particularly for the two dominant error types discussed above: function signature violations and logical errors. Specifically, on G-TransEval, the average proportions of function signature violations and logical errors are 35.45% and 21.52%, respectively, whereas on TransCoderUni, the corresponding proportions are 19.67% and 34.00%. To investigate the reasons behind these differences, we analyze the data types used in function signatures, which are the main source of function signature violations, as well as the corresponding code complexity, which is closely related to
11
logical errors. The results are shown in Table IX and Table X. The results show that function signatures in G-TransEval have more complex type compositions and make extensive use of compound STL types. This increases the difficulty of cross-language translation because the translation method must not only infer the value types correctly but also understand the corresponding type-mapping rules. In contrast, the type structures in TransCoder-Uni are relatively simple and mainly consist of primitive types and C-style arrays, with primitive types alone accounting for 75.15%. This difference in type complexity directly contributes to the substantially higher proportion of function signature violations in G-TransEval and also explains the cross-language translation difficulty observed in RQ2. Similarly, the code complexity statistics show that TransCoder-Uni has a higher average cyclomatic complexity. Tasks with medium or higher difficulty account for nearly 30% of the cases in TransCoder-Uni, compared with only 20.20% in G-TransEval. This indicates that, although its type system is simpler, TransCoder-Uni contains more translation challenges involving complex logic, leading to more logical errors. This also explains why, on TransCoder-Uni, the performance gap between using Python, an information-sparse language, as the target language and using C++, an information-rich language, as the target language is smaller than that on G-TransEval. TransCoder-Uni contains more complex logic but simpler data types, which reduces the impact of inaccurate type translation. Finding 7: As observed in Finding 3, the performance difference between translating from statically to dynamically typed languages and translating in the reverse direction is closely related to the complexity of type inference and type mapping. More specifically, datasets containing more diverse and complex types are more susceptible to language effects. 2) RQ4.2 Class-Level Error Analysis: To further examine whether the failure patterns identified above persist at a larger translation granularity, we extend the error analysis to classlevel translation on ClassEval-T. In this setting, we annotate 150 failed outputs, with 30 cases generated by each studied method. The distribution of class-level error cases is shown in Figure 6. The results show a more pronounced deviation in semantic correctness: static semantic errors account for 70.67 of all failed cases, while logical errors account for 19.33%. In comparison, syntax errors and runtime errors account for only 8.00% and 2.00% of the cases, respectively. Thus, compared with method-level translation, class-level translation is less affected by syntactic problems but more constrained by semantic consistency, particularly the preservation of interfaces, member bindings, and cross-method dependencies. Among static semantic errors, function signature violations remain the dominant subcategory, accounting for 75.47% of all static semantic errors and 53.33% of all class-level failed cases. The remaining static semantic errors mainly involve declaration or binding violations and undefined or undeclared identifiers. These results indicate that class-level translation
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
12
TransCoder-Uni
Metric
G-TransEval
TransCoder-Uni
Statistical Overview
Category
TABLE X: Cyclomatic Complexity of the code for two datasets
Avg. Complexity Median Std. Dev. Max / Min 90th Percentile
4.14 4.00 2.63 19 / 1 7.00
4.57 4.00 2.77 19 / 1 8.00
Complexity Distri. (%)
TABLE IX: Statistical comparison of type distribution between the two datasets.
Simple (1–5) Medium (6–10) Complex (11–20) Very Complex (>20)
79.8 17.2 3.0 0.0
70.3 26.9 2.8 0.0
G-TransEval
Count
% of Total
Count
% of Total
Basic Types STL Types C-style Arrays Pointers/References
1282 122 289 13
75.15% 7.15% 16.94% 0.76%
633 426 0 0
59.77% 40.23% 0% 0%
Total
1706
100%
1059
100%
is not merely a problem of mapping individual types. It also requires maintaining consistency across fields, methods, and class-level scopes. In other words, as the translation unit becomes larger, the model must preserve not only the correctness of local statements but also the structural relationships among the members of the class. We further analyze the distribution of error types for each method in the class-level setting. For UniTrans and the three commercial models, failures are primarily dominated by static semantic errors, most of which are function signature violations. In contrast, ExeCoder, which uses a weaker underlying model, produces a more diverse range of error types, including more syntax errors and more frequent binding-related failures. This distribution suggests that stronger models are more capable of reducing syntax and runtime errors, while their remaining failures are more concentrated in static semantic and logical errors. A similar trend can also be observed in the method-level error analysis. Overall, class-level and method-level code translation exhibit similar distributions of broad error categories, with static semantic errors and logical errors being the dominant types in both settings. However, class-level translation substantially amplifies the importance of type consistency and symbol binding. As the translation granularity increases, the main challenge shifts from local statement-level translation to maintaining syntactic and semantic consistency across the entire class. This challenge involves not only longer contexts and greater semantic complexity, but also stronger consistency requirements among class members, methods, and properties. These factors help explain why class-level translation becomes particularly difficult when the target language imposes stricter type and structural constraints. Finding 8: Compared with method-level translation, class-level translation exhibits a similar overall error profile but substantially amplifies static semantic consistency problems. In particular, function signature violations account for 53.33% of all class-level failed cases. This indicates that preserving interface consistency, symbol bindings, and cross-member dependencies becomes a primary bottleneck as the translation granularity increases.
VI. I MPLICATIONS Our comprehensive evaluation yields the following implications for future research in code translation. 1) Future research should move beyond isolated methodlevel translation. In our experiments, LLMs and LLM-based
methods generally outperform learning-based methods on correctness metrics in method-level translation. However, this advantage does not imply that method-level translation has been fully solved. More importantly, class-level translation remains substantially more difficult: the best-performing method achieves an average CA of only 24.82%, despite obtaining an average CSR of 89.39%. This large gap shows that generating compilable code is considerably easier than preserving semantic correctness in larger translation units. Future research should therefore investigate class-level, repository-level, and other practically relevant settings that require the preservation of inter-method dependencies, class structure, and global program semantics. 2) Code translation should be evaluated using both correctness and similarity metrics. Our results show that learning-based methods may achieve CodeBLEU scores comparable to those of LLM-based methods, even though their CA and CSR are substantially lower. This indicates that similarity metrics alone cannot reliably reflect compilation success or functional correctness. At the same time, the smaller CodeBLEU gap between passed and failed outputs generated by LLM-based methods suggests that functionally correct translations may not always closely resemble reference implementations at the surface level. Future evaluations should therefore report correctness-oriented metrics together with similarity metrics and should avoid using CodeBLEU as a substitute for executable or test-based validation. 3) Translation difficulty depends on both language direction and benchmark characteristics. Translation direction has a clear impact on performance: translating from information-rich languages (e.g., Java and C++) to information-sparse languages (e.g., Python and JavaScript) is generally easier than the reverse direction, which requires additional type inference and structural completion. Future methods should explicitly consider the challenges posed by different translation directions, particularly the type inference and type mapping issues involved when translating from information-sparse languages to information-rich languages, while also accounting for the type complexity and logical complexity of the target benchmark dataset. 4) Static semantic consistency and logical equivalence remain the main bottlenecks. Among the sampled method-level failed cases, Static semantic errors (49.37%) and logical errors (27.46%) account for the vast majority of all failed cases. Function signature violations constitute 70.18% of the static semantic errors and are especially frequent on G-TransEval,
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
TABLE XI: Data-leakage results and gaps against the original benchmarks. Model DS-V4-Flash GPT-3.5-turbo GPT-4o-mini
C++ → Python
C++ → Java
94.00 / -5.25 / +4.70 94.00 / -2.17 / +7.58 92.00 / -2.75 / +6.58
96.00 / -3.92 / +3.95 90.00 / -9.17 / -1.70 94.00 / -2.67 / +7.28
Each cell reports Data-leakage Result / ∆G / ∆T, where ∆G and ∆T denote gaps against G-TransEval and TransCoder-Uni, respectively. All values are percentages or percentage-point gaps.
which contains more diverse and complex type compositions. At the class level, static semantic errors account for 70.67% of the sampled failures, and function signature violations alone account for 53.33% of all class-level failed cases. These findings highlight the need for stronger type-mapping constraints, interface checking, symbol-binding analysis, and semantic reasoning during translation. Future approaches should also improve their ability to preserve control-flow structure and algorithmic logic, since compilable code may still produce incorrect results. 5) Class-level translation requires global consistency rather than local code generation. The class-level results show that increasing the translation granularity amplifies the influence of language-specific type systems, object models, declarations, and member dependencies. In particular, translations targeting C++ are considerably more difficult than those targeting Python or Java in several settings. This indicates that class-level translation cannot be addressed merely by extending the context window or translating methods independently. Future systems should maintain consistency across fields, method signatures, method bodies, inheritance relations, and cross-method references. Techniques such as structure-aware representations, dependency-aware translation, incremental verification, and class-level compilation or testing may therefore be important directions for improving practical code translation. VII. T HREATS TO VALIDITY Internal Validity. Our analysis involves expert judgment. Although three evaluators independently annotated the cases and reconciliation to reduce subjectivity (as mentioned in Section V-D), the results may still be influenced by personal experience. Therefore, misinterpretations of translated code may introduce bias into the classification of error causes. To mitigate this threat, we used Fleiss’ kappa [63] to measure inter-rater agreement and assess the consistency of the classification results. We also adopted an inductive open-coding procedure [64], in which the raters independently examined all error cases and resolved disagreements through discussion. Another internal threat is data leakage. To mitigate this risk, for the learning-based methods under study, we manually cleaned the training samples that overlap with the test benchmarks. The open-source large language models used in this study (i.e., StarCoder2 and Magicoder) are primarily trained on data from GitHub repositories [57], [58], while our test benchmarks are sourced from the GeeksforGeeks platform, making direct overlap unlikely. This is also consistent with
13
the discussions in prior works [23], [27]. For the LLMs and LLM-based methods, we manually constructed a dataset containing 50 C++ code snippets along with their test suites, all selected from problems that appeared on Luogu [65] after the knowledge cutoff dates of the commercial LLMs. These problems appeared in the three commercial LLMs only after the cutoff date. On this dataset, we conducted experiments in both translation directions for the three LLMs we used, with results shown in the figure. Compared with the original results, we observed only limited increases or decreases. We consider this acceptable, as the data-leakage dataset differs from the original function-level dataset in terms of context length and problem difficulty. In summary, we believe that the impact of data leakage on the experiments in this paper is limited. External Validity. Our findings may not fully generalize beyond the selected benchmarks, programming languages, and methods. Although we include diverse datasets and representative recent approaches, they may not cover all real-world translation scenarios, language pairs, or future models. To mitigate this threat, we evaluate multiple benchmarks with different properties and compare representative methods under a unified setting. VIII. C ONCLUSION In this paper, we conducted a large-scale empirical study of 11 representative code translation techniques, including learning-based methods, LLM-based methods, and generalpurpose LLMs. We evaluated them under unified settings on two method-level multilingual benchmarks and one class-level benchmark using both correctness and similarity metrics. The results show that LLMs and LLM-based methods generally outperform learning-based methods in correctness, although their advantage is less evident on similarity metrics. Translation direction also affects performance, particularly on G-TransEval. Furthermore, class-level translation remains substantially more difficult than method-level translation, especially in preserving semantic correctness and structural consistency. Our failure analysis shows that static semantic errors and logical errors are the dominant challenges, with function signature violations being particularly common. Overall, our findings highlight the need for future code translation systems to improve type-aware, structure-aware, and context-aware translation, while combining correctnessoriented and similarity-oriented evaluation. IX. DATA AVAILABILITY The data and code supporting the findings of this study are available in our open-source repository: https://doi.org/10. 5281/zenodo.21973888. R EFERENCES [1] F. Liu, J. Li, and L. Zhang, “Syntax and domain aware model for unsupervised program translation,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 755–767. [2] A. T. Nguyen, T. T. Nguyen, and T. N. Nguyen, “Divide-and-conquer approach for multi-phase statistical migration for source code,” in Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’15. IEEE Press, 2015, p. 585–596. [Online]. Available: https://doi.org/10.1109/ASE.2015.74
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
[3] W. Wu, Y.-G. Guéhéneuc, G. Antoniol, and M. Kim, “Aura: a hybrid approach to identify framework evolution,” in 2010 ACM/IEEE 32nd International Conference on Software Engineering, vol. 1, 2010, pp. 325–334. [4] H. Zhong, S. Thummalapenta, T. Xie, L. Zhang, and Q. Wang, “Mining api mapping for language migration,” in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1, 2010, pp. 195–204. [5] A. Z. Yang, Y. Takashima, B. Paulsen, J. Dodds, and D. Kroening, “Vert: Polyglot verified equivalent rust transpilation with large language models,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Press, 2026, p. 1453–1463. [Online]. Available: https://doi.org/10.1109/ASE63991. 2025.00123 [6] A. R. Ibrahimzada, K. Ke, M. Pawagi, M. S. Abid, R. Pan, S. Sinha, and R. Jabbarvand, “Alphatrans: A neuro-symbolic compositional approach for repository-level code translation and validation,” 2025. [Online]. Available: https://arxiv.org/abs/2410.24117 [7] G. Ou, M. Liu, Y. Chen, Y. Wang, X. Peng, and Z. Zheng, “Rustrepotrans: Repository-level context code translation benchmark targeting rust,” 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 610–622, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:277505320 [8] A. T. Nguyen, T. T. Nguyen, and T. N. Nguyen, “Lexical statistical machine translation for language migration,” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2013. New York, NY, USA: Association for Computing Machinery, 2013, p. 651–654. [Online]. Available: https://doi.org/10.1145/2491411.2494584 [9] B. Roziere, M.-A. Lachaux, L. Chanussot, and G. Lample, “Unsupervised translation of programming languages,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020. [10] Mono Project, “Sharpen: Automated java to c# conversion,” https:// github.com/mono/sharpen, originally created by db4o; online repository; accessed 2026-03-25. [11] Immunant, “Migrate c code to rust,” https://github.com/immunant/c2rust, online; accessed 2026-03-25. [12] B. Wang, A. Kolluri, I. Nikolić, T. Baluta, and P. Saxena, “User-customizable transpilation of scripting languages,” Proc. ACM Program. Lang., vol. 7, no. OOPSLA1, Apr. 2023. [Online]. Available: https://doi.org/10.1145/3586034 [13] alexandrefau, “j2csharp,” 2015, accessed: 2026-03-25. [Online]. Available: https://sourceforge.net/projects/j2cstranslator/ [14] natural, “java2python,” 2012, accessed: 2026-03-25. [Online]. Available: https://github.com/natural/java2python/ [15] Go Transpile, “cxgo,” 2023, accessed: 2026-03-25. [Online]. Available: https://github.com/gotranspile/cxgo [16] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics, Nov. 2020, pp. 1536–1547. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.139/ [17] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in EMNLP, 2021. [18] Y. Wang, H. Le, A. Gotmare, N. Bui, J. Li, and S. Hoi, “CodeT5+: Open code large language models for code understanding and generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 1069–1088. [Online]. Available: https://aclanthology.org/2023.emnlp-main.68/ [19] S. Tipirneni, M. Zhu, and C. K. Reddy, “Structcoder: Structure-aware transformer for code generation,” vol. 18, no. 3, Jan. 2024. [Online]. Available: https://doi.org/10.1145/3636430 [20] Y. Du, H. Sun, and M. Li, “A joint learning model with variational interaction for multilingual program translation,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24, New York, NY, USA, 2024, p. 1907–1918. [Online]. Available: https://doi.org/10.1145/3691620.3695553 [21] Q. Sun, N. Chen, J. Wang, M. Gao, and X. Li, “TransCoder: Towards unified transferable code representation learning inspired by human skills,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). Torino, Italia: ELRA and
14
ICCL, May 2024, pp. 16 713–16 726. [Online]. Available: https: //aclanthology.org/2024.lrec-main.1453/ [22] S. Chakraborty, T. Ahmed, Y. Ding, P. T. Devanbu, and B. Ray, “Natgen: generative pre-training by “naturalizing” source code,” in Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, 2022, pp. 18–30. [23] B. Roziere, J. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample, “Leveraging automated unit tests for unsupervised code translation,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=cmt-6KtR4c4 [24] S. Liu, B. Wu, X. Xie, G. Meng, and Y. Liu, “Contrabert: Enhancing code pre-trained models via contrastive learning,” in Proceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23. IEEE Press, 2023, p. 2476–2487. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00207 [25] M. Macedo, Y. Tian, P. Nie, F. R. Cogo, and B. Adams, “Intertrans: Leveraging transitive intermediate translations to enhance llm-based code translation,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 1153–1164. [26] P. Jana, P. Jha, H. Ju, G. Kishore, A. Mahajan, and V. Ganesh, “Cotran: An llm-based code translator using reinforcement learning with feedback from compiler and symbolic execution,” in European Conference on Artificial Intelligence, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:259138621 [27] Z. Yang, F. Liu, Z. Yu, J. W. Keung, J. Li, S. Liu, Y. Hong, X. Ma, Z. Jin, and G. Li, “Exploring and unleashing the power of large language models in automated code translation,” Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3660778 [28] Z. Yuan, W. Chen, H. Wang, K. Yu, X. Peng, and Y. Lou, “Semantic alignment-enhanced code translation via an llm-based multiagent system,” 2024. [Online]. Available: https://api.semanticscholar. org/CorpusID:272988134 [29] X. Chen, J. Xue, X. Xie, C. Liang, and X. Ju, “A systematic literature review on neural code translation,” ArXiv, vol. abs/2505.07425, 2025. [Online]. Available: https://api.semanticscholar.org/CorpusID: 278501698 [30] P. Xue, L. Wu, Z. Yang, C. Wang, X. Li, Y. Zhang, J. Li, R. Jin, Y. Pei, Z. Shen, X. Lyu, and J. W. Keung, “Classeval-t: Evaluating large language models in class-level code translation,” Proc. ACM Softw. Eng., vol. 2, no. ISSTA, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3728940 [31] M. Jiao, T. Yu, X. Li, G. Qiu, X. Gu, and B. Shen, “On the evaluation of neural code translation: Taxonomy and benchmark,” 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1529–1541, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:261030272 [32] R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, and R. Jabbarvand, “Lost in translation: A study of bugs introduced by large language models while translating code,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639226 [33] H. F. Eniser, H. Zhang, C. David, M. Wang, M. Christakis, B. Paulsen, J. Dodds, and D. Kroening, “Towards translating real-world code with llms: A study of translating to rust,” ArXiv, vol. abs/2405.11514, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID: 269922036 [34] Q. Tao, T. Yu, X. Gu, and B. Shen, “Unraveling the potential of large language models in code translation: How far are we?” 2024 31st Asia-Pacific Software Engineering Conference (APSEC), pp. 353–362, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID: 273345538 [35] S. Bhatia, S. Kohli, S. A. Seshia, and A. Cheung, “Building code transpilers for domain-specific languages using program synthesis (experience paper),” in 37th European Conference on Object-Oriented Programming (ECOOP 2023). Schloss Dagstuhl–Leibniz-Zentrum für Informatik, 2023, pp. 38–1. [36] C. Verhoef and A. A. Terekhov, “ The Realities of Language Conversions ,” IEEE Software, vol. 17, no. 06, pp. 111–124, Nov. 2000. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/52.895180 [37] S. Karaivanov, V. Raychev, and M. Vechev, “Phrase-based statistical translation of programming languages,” in Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software, ser. Onward! 2014. New
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
York, NY, USA: Association for Computing Machinery, 2014, p. 173–184. [Online]. Available: https://doi.org/10.1145/2661136.2661148 [38] X. Chen, C. Liu, and D. Song, “Tree-to-tree neural networks for program translation,” Advances in neural information processing systems, vol. 31, 2018. [39] M. Szafraniec, B. Roziere, H. J. Leather, P. Labatut, F. Charton, and G. Synnaeve, “Code translation with compiler representations,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=XomEU3eNeSQ [40] W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pretraining for program understanding and generation,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics, Jun. 2021, pp. 2655–2668. [Online]. Available: https://aclanthology.org/2021.naacl-main.211/ [41] W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Summarize and generate to back-translate: Unsupervised translation of programming languages,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Dubrovnik, Croatia: Association for Computational Linguistics, May 2023, pp. 1528–1542. [Online]. Available: https: //aclanthology.org/2023.eacl-main.112/ [42] Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, L. Shen, Z. Wang, A. Wang, Y. Li, T. Su, Z. Yang, and J. Tang, “Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,” ser. KDD ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 5673–5684. [Online]. Available: https://doi.org/10.1145/3580305.3599790 [43] X. Li, X. Li, S. Yuan, X. Gu, Y. Chen, Y. Chen, and B. Shen, “Few-shot code translation via task-adapted prompt learning,” Journal of Systems and Software, vol. 212, p. 112002, jun 2024. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0164121224000451 [44] B. Liu, C. Chen, Z. Gong, C. Liao, H. Wang, Z. Lei, M. Liang, D. Chen, M. Shen, H. Zhou, W. Jiang, H. Yu, and J. Li, “Mftcoder: Boosting code llms with multitask fine-tuning,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 5430–5441. [Online]. Available: https://doi.org/10.1145/3637528.3671609 [45] P. Di, J. Li, H. Yu, W. Jiang, W. Cai, Y. Cao, C. Chen, D. Chen, H. Chen, L. Chen, G. Fan, J. Gong, Z. Gong, W. Hu, T. Guo, Z. Lei, T. Li, Z. Li, M. Liang, C. Liao, B. Liu, J. Liu, Z. Liu, S. Lu, M. Shen, G. Wang, H. Wang, Z. Wang, Z. Xu, J. Yang, Q. Ye, G. Zhang, Y. Zhang, Z. Zhao, X. Zheng, H. Zhou, L. Zhu, and X. Zhu, “Codefuse-13b: A pretrained multi-lingual code large language model,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, ser. ICSE-SEIP ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 418–429. [Online]. Available: https://doi.org/10.1145/3639477.3639719 [46] M. He, Y. Chen, F. Yang, P. Zhao, W. Yin, Y. Kang, Q. Lin, S. Rajmohan, and D. Zhang, “ExeCoder: Empowering large language models with executability representation for code translation,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 7099–7125. [Online]. Available: https://aclanthology.org/2025.emnlp-main.362/ [47] R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. A. Zolotov, J. Dolby, J. Chen, M. R. Choudhury, L. Decker, V. Thost, L. Buratti, S. Pujar, and U. Finkler, “Codenet: A largescale ai for code dataset for learning a diversity of coding tasks,” in NeurIPS Datasets and Benchmarks, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:235195915 [48] W. U. Ahmad, M. G. R. Tushar, S. Chakraborty, and K.-W. Chang, “AVATAR: A parallel corpus for Java-python program translation,” in Findings of the Association for Computational Linguistics: ACL 2023. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 2268–2281. [Online]. Available: https://aclanthology.org/ 2023.findings-acl.143/ [49] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. GONG, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. LIU, “Codexglue: A machine learning benchmark dataset for code understanding and generation,” in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung, Eds., vol. 1, 2021. [Online]. Avail-
15
able: https://datasets-benchmarks-proceedings.neurips.cc/paper files/ paper/2021/file/c16a5320fa475530d9583c34fd356ef5-Paper-round1.pdf [50] M. Zhu, K. Suresh, and C. K. Reddy, “Multilingual code snippets training for program translation,” in Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 2022, pp. 11 783– 11 790. [Online]. Available: https://doi.org/10.1609/aaai.v36i10.21434 [51] M. Zhu, A. Jain, K. Suresh, R. Ravindran, S. Tipirneni, and C. K. Reddy, “Xlcost: A benchmark dataset for cross-lingual code intelligence,” ArXiv, vol. abs/2206.08474, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:249848115 [52] P. Jana, P. Jha, H. Ju, G. Kishore, A. Mahajan, and V. Ganesh, CoTran: An LLM-Based Code Translator Using Reinforcement Learning with Feedback from Compiler and Symbolic Execution. IOS Press, Oct. 2024. [Online]. Available: http://dx.doi.org/10.3233/FAIA240968 [53] Z. Gong, Z. Sun, D. Huang, Q. Liang, J. M. Zhang, and D. Hao, “Tracy: Benchmarking execution efficiency of llm-based code translation,” arXiv preprint arXiv:2508.11468, 2025. [54] Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan, “Automated repair of programs from large language models,” 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1469–1481, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:255372224 [55] F. Li, J. Jiang, J. Sun, and H. Zhang, “Hybrid automated program repair by combining large language models and program analysis,” vol. 34, no. 7, Aug. 2025. [Online]. Available: https://doi.org/10.1145/3715004 [56] C. S. Xia, Y. Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 1482–1494. [57] Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. Zhang, “Magicoder: empowering code generation with oss-instruct,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024. [58] R. Li, L. B. allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. LI, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, J. Lamy-Poirier, J. Monteiro, N. Gontier, M.-H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, J. T. Stillerman, S. S. Patel, D. Abulkhanov, M. Zocca, M. Dey, Z. Zhang, U. Bhattacharyya, W. Yu, S. Luccioni, P. Villegas, F. Zhdanov, T. Lee, N. Timor, J. Ding, C. S. Schlesinger, H. Schoelkopf, J. Ebert, T. Dao, M. Mishra, A. Gu, C. J. Anderson, B. Dolan-Gavitt, D. Contractor, S. Reddy, D. Fried, D. Bahdanau, Y. Jernite, C. M. Ferrandis, S. Hughes, T. Wolf, A. Guha, L. V. Werra, and H. de Vries, “Starcoder: may the source be with you!” Transactions on Machine Learning Research, 2023, reproducibility Certification. [Online]. Available: https://openreview.net/forum?id=KoFOg41haE [59] OpenAI, “Gpt-3.5-turbo,” 2026, accessed: 2026-02-09. [Online]. Available: https://platform.openai.com/docs/models/gpt-3.5-turbo [60] OpenAI, “Gpt-4o-mini,” 2026, accessed: 2026-02-09. [Online]. Available: https://platform.openai.com/docs/models/gpt-4o-mini [61] DeepSeek, “Deepseek-v4-f,” 2026, accessed: 2026-02-09. [Online]. Available: https://www.deepseek.com/ [62] S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” ArXiv, vol. abs/2009.10297, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:221836101 [63] J. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological Bulletin, vol. 76, pp. 378–382, 11 1971. [64] N. K. Denzin and Y. S. Lincoln, The Sage handbook of qualitative research. sage, 2011. [65] Luogu, “Luogu online judge,” https://www.luogu.com.cn/, accessed: August 13, 2026.
Related documents
- An Extensive Empirical Study on Code Translation Technique#659325
- An Extensive Empirical Study on Code Translation Technique#657189
- RCL: A Retrieval-Confidence Layer for Detecting Insufficient Context in Enterprise Retrieval-Augmented Code Generation#998591
- Review of Translation and Relevance by EA Gutt#750690
- Development of multilingual and code-mixed translation app for Malaysian#432265
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive