Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We? KAZUKI KUSAMA, Kyushu University, Japan HONGLIN SHU, Kyushu University, Japan MASANARI KONDO, Kyushu University, Japan YASUTAKA KAMEI, Kyushu University, Japan
arXiv:2605.25374v1 [cs.SE] 25 May 2026
Log statements capture critical information for software maintenance activities such as testing, debugging, and failure analysis. Because of this importance, developers must carefully design log statements, which requires significant effort. To support developers, various end-to-end automated log statement generation approaches have been proposed, whereas these approaches have mainly been evaluated within a single programming language environment and their effectiveness in multilingual environments remains underexplored. In this paper, we therefore comparatively evaluate three state-of-the-art log statement generation approaches and five large language models (LLMs) across multiple programming languages. For this purpose, we constructed a multilingual benchmark comprising 150,000 instances across five programming languages. Our empirical results demonstrate that UniLog, a state-of-the-art approach, achieves the best overall performance, maintaining high effectiveness even in multilingual environments. We also observe substantial variance in the difficulty of log generation across languages: Python presents a greater challenge, whereas JavaScript yields comparatively better performance. Detailed analysis reveals that these disparities stem from variations in log insertion distributions and language-specific logging idioms. Our findings indicate that simply scaling model size or the volume of training data is insufficient for multilingual log generation; rather, designing approaches tailored to the specific characteristics of target languages is crucial. These findings suggest that future automated logging techniques should explicitly account for language-specific logging characteristics to achieve robust performance in multilingual software development environments. CCS Concepts: • Software and its engineering → Software maintenance tools. Additional Key Words and Phrases: Log Statement Generation, Multilingual Software Engineering, Large Language Models, Empirical Study, Automated Logging ACM Reference Format: Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei. 2026. Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?. ACM Trans. Softw. Eng. Methodol. 1, 1 (May 2026), 38 pages. https://doi.org/10.1145/ nnnnnnn.nnnnnnn
1
Introduction
Integrating log statements into code is a fundamental practice for capturing the dynamic internal behavior of software systems. These log statements serve as an indispensable resource, driving critical maintenance activities such as Authors’ Contact Information: Kazuki Kusama, Kyushu University, Nishi-ku, Fukuoka, Japan, [email protected]; Honglin Shu, Kyushu University, Nishi-ku, Fukuoka, Japan, [email protected]; Masanari Kondo, Kyushu University, Nishi-ku, Fukuoka, Japan, kondo@ait. kyushu-u.ac.jp; Yasutaka Kamei, Kyushu University, Nishi-ku, Fukuoka, Japan, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM Manuscript submitted to ACM
1
2
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei
testing, debugging, and failure analysis [7, 19]. To ensure that these log statements provide actionable insights while avoiding the system overhead of excessive logging and the information loss of insufficient logging, developers must carefully orchestrate their design by pinpointing optimal insertion locations, assigning appropriate severity levels, and formulating context-rich messages. Because this design process inherently relies on the manual judgment of developers, it is notoriously challenging and imposes a substantial burden on developer time and effort [15, 25, 61, 64]. To alleviate the burden of manual log statement design, prior research has actively explored automated log statement generation [18]. Previously, this process was typically decomposed into discrete subtasks, such as identifying insertion locations, determining appropriate severity levels, and generating message content, and developed specialized automation techniques for each [32, 35]. Advances in language models have led researchers to propose end-to-end approaches that seamlessly integrate these subtasks to holistically automate log generation [37, 56, 57]. For instance, UniLog [57] is an end-to-end approach for log statement generation that combines retrieval-based few-shot prompting with a warmup strategy to achieve strong performance. Despite significant advances in automated log statement generation with language models, current evaluation approaches still suffer from three important limitations: (I) The performance of log statement generation approaches in multilingual environments remains underexplored. Existing end-to-end log statement generation studies have been evaluated primarily on a single programming language, especially Java, and it remains unclear whether these approaches are equally effective in multilingual environments [37, 56, 57]. Modern software development commonly involves the use of multiple programming languages, such as Python and JavaScript [28]. This makes it necessary to move beyond single-language settings and evaluate the performance of log statement generation approaches across multiple programming languages. (II) The impact of training strategies on multilingual log statement generation remains unclear. Existing log statement generation approaches adopt different training and adaptation strategies, such as fine-tuning pre-trained language models, retrieval-based prompting, and lightweight warmup. In multilingual environments, it remains unclear whether models should be adapted separately for each programming language or trained jointly across multiple languages. Language-specific adaptation may better capture local logging practices, vocabulary, and coding styles, whereas multilingual training may help models learn common patterns shared across languages. Existing studies have not sufficiently investigated how such training strategies affect performance in multilingual log statement generation. Without clarifying this point, determining how training data and model adaptation should be designed for practical multilingual log statement generation remains difficult. (III) It remains unclear why automated log statement generation exhibits different performance across programming languages. Even if multilingual evaluation reveals performance differences across programming languages, such results alone do not explain why prediction is easier in some languages and more difficult in others. Log statement generation involves multiple decisions, including where to insert logs, which log levels to assign, and what message content to generate. These decisions may be influenced by language-specific factors, such as control structures, library usage, coding styles, and logging idioms. However, existing studies have not sufficiently analyzed how such language-specific logging characteristics are related to performance differences across programming languages. Without clarifying these factors, it remains difficult to explain cross-language performance gaps and to derive design guidelines for log statement generation approaches suitable for multilingual environments. To address these three limitations, this study empirically investigates end-to-end log statement generation in multilingual environments. Specifically, we construct a comprehensive multilingual benchmark from public GitHub Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
3
repositories across five programming languages: Java, Python, JavaScript, TypeScript, and C#. For each language, the benchmark contains 24,000 training instances, 3,000 validation instances, and 3,000 test instances, resulting in a total of 150,000 instances. Using this benchmark, we comparatively evaluate the performance of three representative existing approaches and five large language models (LLMs) under unified experimental conditions, including the same target languages, data construction criteria, and evaluation metrics. We also analyze the effects of different training strategies and examine language-specific logging characteristics, including log position, log level, and log message, thereby clarifying cross-language differences and the factors underlying them. We define the following three research questions (RQs): (RQ1) What is the performance of log statement generation approaches in multilingual environments? Approach: We evaluate three representative existing approaches, LANCE [37], FastLog [56], and UniLog [57]. Since general-purpose LLMs have shown promising performance in log statement generation, we also evaluate five LLMs in multilingual environments: Llama3, Qwen2.5-Coder, Mistral, DeepSeek-V3, and GPT-4.1 mini. Results: UniLog achieved the highest performance among the existing approaches in multilingual environments and perfectly predicted log statements with an average of 20.35% across all languages. Among the evaluated LLMs, DeepSeek-V3 achieved the highest performance and perfectly predicted log statements with an average of 10.96% across all languages. In addition, JavaScript showed the highest overall performance, whereas Python showed the lowest overall performance, which indicates that prediction difficulty differs across languages. (RQ2) What is the impact of training strategies on log statement generation in multilingual environments? Approach: We evaluate multiple training strategies, including monolingual and multilingual training, warmup-based adaptation, and training strategies used in representative end-to-end approaches. Results: Single-language training showed overall better performance than mixed-language training. UniLog also showed strong performance, although it requires less training data. (RQ3) How do language-specific characteristics affect automated log statement generation in multilingual environments? Approach: We examine insertion locations, log levels, and message content across programming languages to clarify the factors underlying cross-language performance differences. Results: Language-specific logging characteristics, such as insertion locations, log levels, and message content, differ across programming languages. For example, log statements inside loops, which are difficult to predict, are more common in Python and less common in JavaScript. This tendency partially explains the performance differences across languages. The main contributions of this study are as follows: • We release a large-scale multilingual benchmark 1 for end-to-end log statement generation, collected from public GitHub repositories and spanning five programming languages: Java, Python, JavaScript, TypeScript, and C#. For each language, the benchmark comprises 24,000 training, 3,000 validation, and 3,000 test instances, amounting to 150,000 instances in total. • We conduct an empirical study to evaluate the multilingual adaptability of current log generation approaches. Specifically, we benchmark three representative existing approaches (LANCE, FastLog, and UniLog) against five state-of-the-art LLMs (Llama3, Qwen2.5-Coder, Mistral, DeepSeek-V3, and GPT-4.1 mini) using a unified dataset and standardized evaluation metrics. 1 https://doi.org/10.5281/zenodo.20279312
Manuscript submitted to ACM
4
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei • To ensure reproducibility and transparency, we release the dataset, experimental code, and results used in this study. This enables the research community to verify the findings of this study and promote further research.
2
Related Work
This section reviews studies on automated log statement generation and studies on multilingual research in software engineering. 2.1
Log Statement Generation
Previous log statement generation research first focused on automating the subtasks [11, 12, 24, 32, 33]. For example, Li et al. [32] proposed a deep learning framework that estimates log insertion locations at the code block level by incorporating syntactic and semantic information from source code. For log level prediction, Li et al. [33] proposed DeepLV, which models log level recommendation as an ordinal classification task using syntactic context and log message features extracted from source code. Ding et al. [11] proposed LoGenText, which generates natural language log messages based on neural machine translation. Around 2022, research more focused on end-to-end log statement generation using language models [31, 37, 56, 57]. Mastropaolo et al. [37] proposed LANCE, which is the first pre-trained language model (PLM)-based end-to-end approach built on T5 with fine-tuning. LEONID, which extends LANCE, has also been proposed [36]. LEONID integrates deep learning and information retrieval, which is abbreviated as IR, to improve accuracy, but it involves a substantial increase in computational cost while achieving only limited performance improvement. Xie et al. [56] proposed FastLog, which is another PLM-based end-to-end log statement generation approach. FastLog introduces a design that splits the input code and predicts insertion locations at the token level in order to improve the accuracy of log insertion location prediction. Xu et al. [57] proposed UniLog, which is an LLM-based end-to-end log statement generation approach that combines few-shot prompting with a warmup strategy, which is a lightweight fine-tuning approach. UniLog reported high performance in log insertion location and log level prediction and demonstrated the effectiveness of retrieval-based demonstrations and lightweight model adaptation. Li et al. [31] proposed LOGIMPROVER, which is an LLM-based end-to-end logging automation framework for logging quality improvement. LOGIMPROVER combines candidate identification, contextualized refinement, false positive pruning, and holistic logging patch generation to improve logging quality. Their results demonstrated the effectiveness of this end-to-end framework in large-scale codebases. In addition to these end-to-end approaches, Li et al. [30] conducted the first empirical study on the effectiveness of LLMs for automated logging generation. They evaluated multiple LLMs on both code collected from open-source projects and semantically equivalent transformed variants, and showed that, although LLMs are promising for logging generation, their performance and generalization capability remain limited. While there are various automated log statement generation approaches, prior studies have mainly evaluated these approaches in limited language settings, primarily on Java and, in some cases, Go. As a result, empirical evidence remains limited regarding how current log statement generation approaches perform across multiple programming languages. 2.2
Multilingual Software Engineering Research
In real software development, it is common to use multiple programming languages together [28, 29]. Li et al. [29] characterized how open-source multilingual systems are constructed and how programming languages are selected in such systems, providing further evidence that multilingual software development is a common and practically important Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
5
setting. Prior work has also shown that multilingual software development involves diverse issues and challenges arising from cross-language interfacing, data handling, and the complexity of language-specific features [59]. In addition, Yang et al. [58] argued that analyzing multilingual code holistically is important, while fully language-agnostic designs may face practicality challenges. To demonstrate the practical usefulness of approaches in software engineering, it is important not only to confirm effectiveness on a single language but also to evaluate generality across multiple languages. From this perspective, benchmark construction and empirical evaluation for software engineering tasks in multilingual environments have been actively studied [50, 51, 54, 60, 62]. Zan et al. [62] proposed SWE-bench-Multi, which extends SWE-bench, a benchmark widely used in the context of automated program repair, to eight languages, and they provided a benchmark for multilingual automated program repair. Their empirical study showed that existing program repair approaches that were effective on SWE-bench often generalize poorly to languages other than Python. As possible reasons, they reported that existing approaches are mainly optimized for Python and that task difficulty differs across languages. Wang et al. [54] empirically evaluated the performance of PLMs and LLMs across seven programming languages in the context of automated vulnerability repair. They found that repair success rates vary substantially across programming languages, with the highest success rate of 31.59% on Go, whereas the success rate on C++ was only 6.73%. Yang et al. [60] addressed multilingual bug detection and localization by proposing a deep learning-based technique for detecting and localizing multilingual bugs. Their work highlighted that bug detection and localization support has long been centered on single-language software despite the prevalence of multilingual systems, and demonstrated the importance of modeling cross-language structures in multilingual settings. Shu et al. [51] evaluated the performance of PLMs and LLMs across seven programming languages in the context of automated vulnerability detection. They reported that detection performance also varies across programming languages, with the highest accuracy of 80.82% on Go, whereas the lowest accuracy of 66.26% on Python, and they confirmed clear cross-language differences in multilingual environments. These studies indicate that multilingual evaluation is important in software engineering and support the necessity of verifying the effectiveness of approaches across different programming languages. As discussed, existing log statement generation research has mainly evaluated approaches in single-language settings. Hence, in this paper, we aim to construct a multilingual benchmark for log statement generation and conduct a large-scale empirical study to evaluate the performance of log statement generation approaches. To deeply understand the log statement generation on multilingual environments, we investigate not only the performance, but also the impact of LLM training strategies and language-specific logging characteristics across multiple programming languages.
3
EXPERIMENTAL SETUP
This section describes the experimental design of this study, as shown in Figure 1. We first provide an overview of the evaluated log statement generation approaches and training strategies. We then describe the implementation details and evaluation metrics used in this study. We evaluate the performance of log statement generation approaches, including existing end-to-end approaches and LLMs, in multilingual environments (RQ1). Next, we investigate the effect of different training strategies on multilingual log statement generation performance (RQ2). Finally, we analyze language-specific characteristics of log statements to identify factors that affect prediction difficulty across programming languages (RQ3). Manuscript submitted to ACM
6
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei Benchmark Construction
RQ1: Performance Analysis Log Statement Generation Approaches
Collect Public Repositories Public repositories across five languages
Log Position Log Level
Multilingual Benchmark
Log Message Generate Log Statements
Compare the effectiveness of existing approaches and LLMs across 5 programming languages.
Filter Projects and Libraries Project criteria and logging-library filters
RQ2: Training Strategy Analysis Training Strategies
Log Position
Extract Method and Logs Methods and log statements
Log Level Multilingual Benchmark
Monolingual Training
Log Message
Multilingual Training
Analyze the impact of different training strategies on log statement generation performance. Construct and Split Instances
RQ3: Language-Specific Logging Analysis
Leave-one-out and train/valid/test split
Fine-grained Logging Analysis Position Analysis
Multilingual Benchmark 150,000 instances across five languages (24k / 3k /3k per language)
Level Analysis
Message Analysis
Multilingual Benchmark
Insights into Cross-language Difficulty Differences
Identify where performance differences arise and associated language-specific characteristics.
Fig. 1. The overview of the experimental design.
3.1
Experimental Techniques and Settings
In this study, we evaluate the performance of existing end-to-end approaches and LLMs, as well as the impact of training strategies, on multilingual log statement generation. This section describes the approaches and settings compared in this study. 3.1.1 Existing End-to-End Log Statement Generation Approaches. To evaluate how effectively existing end-to-end log statement generation approaches work in multilingual environments, we conducted a literature review of related papers published in the software engineering community and selected baselines. Specifically, we selected the following three approaches, which are widely used as baselines in log statement generation research [52, 63]: • LANCE [37] is the first end-to-end approach leveraging a pre-trained language model (PLM) for automated log statement insertion. Built upon the T5 architecture, LANCE first acquires a general understanding of Java syntax and typical logging locations through a dedicated pre-training phase. It is subsequently fine-tuned to accurately synthesize and inject contextually appropriate log statements directly into the source code. • FastLog [56] extends the foundational concepts of LANCE [37] by introducing a decoupled, two-stage PLM architecture. In the first stage, the model explicitly predicts optimal log insertion locations; in the subsequent stage, it synthesizes the corresponding log statements for those specific sites. This architectural refinement enables FastLog to achieve both higher accuracy and significantly faster generation speeds compared to its predecessor. Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
7
• UniLog [57] represents an LLM-driven framework for end-to-end log statement generation. UniLog incorporates lightweight parameter tuning, which is referred to as warmup, and retrieval-based few-shot prompting to generate log insertion locations and log statements. 3.1.2 Evaluated LLMs. We studied five representative LLMs. These models have been widely adopted in the literature on various software engineering tasks [1, 23, 51, 55, 62, 63]. In this study, we categorize them into two types based on how they are used, which we refer to as self-hosted and API-based. Self-hosted category refers to LLMs that can be deployed locally within our own computational infrastructure in the experimental environment of this study (Section 3.2), whereas API-based refers to LLMs that are accessed through external service APIs. We strategically selected three self-hosted LLMs with parameter counts ranging from 7B to 8B. This choice is motivated by deployment constraints: many researchers lack the high-end computational infrastructure necessary to deploy or instruction-tune significantly larger models. The details of these LLMs are as follows: • Llama3 (self-hosted) [17] is a family of instruction-tuned LLMs released by Meta and designed for tasks that require multilingual dialogue, coding, and long-context processing. It has shown strong performance on a variety of tasks, including mathematical reasoning and code generation. In this study, we use Llama3.1-8B-Instruct. • Qwen2.5-Coder (self-hosted) [21] is a code-specialized LLM in the Qwen2.5 [47] family. It substantially improves code-related capabilities while maintaining the strengths of Qwen2.5 in mathematics and general abilities. In this study, we use Qwen2.5-Coder-7B. • Mistral (self-hosted) [22] is an LLM released by Mistral AI and targets a wide range of tasks, including dialogue generation, reasoning, and code generation. Mistral is known for its efficient performance, which is supported by fast inference speed and broad context handling. In this study, we use Mistral-7B-Instruct-v0.3. • GPT-4.1 mini (API-based) [41] is a small LLM provided by OpenAI that is fast and low-cost and is suitable for tasks that require instruction following. It substantially outperforms GPT-4o while reducing latency and cost. • DeepSeek-V3 (API-based) [10] is a Mixture-of-Experts (MoE) LLM that was pre-trained on more than 14T tokens and supports a 128k context window. It has been reported to achieve state-of-the-art performance among open-source models and to outperform strong models such as GPT-4o [40] and Claude3.5 [4]. 3.1.3 Strategies for LoRA Fine-tuning. In this study, we investigate the effect of training strategies on multilingual log statement generation. Specifically, we compare training strategies for parameter-efficient fine-tuning (PEFT) with LoRA [20]. LoRA is an approach that adapts a foundation model to a task with a small number of additional parameters by freezing the weights of the base model and learning only low-rank matrices. Prior work has reported that LoRA achieves better performance on log statement generation tasks than other PEFT approaches, such as Prompt tuning and Prefix tuning [63]. In this study, we compare the following two strategies as learning units for LoRA. • Monolingual LoRA (Mono-LoRA): We perform LoRA fine-tuning independently for each language. In this setting, language-specific adaptation parameters are learned for each language in order to acquire log statement generation capability specialized for that language. This allows us to evaluate the effect of locally incorporating language-specific vocabulary, expressions, and logging practices. • Multilingual LoRA (Multi-LoRA): We integrate the training data of the five languages and train a single LoRA. After training, the same adaptation parameters are applied to all languages during inference. In this setting, we Manuscript submitted to ACM
8
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei Output
Input
@Override public IAtom getAtom(int idx) { return super.getAtom(idx); }
Predict
@Override public IAtom getAtom(int idx) { logger.debug(“Getting atom at: ”, idx); return super.getAtom(idx); }
Fig. 2. Input-output design of LANCE for fine-tuning and inference.
Stage-1
Input: public IAtom getAtom ( int idx ) { return super . getAtom ( idx ) ; }
evaluate whether cross-language log statement generation capability can be acquired by capturing code patterns Token and log insertion strategies that are shared across multiple languages. classification
3.2
Implementation Stage-2
Output: 0.01 0.01
0.01
......
0.95
0.0
......
Input: public IAtom getAtom ( int idx ) { <mask> return super . getAtom ( idx ) ; }
This subsection describes the implementation details, task design, and prompt design for the existing end-to-end log logger.debug(“Getting atom at: ”, idx); Output:
Seq2Seq text Final Output: public IAtom getAtom ( int idx ) {logger.debug(“Getting atom at: ”, idx); LANCE.generation Figure 2 illustrates the input andsuper output formats (ofidx LANCE return . getAtom ) ; } for both fine-tuning and inference. LANCE
statement generation approaches and LLMs.
receives a source method with the target log statement removed as input and generates the complete method with the predicted log statement inserted. Since LANCE originally targets Java, we adapted the framework to support multilingual scenarios by modifying the backbone model and training strategy, as described below. • CodeT5+ Backbone:Input We replace the general-text T5 backbone with CodeT5+. Because CodeT5+ is explicitly pre-trained on a massive multilingual code corpus, unlike T5, which primarily targets natural language, this Select <line#> and insert log level and log
substitution eliminates message after <line#> the prohibitive cost of collecting and pre-training on language-specific code corpora. • Omission of Pre-training: We bypass the resource-intensive pre-training phase of LANCE and rely exclusively Output Example: <code method>
Predict on fine-tuning. is twofold: (1) the CodeT5+ backbone already encapsulates prior Label: <line#> This <log decision statement> <line3> logger.debug(“Getting atomknowledge at: ”, idx); across
Example: <code method> various programming languages, and (2) prior ablation studies [37] indicate that this specific pre-training step
Label: <line#> <log statement> ... Query: <code method> To accommodate these changes, we aligned our hyperparameters with standard CodeT5+ configurations, specifically
yields only marginal performance gains.
setting the learning rate to 2e-5 and the batch size to 16. To empirically validate these adjustments, we fine-tuned CodeT5+ on the original LANCE dataset and successfully reproduced the reported performance of the paper within a margin of three points. FastLog. Figure 3 illustrates the input and output formats of FastLog during both fine-tuning and inference. FastLog automates log statement insertion via a sequential, two-stage pipeline. In Stage-1, the model performs a token classification task over each token in the input method, predicting whether a log statement should be inserted immediately after that token; during inference, the token with the highest probability of being classified as 1 is selected as the insertion location. In Stage-2, a <mask> token is inserted as a placeholder immediately after the predicted insertion location, and the log statement body is generated in a sequence-to-sequence manner to fill the <mask>. Finally, the <mask> is replaced with the generated log statement, yielding the complete method with the inserted log statement without regenerating the surrounding non-log code. In this study, we leveraged the official FastLog replication package to fine-tune two distinct PLBART-based [2] models: a token classification model for Stage-1 and a log generation model for Stage-2. Adhering closely to the original configurations of the authors, we applied a learning rate of 2e-5 and a batch size of 8. The Stage-1 model was trained for 10 epochs, while the Stage-2 model underwent 30 epochs of fine-tuning. Manuscript submitted to ACM
Output
Input
@Override public IAtom getAtom(int idx) { return super.getAtom(idx); }
Predict
@Override public IAtom getAtom(int idx) { logger.debug(“Getting atom at: ”, idx); return super.getAtom(idx); }
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
Stage-1 Token classification Stage-2 Seq2Seq text generation
9
Input: public IAtom getAtom ( int idx ) { return super . getAtom ( idx ) ; } Output: 0.01 0.01
0.01
......
...... ...... 0.95 0.95 0.0 0.0
Input: public IAtom getAtom ( int idx ) { <mask> return super . getAtom ( idx ) ; } Output: logger.debug(“Getting atom at: ”, idx); Final Output: public IAtom getAtom ( int idx ) {logger.debug(“Getting atom at: ”, idx); return super . getAtom ( idx ) ; }
Fig. 3. Input-output design of FastLog for fine-tuning and inference. Input
Select <line#> and insert log level and log message after <line#> Example: <code method> Label: <line#> <log statement> Example: <code method> Label: <line#> <log statement> ... Query: <code method>
Output
Predict
<line3> logger.debug(“Getting atom at: ”, idx);
Fig. 4. Input-output design of UniLog and LLMs for warmup and inference.
UniLog. Figure 4 illustrates the input and output formats of UniLog, an end-to-end LLM-based approach for log statement generation by lightweight parameter tuning (Warmup) and retrieval-based few-shot prompting. Given a source method with the target log statement removed, UniLog simultaneously predicts the optimal insertion location (line ID) and generates the corresponding log statement (level and message). Because the original backbone of UniLog, Codex, is deprecated, we use GPT-4.1 mini as the backbone in our reimplementation due to its favorable balance between cost-efficiency and performance. The implementation of UniLog centers on two primary mechanisms: (1) Retrieval-Augmented Prompting. For both warmup and inference, prompts consist of a task instruction, the target method (Query), and the top-five most similar training examples retrieved via cosine similarity. To build spatial awareness, code lines in the prompt are explicitly annotated with <line#> tags so that the model can link code context directly to the predicted line ID. (2) Warmup. Prior to inference, the model undergoes parameter tuning using 500 randomly selected validation queries. This allows the LLM to adapt specifically to the logging patterns of the target dataset. During inference, the same prompt construction is applied to the test data to evaluate the generated line IDs and log statements. General-purpose LLMs. To formulate the task and construct the prompts for the evaluated LLMs, we mirror the architecture established by UniLog, since it is an LLM-based baseline in our study and using the same task setting enables a fairer comparison among LLM-based approaches. Specifically, the model receives a source method with the target log statement removed as input, and it outputs both the predicted insertion location and the complete log statement (comprising the severity level and message content). Manuscript submitted to ACM
10
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei We employ retrieval-based few-shot prompting, a strategy demonstrated by Zhong et al. [63] to consistently
outperform alternative techniques such as Chain-of-Thought. Adhering to the prompt structure of UniLog, which encompasses a task instruction, an Example with its corresponding Label, and a Query, we specifically utilize a 1-shot setting [63]. For each test method (the Query), we retrieve the single most relevant training instance based on cosine similarity to serve as the Example. To guarantee reproducibility across generations, the decoding temperature is strictly set to 0. To investigate the impact of training strategies on LLM performance in RQ2, we fine-tune the models using Low-Rank Adaptation (LoRA). The input-output formulation and prompt template remain identical to those used during inference, with queries sourced from the validation dataset. For the LoRA hyperparameters, we configure the rank 𝑟 to 16, the scaling coefficient 𝛼 to 32, the learning rate to 1e-4, and the batch size to 64. Experiment Environment. GPT-4.1 mini was used through the OpenAI API. Specifically, we used gpt-4.1-mini2025-04-14 as the experimental model version of GPT-4.1 mini. DeepSeek-V3 was used through the DeepSeek API. All open-source LLMs were downloaded from Hugging Face, specifically Llama3, Qwen2.5-Coder, and Mistral. For all LLMs, experiments were conducted on a machine equipped with 251GiB RAM, Ubuntu 24.04.3 LTS, and an RTX 6000 Ada Generation GPU with 48GB, with an Intel(R) Xeon(R) w7-2595X 26-core processor. 3.3
Evaluation Metrics
To evaluate the performance of automated log statement generation, we adopt evaluation metrics that are commonly used in end-to-end log statement generation research [56, 57]. Specifically, we evaluate the correctness of the insertion location, log level, and message content of each log statement using the following measures: • Position Accuracy: The proportion of cases in which the predicted insertion location exactly matches the ground truth. • Level Accuracy: The proportion of cases in which the predicted log level matches the ground truth. • Message Accuracy: The proportion of cases in which the generated message exactly matches the ground truth message. • All Accuracy: The proportion of cases in which the insertion location, log level, and message all simultaneously match the ground truth. Because Message Accuracy is based on exact matching, it does not evaluate cases in which the predicted message and the reference message convey the same meaning but differ in expression. To evaluate how appropriately log messages are predicted, we additionally use the following text similarity metrics: • BLEU [43] is an automatic evaluation metric that is widely used in machine translation and measures n-gram overlap between a generated sentence and a reference sentence. In this study, we report sentence-level BLEU with smoothing on a scale from 0 to 100. Following the standard definition, BLEU integrates 1 to 4 gram precision with uniform weights of 0.25 each and includes a brevity penalty. • ROUGE [34] is an automatic evaluation metric that is widely used in summarization and measures lexical overlap between a generated sentence and a reference sentence. In this study, we report sentence-level ROUGE-L based on the longest common subsequence, on a scale from 0 to 100. ROUGE-L evaluates overlap while preserving order and provides a perspective different from that of BLEU. These n-gram-based metrics have also been widely used in prior work to measure the similarity between generated log messages and target messages [52, 56]. Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
11
Table 1. Statistics of the full dataset across five programming languages.
4
Language
# Methods
# Instances
Train
Valid
Test
Java Python JavaScript TypeScript C#
419,404 197,394 20,560 36,530 60,708
667,860 417,273 34,539 64,613 119,412
534,288 333,818 27,631 51,690 95,529
66,786 41,727 3,453 6,461 11,941
66,786 41,728 3,455 6,462 11,942
Benchmark Construction
To construct our multilingual dataset, we mined open-source repositories hosted on GitHub across five prominent programming languages: Java, Python, JavaScript, TypeScript, and C#. We selected these five languages because they were the top five most commonly used programming languages on GitHub in 2025, according to the 2025 GitHub Octoverse report. 2 Project Mining. We leveraged the repository mining tool provided by Dabic et al. [9] to systematically extract candidate projects across the five target languages. To ensure dataset quality and align with established methodologies [37], we filtered these repositories based on the following criteria: • Having at least 500 commits, 10 contributors, and 10 stars, to avoid toy or personal projects. • Not being forked to reduce the chance of mining duplicated code. Following this rigorous filtering phase, we retained 9,018 candidate projects for Java, 17,239 for Python, 12,728 for JavaScript, 11,581 for TypeScript, and 3,750 for C#. Logging Dependency Filtering. To ensure dataset consistency, we restricted our selection to projects utilizing representative logging libraries or standard logging mechanisms specific to each language. Because substantial variations in logging APIs and severity schemes can destabilize log extraction and level normalization, filtering out non-standard implementations is crucial to prevent heterogeneous training targets. Consequently, we limited our dataset to projects explicitly relying on Log4j [53] for Java, the standard logging module [14] for Python, Winston [48] or Pino [44] for JavaScript and TypeScript, and NLog [39], log4net [5], or Serilog [49] for C#. We identified these dependencies primarily by analyzing language-specific build configuration files (e.g., pom.xml for Java, package.json for JavaScript and TypeScript, and .csproj for C#), supplementing this process when necessary by scanning source code for explicit import statements in Java and Python, require calls in JavaScript and TypeScript, and using directives in C#. Method Extraction and Filtering. Log statement generation is typically formulated as a method-level task, where the model generates log statements for a given method [37, 56, 57]. We systematically extracted individual methods from the curated projects. In alignment with established practices [37], we retained only methods that satisfy #tokens ≤ 512 and #tokens > 10, where #tokens denotes the number of tokens after excluding comments. This constraint is crucial for mitigating the computational overhead associated with LLM training and inference. To eliminate the risk of data leakage, we computed the hash value of each method and removed duplicate methods with identical hashes. Log Statement Identification. To identify log statements within the filtered methods, we implemented a pattern matching approach based on pairs of logger identifiers and log level names. Specifically, we identified method calls as log statements when log-level method names (e.g., info, debug, and error) were invoked on frequently used logger identifiers (e.g., log, logger, and logging). To reduce missed extractions caused by notational variation across languages 2 https://github.blog/news-insights/octoverse/octoverse-a-new-developer-joins-github-every-second-as-ai-leads-typescript-to-1/
Manuscript submitted to ACM
Example: <code method> Label: <line#> <log statement> ... Query: <code method>
12
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei Instance 1
Input:
Original Method
void processUser(User user) { logger.info(“start processing”); if (user == null) { logger.error(“user is null”); return; } }
void processUser(User user) { if (user == null) { logger.error(“user is null”); return; } } Output: logger.info(“start processing”);
Instance 2
Input: void processUser(User user) { logger.info(“start processing”) if (user == null) { return; } } Output: logger.error(“user is null”);
Fig. 5. Illustration of the leave-one-out instance construction process for log statements.
and libraries, we also normalized differences in letter case and synonymous expressions, such as warning/warn and information/info, treating them as equivalent categories. The # Methods column in Table 1 reports the number of methods obtained for each language. Code Formatting. To reduce stylistic inconsistency and ensure data quality, we reformatted the extracted methods using automatic code formatting, following prior work [57]. While the prior work applied Google Java Format to Java code, we extended this preprocessing step to a multilingual setting. Specifically, we applied Google Java Format [16] to Java, and for the other languages, we selected widely used formatting tools with strong adoption in practice: Black [46] for Python, Prettier [45] for JavaScript and TypeScript, and dotnet format [38] for C#. Instance Construction. We generated our final datasets by converting the formatted methods into discrete data instances. Building upon established methodologies from LANCE [37] and UniLog [57], we structure each instance as an input-output pair comprising a source method and a single target log statement. For methods containing multiple log statements, we employ a leave-one-out extraction strategy. Specifically, only the focal target log statement is removed, while all other log statements remain intact to provide vital contextual information for the model. Figure 5 illustrates this leave-one-out extraction process. In this example, the original method contains two log statements; therefore, two instances are generated by removing one log statement at a time. In each instance, the removed log statement serves as the expected output, while the remaining log statement is kept in the input method. By iteratively repeating this process, a single source method containing 𝑛 log statements generates exactly 𝑛 distinct instances. The final yield of instances per language is detailed in the # Instances column of Table 1. Dataset Split. Following prior work [63], we partitioned the final dataset into training, validation, and test sets using a target ratio of 8:1:1. The exact number of instances per set for each language is detailed in the Train, Valid, and Test columns of Table 1. To prevent data leakage, where highly similar code snippets from the same file inadvertently appear Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
13
Table 2. Log level distribution across five languages. Each cell reports the number of instances (#), percentage (%), and an in-cell bar proportional to the percentage. Java
Level #
Python
C#
JavaScript
TypeScript
%
#
%
#
%
#
%
#
%
Critical/Fatal Error Warn Info Debug Trace/Verbose
129 0.43 7,740 25.8 3,824 12.7 8,829 29.4 8,276 27.6 1,202 4.0
286 4,754 3,887 14,543 6,530 0
1.0 15.8 13.0 48.5 21.8 0.0
339 10,183 3,111 5,537 7,407 3,423
1.1 33.9 10.4 18.5 24.7 11.4
67 0.2 11,277 37.6 2,996 10.0 8,106 27.0 7,120 23.7 434 1.4
56 9,354 3,555 8,892 7,454 689
0.2 31.2 11.9 29.6 24.8 2.3
Total
30,000
30,000
30,000
30,000
30,000
across multiple splits, we enforced a strict file-level splitting strategy. This guarantees that all instances originating from a single file are assigned exclusively to one of the three sets. To avoid confounding the effect of programming language with differences in data volume, we randomly sampled exactly 24,000 training, 3,000 validation, and 3,000 test instances per language. Although the natural distribution of programming languages in real-world repositories is inherently imbalanced, we maintained a unified instance distribution across the five languages to enable a fair comparison of multilingual performance. In particular, this design helps disentangle whether performance differences arise from language-specific characteristics or simply from differences in the number of fine-tuning instances available for each language. This yielded a perfectly balanced multilingual benchmark comprising 30,000 instances per language (150,000 total instances). To ensure reproducibility and facilitate future research, both the full dataset and this balanced experimental subset are publicly available in our replication package. Log Level Normalization. Table 2 shows the distribution of log levels in the dataset for each language. Following prior work [52], we identified representative level expressions, such as info, fatal, and debug, using regular expressions, normalized notational variants that are semantically equivalent (e.g., synonyms and case differences), and mapped them to the following six standardized levels: Critical/Fatal, Error, Warn, Info, Debug, and Trace/Verbose. Critical/Fatal represents the most severe situations, capturing events that may directly lead to system shutdown or unrecoverable failures. Error denotes the next severity tier, recording problems that may cause system failures or prevent continued processing. Warn captures signs of potential degradation — such as unexpected states or inconsistencies — that do not immediately halt processing but may escalate to Error. Info records information pertaining to system operating status, such as progress updates and state transitions. Debug captures diagnostic information used to identify root causes during development and troubleshooting. Trace/Verbose is the most granular level, recording fine-grained details that enable tracing of program execution flow, such as method calls, branch traversal, and internal state changes. As shown in Table 2, Error, Info, and Debug account for the majority of log statements across all languages. 5
Results
This section presents the experimental results for the three RQs. 5.1 RQ1: What is the performance of log statement generation approaches in multilingual environments? Approach. In RQ1, we evaluate the performance of log statement generation approaches in multilingual environments. Specifically, we compare the performance of existing end-to-end log statement generation approaches and the generalpurpose LLMs. Manuscript submitted to ACM
14
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei For the existing end-to-end approaches, we first applied each approach to the training set of each programming
language for either fine-tuning or warmup. Upon completion of this tuning phase, we evaluated the adapted approaches on the unseen test set. For the LLMs, each model was evaluated under the same task setting using few-shot prompting with data from the same programming language as the target of evaluation. Results. Table 3 presents the evaluation results for each approach, each programming language, and the average across the five languages. Boldface indicates the best value for each combination of language and evaluation metric. Underline indicates the best value among the five LLMs. Observation 1: UniLog achieves the best overall performance among all evaluated approaches in multilingual environments. On average across the five languages, UniLog achieved the best values in Position Accuracy (75.23%), Level Accuracy (71.26%), Message Accuracy (22.74%), BLEU (26.78), and All Accuracy (20.35%). All Accuracy is the strictest metric because it requires the insertion location, level, and message content to all be correct at the same time. Even at the language level, UniLog achieved the best All Accuracy in all five languages, with 11.87% for Java, 8.30% for Python, 44.83% for JavaScript, 15.17% for TypeScript, and 21.60% for C#. These results suggest that UniLog not only predicts each element correctly in isolation, but also captures the dependencies among where to insert, which level to assign, and what content to include relatively stably even in multilingual settings. Among the evaluated LLMs, DeepSeek-V3 achieved the best overall performance. On average across the five languages, DeepSeek-V3 achieved the highest values among LLMs in Position Accuracy (50.62%), Level Accuracy (64.39%), Message Accuracy (13.47%), BLEU (18.74), ROUGE (34.22), and All Accuracy (10.96%). This indicates that DeepSeek-V3 is the most effective LLM for multilingual log statement generation among the evaluated models. At the same time, UniLog achieves substantially higher All Accuracy than DeepSeek-V3, and this difference is statistically significant on the combined 15,000 test instances (20.35% vs. 10.96%; difference = 9.39 percentage points; McNemar test, 𝑝 < .001; odds ratio = 8.30). This result highlights the effectiveness of task-specific design for log statement generation, particularly the combination of retrieval-based prompting and warmup, in multilingual environments. Observation 2: The impact of cross-language differences varies by approach, and an approach that performs well in one language may vary in effectiveness across others. The results show that the effect of programming language differs across approaches. For example, FastLog shows performance relatively close to UniLog in Java, with Position Accuracy of 74.20%, Level Accuracy of 60.87%, and BLEU of 19.33, where the BLEU score even exceeds that of UniLog at 17.25. In Python, FastLog achieves only 38.90% in Position Accuracy and 44.03% in Level Accuracy, which are substantially lower than 61.20% and 62.10% for UniLog. These results indicate that even if an approach is competitive in one language, its effectiveness may still vary across other languages. Overall, these findings suggest that judging the effectiveness of log statement generation approaches based only on results from a single language is insufficient. In particular, in multilingual environments, the robustness of each approach can vary substantially depending on the target language, and therefore cross-language comparison is essential. Observation 3: The difficulty of log statement generation varies substantially across programming languages. Across both existing approaches and LLMs, JavaScript tends to show the highest performance, whereas Python tends to show the lowest performance. For example, UniLog achieved an All Accuracy of 44.83% for JavaScript, but only 8.30% for Python. Similarly, DeepSeek-V3 achieved an All Accuracy of 20.27% for JavaScript, whereas its performance on Python was much lower. This tendency indicates that prediction difficulty differs substantially across programming languages. Such differences may be influenced by language-specific factors, such as syntax, implementation practices, logging APIs, and coding styles. They may also be affected by the number of candidate insertion locations and the degree to which Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
15
Table 3. The performance of log statement generation approaches in multilingual settings. Position
Level
Message
All
Language
Approach
Acc
Acc
Acc
BLEU
ROUGE
Acc
Java
LANCE FastLog UniLog Llama3 Qwen2.5-Coder Mistral GPT-4.1 mini DeepSeek-V3
61.77 74.20 75.97 21.27 35.37 25.97 35.27 44.67
56.43 60.87 67.97 42.83 52.03 50.10 55.70 60.77
6.13 9.27 14.33 6.93 7.13 4.60 4.53 8.20
7.48 19.33 17.25 10.07 10.60 8.87 9.49 13.08
9.05 45.48 34.01 23.70 24.91 22.56 24.84 29.66
4.83 7.27 11.87 3.47 4.40 3.03 3.60 6.30
Python
LANCE FastLog UniLog Llama3 Qwen2.5-Coder Mistral GPT-4.1 mini DeepSeek-V3
39.97 38.90 61.20 9.40 19.23 13.27 20.70 33.30
43.87 44.03 62.10 43.77 43.23 49.67 51.23 55.23
3.37 3.17 11.33 4.23 4.27 2.57 1.87 5.00
3.55 8.23 13.23 4.71 5.71 5.02 4.69 7.71
4.81 32.94 26.76 12.01 15.53 13.94 14.46 18.70
2.60 2.03 8.30 1.97 1.80 0.73 0.97 3.10
JavaScript
LANCE FastLog UniLog Llama3 Qwen2.5-Coder Mistral GPT-4.1 mini DeepSeek-V3
67.67 77.70 84.17 39.00 41.40 34.60 42.37 69.77
62.07 74.40 81.93 69.97 51.13 59.83 58.63 76.83
10.77 23.17 46.83 24.40 8.07 9.80 7.20 23.90
11.83 38.41 51.59 31.49 21.81 17.30 14.39 32.57
14.57 59.98 67.08 48.31 37.84 34.01 31.76 50.87
9.73 20.90 44.83 11.83 8.07 5.20 4.07 20.27
TypeScript
LANCE FastLog UniLog Llama3 Qwen2.5-Coder Mistral GPT-4.1 mini DeepSeek-V3
58.27 68.63 78.53 27.77 30.07 27.60 32.30 45.47
56.57 62.83 72.83 55.50 47.50 51.47 52.33 63.17
4.20 6.90 17.03 9.73 8.33 6.77 6.60 12.03
4.86 20.01 24.96 15.74 14.21 12.88 12.94 19.23
6.65 44.27 43.19 29.37 29.43 27.20 27.73 35.19
3.37 6.23 15.17 6.47 4.67 3.80 4.50 9.20
C#
LANCE FastLog UniLog Llama3 Qwen2.5-Coder Mistral GPT-4.1 mini DeepSeek-V3
59.03 65.37 76.27 17.40 28.77 14.00 46.57 59.87
50.50 63.93 71.47 52.80 46.87 49.30 53.57 65.93
8.93 11.67 24.17 15.70 12.80 9.33 11.07 18.20
11.18 23.02 26.85 17.80 15.45 11.98 13.99 21.09
13.41 48.90 44.67 31.27 29.50 25.71 27.95 36.69
7.60 10.13 21.60 6.33 7.13 3.67 9.13 15.93
Average
LANCE FastLog UniLog Llama3 Qwen2.5-Coder Mistral GPT-4.1 mini DeepSeek-V3
57.34 64.96 75.23 22.97 30.97 23.09 35.44 50.62
53.89 61.21 71.26 52.97 48.15 52.07 54.29 64.39
6.68 10.84 22.74 12.20 8.12 6.61 6.25 13.47
7.78 21.80 26.78 15.96 13.56 11.21 11.10 18.74
9.70 46.31 43.14 28.93 27.44 24.68 25.35 34.22
5.63 9.31 20.35 6.01 5.21 3.29 4.45 10.96 Manuscript submitted to ACM
16
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei
log messages are formulaic. In particular, Python shows low Message Accuracy for many approaches, which suggests that the diversity and context dependence of message content may be a bottleneck for overall performance. These findings indicate that evaluating log statement generation only on a single programming language is insufficient, and that multilingual evaluation is necessary to understand the practical effectiveness and limitations of each approach. To further clarify the factors behind these cross-language differences, RQ3 provides a detailed analysis of language-specific characteristics of log statements. Observation 4: Self-hosted LLMs can be competitive with API-based LLMs on some subtasks, but their end-to-end performance remains limited. Although DeepSeek-V3 achieved the best overall performance among the evaluated LLMs, self-hosted LLMs also showed competitive performance on some individual metrics. For example, Qwen2.5-Coder achieved an average Position Accuracy of 30.97%, which is close to GPT-4.1 mini at 35.44%. For Level Accuracy, Llama3 achieved 52.97%, whereas GPT-4.1 mini achieved 54.29%, which are nearly the same. On messagerelated metrics, Llama3 achieved a Message Accuracy of 12.20%, BLEU of 15.96, and ROUGE of 28.93, all of which are higher than 6.25%, 11.10, and 25.35 for GPT-4.1 mini, respectively. These results suggest that self-hosted LLMs can be viable options for specific subtasks, especially when cost, privacy, or deployment constraints are important. Their All Accuracy remains lower than that of DeepSeek-V3 and much lower than that of UniLog. When applying LLMs to multilingual log statement generation, it is important to consider not only overall performance but also subtask-specific strengths and practical deployment constraints.
RQ1 Summary: In the unified performance comparison of log statement generation approaches in multilingual environments, UniLog achieved the best overall performance among all evaluated approaches, with an average All Accuracy of 20.35% across the five languages. Among the evaluated LLMs, DeepSeek-V3 achieved the highest performance, with an average All Accuracy of 10.96%. The comparison between UniLog and DeepSeek-V3 highlights the effectiveness of task-specific design, retrieval-based prompting, and warmup for multilingual log statement generation. We also observed clear differences in prediction difficulty across programming languages: JavaScript consistently showed the highest performance, whereas Python showed the lowest performance. Self-hosted LLMs can be competitive with API-based LLMs on some individual metrics, but their end-to-end performance remains limited. Overall, these results indicate that both the choice of approach and the target programming language strongly affect the performance of multilingual log statement generation.
5.2
RQ2: What is the impact of training strategies on log statement generation in multilingual environments?
Approach. To compare the two training strategies (i) Monolingual LoRA (Mono-LoRA) and (ii) Multilingual LoRA (Multi-LoRA), we adopt Llama3 as the backbone model because it achieved the highest average All Accuracy among the self-hosted models in RQ1. Self-hosted is a necessary condition for adapting the training strategy by LoRA. For Mono-LoRA, we train a separate LoRA for each language by using 24,000 training instances from each language, and we evaluate the effect of language-specific adaptation. For Multi-LoRA, we prepare two settings in order to also examine the effect of training data size in multilingual joint training. The first setting trains a single LoRA on a total of 24,000 instances, with 4,800 instances from each language. This setting enables a fair comparison of the learning unit itself, monolingual versus multilingual joint training, by matching the total amount of training data with Mono-LoRA. Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
17
Table 4. Summary of training configurations for the approaches used in RQ2.
Approach Llama3 (no LoRA) Mono-UniLog-L Multi-UniLog-L Mono-FastLog Multi-FastLog Mono-LoRA Multi-LoRA-24k Multi-LoRA-120k
Training Setting
Total
/Lang.
– Monolingual Multilingual Monolingual Multilingual Monolingual Multilingual Multilingual
0 500 500 24,000 24,000 24,000 24,000 120,000
0 500 100 24,000 4,800 24,000 4,800 24,000
The second setting trains a single LoRA on a total of 120,000 instances, with 24,000 instances from each language. This setting evaluates the effect of scaling up the training data in multilingual joint training. In addition, we also evaluate UniLog and FastLog for comparison with existing approaches. In particular, although UniLog uses GPT-4.1 mini by default, in this study we evaluate an implementation that uses Llama3 in order to make the comparison with the LoRA-trained models as fair as possible. We refer to this implementation as UniLog-L. For UniLog-L, we also construct Mono-UniLog-L, which performs warmup independently for each language, and MultiUniLog-L, which performs warmup on the integrated data of the five languages, in correspondence with Mono-LoRA and Multi-LoRA, and compare the effect of the learning unit. Similarly, for FastLog, we construct Mono-FastLog and Multi-FastLog. Table 4 shows the training settings of these approaches. The Training Setting column indicates whether training is performed monolingually or multilingually. The Total column indicates the total number of training instances across all five languages, whereas the /Lang. column indicates the number of training instances allocated to each language. Results. Table 5 presents the language-wise performance and the average performance across the five languages for all approaches evaluated in RQ2. Boldface indicates the best value for each combination of programming language and evaluation metric. Underline indicates the second-best value. Observation 5: Even a small amount of language-specific warmup can outperform much larger training setups. As shown in Table 5, Mono-UniLog-L achieves an average All Accuracy of 16.24% across the five languages, which is the highest among all evaluated approaches. It also records the best values for Level Accuracy at 69.23%, Message Accuracy at 20.29%, and BLEU at 24.81, which indicates that it is particularly strong at predicting log levels and message content. At the language level, it also achieves the best All Accuracy, with 34.77% for JavaScript and 19.60% for C#. Although Mono-UniLog-L achieves the highest performance in this way, what is noteworthy is its amount of training data. As shown in Table 4, Mono-UniLog-L is trained with only 500 instances, yet it outperforms Mono-LoRA at 15.13%, which is trained with 24,000 instances per language, and Multi-LoRA-120k at 15.94%, which is trained with 120,000 instances in total. These results indicate that, in multilingual log statement generation, UniLog-style adaptation with a small amount of warmup for each language can achieve both very high data efficiency and strong overall performance. Observation 6: When the total amount of training data is controlled, monolingual training consistently outperforms multilingual training. As shown in Table 5 and Table 4, when approaches are compared under the same total amount of training data, monolingual training consistently outperforms multilingual training. For the UniLog-based approaches, Mono-UniLog-L achieves an average All Accuracy of 16.24% across the five languages, which is higher than Multi-UniLog-L at 12.80%. Similarly, for the FastLog-based approaches, Mono-FastLog achieves an All Manuscript submitted to ACM
18
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei Table 5. The performance of different training strategies in multilingual log statement generation. Position
Level
Message
All
Language
Methods
Acc
Acc
Acc
BLEU
ROUGE
Acc
Java
Llama3 (no LoRA) Mono-UniLog-L Multi-UniLog-L Mono-FastLog Multi-FastLog Mono-LoRA Multi-LoRA-24k Multi-LoRA-120k
21.27 66.90 59.90 74.20 71.20 72.13 65.23 65.87
42.83 66.60 63.27 60.87 59.83 65.37 62.90 63.23
6.93 11.60 12.53 9.27 6.47 12.37 11.07 11.67
10.07 16.42 16.10 19.33 15.91 16.29 15.27 15.78
23.70 34.37 34.67 45.48 40.63 34.26 32.56 33.17
3.47 8.53 8.10 7.27 4.87 10.73 8.57 9.70
Python
Llama3 (no LoRA) Mono-UniLog-L Multi-UniLog-L Mono-FastLog Multi-FastLog Mono-LoRA Multi-LoRA-24k Multi-LoRA-120k
9.40 35.27 30.90 38.90 37.33 52.93 50.17 52.30
43.77 56.50 56.20 44.03 47.53 57.73 56.37 57.67
4.23 9.17 8.33 3.17 2.87 9.93 9.03 9.93
4.71 10.77 9.91 8.23 7.92 11.44 10.81 11.46
12.01 22.64 21.31 32.94 30.11 22.83 21.67 22.38
1.97 5.07 4.43 2.03 1.70 6.90 6.27 7.23
JavaScript
Llama3 (no LoRA) Mono-UniLog-L Multi-UniLog-L Mono-FastLog Multi-FastLog Mono-LoRA Multi-LoRA-24k Multi-LoRA-120k
39.00 73.30 68.53 77.70 70.80 75.20 63.10 73.27
69.97 82.40 79.37 74.40 64.13 78.30 76.53 79.77
24.40 41.70 34.97 23.17 12.00 33.47 32.63 36.03
31.49 48.04 41.55 38.41 22.96 40.74 40.55 43.27
48.31 64.56 57.89 59.98 45.74 58.26 57.95 60.37
11.83 34.77 27.50 20.90 10.43 28.30 22.50 30.07
TypeScript
Llama3 (no LoRA) Mono-UniLog-L Multi-UniLog-L Mono-FastLog Multi-FastLog Mono-LoRA Multi-LoRA-24k Multi-LoRA-120k
27.77 64.63 53.83 68.63 64.20 73.77 68.57 72.97
55.50 71.90 68.30 62.83 58.13 69.63 68.13 70.07
9.73 15.43 13.03 6.90 3.60 14.47 13.73 14.80
15.74 22.72 20.14 20.01 14.50 21.71 20.50 21.76
29.37 38.62 35.50 44.27 38.42 38.13 36.65 37.78
6.47 13.23 9.87 6.23 3.13 13.03 11.93 13.73
C#
Llama3 (no LoRA) Mono-UniLog-L Multi-UniLog-L Mono-FastLog Multi-FastLog Mono-LoRA Multi-LoRA-24k Multi-LoRA-120k
17.40 65.90 43.73 65.37 62.77 63.23 65.00 70.60
52.80 68.73 66.63 63.93 56.70 66.77 65.93 69.23
15.70 23.57 20.97 11.67 7.27 20.47 20.77 22.53
17.80 26.11 23.35 23.02 15.43 23.18 23.26 24.81
31.27 42.05 38.86 48.90 41.31 38.06 37.97 40.19
6.33 19.60 14.10 10.13 5.97 16.67 16.23 18.97
Average
Llama3 (no LoRA) Mono-UniLog-L Multi-UniLog-L Mono-FastLog Multi-FastLog Mono-LoRA Multi-LoRA-24k Multi-LoRA-120k
22.97 61.20 51.38 64.96 61.26 67.45 62.41 67.00
52.97 69.23 66.75 61.21 57.26 67.56 65.97 67.99
12.20 20.29 17.97 10.84 6.44 18.14 17.45 18.99
15.96 24.81 22.21 21.80 15.34 22.67 22.08 23.42
28.93 40.45 37.65 46.31 39.24 38.31 37.36 38.78
6.01 16.24 12.80 9.31 5.22 15.13 13.10 15.94
Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
19
Accuracy of 9.31%, which is higher than Multi-FastLog at 5.22%. The same tendency is observed for the LoRA-based approaches. When the total amount of training data is fixed at 24,000 instances, Mono-LoRA achieves an All Accuracy of 15.13%, which is higher than Multi-LoRA-24k at 13.10%. This tendency is also consistent with the difference in the number of training instances allocated to each language. For example, in the LoRA setting, Mono-LoRA is trained with 24,000 instances for each language, whereas Multi-LoRA-24k shares a total of 24,000 instances across the five languages, which means that only 4,800 instances are used for each language. The same holds for FastLog, where Mono-FastLog uses 24,000 instances per language, whereas Multi-FastLog uses 4,800 instances per language. For the UniLog-based approaches as well, Mono-UniLog-L uses 500 instances per language, whereas Multi-UniLog-L uses 500 instances in total, which corresponds to 100 instances per language. These results suggest that, under a fixed total training budget, the disadvantage caused by the reduction in training data for each language outweighs the benefit of sharing information through multilingual integration. In multilingual log statement generation, monolingual training is more advantageous when the total amount of training data is limited. This comparison controls for the total amount of training data rather than the full computational or operational cost, since monolingual training requires separate training for each language whereas multilingual training uses a single shared model. Observation 7: Position prediction is more sensitive to training data scale than level and message prediction. As shown in Table 4 and Table 5, Position Accuracy tends to improve overall as the amount of training data for each language increases. For example, when focusing on the multilingual training settings, the average Position Accuracy across the five languages is 51.38% for Multi-UniLog-L, which uses only 100 instances per language. It increases to 61.26% for Multi-FastLog, which uses 4,800 instances per language, to 62.41% for Multi-LoRA-24k, and to 67.00% for Multi-LoRA-120k, which uses 24,000 instances per language. A similar tendency is also observed in monolingual training. Mono-UniLog-L, which uses 500 instances for each language, achieves a Position Accuracy of 61.20%, whereas Mono-FastLog and Mono-LoRA, which use 24,000 instances for each language, achieve 64.96% and 67.45%, respectively. These results show that increasing the amount of training data is consistently associated with improvement in position prediction. This tendency can also be confirmed by comparisons within the same approach. For example, in Multi-LoRA, when the number of training instances for each language is increased from 4,800 to 24,000, Position Accuracy improves from 62.41% to 67.00%, which is an increase of 4.59 points and the largest gain among all evaluation metrics. Under the same comparison, Level Accuracy improves from 65.97% to 67.99%, which is an increase of 2.02 points, Message Accuracy improves from 17.45% to 18.99%, which is an increase of 1.54 points, BLEU improves from 22.08 to 23.42, which is an increase of 1.34, and ROUGE improves from 37.36 to 38.78, which is an increase of 1.42. All Accuracy also improves from 13.10% to 15.94%, but this improvement is likely supported mainly by the gain in position prediction. When looking across all approaches, the performance of level and message prediction is not explained only by the amount of training data, and the effect of approach design is also large. For example, Mono-UniLog-L, which is trained with only 500 instances per language, achieves Level Accuracy of 69.23% and Message Accuracy of 20.29%, which are higher than those of Mono-FastLog and Mono-LoRA, both of which use 24,000 instances per language. In other words, for log level and message content prediction, increasing the amount of training data has a certain effect, but its influence is not as dominant as it is for position prediction, and the results are more strongly affected by the adaptation strategy and training design of each approach. Overall, these findings indicate that increasing the amount of training data has a certain effect on all three elements in multilingual log statement generation, but the effect is strongest for deciding where to insert a log statement. For log level and message content prediction, the design of the approach itself appears to play a more important role in addition to training scale. Manuscript submitted to ACM
20
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei Table 6. Six categories of logging locations ID
Category
Description
Category 1 Category 2 Category 3 Category 4 Category 5 Category 6
Try-Catch Block Branching Block Looping Block Method Start Method End Domain-Specific Methods
Within a try-catch structure, including a try block or a catch block. Within a branching structure, including blocks of if, else, switch, etc. Within an iteration structure, including blocks of for, while, do, etc. Near the beginning of a method body. Near the end of a method body. Within a domain-specific method, such as a method for handling a specific request.
RQ2 Summary: In the comparison of training strategies for multilingual log statement generation, MonoUniLog-L achieved the best overall performance, with an average All Accuracy of 16.24% across the five languages. In particular, although Mono-UniLog-L uses only 500 instances for additional training, it achieved higher performance than models using LoRA with 120k instances. In addition, when compared under the same total training budget, monolingual training consistently outperformed multilingual joint training. Increasing the amount of training data had a particularly strong effect on Position Accuracy. For example, in Multi-LoRA, increasing the number of instances per language from 4,800 to 24,000 substantially improved Position Accuracy from 62.41% to 67.00%, whereas the improvements in Level Accuracy and Message Accuracy were relatively small. These results indicate that, in multilingual log statement generation, language-specific adaptation is the most effective strategy under a limited training budget. They also indicate that scaling up training data especially contributes to improving log insertion location prediction, whereas the prediction of log level and message content is more strongly affected by the design of the approach itself.
5.3
RQ3: How do language-specific characteristics affect automated log statement generation in multilingual environments?
Approach. In RQ3, we analyze whether the cross-language performance differences observed in RQ1 can be explained by language-specific logging characteristics. Specifically, we use log insertion categories as the main axis of analysis and examine the three elements that constitute log statement generation: insertion location, log level, and log message. To conduct a systematic analysis, we classify log insertion locations into six categories and use these categories to analyze position, level, and message characteristics across languages. The category definitions are based on the framework proposed by Li et al. [32], which systematized log insertion locations in Java methods, and we adopt the following six categories: Try-Catch Block, Branching Block, Looping Block, Method Start, Method End, and DomainSpecific Methods. The definitions of these six categories are shown in Table 6. In the following analysis, we analyze the characteristics of log insertion locations, levels, and message content on the basis of these categories. The category classification was performed with GPT-5.2 [42] on all 15,000 instances in the test set. We selected GPT-5.2 because it was the most advanced model available through the OpenAI API at the time of the experiments. Such state-of-the-art LLMs have shown strong performance in annotating software engineering artifacts [3, 26], making them suitable for this task. Because the validity of this classification affects the subsequent analysis, we manually verified the classification results. Assuming a population size of 15,000, a 95% confidence level, and a margin of error of ±5%, we calculated a required sample size of 375 and randomly sampled these instances. The first and second authors independently assigned categories to the sampled instances, and the inter-annotator agreement reached a Cohen’s Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
Java
50.0
42.4
16.5
37.2
29.7
52.4
Python
38.1
34.6
17.9
27.0
31.5
37.6
JS
77.1
68.2
50.0
59.3
57.9
72.3
50
TS
48.9
37.2
26.7
47.8
29.1
54.7
40
CS
70.6
55.0
43.2
51.4
29.4
63.0
30
ALL
56.4
49.0
26.3
42.7
32.6
57.5
20
cat1
cat2
cat3
cat4
cat5
cat6
21
70 60
Fig. 6. Heatmap of Position Acc. (%) across Languages and Categories.
kappa of 0.95. When their labels differed, they resolved the disagreements through discussion and determined the final adjudicated labels. We then compared the GPT-5.2-assigned labels with these adjudicated labels and found an agreement rate of 88.8%. These results indicate that the GPT-5.2-based category classification is sufficiently reliable for the RQ3 analysis. For the accuracy analysis in RQ3, we use the prediction results of DeepSeek-V3, which is the LLM that achieved the highest performance in RQ1. 5.3.1 Results for Position. In this subsection, we analyze factors that cause the prediction difficulty of log insertion location (i.e., Position) to differ across languages. Figure 6 shows Position Accuracy for each combination of programming language, which includes five languages and ALL, and position category. Observation 8: The difficulty differs substantially across position categories, and in particular Category 3, which is Looping Block, is systematically the most difficult. When focusing on the ALL row in Figure 6, the difficulty of position prediction differs substantially across categories. The lowest accuracy is observed for Category 3, which is Looping Block, where Position Accuracy remains at 26.3%. Next, Category 5, which is Method End, is also low at 32.6%, which indicates that identifying insertion locations is difficult even near the end of a method. Category 1, which is Try-Catch Block, and Category 6, which is Domain-Specific Methods, are relatively high, at 56.4% and 57.5%, respectively. These results show that, even for the same position prediction task, the difficulty systematically changes depending on structural context. These findings suggest that position prediction is not uniformly difficult, and that in contexts such as loops and method ends, where multiple reasonable candidate locations are likely to exist, insertion ambiguity increases and the task becomes inherently more difficult for models. Observation 9: The distribution of position categories differs across languages, and the proportion of difficult categories varies across languages. Next, we examine the extent to which cross-language differences in accuracy can be explained by distributional factors. Specifically, the proportion of difficult categories contained in each language. Figure 7 shows the occurrence ratios of Category 1 to Category 6 in each language. For example, in Python, the proportions of Category 3, which is the most difficult, and Category 5, which is also low accuracy, are both 5.6%, which are higher than those in ALL, where Category 3 is 2.8% and Category 5 is 3.0%. In JavaScript, Category 3 is 1.2% Manuscript submitted to ACM
22
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei
Java
37.9
26.7
2.8
16.9
2.5
13.2
35
Python
22.7
29.1
5.6
20.0
5.6
17.0
30
JS
21.8
38.6
1.2
12.0
1.3
25.1
TS
35.1
28.4
1.5
15.3
2.9
16.8
CS
33.4
31.0
2.7
14.7
2.8
15.4
10
ALL
30.2
30.7
2.8
15.8
3.0
17.5
5
cat1
cat2
cat3
cat4
cat5
cat6
25 20 15
Fig. 7. Heatmap of Category Distribution across Languages and Categories.
Table 7. Loop density statistics by programming language. Each row reports the average loop density (avg_loop_per_line) and the proportion of methods containing at least one loop (loop_present_rate).
Language
avg_loop_per_line
loop_present_rate (%)
Java Python JavaScript TypeScript C# All
0.289 0.341 0.075 0.076 0.147 0.185
21.33 23.00 6.30 6.00 11.67 13.62
and Category 5 is 1.3%, which indicates relatively lower exposure to difficult categories. This difference indicates that category distributions can contribute to cross-language differences in the difficulty of position prediction. Observation 10: Loop density is negatively associated with position accuracy and can serve as an auxiliary factor that explains residuals. We further focus on loop density, which is denoted as loop_per_line, as a lightweight structural factor that can help explain cross-language differences in the difficulty of position prediction. Loop density is a metric that represents the average number of loop structures per line of code. Table 7 shows the average loop density in each language and the proportion of methods that contain loops. Our analysis found a moderate negative correlation between loop_per_line and Position Accuracy, with Pearson 𝑟 = −0.46. This is consistent with Observation 8, which indicates that Category 3, which is Looping Block, is the most difficult category. As shown in Table 7, Python has the highest average loop density, at 0.341, and also has the highest proportion of methods that contain loops, at 23.0%. JavaScript has the lowest average loop density, at 0.075, and also has a small proportion of methods that contain loops, at 6.30%. In this way, the fact that Python, which has large residuals that are not explained by category distributions, also has high loop density suggests that having more loop structures can be one factor that increases the difficulty of position prediction. Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
Java
55.9
61.9
61.2
62.2
64.9
69.9
Python
40.2
46.1
58.9
70.8
72.6
65.7
JS
79.7
75.1
63.9
74.9
76.3
78.5
TS
63.6
52.2
64.4
71.7
74.4
70.9
CS
67.0
62.1
59.3
65.2
65.9
73.4
ALL
61.2
60.5
60.5
68.7
70.7
72.4
cat1
cat2
cat3
cat4
cat5
cat6
23
75 70 65 60 55 50 45
Fig. 8. Heatmap of Level Acc. (%) across Languages and Categories.
Implication (Position): We found that cross-language differences in position prediction arise from a combination of (i) category-specific difficulty differences, (ii) differences in category distributions across languages, and (iii) language-specific factors, such as the prevalence of loop structures. In particular, we found that contexts with more loop structures, which correspond to Category 3, increase the difficulty of position prediction, and that Python contains this category relatively more frequently, which can be one factor behind the difficulty of position prediction in Python. Improving position prediction accuracy requires considering both strengthening structural representations with an emphasis on difficult categories, especially Category 3, and language adaptation that incorporates language-specific syntax and practices. 5.3.2
Results for Level. In this subsection, we analyze factors that cause the prediction difficulty of log levels, which
are represented in six bands, to differ across languages. Figure 8 shows Level Accuracy for each combination of language, which includes five languages and ALL, and position category. Observation 11: Cross-language differences in Level Accuracy remain, but category dependence is weaker than for Position. When focusing on the ALL row in Figure 8, Level Accuracy is generally within the range of 60.5–72.4%, and there is no extreme difference in difficulty across categories, unlike what was observed for Position. Cross-language differences are clear. JavaScript shows high accuracy across all categories, at 63.9–79.7%, whereas Python is particularly low for Category 1, which is Try-Catch Block, and Category 2, which is Branching Block, at 40.2% and 46.1%, respectively. This tendency suggests that the difficulty of level prediction is not determined sufficiently by category information alone, and may be strongly affected by language-specific factors, such as operational conventions and practices in each language. Observation 12: The distribution of log levels within the same position category differs across languages. Observation 11 confirms that there is a gap in level prediction accuracy between Python and JavaScript for Category 1 and Category 2. By comparing the level-band distributions in Category 1 and Category 2, we analyze whether one factor behind the cross-language difference is a difference in the typical levels within the same category. Figure 9 shows the level-band distributions for Category 1 and Category 2 in Python and JavaScript. For Category 1, Error is dominant Manuscript submitted to ACM
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei
Python
JavaScript
100
cat1
0.0
15.4
13.6
25.2
44.7
1.0
0.3
5.8
4.9
9.9
78.8
0.3
cat2
0.0
14.4
33.9
32.1
18.3
1.1
0.9
14.7
18.7
15.1
50.5
0.1
cat3
0.0
29.2
58.3
7.7
4.2
0.6
8.3
27.8
44.4
13.9
5.6
0.0
60
cat4
0.0
30.3
65.7
2.3
1.5
0.2
1.1
41.5
40.1
6.1
10.9
0.3
40
cat5
0.0
22.0
74.4
2.4
1.2
0.0
2.6
10.5
78.9
7.9
0.0
0.0
cat6
0.0
31.2
56.3
8.6
3.5
0.4
1.9
29.4
29.8
6.8
32.0
0.1
trace debug info warn error fatal
trace debug info warn error fatal
80 Percentage (%)
24
20 0
Fig. 9. Ground-truth log level-band distributions by position category for Python (left) and JavaScript (right).
in JavaScript at 78.8%, whereas in Python it accounts for only 44.7%, and Warn at 25.2%, Debug at 15.4%, and Info at 13.6% are also mixed in. For Category 2 as well, Error accounts for more than half in JavaScript, at 50.5%, whereas in Python it is much smaller at 18.3%, and Info at 33.9% and Warn at 32.1% are the major bands. In this way, even within the same category, the typical level can differ by language. As a result, estimation based only on category information is less likely to work well in Python, which may amplify cross-language differences. Implication (Level): Because the frequent log levels differ across categories, and because the typical level can also differ across languages even within the same category, the difficulty of level prediction may be more strongly affected by language-specific factors than by category dependence. For example, in categories with clear cross-language differences, such as Category 1, which is Try-Catch Block, and Category 2, which is Branching Block, a uniform language-independent heuristic is likely to be insufficient, and language adaptation that reflects language-specific operational conventions and practices may be effective. 5.3.3
Results for Message. In this subsection, we analyze factors that cause the prediction difficulty of log message
content (i.e., Message) to differ across languages. Because exact matching, Message Accuracy, does not sufficiently capture semantically similar generations, we use BLEU, which is based on surface similarity, as the main metric in this analysis. Figure 10 shows BLEU for each combination of language, which includes five languages and ALL, and position category. Observation 13: Message prediction shows the largest cross-language differences among the three elements, and the same tendency appears for both exact matching and semantic similarity. As shown in Figure 10, BLEU differs substantially across languages, and the cross-language gap is larger than that observed for position and level. JavaScript shows particularly high scores for Category 1 and Category 2, with 41.7 for Category 1 and 37.5 for Category 2, which indicates that it is easier to generate surface expressions that are close to the reference messages even within the same category. Python is consistently low across all categories. These results indicate that message prediction strongly Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
25
40
Java
11.7
12.1
9.0
12.2
18.5
20.0
Python
6.9
7.1
5.6
8.8
7.4
9.2
JS
41.7
37.5
26.2
19.5
18.2
24.4
TS
22.3
15.2
9.4
20.0
18.5
20.0
20
CS
23.4
18.6
11.8
18.7
16.7
25.8
15
ALL
20.4
19.4
9.7
15.2
14.0
20.2
10
cat1
cat2
cat3
cat4
cat5
cat6
35 30 25
Fig. 10. Heatmap of BLEU score across Languages and Categories.
Java 0.650 0.770 0.822 0.843 0.846 0.813
0.95
Python 0.855 0.893 0.971 0.924 0.948 0.949
0.90
JS 0.666 0.623 0.877 0.777 0.946 0.733
0.85
TS 0.715 0.855 0.996 0.874 0.956 0.859 CS 0.672 0.785 0.945 0.885 0.969 0.779 ALL 0.660 0.736 0.917 0.844 0.921 0.791 cat1
cat2
cat3
cat4
cat5
0.80 0.75 0.70 0.65
cat6
Fig. 11. Heatmap of distinct2 for ground-truth log message diversity across languages and categories.
depends on language-specific expression habits and lexical choices, and that message prediction is the element in which cross-language differences are most strongly amplified. Observation 14: There are also difficulty differences across categories, and Category 3, which is Looping Block, is the most difficult even for message prediction. When focusing on the ALL row in Figure 10, we can observe difficulty differences across categories. Although Category 1 shows the highest BLEU score, at 20.4, Category 3 remains the lowest, at 9.7. This is consistent with the result for Position, where Category 3 was also the most difficult category. One possible reason is that, within loops, the purpose and situation of logging become more diverse, which makes the message content that should be written more situation-dependent. Observation 15: Message diversity shows a negative correlation with accuracy. To analyze language-specific factors behind cross-language differences in message prediction accuracy, we compute distinct2 to quantify how template-like the log messages are (i.e., repetitiveness). distinct2 is widely used as a diversity metric in natural language generation [6, 8, 27], and it represents the proportion of unique consecutive 2-grams in the ground-truth messages. A Manuscript submitted to ACM
26
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei Table 8. Target-wise comparison between in-language and cross-language LoRA models based on All Accuracy.
Target
In-lang
Cross-lang Avg.
Diff
Best Source
Java Python JavaScript TypeScript C#
10.73 6.90 28.30 13.03 16.67
8.84 5.48 22.19 11.28 16.65
1.89 1.42 6.11 1.75 0.02
Python Java TypeScript Python JavaScript
higher value indicates greater message diversity, which means lower template-likeness, whereas a lower value indicates a stronger tendency toward template-like and repetitive messages. Figure 11 shows distinct2 for each language and position category. As shown in Figure 11, Category 1 and Category 2 in JavaScript, where high BLEU scores were observed, have particularly low distinct2, which indicates that the messages tend to be more formulaic. Python shows high distinct2 in many categories, which indicates that the log messages tend to be more diverse. These findings are consistent with the difficulty of message prediction in Python and with the high accuracy observed for Category 1 and Category 2 in JavaScript. Implication (Message): We found that cross-language differences in message prediction are strongly related to the diversity of log messages, where higher diversity is associated with lower accuracy. In particular, Python has diverse messages, which suggests that simple template learning or matching frequent phrases is unlikely to achieve sufficient performance. JavaScript has relatively formulaic messages in some categories, which makes it easier to achieve high accuracy. For future automated log message generation, it may be effective to adapt the approach to each language and category by (i) leveraging templating and phrase learning for repetitive categories, while (ii) combining abstraction, such as handling identifiers and values, and retrieval of similar examples, such as RAG, for languages with high diversity, especially Python.
6
Discussion
In this section, we discuss three additional analyses that complement the findings of the main RQs. First, motivated by the results on training strategies, we analyze cross-language generalization in log statement generation to examine how well knowledge learned from one programming language transfers to another. Second, we investigate the effectiveness of LLMs in a more realistic setting where multiple log statements may appear within a single function, which is not captured by the single-log insertion setting used in the main experiments. Third, because exact-match and n-gram-based metrics do not fully capture the semantic quality of generated log messages, we conduct a qualitative analysis with LLM-as-a-Judge. 6.1
Cross-Language Generalization in Log Statement Generation
Motivation. In the experiments so far, we have mainly evaluated the effectiveness of LoRA models under languagespecific training settings for each programming language. In real development environments, securing sufficient training data for every language is often difficult. Clarifying the extent to which a model trained on one language can generalize to other languages is important for considering the practicality of LLM-based log statement generation approaches. In Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
Java 10.27
9.87
8.80
9.07
8.40
6.40
6.90
5.43
6.10
4.00
JS 22.97
20.17
28.30
25.93
19.70
TS 11.03
11.67
11.53
13.03
10.90
CS 16.10
15.60
17.53
17.37
16.67
Python JS TS Train Language
CS
Test Language
Python
Java
27
25 20 15 10 5
Fig. 12. Heatmap of All Accuracy for cross-language log statement generation.
this section, we analyze cross-language generalization in log statement generation to investigate how beneficial training on the target language is and whether knowledge acquired from one language can be transferred to another language. Approach. To investigate cross-language generalization, we use the five Mono-LoRA models introduced in RQ2. These models are LoRA models fine-tuned separately for each programming language. Each LoRA adapter retains adaptation knowledge obtained from a single training language, making these models suitable for analyzing the extent to which such knowledge transfers to other languages. Llama3 is used as the backbone model for these models. Each Mono-LoRA model is obtained by independently fine-tuning a LoRA adapter using 24,000 training instances from one programming language. In this analysis, we evaluate these five Mono-LoRA models on the test sets of all five languages. We regard the language used for LoRA fine-tuning as the training language and the language used for evaluation as the test language. This design yields 25 combinations of training and test languages across Java, Python, JavaScript, TypeScript, and C#, enabling us to systematically examine how differences in the training language affect performance on each target language. For each combination, we evaluate the generated log statements using All Accuracy. We distinguish between the in-language setting, where the training and test languages are identical, and the crosslanguage setting, where they are different. This comparison quantifies the benefit of training directly on the target language compared with transferring from other languages. For each target language, we compare the in-language performance with the average performance of the cross-language models trained on the other four languages. For each target language, we also identify the Best Source: the source language that achieves the highest performance among the cross-language models. This allows us to analyze not only whether cross-language transfer is possible, but also which training source language transfers most effectively to each target language. Results. Figure 12 shows a heatmap of All Accuracy for each combination of training language and test language. Table 8 compares, for each target language, the in-language performance with the cross-language average. Here, inlanguage refers to the performance of the model trained on the same language as the target language, and cross-language average refers to the average performance of models trained on the other four languages. Diff represents the difference obtained by subtracting the cross-language average from the in-language performance, and Best Source indicates the source language that achieved the highest performance in the cross-language setting. Manuscript submitted to ACM
28
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei In-language fine-tuning achieves the highest All Accuracy for most target languages. For four languages
except C#, the in-language setting, where the training language and test language are identical, achieved the highest All Accuracy, as shown in Figure 12. This result suggests that cross-language transfer is effective to some extent, while models trained directly on the target language are basically the most effective for log statement generation. The advantage of in-language fine-tuning differs substantially across target languages. The absolute level of performance differs depending on the target language. For example, JavaScript shows relatively high values overall, whereas Python remains at relatively low values overall. A target-wise comparison is therefore needed to evaluate the advantage of the in-language setting fairly. As shown in Table 8, the in-language model outperformed the average of the cross-language models for all target languages. The most notable case was JavaScript, where the in-language model achieved an All Accuracy of 28.30 and outperformed the cross-language average of 22.19 by 6.11 points. This result suggests that, in JavaScript, language-specific knowledge and logging practices obtained from training data in the same language are particularly important. For Java, Python, and TypeScript, the differences were only 1.89, 1.42, and 1.75 points, respectively, indicating that the advantage of in-language training was limited. For C#, the in-language performance was 16.67, whereas the cross-language average was 16.65, and the difference was only 0.02 points. This indicates that, for C#, models trained on other languages can achieve almost equivalent performance. Cross-language transferability is not explained solely by simple language similarity. From the perspective of Best Source, cross-language transfer may be influenced to some extent by closeness between languages. For example, the best cross-language source for JavaScript was TypeScript, which is likely to reflect the fact that JavaScript and TypeScript are close in terms of syntax and development style. We also observed that Java was the Best Source for Python and Python was the Best Source for Java, which indicates that generalization performance is not explained only by simple language similarity. Transferability in log statement generation is likely affected not only by superficial syntactic similarity but also by structural characteristics of log insertion locations, such as branching, exception handling, and state updates, as well as by biases in the log patterns contained in each language dataset. These results indicate that in-language fine-tuning is overall the most effective strategy for log statement generation. The magnitude of its advantage differs substantially across target languages. In JavaScript, the effect of in-language adaptation was clearly observed. In C#, sufficiently competitive performance was obtained even with cross-language transfer alone. This suggests that large-scale language-specific data may be unnecessary in some cases. The results also show that, for some languages, adaptation specialized to the target language remains important. In the future, in addition to training separate models for each language, an important challenge will be how to balance logging knowledge that can be shared across languages with knowledge that is specific to each target language. 6.2
Effectiveness of LLMs in Multi-Log-Statement Settings
Motivation. In the experiments so far, we have evaluated the performance of LLMs for log statement generation under the setting where one log statement is inserted into one function. In actual development environments, it is common for multiple log statements to be inserted within a single function. Evaluation that targets only single-log insertion may fail to capture the practical difficulty of log statement generation. In a setting where a function may contain multiple log statements, the model needs to determine not only where to insert log statements, but also how many log statements should be inserted and how their roles should be differentiated within the same function. In this section, we additionally analyze a setting in which a single function may contain multiple log statements, as a more realistic setting. Approach. To analyze this setting, we constructed a new dataset by following the same procedure as in Section 4. For each language, we collected 200 functions that contain at least one log statement. The statistics of this dataset are Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
29
Table 9. Statistics of the multi-log-statement dataset
Language
Single-Log Functions
Multi-Log Functions
Total Log Statements
Java Python JavaScript TypeScript C#
134 97 117 109 123
66 103 83 91 77
342 432 345 373 361
Total
580
420
1,853
Table 10. Overall performance of Llama3 on the multi-log-statement setting
Position
Language
Level
Message
Function
Precision
Recall
F1
Acc.
Acc.
BLEU-4
ROUGE-L
Position
Position+Level
Perfect
Java Python JavaScript TypeScript C#
19.11 22.21 24.81 25.86 22.09
54.09 46.99 47.54 54.69 45.15
28.24 30.16 32.60 35.11 29.66
63.78 62.56 51.22 54.41 23.93
0.54 0.99 1.83 0.98 0.00
0.19 2.11 2.46 2.54 0.27
20.26 19.12 17.12 20.29 7.73
6.00 0.00 7.50 5.00 2.00
3.00 0.00 3.00 3.00 0.50
0.50 0.00 0.00 0.00 0.00
Average
22.82
49.69
31.15
51.18
0.87
1.51
16.90
4.10
1.90
0.10
shown in Table 9. This table reports, for each language, the number of functions that contain only one log statement, which is denoted as Single-Log Functions, the number of functions that contain multiple log statements, which is denoted as Multi-Log Functions, and the total number of contained log statements, which is denoted as Total Log Statements. For the evaluation in Table 10, we use all 200 functions for each language. We evaluate both Single-Log Functions and Multi-Log Functions. In practical development environments, the model is not given in advance how many log statements should be included in the target function, and must determine whether to insert a single log statement or multiple log statements. The purpose of this analysis is not to compare the superiority of specific models, but to confirm the inherent difficulty of a setting in which a function may contain multiple log statements. We conduct an additional analysis using Llama3, which is one of the representative open-source LLMs consistently used in the main experiments. In a setting where a function may contain multiple log statements, the number of log statements to be generated is not fixed, so evaluating position prediction only by exact-match accuracy is insufficient. Following prior work [13], we use Precision, Recall, and F1-score for log position prediction. These metrics allow us to evaluate whether the predicted log statements are excessive and may cause system overhead, or insufficient and may miss necessary information. The metrics related to log levels and messages are calculated only for log statements whose positions are predicted correctly. Level Accuracy represents the proportion of log statements inserted at correct positions whose levels also match. Message Accuracy, BLEU, and ROUGE represent the agreement and similarity of messages generated for correct positions. We also use Function-level metrics to evaluate the extent to which the model can consistently generate the required log statements for an entire function. Function-level Position represents the proportion of functions for which all log positions in the function are predicted correctly. Position+Level represents the proportion of functions for which both all positions and the levels of the corresponding log statements are predicted correctly. Perfect represents the proportion of functions for which position, level, and message all match for every log statement in the function. These Manuscript submitted to ACM
30
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei
function-level metrics are necessary because, in practical use, partially correct predictions may still be insufficient if the set of log statements generated for the entire function is inconsistent. Results. Table 10 shows the results of Llama3 in this setting. Llama3 can capture some correct log insertion locations, but tends to overpredict unnecessary locations in a setting where a function may contain multiple log statements. When averaged across the five languages, Precision, Recall, and F1 for position prediction are 22.82%, 49.69%, and 31.15%, respectively. Recall is substantially higher than Precision. This result indicates that Llama3 can capture some correct log insertion locations in a function to some extent, while it also frequently overpredicts unnecessary locations. The model tends to retrieve correct positions broadly, but has difficulty narrowing the predicted positions appropriately. For each language, Precision remains between 19.11% and 25.86%, and the maximum F1 is only 35.11%. Generating appropriate log levels and messages becomes even more difficult when multiple log statements must be generated within the same function. For level prediction, the average is 51.18%, while C# remains at only 23.93%, indicating substantial variation across languages. Message generation is even more difficult. The average Message Accuracy is 0.87%, BLEU-4 is 1.51, and ROUGE-L is 16.90. Message Accuracy is below 2% for all languages. This indicates that, when generating multiple log statements appropriately within the same function, it is extremely difficult not only to determine positions but also to generate message content consistently according to the role of each log statement. At the function level, Llama3 rarely generates position, level, and message consistently for the entire function. The function-level results clarify the difficulty of this setting. On average across the five languages, Functionlevel Position is only 4.10%, Position+Level is 1.90%, and Perfect is 0.10%. For each language, Function-level Position is 7.50% for JavaScript, 6.00% for Java, and 5.00% for TypeScript, whereas it is 0.00% for Python. This indicates that matching all log positions in a function exactly is already very difficult. Position+Level is at most 3.00%, and Perfect is 0.00% for all languages except Java, which reaches only 0.50%. Even if some individual log elements can be predicted partially correctly, the model can hardly generate position, level, and message consistently for the entire function at the same time. This result indicates that the end-to-end generation framework used in the single-log setting does not sufficiently handle the more realistic scenario of inserting multiple log statements. These results suggest that the difficulty of this setting does not arise simply because the number of prediction targets increases. It stems from the need to handle, at the same time, the positional relationships among multiple log statements, the division of roles among them, and the consistency of the function as a whole. In this setting, the model must determine not only where log statements should be inserted, but also how many are necessary, which level should be assigned to each log statement, and how to generate messages that are not redundant with one another. Conventional generation strategies that assume a single log statement have inherent limitations. In the future, stepwise generation approaches, which first identify candidate log insertion locations and then generate the level and message for each location, as well as structured approaches that explicitly consider the control flow of the entire function and the relationships among multiple events, are likely to become important. 6.3
Qualitative Analysis of Generated Log Messages with LLM-as-a-Judge
Motivation. As demonstrated in the preceding performance analyses, the evaluated log statement generation approaches, including existing end-to-end approaches and LLMs, struggle to achieve high Message Accuracy. This limitation stems from the strict reliance of the metric on exact string matching against the target log messages; consequently, semantically valid logs are unfairly penalized as incorrect predictions simply because they differ in surface-level Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
31
phrasing. Conventional metrics like BLEU and ROUGE primarily measure lexical overlap, and they do not fully capture the semantic equivalence between generated and target log messages. To fill this gap, we conducted a qualitative analysis with an LLM-as-a-Judge to more accurately assess message quality. Specifically, our goal was to determine whether a generated log message, despite differing in its exact wording, preserves similar semantics and can serve as a viable alternative to the target log message. Approach. To achieve this, we utilized 15,000 log messages generated by the best-performing approach, UniLog, comparing them directly against the target log messages from the test set. We employed an LLM-as-a-Judge methodology to assess the semantic equivalence between these pairs. To systematically classify this semantic similarity, we adopted the taxonomy proposed by Mastropaolo et al. [37], categorizing each generated instance into one of the following three classes: • Same Information. The generated log message is semantically equivalent to the target one, despite differing in its exact lexical expression. • Meaningful. The generated log message is contextually coherent and possesses a concrete meaning, but diverges semantically from the intended target message. • Meaningless. The generated log message is incomprehensible, incoherent, or entirely irrelevant to the surrounding code context. Based on this taxonomy, we employed GPT-5.2 as an automated annotator to compare the generated log messages against the target ground truth, classifying each generation into one of these three categories. The prompt used for this automated annotation is shown in Figure 13. To empirically validate the LLM-as-a-Judge methodology, we manually verified the automated classifications against a rigorous human evaluation. Specifically, we randomly sampled 375 instances from the full corpus of 15,000, a size determined to guarantee a 95% confidence level with a ±5% margin of error. The first and second authors served as two independent annotators and classified each instance according to our established taxonomy, achieving an almost perfect inter-annotator agreement with a Cohen’s Kappa coefficient of 0.93. Any isolated labeling discrepancies were subsequently resolved through discussion to establish a definitive consensus. When comparing these finalized human annotations against the labels assigned by GPT-5.2, we observed a 92.5% agreement rate. This robust alignment confirms that our LLM-as-a-Judge methodology serves as a reliable proxy for human judgment in evaluating the semantic similarity between generated and target log messages. Results. A semantics-aware evaluation reveals that UniLog generates appropriate log messages more often than exact-match Message Accuracy suggests. The analysis revealed that, among the 15,000 log messages generated by UniLog, 5,987 (39.9%) preserved the same semantic message as the target messages despite surface-level differences, indicating semantic equivalence. An additional 8,962 (59.8%) contained meaningful but non-equivalent messages, while the remaining 51 (0.3%) were classified as meaningless. Although the Message Accuracy of UniLog was 22.74%, a semantics-aware evaluation reveals that UniLog generated appropriate messages for 39.9% of instances, nearly double what exact-match metrics suggest. These findings indicate that UniLog is capable of generating log messages that are semantically similar to the targets while differing only in surface expression, and that such messages can serve as alternatives. Consequently, automatic evaluation metrics that rely solely on exact match may substantially underestimate the true capability of UniLog in log message generation. To further understand the log message generation capability of UniLog, Figure 14 presents representative examples of the three categories: Same Information, Meaningful, and Meaningless. The first example is a Same Information case. In this example, the synthetic log message (e.g., logger.Error("Error whilst closing handle. Type: {0}, Handle: Manuscript submitted to ACM
32
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei You are a classifier for log message quality. Pick EXACTLY ONE category. Categories: cat1: Same information: the generated log message is semantically equivalent to the target one. This includes cases where the two log messages express the same information with different wordings. The generated log message may also be more expressive than the target while still conveying the same essential information. cat2: Meaningful: the generated log message includes an articulated message that can be understood, but it is not equivalent to the one present in the target log message. cat3: Meaningless: the generated log message includes a message that is meaningless considering the context and/or the logging message cannot be comprehended. Guidance: - Evaluate only the generated log message. - Compare the generated log message with the target log message using the given source code context. - Do NOT evaluate logging position, log level, syntax, formatting, or surrounding code structure. - The goal is to judge whether the generated log message, while different from the target one, represents a good alternative to the reference one. - Choose cat1 when the generated log message is semantically equivalent to the target log message, even if the wording is different. - Choose cat2 when the generated log message is understandable and meaningful in context, but not semantically equivalent to the target log message. - Choose cat3 when the generated log message is meaningless in context and/or cannot be comprehended. - Be conservative when assigning semantic equivalence. - Return JSON that matches the given schema. Target log message: {target log message} Generated log message: {generated log message} Source code context: {source code context}
Fig. 13. Prompt used to classify generated log messages in the LLM-as-a-Judge evaluation.
{1}", requestType, handle, exception);) can be regarded as semantically equivalent to the developer-written log message (e.g., logger.Error("${requestType} ${handle} error whilst closing handle.", exception);), although the expression format is different, because both record an error while closing a handle and identify the target associated with that error. This suggests that UniLog can generate log messages that are practically equivalent in meaning even when their surface expressions differ, based on the given code context. At the same time, this example shows that Message Accuracy, which is based on exact match, would treat such a semantically appropriate output as incorrect. The second example is a Meaningful case. In this example, the synthetic log message differs from the developer-written log message (e.g., "Unimplemented NID function sceKernelRegisterThreadEventHandler [0x0C106E53]") in the identifier at the end (e.g., "Unimplemented NID function sceKernelRegisterThreadEventHandler [0xCA145E2E]"). The generated message is not semantically equivalent to the target. It still correctly captures that the log concerns the unimplemented function sceKernelRegisterThreadEventHandler, and it is sufficiently understandable in light of the code context. In other words, although this generated message does not exactly match the reference, it still carries meaningful information and utility, and is therefore classified as Meaningful. This example also highlights a limitation of automatic evaluation metrics based on n-grams, such as BLEU and ROUGE. Because the generated log Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
33
Example of Taxonomy-based Classification of Generated Log Messages
❶ Same Information
public static void ErrorWhilstClosingHandle(this IInternalLogger logger, uv_req_type requestType, IntPtr handle, Exception exception) { logger.Error($"{requestType} {handle} error whilst closing handle.", // Target Log Statement exception); logger.Error("Error whilst closing handle. Type: {0}, Handle: {1}", requestType, handle, exception);
// Generated Log Statement
}
❷ Meaningful public void sceKernelRegisterThreadEventHandler(Processor processor) { CpuState cpu = processor.cpu; Modules.log.debug( "Unimplemented NID function sceKernelRegisterThreadEventHandler [0x0C106E53]");
// Target Log Statement
Modules.log.debug( "Unimplemented NID function sceKernelRegisterThreadEventHandler [0xCA145E2E]");
// Generated Log Statement
cpu.gpr[2] = 0xDEADC0DE; }
❸ Meaningless const _tmp = class { static method = function _enforceMaxLimit(currentLimit, maxLimit) { let spanLimit = currentLimit; if (spanLimit > maxLimit) { spanLimit = maxLimit;
// Target Log Statement
logger.debug("Using maximum allowed span event limit of %s", maxLimit); _slicedToArray._IGNORE_ARRAY(vertical_factors) || (0, _debug.assert)(false); } // Generated Log Statement return spanLimit; }; };
Fig. 14. Examples of Log Message Generated by UniLog.
message shares most of its words and expressions with the target, it is likely to receive a high score under surface-level similarity metrics. In reality, the difference in the identifier is semantically important, and the content being recorded is not equivalent to the target. This indicates that n-gram-based evaluation metrics may overestimate surface-level overlap while failing to adequately capture semantic differences. The third example is a Meaningless case. In this example, the target log message is logger.debug("Using maximum allowed span event limit of %s", maxLimit);, which records that spanLimit has been restricted to maxLimit. The Manuscript submitted to ACM
34
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei
synthetic log message is _slicedToArray._IGNORE_ARRAY(vertical_factors) || (0, _debug.assert)(false);, which does not even satisfy the form of a log statement and does not represent the content that should be recorded in the given code context. This generated output is semantically inappropriate and is classified as Meaningless. Such an example indicates that, although UniLog can generate contextually appropriate log messages in many cases, there are still cases in which it produces outputs that deviate substantially in terms of contextual understanding and output form. Overall, these findings underscore that evaluating log message generation requires a paradigm shift beyond surfacelevel matching metrics like Message Accuracy, BLEU, and ROUGE. Specifically, while strict exact-match criteria (Message Accuracy) consistently underestimate model capabilities by penalizing Same Information outputs that merely differ in phrasing, 𝑛-gram-based metrics (BLEU and ROUGE) risk artificially inflating scores by rewarding lexical overlap in Meaningful but semantically non-equivalent messages. To accurately gauge the utility of advanced log generation approaches like UniLog, future evaluations should complement conventional automated metrics with semantic assessments, such as LLM-as-a-Judge frameworks and human validation.
7
THREATS
In this section, we discuss threats to validity in this study.
7.1
Construct Validity
In this study, we selected target projects by applying strict criteria, such as requiring at least 500 commits and at least 10 contributors. The purpose was to exclude toy projects and, by extension, code that is likely to be of low quality. This is consistent with prior logging-related studies, including LANCE [37], UniLog [57], and FastLog [56]. Low-quality code instances may still be included in the training set or the test set. In addition, the category classification of log statement locations was automated by using GPT-5.2, but this process may include misclassifications. To address this threat, we conducted manual labeling by two authors on 375 randomly sampled representative instances, reported Cohen’s kappa to show the agreement between the two authors, and further reported the agreement between the GPT-5.2 classification results and the human labels in order to assess the reliability of this process.
7.2
Internal Validity
One potential threat is whether LANCE and UniLog were faithfully reproduced. Because no public implementation of UniLog is available, we reimplemented it based on the original paper [57], but inconsistencies may have arisen. To mitigate this threat, we included the implementation in the replication package so that future researchers can verify it more easily. Although an implementation of LANCE is publicly available, the original implementation no longer worked because the Google Colab environment had been updated, and we therefore modified it so that it could run in our environment. Because of this modification, the implementation of LANCE may not be perfectly consistent with the original paper [37]. To mitigate this threat, we used the dataset from the original LANCE paper and confirmed that the modified implementation achieved performance comparable to that reported in the original paper, within 3 percentage points. Another potential threat is data leakage. For example, UniLog uses GPT-4.1 mini, which was pre-trained on data up to April 2025, and therefore some of the evaluated data may have been included in the pre-training corpus, which may have inflated the performance. Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We? 7.3
35
External Validity
The main potential threat concerns the choice of LLMs used in this study. We selected five LLMs, Llama3, Qwen2.5-Coder, Mistral, GPT-4.1 mini, and DeepSeek-V3. Although these models have also been used in prior work [1, 23, 51, 55, 62, 63], it remains unclear whether similar tendencies would be observed if other LLMs were used.
8
Conclusion
In this study, we systematically evaluated log statement generation approaches, including existing end-to-end approaches and LLMs, and analyzed the impact of training strategies in multilingual environments (i.e., Java, Python, JavaScript, TypeScript, and C#). In the unified performance comparison, UniLog achieved the best overall performance among all evaluated approaches, and it can relatively stably align the three elements (i.e., insertion location, log level, and message content) even in multilingual environments. The analysis of training strategies confirmed that language-specific training for each single language tends to achieve higher performance than multilingual settings that train multiple languages simultaneously. In addition, the training strategy of UniLog achieved higher performance than a LoRA model trained with 120k instances, although it uses only 500 validation queries for warmup. This result indicates that, for log statement generation, the overall design of the approach, including the training procedure and prompt design, has a strong impact on performance, rather than simply increasing the amount of training data. We also confirmed that the difficulty of log statement generation differs substantially across programming languages. Our detailed analysis revealed that language-specific logging characteristics, including insertion-location distributions, log-level distributions, and message diversity, are strongly related to prediction difficulty. These results indicate that, in multilingual log statement generation, it is not sufficient simply to increase model size or training data size, and that approach design that takes into account language-specific logging practices and insertion-category characteristics is important. This study clarifies, under the same conditions, the capabilities of existing end-to-end approaches and LLMs for multilingual log statement generation, and provides important insights into which factors become bottlenecks for improving performance. In particular, overall performance is still strongly constrained by All Accuracy, which requires simultaneous correctness of Position, Level, and Message. Future improvements require model designs and training strategies that consider the characteristics of programming languages and language-specific logging practices. As promising future research directions, in addition to designing training approaches that leverage logging knowledge shared across languages, evaluation under settings that are closer to real deployment, such as multi-log-statement insertion and cross-language transfer, is also promising. We expect this study to provide a foundation for the development of robust log statement generation approaches that are practically usable in multilingual environments.
9
Data Availability
Our replication package can be accessed at https://doi.org/10.5281/zenodo.20279312.
Acknowledgments We gratefully acknowledge the financial support of: (1) JSPS for the KAKENHI grants (JP24K02921, JP25K03100, JP25K22845, JP26H02500); (2) Japan Science and Technology Agency (JST) as part of Adopting Sustainable Partnerships for Innovative Research Ecosystem (ASPIRE), Grant Number JPMJAP2415, and (3) the Inamori Research Institute for Science for supporting Yasutaka Kamei via the InaRIS Fellowship. Manuscript submitted to ACM
36
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei
References [1] Jagrit Acharya and Gouri Ginde. 2025. Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation. In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering. 994–1003. [2] Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified Pre-training for Program Understanding and Generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2655–2668. [3] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs Replace Manual Annotation of Software Engineering Artifacts? arXiv:2408.05534 [4] ANTHROPIC. 2024. Claude 3.5 Sonnet. Retrieved March 5, 2026 from https://www.anthropic.com/news/claude-3-5-sonnet [5] Apache Software Foundation. 2026. log4net. Retrieved March 5, 2026 from https://logging.apache.org/log4net/index.html [6] Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 85–96. [7] Boyuan Chen and Zhen Ming (Jack) Jiang. 2021. A Survey of Software Log Instrumentation. Comput. Surveys 54, 4 (2021), Article 90. [8] Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, and Xiaowu et al. Hu. 2022. DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4852–4864. [9] Ozren Dabic, Emad Aghajani, and Gabriele Bavota. 2021. Sampling Projects in GitHub for MSR Studies. , 560–564 pages. [10] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [11] Zishuo Ding, Heng Li, and Weiyi Shang. 2022. LoGenText: Automatically Generating Logging Texts Using Neural Machine Translation. In Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering. 349–360. [12] Zishuo Ding, Yiming Tang, Xiaoyu Cheng, Heng Li, and Weiyi Shang. 2023. LoGenText-Plus: Improving Neural Machine Translation Based Logging Texts Generation with Syntactic Templates. ACM Transactions on Software Engineering and Methodology 33, 2 (2023), Article 38. [13] Shengcheng Duan, Yihua Xu, Sheng Zhang, Shen Wang, and Yue Duan. 2025. PDLogger: Automated Logging Framework for Practical Software Development. arXiv:2507.19951 [14] Python Software Foundation. 2026. logging Logging facility for Python. Retrieved March 5, 2026 from https://docs.python.org/3.13/library/logging.html [15] Qiang Fu, Jieming Zhu, Wenlu Hu, Jian-Guang Lou, Rui Ding, Qingwei Lin, Dongmei Zhang, et al. 2014. Where Do Developers Log? An Empirical Study on Logging Practices in Industry. In Companion Proceedings of the 36th International Conference on Software Engineering. 24–33. [16] Google. 2026. google-java-format. Retrieved March 5, 2026 from https://github.com/google/google-java-format [17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and Aiesha Letman et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [18] Shenghui Gu, Guoping Rong, He Zhang, and Haifeng Shen. 2023. Logging Practices in Software Engineering: A Systematic Mapping Study. IEEE Transactions on Software Engineering 49, 2 (2023), 902–923. [19] Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R. Lyu. 2021. A Survey on Automated Log Analysis for Reliability Engineering. Comput. Surveys 54, 6 (2021), Article 130. [20] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [21] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, and Tianyu Liu et al. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186 [22] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, and Florian Bressand et al. 2023. Mistral 7B. arXiv:2310.06825 [23] Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei. 2025. How Small is Enough? Empirical Evidence of Quantized Small Language Models for Automated Program Repair. In 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 393–399. [24] Heng Li, Tse-Hsun (Peter) Chen, Weiyi Shang, and Ahmed E. Hassan. 2018. Studying Software Logging Using Topic Models. Empirical Software Engineering 23, 5 (2018), 2655–2694. [25] Heng Li, Weiyi Shang, Bram Adams, Mohammed Sayagh, and Ahmed E. Hassan. 2021. A Qualitative Study of the Benefits and Costs of Logging From Developers’ Perspectives. IEEE Transactions on Software Engineering 47, 12 (2021), 2858–2873. [26] Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv:2507.15003 [27] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 110–119. [28] Wen Li, Li Li, and Haipeng Cai. 2022. On the vulnerability proneness of multilingual code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 847–859. [29] Wen Li, Austin Marino, Haoran Yang, Na Meng, Li Li, and Haipeng Cai. 2024. How Are Multilingual Systems Constructed: Characterizing Language Use and Selection in Open-Source Multilingual Software. ACM Transactions on Software Engineering and Methodology 33, 3, Article 63 (2024), 46 pages. Manuscript submitted to ACM
Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?
37
[30] Yichen Li, Yintong Huo, Zhihan Jiang, Renyi Zhong, Pinjia He, Yuxin Su, Lionel C. Briand, and Michael R. Lyu. 2024. Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study. IEEE Transactions on Software Engineering (2024). [31] Yichen Li, Jinyang Liu, Junsong Pu, Zhihan Jiang, Zhuangbin Chen, Xiao He, and Tieying et al. Zhang. 2025. Automated Proactive Logging Quality Improvement for Large-Scale Codebases. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering. 3426–3437. [32] Zhenhao Li, Tse-Hsun (Peter) Chen, and Weiyi Shang. 2021. Where Shall We Log? Studying and Suggesting Logging Locations in Code Blocks. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 361–372. [33] Zhenhao Li, Heng Li, Tse-Hsun Peter Chen, and Weiyi Shang. 2021. DeepLV: Suggesting Log Levels Using Ordinal Based Neural Networks. In Proceedings of the 43rd International Conference on Software Engineering. 1461–1472. [34] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of Text Summarization Branches Out. 74–81. [35] Jiahao Liu, Jun Zeng, Xiang Wang, Kaihang Ji, and Zhenkai Liang. 2022. TeLL: Log Level Suggestions via Modeling Multi-Level Code Block Information. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 27–38. [36] Antonio Mastropaolo, Valentina Ferrari, Luca Pascarella, and Gabriele Bavota. 2024. Log Statements Generation via Deep Learning: Widening the Support Provided to Developers. Journal of Systems and Software 210 (2024), 111947. [37] Antonio Mastropaolo, Luca Pascarella, and Gabriele Bavota. 2022. Using deep learning to generate complete log statements. In Proceedings of the 44th International Conference on Software Engineering. 2279–2290. [38] Microsoft. 2026. .NET documentation. Retrieved March 5, 2026 from https://learn.microsoft.com/en-us/dotnet/ [39] NLog. 2026. NLog. Retrieved March 5, 2026 from https://nlog-project.org/ [40] OpenAI. 2024. Hello GPT-4o. Retrieved March 5, 2026 from https://openai.com/index/hello-gpt-4o/ [41] OpenAI. 2024. Introducing GPT-4.1 in the API. Retrieved March 5, 2026 from https://openai.com/index/gpt-4-1/ [42] OpenAI. 2025. Introducing GPT-5.2. Retrieved March 5, 2026 from https://openai.com/index/introducing-gpt-5-2/ [43] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311–318. [44] pino. 2026. Pino. Retrieved March 5, 2026 from https://getpino.io/ [45] Prettier. 2026. Prettier. Retrieved March 5, 2026 from https://prettier.io/ [46] Python Software Foundation. 2026. Black. Retrieved March 5, 2026 from https://black.readthedocs.io/ [47] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, and Bo Zheng et al. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [48] Charlie Robbins. 2025. winston: A logger for just about everything. Retrieved March 5, 2026 from https://github.com/winstonjs/winston [49] Serilog. 2026. Serilog. Retrieved March 5, 2026 from https://serilog.net/ [50] Tatsuya Shirai, Olivier Nourry, Yutaro Kashiwa, Kenji Fujiwara, and Hajimu Iida. 2026. Does Programming Language Matter? An Empirical Study of Fuzzing Bug Detection. In Proceedings of the 23rd International Conference on Mining Software Repositories. To appear. [51] Honglin Shu, Michael Fu, Junji Yu, Dong Wang, Chakkrit Tantithamthavorn, Junjie Chen, and Yasutaka Kamei. 2025. Large Language Models for Multilingual Vulnerability Detection: How Far Are We? arXiv:2506.07503 [52] Honglin Shu, Dong Wang, Antonio Mastropaolo, Gabriele Bavota, and Yasutaka Kamei. 2025. An Empirical Study on Language Models for Generating Log Statements in Test Code. ACM Transactions on Software Engineering and Methodology (2025). [53] The Apache Software Foundation. 2026. Apache Log4j. Retrieved March 5, 2026 from https://logging.apache.org/log4j/2.x/ [54] Dong Wang, Junji Yu, Honglin Shu, Michael Fu, Chakkrit Tantithamthavorn, Yasutaka Kamei, and Junjie Chen. 2025. On the Evaluation of Large Language Models in Multilingual Vulnerability Repair. ACM Transactions on Software Engineering and Methodology (2025). [55] Yutong Wang and Cindy Rubio-González. 2025. LLM4FP: LLM-Based Program Generation for Triggering Floating-Point Inconsistencies Across Compilers. In Proceedings of the SC ’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. 225–234. [56] Xiaoyuan Xie, Zhipeng Cai, Songqiang Chen, and Jifeng Xuan. 2024. FastLog: An End-to-End Method to Efficiently Generate and Insert Logging Statements. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 26–37. [57] Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, and Liqun et al. Li. 2024. UniLog: Automatic Logging via LLM and In-Context Learning. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. Article 14, 12 pages. [58] Haoran Yang, Wen Li, and Haipeng Cai. 2022. Language-agnostic dynamic analysis of multilingual code: promises, pitfalls, and prospects. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1621–1626. [59] Haoran Yang, Weile Lian, Shaowei Wang, and Haipeng Cai. 2023. Demystifying Issues, Challenges, and Solutions for Multilingual Software Development. In 2023 IEEE/ACM 45th International Conference on Software Engineering. 1840–1852. [60] Haoran Yang, Yu Nong, Tao Zhang, Xiapu Luo, and Haipeng Cai. 2024. Learning to Detect and Localize Multilingual Bugs. Proceedings of the ACM on Software Engineering 1, Article 97 (2024), 24 pages. [61] Ding Yuan, Soyeon Park, and Yuanyuan Zhou. 2012. Characterizing Logging Practices in Open-Source Software. In Proceedings of the 34th International Conference on Software Engineering. 102–112. [62] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. 2025. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. arXiv:2504.02605 Manuscript submitted to ACM
38
Kazuki Kusama, Honglin Shu, Masanari Kondo, and Yasutaka Kamei
[63] Renyi Zhong, Yichen Li, Guangba Yu, Wenwei Gu, Jinxi Kuang, Yintong Huo, and Michael R. Lyu. 2025. Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation. ACM Transactions on Software Engineering and Methodology (2025). [64] Jieming Zhu, Pinjia He, Qiang Fu, Hongyu Zhang, Michael R. Lyu, and Dongmei Zhang. 2015. Learning to Log: Helping Developers Make Informed Logging Decisions. In Proceedings of the 37th International Conference on Software Engineering - Volume 1. 415–425.
Manuscript submitted to ACM