arXiv:2604.08028v1 [cs.SE] 9 Apr 2026
A Comparative Study of Semantic Log Representations for Software Log-based Anomaly Detection Yuqing Wang
Ying Song
Xiaozhou Li
[email protected] University of Helsinki Helsinki, Finland
[email protected] University of Helsinki Helsinki, Finland
[email protected] Free University of Bozen-Bolzano Bolzano, Italy
Nana Reinikainen
Mika V. Mäntylä
University of Helsinki Helsinki, Finland [email protected]
University of Helsinki Helsinki, Finland [email protected]
Abstract
CCS Concepts
Recent deep learning (DL) methods for log anomaly detection increasingly rely on semantic log representation methods that convert the textual content of log events into vector embeddings as input to DL models. However, these DL methods are typically evaluated as end-to-end pipelines, while the impact of different semantic representation methods is not well understood. In this paper, we benchmark widely used semantic log representation methods, including static word embedding methods (Word2Vec, GloVe, and FastText) and the BERT-based contextual embedding method, across diverse DL models for log-event level anomaly detection on three publicly available log datasets: BGL, Thunderbird, and Spirit. We identify an effectiveness–efficiency trade-off under CPU-only deployment settings: the BERT-based method is more effective, but incurs substantially longer log embedding generation time, limiting its practicality; static word embedding methods are efficient but are generally less effective and may yield insufficient detection performance. Motivated by this finding, we propose QTyBERT, a novel semantic log representation method that better balances this trade-off. QTyBERT uses SysBE, a lightweight BERT variant with systemspecific quantization, to efficiently encode log events into vector embeddings on CPUs, and leverages CroSysEh to enhance the semantic expressiveness of these log embeddings. CroSysEh is trained unsupervisedly using unlabeled logs from multiple systems to capture the underlying semantic structure of the standard BERT model’s embedding space. We evaluate QTyBERT against existing semantic log representation methods. Our results show that, for the DL models, using QTyBERT-generated log embeddings achieves detection effectiveness comparable to or better than BERT-generated log embeddings, while bringing log embedding generation time closer to that of static word embedding methods.
• Computer systems organization → Maintainability and maintenance; Reliability; • Software and its engineering → Software maintenance tools.
Authors’ Contact Information: Yuqing Wang, [email protected], University of Helsinki, Helsinki, Finland; Ying Song, [email protected], University of Helsinki, Helsinki, Finland; Xiaozhou Li, [email protected], Free University of BozenBolzano, Bolzano, Italy; Nana Reinikainen, University of Helsinki, Helsinki, Finland, [email protected]; Mika V. Mäntylä, University of Helsinki, Helsinki, Finland, [email protected].
2025. ACM 2994-970X/2025/4-ART https://doi.org/XXXXXXX.XXXXXXX
Keywords semantic log representation, deep learning, anomaly detection, efficiency, embedding, natural language processing ACM Reference Format: Yuqing Wang, Ying Song, Xiaozhou Li, Nana Reinikainen, and Mika V. Mäntylä. 2025. A Comparative Study of Semantic Log Representations for Software Log-based Anomaly Detection. Proc. ACM Softw. Eng. 1, 1 (April 2025), 12 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
As modern software systems become increasingly complex, the potential for anomalies grows [44]. The anomalies may arise from various causes, e.g., misconfigurations, resource contention, or unpredictable workloads [11]. Even a small anomaly may compromise system reliability and performance [17]. Timely and effective anomaly detection is critical to prevent anomalies from escalating into severe failures [11, 39]. Software logs record runtime information and system states, providing a primary source for anomaly detection [44]. However, modern systems generate logs at a massive scale. Recent reports indicate that many systems produce more than 1 TB of logs per day [29]. This makes manual log anomaly detection labor-intensive and error-prone. Deep learning (DL) methods have been widely adopted for automated log anomaly detection. A critical step in these methods is log representation, which converts log events into structured inputs for DL models [11]. Semantic log representation methods have been increasingly adopted in recent DL studies [41, 44]. Compared to traditional methods that represent logs using discrete features (e.g., event identifiers or occurrence counts), semantic log representation methods encode the textual content of log events into vector embeddings that preserve their semantic meaning, thus providing more informative inputs for DL models [11, 41]. Several semantic log representation methods have been proposed, ranging from methods based on static word embedding models (e.g., FastText) to pre-trained language models (e.g., BERT). DL methods built on such representations have shown promising effectiveness across
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
diverse real-world log datasets [11, 16]. However, in prior work, these methods are typically evaluated as end-to-end pipelines that couple semantic representations with DL models, making the reported performance reflect the overall pipeline [41]. It remains unclear how different semantic log representation methods affect the performance of the DL methods. To address the gap, we conduct a comprehensive empirical study to evaluate four widely used semantic log representation methods, including static word embedding methods (Word2Vec, GloVe, and FastText) and the BERT-based contextual embedding method, across a broad set of DL models (covering popular recurrent, convolutional, and attention-based architectures) using publicly available log datasets from three large-scale distributed systems: BGL, Thunderbird (TB), and Spirit. We focus on log event-level anomaly detection, which is well-suited for such distributed software systems where log events are generated in an interleaved manner by different system components that operate independently or participate in inter-component interactions; it enables fine-grained anomaly localization by identifying the responsible components, thereby facilitating root cause analysis [8, 39]. This setting differs from log session-level anomaly detection, which determines whether a session of log events is anomalous or normal. In our datasets, explicit session boundaries are not provided, and constructing sessions would require system-specific heuristics that may introduce confounding factors for our evaluation. We examine both detection effectiveness and computational efficiency, with a particular focus on CPU-only deployment settings that are common in production environments. Although DL models are typically trained using GPUs, not all production environments have dedicated GPU resources; also, log anomaly detection needs to process log events continuously, and provisioning GPUs for sustained inference can significantly increase operational costs [2, 11, 40]. The results of our empirical study reveal a clear effectivenessefficiency trade-off under CPU-only deployment settings: the BERTbased method is more effective, but incurs substantially longer log embedding generation time, limiting its practicality; static word embedding methods are efficient, but are generally less effective and may yield insufficient detection performance. Motivated by this finding, we propose QTyBERT, a novel semantic log representation method that better balances this effectiveness–efficiency trade-off in CPU deployment settings. The key idea behind QTyBERT is to use a lightweight BERT variant to efficiently generate log embeddings while ensuring that these embeddings achieve semantic expressiveness comparable to that of the standard BERT model. Although lightweight variants of BERT have been widely explored in the natural language processing (NLP) community as efficient alternatives for BERT-style contextual embedding generation [14, 31], their applicability to semantic log representation remains unexplored. QTyBERT consists of two components: a System-specific Base Encoder (SysBE), which converts log events into vector embeddings, and a Cross-System Embedding Enhancement module (CroSysEh), which operates on these log embeddings to improve their semantic expressiveness. SysBE is constructed by applying system-specific quantization to a lightweight BERT variant, enabling efficient log embedding generation on CPUs. CroSysEh is trained in an unsupervised manner using unlabeled logs from multiple systems to capture
Wang et al.
the underlying semantic structure of the standard BERT model’s embedding space, compensating for the semantic loss introduced by the compact design of SysBE. We evaluate QTyBERT against existing semantic log representation methods under the same experimental settings as our empirical study. Our results show that, for the same DL models, QTyBERTgenerated log embeddings achieves detection effectiveness comparable to or better than BERT-generated log embeddings, with F1-score differences within ±1% in most cases and improvements of up to 21.53%. QTyBERT reduces log embedding generation time by more than 94% compared to BERT, achieving sub-millisecond latency per log event, bringing its efficiency much closer to that of static word embedding methods. In summary, our main contributions are highlighted as follows: • We conduct a comprehensive empirical study to benchmark widely used semantic log representation methods across a broad set of DL models for log event-level anomaly detection using publicly available log datasets. • Our empirical study identifies a clear trade-off between static word embedding and BERT-based methods in detection effectiveness and log embedding generation efficiency under CPU-only deployment settings. • We propose QTyBERT, a novel semantic log representation method that better balances this effectiveness–efficiency trade-off, and evaluate it against existing semantic log representation methods using publicly available log datasets.
2 Background 2.1 Deep Learning-based Log Anomaly Detection with Semantic Log Representation 2.1.1 Semantic Log Representation. Early studies use static word embedding methods, which first parse log messages into structured log templates, decompose these templates into word tokens, and then encode word tokens into vector representations using pretrained static embedding models. The widely used static embedding models include Word2Vec [25], GloVe [28], and FastText [15]. For instance, Word2Vec is used in TinyLog [21], LightLog [40], and EdgeLog [2], where a Word2Vec model is either trained on the target system log templates or pre-trained and then applied to encode each template into vector embeddings. GloVe is adopted in LogTransfer [3] and LogPal [34], while FastText is employed in LogRobust [44] and RT-Log [13], both using pre-trained word embeddings trained on large-scale corpora (e.g., Common Crawl) to encode word tokens of log templates into vector embeddings. Static word embedding methods are computationally efficient because they use pre-trained word embeddings to represent words in log templates. Generating such log embeddings mainly involves dictionary lookup and aggregation operations, making them suitable for resource-constrained environments such as CPU-only deployments [15, 25, 28]. However, these methods have two key limitations. First, they rely on a fixed word vocabulary learned from training data and thus struggle to handle out-of-vocabulary (OOV) words, which are common in software logs [16, 18]. Examples of OOV words include system modules (e.g., ‘kubelet’, ‘etcd’), kernelrelated processes (e.g., ‘ksoftirqd’, ‘rcu_sched’) [39]. Second, static word embedding methods depend on log parsing, which separates
A Comparative Study of Semantic Log Representations for Software Log-based Anomaly Detection
static (template) and variable (parameters) part of the log, e.g., in log message “User connected to 192.168.0.1” the template is “User connected to" and parameter is “192.168.0.1”. The quality of log embeddings from these methods depends on the accuracy of log parsing, which can affect the effectiveness of downstream anomaly detection [16, 18]. Even widely used parsers such as Drain [9] may produce parsing errors due to inconsistent log formats, nested data structures, or missing values [5, 32]. Recent studies have shifted to the BERT-based method for contextual embedding (e.g., NeuralLog [16], CNN [30], CroSysLog [39]), which use the pre-trained language model BERT to encode raw log events into vector representations. This BERT-based method addresses the limitations of static word embedding methods. It does not require log parsing and can directly process raw log events. It first tokenizes each log event into subword tokens and then encodes these subword tokens using BERT’s self-attention mechanism, capturing semantic relationships among subword tokens. This mechanism allows to handle OOV words by decomposing them into subwords. With this mechanism, the embedding of each subword token is contextualized, i.e., it is dynamically generated based on its surrounding subword tokens. However, generating such BERT-based log embeddings is computationally expensive. In practice, BERT inference for embedding generation is typically accelerated using GPUs [4, 18, 35]. 2.1.2 Deep Learning Models. Software system logs are sequential, as log events are generated over time during system execution [11, 44]. Log events exhibit temporal correlations that reflect system behavior. DL-based sequence models are therefore widely adopted to capture such temporal dependencies. Recurrent neural network (RNN) variants are the most commonly used DL models. For example, CroSysLog [39] and LogAnomaly [22] use LSTM, while the study [33] uses GRU. SwissLog [19] and LogRobust [44] use the Attention-based BiLSTM (AttBiLSTM), which extends LSTM with a bidirectional encoder and an attention mechanism to capture both forward and backward dependencies among log events and focus on the most relevant ones. NeuralLog [16] and HitAnomaly [12] use a Transformer-encoder (TransEnc), which replaces recurrence with self-attention to capture long-range dependencies across log events. CNN has also been applied, e.g., in the studies [8, 20, 30]. Unlike RNN models, CNNs apply convolutional filters over log event sequences to capture local patterns among neighboring log events.
2.2
Related Work
2.2.1 Effect of Semantic Log Representation Methods. The studies on how semantic log representation methods affect DL-based log anomaly detection are scarce. The only closely related work is by Wu et al. [41], who investigate the impact of log representation methods on log session-level anomaly detection. Their results show that, BERT-generated log embeddings achieve the highest effectiveness when used with DL models, while classical log representation methods such as MCV outperform semantic-based ones when used with traditional ML models. Our empirical study addresses several important aspects not considered in their study. First, our study investigates log event-level anomaly detection, which is not explored in their study. Second, their study evaluates three semantic log
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
representation methods (Word2Vec, FastText, and BERT), whereas our study additionally includes GloVe, which is also widely used in existing DL-based log anomaly detection. Third, we evaluate each log representation method on a broader set of DL models, covering commonly used ones, including RNN, GRU, LSTM, AttBiLSTM, TransEnc, and CNN, whereas their study only covers MLP, CNN and LSTM. Last, we benchmark the computational efficiency of each semantic log representation method, which is a crucial practical concern when deploying these methods in production environments but was not previously evaluated.
2.2.2 Efficient BERT-style Log Embedding Generation. Efforts to address the high computational cost of BERT-based log embedding generation remain limited. The only related work is LAnoBERT [18], which introduces a log dictionary-based inference mechanism to avoid redundant embedding computation for previously seen log events, but the computational cost of generating embeddings for new log events remains high. In the NLP community, lightweight variants of BERT, such as DistilBERT [31] and TinyBERT [14], have been proposed to accelerate BERT-style contextual embedding generation in resourceconstrained environments. These variants compress the standard BERT model using techniques such as knowledge distillation and architectural compression, resulting in fewer model layers and parameters and thus reducing computational cost during embedding generation [6]. However, this efficiency comes at the cost of semantic loss, as their ability to capture complex semantic and contextual relationships among subword tokens is weakened compared to the standard BERT [6, 14]. As such, applying these variants to domainspecific tasks typically requires fine-tuning [14, 31], which involves task-specific training with domain data and updating model parameters. This process incurs additional training costs and must be repeated for each task. These variants have been widely adopted as efficient alternatives to the standard BERT for NLP tasks, e.g., text classification and question answering [14]. However, their applicability to semantic log representation in log anomaly detection remains unexplored. This motivates us to develop QTyBERT. Our QTyBERT addresses the gaps from two aspects. First, inspired by lightweight BERT variants in NLP, QTyBERT extends this idea to efficient log embedding generation through SysBE, a lightweight BERT variant with system-specific quantization. Unlike LAnoBERT, SysBE directly accelerates log embedding generation on CPUs. Second, to compensate for the semantic loss introduced by the compact design of SysBE, QTyBERT employs CroSysEh, which operates on log embeddings generated by SysBE to improve their semantic expressiveness, eliminating per-system fine-tuning and reducing such training costs in multi-system settings.
3
Empirical Study
Our empirical study is guided by the research question: • RQ1: How do different semantic log representation methods impact the effectiveness and efficiency of log event-level anomaly detection, when serving input for DL models?
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
3.1
Wang et al.
Experimental Setup
3.1.1 Datasets. For a comprehensive evaluation, we use software log datasets of three large-scale distributed supercomputing systems: BGL, TB, and Spirit, sourced from the USENIX CFDR repository [26, 36]. BGL is the IBM Blue Gene/L system at Lawrence Livermore National Laboratory. TB and Spirit are high-performance Linux clusters operated by Sandia National Laboratories. Each dataset includes log event level binary labels (normal vs. anomalous). We used two chronological log sequences from each system: one sequence as the training set, and the other as the testing set. Table 1 summarizes the statistics of these sets for each system. For each system, the testing set is temporally subsequent to the training set to preserve chronological order; these two sets do not overlap, there is a temporal gap of 4-6 months between the training set and the testing set to break short-range autocorrelation and avoid near-duplicate patterns around the boundary between the two sets, thereby improving the validity of the evaluation [1, 10]. Table 1: Statistics of training and testing sets. System
Set
# Log events
# Anomaly
BGL
Training Testing
1,885,397 471,349
227,994 (12.09%) 37,000 (7.85%)
TB
Training Testing
997,677 1,396,747
69,838 (7.00%) 184,231 (13.19%)
Spirit
Training Testing
499,095 499,095
149,728 (30.00%) 19,964 (4.00%)
3.1.2 Pre-processing. For each system, we utilize LogLead [23] to process raw log files, extracting individual log events and organizing them into dataframes that capture key attributes such as timestamp, severity level, reporting component, log message, and anomaly label, if these are available. Since the attributes vary across datasets, we remove log events with missing values in the attributes defined by each dataset, and then sort the remaining log events in chronological order to reflect the operational sequence. For each log event, we concatenate the textual attributes (i.e., reporting component, severity level, and log message) into a single text sequence to represent this log event. The concatenated sequence is then preprocessed by lowercasing, removing non-alphabetic characters, and masking sensitive variables, e.g., replacing “192.168.1.*" with “ip address", or “/var/app/config/settings.yaml” to “file path”. This design differs from conventional log session-level anomaly detection, where log messages alone are used to represent log events, as anomaly signals are typically captured from patterns in log sequences [11]. Since we focus on log event-level anomaly detection in distributed systems, where log events are generated by different system components that operate independently or participate in inter-component interactions, each log event is expected to carry sufficient information for anomaly detection. Therefore, we retain additional textual fields such as reporting component and severity level to preserve component-level operational context, consistent with prior work on this topic [16, 39].
3.1.3 Semantic Log Representation Methods. We evaluate four widely used semantic log representation methods: three static word embedding methods (Word2Vec, FastText, and GloVe), and the BERT-based contextual embedding method. We follow prior studies reviewed in Section 2.1.1 to implement these methods and ensure a fair comparison across them. For the static word embedding methods, we use the pre-trained FastText model (300-dimensional, trained on Common Crawl) [15], pre-trained GloVe model (300-dimensional, trained on Wikipedia and Gigaword) [28], and train Word2Vec on each system’s training set. For each static word embedding method, we obtain log event-level embeddings as follows: we first parse log events into log templates using Drain [9], tokenize log templates into word tokens, obtain word token embeddings using the corresponding static word embedding model, and then aggregate the token embeddings using TF-IDF weighting. We keep the log parser, tokenization strategy, and aggregation approach fixed across these methods to avoid introducing confounding factors that affect our evaluation results. For the BERT-based method, we implement it using a neural representation approach following prior studies [16, 30, 39]. Specifically, we use the BERT-base model [7], which consists of 12 Transformer encoder layers with 768 hidden units and 12 attention heads. We obtain log event-level embeddings as follows: we tokenize log events into subword tokens using WordPiece technique [42], feed subword tokens into BERT-base to obtain contextualized subword embeddings, and then aggregate these subword embeddings using mean pooling over the final hidden layer. 3.1.4 Deep Learning Models. We select commonly used DL models in prior log anomaly detection studies, including all discussed in Section 2.1.2: GRU, LSTM, AttBiLSTM, CNN, and TransEnc. In addition, we include a vanilla RNN as a simple recurrent baseline to assess the benefits of more complex recurrent architectures. For each system, we train all DL models on its training set using a consistent supervised setting and evaluate them on its testing set. This ensures that the comparison focuses solely on the effect of different log representation methods, rather than differences caused by unsupervised detection objectives or thresholding strategies. These DL models use log embeddings generated by each log representation method (Section 3.1.3) as input during both training and testing. Following prior work on log event-level anomaly detection [39], these DL models take fixed-size windows of log event embeddings as inputs. Specifically, for each system 𝑠 𝑗 , its log events are ordered chronologically as 𝐿 ( 𝑗 ) = {𝑒 1, 𝑒 2, . . . , 𝑒 𝑁 }, where each 𝑒𝑘 denotes the embedding of the 𝑘-th log event produced by a certain log representation method. We partition 𝐿 ( 𝑗 ) into non-overlapping windows of size 𝑚, where each window consists of 𝑚 consecutive log event embeddings and serves as an input to the DL models. 3.1.5 Implementation Details. We perform the model training on a computing server with 16 CPU cores and a single NVIDIA Ampere A100 GPU with 40 GB of memory. All DL models are trained for a fixed number of epochs, and each model is tuned to obtain its optimal performance under our experimental setting. During testing, we simulate CPU-only environments with 4-core or 8-core CPU allocations without GPU resources. These environments are configured to ensure full utilization of the allocated CPU cores. We monitor CPU utilization throughout the evaluation process.
A Comparative Study of Semantic Log Representations for Software Log-based Anomaly Detection
3.1.6 Metrics. For each DL model, we compare different semantic log representation methods in terms of anomaly detection effectiveness using Precision, Recall, and F1-score. These metrics are computed based on True Positives (TP), False Positives (FP), and False Negatives (FN). Precision is defined as the proportion of correctly identified anomalies among all predicted anomalies, i.e., Pre𝑇𝑃 cision = 𝑇 𝑃+𝐹 𝑃 . Recall measures how many actual anomalies were 𝑃 correctly detected, i.e., Recall = 𝑇 𝑃𝑇+𝐹 𝑁 . F1-score, as the harmonic mean of Precision and Recall, is given by F1-score = 2·Precision·Recall Precision+Recall . We adopt these metrics because log anomaly detection is a binary classification task where the normal and abnormal classes are often imbalanced. In such cases, Precision quantifies the false alarm rate, Recall ensures that actual anomalies are not missed, and the F1-score offers a balanced summary of both. For efficiency, we compare each log representation method in terms of the time required to generate embeddings for log events in the testing set of each system, as well as the detection latency of each DL model using these log embeddings.
3.2
Study results and analysis
3.2.1 Effectiveness. The DL models consistently achieve higher effectiveness when using BERT-based log embeddings than those generated by static word embedding methods, with the impact being the most pronounced on BGL. As shown in Table 2, static word embedding methods only achieve F1-scores of 55.05%–67.87% on BGL across all DL models; however, replacing them with BERT-based log embeddings improves F1-scores by approximately 13%–31% for each model. On TB and Spirit, BERT-based log embeddings remain more effective in most cases, although the performance gap becomes smaller, generally within 9% F1-score across DL models. These findings are consistent with Wu et al. [41], who observe similar results for log session-level anomaly detection. In contrast, the performance differences among static embedding methods are limited. Using FastText-, GloVe-, and Word2Vec-based log embeddings, the maximum F1-score deviation on each DL model is small (typically within about 3%), indicating that the choice among static embedding methods has only a limited impact. 3.2.2 Efficiency. Log Embedding Generation. Static word embedding methods require substantially less log embedding generation time than the BERT-based method under CPU-only environments. As reported in Table 4, Word2Vec is the fastest across all systems. Under the 8-core CPU setting, Word2Vec requires only 0.05–0.12 ms per log event, whereas BERT requires 4.38–7.44 ms, resulting in approximately 37×–149× longer embedding generation time for BERT. Under the 4-core CPU setting, this gap further widens to approximately 74×–312×. Detection latency. Compared with log embedding generation time, downstream detection latency is much less affected by the choice of semantic log representation methods. Since DL models using FastText-, GloVe-, and Word2Vec-based log embeddings exhibit very similar detection latency, we report their average latency (Static Avg) along with the maximum deviation (Δmax ) to simplify comparison in Table 5. For most DL models (LSTM, GRU, CNN, and RNN), using BERT-based log embeddings incurs only approximately 1.05×–1.20× the detection latency compared to those produced by static word embedding methods. The gap becomes more noticeable
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
for DL models with more complex architecture (TransEnc and AttnBiLSTM), where the increase ranges from approximately 1.13× to 1.9×, depending on the system and CPU configuration. This difference mainly stems from variations in embedding dimensionality across log representation methods: 768 for BERT vs. 300 for static word embedding methods, see Section 3.1.3. Since the computational cost of linear transformations and attention mechanisms scales with the input dimensionality, the higher-dimensional BERTbased log embeddings result in increased processing time in DL models. 3.2.3 Trade-off. Our above results show that the choice of semantic log representation methods affects the performance of DL-based log event-level anomaly detection. BERT-based and static word embedding methods exhibit a clear trade-off between detection effectiveness and log embedding generation efficiency. BERT-based log embeddings generally lead to higher detection effectiveness, but their substantially higher generation time may limit their practicality in CPU-only environments. In contrast, static word embedding methods are efficient and well-suited for CPU-only deployment settings, but their log embeddings are generally less effective and may yield insufficient detection performances.
4 QTyBERT for semantic log representation 4.1 Design Figure 1 shows the overall workflow of QTyBERT. During application in a target system, SysBE produces log embeddings, which are then processed by CroSysEh to obtain the final log representations. We explain how each component is built in the following subsections. CroSysEh trained once for multiple systems
generate
Multi-system unlabeled log events
❄ TinyBERT
Target system unlabeled log events
Log embeddings
SysBE
log events
quantization
CroSysEh
apply
SysBE obtain
Final log embeddings
Log embeddings generate
map to
apply
apply
Small calibration calibrate dataset
Target system application
Mapped log embeddings compare loss
generate
SysBE constructed per target system sample
❄ Frozen
Trainable
Log embeddings
❄ BERT
CroSysEh
map to
Figure 1: An overview of QTyBERT
4.1.1 SysBE. To build SysBE, we conduct a preliminary study on existing lightweight BERT variants for efficient BERT-style contextual embedding generation in CPU-only environments. Through a review of relevant studies and publicly available implementations, we identify several candidate models (e.g., TinyBERT [14], DistilBERT [31], MiniLM [38]) that retain the standard BERT embedding
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
pipeline, particularly subword tokenization and contextualized subword representations, thereby enabling fair comparison and avoiding the introduction of confounding factors. We evaluate these models under the same experimental settings as in our empirical study. TinyBERT achieves the best effectiveness among the candidates across DL models, while exhibiting comparable embedding generation latency. We therefore select TinyBERT to build SysBE. Specifically, we use a TinyBERT model consisting of 4 Transformer encoder layers with 312 hidden units and 12 attention heads. Let M denote the original TinyBERT. For each system 𝑠 𝑗 , we quantize M to obtain its SysBE in several steps. First, we use a small number of unlabeled log events from 𝑠 𝑗 to build a calibration (𝑗) dataset Dcal . These log events are not required to be temporally consecutive. They are preprocessed using the same steps as those in our empirical study (Section 3.1.2). Second, we collect statistics (including value ranges, means, variances, and outliers) from the activations of M when using M to generate log embeddings for (𝑗) log events in Dcal , and then use these statistics to calibrate the quantization parameters. Third, based on the calibrated parameters, we quantize approximately 20% of the linear layers in the Transformer encoders of M by mapping their FP32 weights to INT8 representations. Here, FP32 and INT8 denote 32-bit floating-point and 8-bit integer numerical representations, respectively. Our quantization keeps M’s embedding layers and activations in FP32 to maintain semantic fidelity, as we empirically observe that aggressive quantization of these components degrades embedding quality, manifested by reduced anomaly detection effectiveness of downstream DL models when operating on the resulting log embeddings. This observation is consistent with prior work that examines how quantize different components of BERT affects the quality of embeddings [24]. We thus obtain the quantized TinyBERT model as the SysBE for 𝑠 𝑗 , denoted as Mq( 𝑗 ) . We export Mq( 𝑗 ) as an ONNX computation graph [27]. The graph includes tensor-level quantization and dequantization operators configured for INT8 precision, which serve as precision bridges between INT8 and FP32 and enable mixed-precision execution. 4.1.2 CroSysEh. We train CroSysEh using unlabeled log events from multiple systems. We consider 𝑁 software systems, each producing log events in chronological order. From each system, we randomly sample 𝑚 unlabeled log events, which are not required to be consecutive. The sampled log events from all systems constitute a cross-system training dataset, denoted as Dcro = 𝑥 1, 𝑥 2, . . . , 𝑥𝑛 , where 𝑥𝑖 is each log event. We pre-process log events in Dcro using the same steps as those in our empirical study (Section 3.1.2) We train CroSysEh in several steps, as outlined in Algorithm 1. For each log event 𝑥𝑖 from Dcro , we use both the frozen standard BERT and the frozen original TinyBERT M to generate the corresponding log embeddings, following the same BERT-based neural representation approach as in our empirical study (Section 3.1.3). We use BERT-base [7] as the standard BERT implementation, consistent with our empirical study setting (Section 3.1.3). For each log event 𝑥𝑖 , we denote its embedding from BERT as the teacher embedding ℎ𝑇 𝑖𝑛 𝑑𝑇 , and the one from M as the student embedding ℎ𝑆 𝑖𝑛 𝑑𝑆 , where the embedding dimensions correspond to the hidden sizes of each model. We use a residual low-rank function to
Wang et al.
map ℎ𝑆 to the embedding space of ℎ𝑇 : ℎ𝑆′ ← 𝜙 (ℎ𝑆 ) = ℎ𝑆 + 𝐵(𝐴(ℎ𝑆 )) where 𝐴 ∈ R𝑟 ×𝑑𝑆 and 𝐵 ∈ R𝑑𝑇 ×𝑟 are trainable projection matrices, and 𝑟 is a small bottleneck dimension that controls the adaptation capacity. The matrices 𝐴 and 𝐵 together parameterize CroSysEh, denoted by 𝜙, which maps each student embedding ℎ𝑆 to the embedding space of ℎ𝑇 . We train CroSysEh 𝜙 by minimizing the mean squared error (MSE) between the mapped embedding ℎ𝑆′ = 𝜙 (ℎ𝑆 ) and the teacher embedding ℎ𝑇 for each log event 𝑥𝑖 in Dcro . The loss function is defined as: ∑︁ 1 2 ℎ𝑆′ − ℎ𝑇 2 L= |Dcro | 𝑥 ∈ D 𝑖
cro
During training, we keep both BERT and M frozen, and optimize only CroSysEh 𝜙 by minimizing the loss L using gradient descent: 𝜙 ← 𝜙 − 𝜂 · ∇𝜙 L where 𝜂 is the learning rate, and ∇𝜙 L denotes the gradient of L with respect to the parameters of 𝜙. After training, we obtain the optimized CroSysEh 𝜙 ′ , which maps M’s log embeddings to the embedding space of BERT. Depending on the source of the sampled log events, 𝜙 ′ can be shared across systems. Algorithm 1 CroSysEh training Require: Log dataset Dcro = {𝑥 1, . . . , 𝑥𝑛 }, frozen BERT, frozen M, trainable CroSysEh 𝜙, learning rate 𝜂, number of epochs 𝐸 1: Initialize 𝜙 (i.e., projection matrices 𝐴 and 𝐵) randomly 2: for epoch = 1 to 𝐸 do 3: for 𝑥𝑖 in Dcro do 4: ℎ𝑇 ← BERT(𝑥𝑖 ) 5: ℎ𝑆 ← M (𝑥𝑖 ) 6: ℎ𝑆′ ← 𝜙 (ℎ𝑆 ) = ℎ𝑆 + 𝐵(𝐴(ℎ𝑆 )) 7: L ← L + ∥ℎ𝑆′ − ℎ𝑇 ∥ 22 8: end for 9: L ← L/|Dcro | 10: Update 𝜙 ← 𝜙 − 𝜂∇𝜙 L 11: end for 12: return Optimized CroSysEh 𝜙 ′
4.2
Experiment setup
To evaluate QTyBERT, we define the research question: • RQ2.Performance: How do QTyBERT perform compared to prior semantic log representation methods when serving input to downstream DL models? To develop QTyBERT, we sample additional unlabeled log events from the same software systems in the USENIX CFDR repository used in our empirical study. Specifically, to build SysBE for each system 𝑠 𝑗 , we randomly select 70 unlabeled log events to form its (𝑗) calibration dataset Dcal to quantize the original TinyBERT M and
obtain the corresponding SysBE Mq( 𝑗 ) ; moreover, we randomly sample 25,000 unlabeled log events from each system, which together constitute the dataset Dcro to train CroSysEh. The sampled log events may overlap with the training set used in our empirical study, but are disjoint from the testing set used in our empirical
A Comparative Study of Semantic Log Representations for Software Log-based Anomaly Detection
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Table 2: Precision, Recall, and F1-score of deep learning models using different semantic log representation methods. DL Model [Log Rep. Method] TransEnc [FastText] TransEnc [Glove] TransEnc [Word2Vec] TransEnc [BERT] TransEnc [QTyBERT] AttBiLSTM [FastText] AttBiLSTM [Glove] AttBiLSTM [Word2Vec] AttBiLSTM [BERT] AttBiLSTM [QTyBERT] LSTM[FastText] LSTM [Glove] LSTM [Word2Vec] LSTM [BERT] LSTM [QTyBERT] GRU [FastText] GRU [Glove] GRU [Word2Vec] GRU [BERT] GRU [QTyBERT] CNN [FastText] CNN [Glove] CNN [Word2Vec] CNN [BERT] CNN [QTyBERT] RNN [FastText] RNN [Glove] RNN [Word2Vec] RNN [BERT] RNN [QTyBERT]
BGL Precision 63.55 67.75 69.81 92.63 93.46 72.97 72.91 68.43 93.33 93.35 96.35 87.60 66.16 90.01 93.85 66.12 81.34 60.62 89.16 94.16 61.61 78.98 81.65 96.06 98.16 47.95 70.17 48.52 86.21 93.65
Recall 64.47 66.07 66.04 88.73 86.76 56.59 59.21 57.42 80.43 90.23 44.60 47.74 54.13 90.31 82.08 55.01 53.87 66.43 92.73 85.04 58.63 56.54 53.59 66.19 69.42 64.64 50.48 66.49 56.63 86.42
TB F1-score 64.01 66.90 67.87 90.63 89.98 63.74 65.35 62.45 86.40 91.77 60.98 61.80 59.55 90.16 87.57 60.05 64.82 63.39 90.91 89.36 60.08 65.90 64.71 78.37 79.52 55.05 58.72 56.09 68.36 89.89
study and occur earlier than the log events in this testing set to preserve chronological order. We construct SysBE on CPUs, and train CroSysEh on a GPU since running BERT on CPU is significantly slow for large-scale log data (see Table 4), using the same hardware configuration as our empirical study (Section 3.1.5). As a result, each SysBE Mq( 𝑗 ) has a storage footprint of 43 MB, which is substantially smaller than BERT (≈440 MB), GloVe (≈1 GB), and FastText (≈4.51 GB), while remaining reasonably compact compared to systemspecific Word2Vec models (1.68–10.44 MB). CroSysEh has a storage footprint of 968 KB. We evaluate QTyBERT against the semantic log representation methods in our empirical study under the same experimental settings (Section 3.1), i.e., using the same log datasets, DL models with fixed-size window strategy, training and testing sets, CPU deployment settings, implementation settings, and evaluation metrics. Specifically, for each system, we encode log events in training and testing sets into embeddings using the corresponding SysBE, and then map these embeddings to the final embedding space through CroSysEh. The final log embeddings are input to the DL models using the fixed-size window strategy for anomaly detection.
Precision 99.91 99.99 99.41 90.26 91.17 99.79 99.97 99.99 99.00 99.33 94.28 99.99 96.50 99.27 99.00 98.94 97.03 86.41 99.39 99.43 99.96 99.98 96.26 99.52 99.71 96.82 95.95 87.12 96.67 99.63
Recall 94.56 94.54 94.55 94.43 93.14 89.04 85.19 87.38 93.80 94.03 83.17 83.85 84.26 97.80 93.95 83.93 85.06 94.97 93.97 93.29 94.26 94.51 94.57 93.62 93.46 79.36 82.17 94.20 93.63 92.57
Spirit F1-score 97.16 97.19 96.92 92.29 92.15 94.11 91.99 93.26 96.32 96.06 88.37 91.21 89.96 98.53 96.41 90.81 90.65 90.49 96.60 96.26 97.02 97.13 95.41 96.48 96.49 87.22 88.53 90.52 95.12 95.97
Precision 98.73 99.34 97.40 98.19 95.51 96.97 98.73 97.47 100.0 99.42 99.38 95.95 98.10 100.0 99.42 98.20 97.48 98.73 100.0 98.30 97.08 97.91 100.0 100.0 97.18 98.75 94.15 100.0 100.0 98.41
Recall 77.22 74.75 74.26 80.69 84.16 79.21 77.22 76.24 82.67 84.65 79.70 82.18 76.73 82.67 85.15 81.19 76.73 77.23 83.67 85.64 82.17 83.74 81.68 83.66 85.15 78.21 79.70 82.17 82.67 92.08
F1-score 86.67 85.31 84.26 88.58 89.47 87.19 86.66 85.56 90.51 91.44 88.46 88.53 86.11 90.51 91.73 88.89 85.87 86.67 91.11 91.53 89.00 90.27 89.91 91.10 90.76 87.29 86.32 90.21 90.51 95.14
5 QTyBERT Experiment Results and Analysis 5.1 RQ2. Performance 5.1.1 Effectiveness. Our QTyBERT generates effective log embeddings that are comparable to those of BERT, and even outperform it in certain cases. As shown in Table 2, for most DL models, using log embedding from QTyBERT instead of BERT leads to F1-score differences within 1%, either slightly higher or lower. A notable exception is RNN on BGL, where using QTyBERT yields a 21.53% higher F1-score than BERT (89.89% vs. 68.36%), and achieves performance comparable to complex DL models TransEnc (89.98%) and AttBiLSTM (91.77%). Furthermore, with QTyBERT-based log embeddings, RNN achieves the highest F1-score on Spirit (95.14%) outperforming all other DL models across different representation methods. These results indicate that QTyBERT generates effective log embeddings, enabling a vanilla RNN to achieve competitive detection effectiveness compared to more complex DL models. To investigate how QTyBERT learns from BERT through its CroSysEh, we perform both visualization (Figure 2) and quantitative comparison (Table 3) of their generated log embeddings. From
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Wang et al.
Table 3: Cosine Similarity and Spearman Correlation of Log Embeddings (QTyBERT vs. BERT)
(a) BGL
(b) TB
System
Cosine (Mean)
Spearman 𝜌
BGL TB SPIRIT
0.0492 0.1194 0.1016
0.7383*** 0.6095*** 0.8089***
***p < 0.001; **p < 0.01; *p < 0.05
Avg F1-score (%)
BGL
(c) Spirit
Figure 2: t-SNE visualizations comparing log embeddings generated by • BERT and • QTyBERT.
TB
QTyBERT
90 85 80 75 70 65 60
98 BERT
92
Static
90 10 −1
100
10 −1
101
Avg F1-score (%)
93
Spirit
Method
100
101
Avg Embedding Time (ms/Log)
QTyBERT BERT
91 89 87
Static
85 10 −1
100
101
Avg Embedding Time (ms/Log)
Figure 3: Trade-off between detection effectiveness (Avg F1score %) and log embedding generation efficiency (ms/Log). •⋄ Static, •⋄ QTyBERT, •⋄ BERT; • 8-core CPU, ⋄ 4-core CPU. 5.1.2 Efficiency. Log Embedding Generation. QTyBERT generates log embedding significantly faster than BERT in CPU-only deployment settings across all three systems, as shown in Table 4. On the 8-core CPU setting, QTyBERT is 12× to 21× faster than BERT, with an average generation time of 0.36 ms per log event compared to 4.38–7.44 ms per log event for BERT. On the 4-core CPU setting, the speedup is even more pronounced, with QTyBERT being 14× to 25× faster than BERT, achieving 0.61–0.64 ms per log
Table 4: Log embedding generation time of different log representation methods.
CPU
BERT
94 Static
Avg Embedding Time (ms/Log)
each system, we randomly sample 50,000 log events and obtain their embeddings with BERT and QTyBERT, respectively. We then apply t-SNE [37] to project log embeddings of these two methods into a two-dimensional space. As shown in Figure 2, for each system, log embeddings generated by these two methods exhibit a high degree of overlap in structure, while maintaining some distributional differences. This is further supported by quantitative results in Table 3. Spearman correlation between log embeddings of QTyBERT and BERT is significantly high (0.6095–0.8089, 𝑝 < 0.001), indicating that their log embeddings have similar structural relationships. The cosine similarity between their log embeddings is low (0.0492–0.1194). This is expected, as quantization in SysBE alters numerical values and CroSysEh learns the shared semantic structure across systems in the embedding space, which may result in different embedding direction and scale. These results indicate that QTyBERT learns the underlying functional semantic structure of BERT’s embedding space rather than replicating its exact embedding values.
QTyBERT
96
BGL
TB
Spirit
Total (s)
Avg (ms)
Total (s)
Avg (ms)
Total (s)
Avg (ms)
8-core
FastText GloVe Word2Vec BERT QTyBERT
111.88 96.44 54.98 2065.88 167.66
0.24 0.20 0.12 4.38 0.36
130.67 124.13 67.50 10392.57 504.22
0.09 0.09 0.05 7.44 0.36
110.23 119.33 57.87 3450.03 178.20
0.22 0.24 0.12 6.91 0.36
4-core
FastText GloVe Word2Vec BERT QTyBERT
141.98 98.13 56.55 4210.95 297.78
0.30 0.21 0.12 8.93 0.63
136.14 126.74 66.30 21779.26 897.74
0.10 0.09 0.05 15.59 0.64
128.08 120.76 58.98 7613.55 303.14
0.26 0.24 0.12 15.25 0.61
A Comparative Study of Semantic Log Representations for Software Log-based Anomaly Detection
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Table 5: Detection latency (in seconds) of DL models using log embeddings from different semantic log representation methods 8-core
DL Model
BGL
TransEnc BiLSTM+WgtAttn LSTM GRU CNN RNN
Static Avg (Δmax ) 22.63 (1.20) 3.94 (1.18) 2.01 (0.43) 2.65 (1.13) 1.37 (0.42) 0.74 (0.15)
BERT 25.61 4.66 2.23 4.01 1.65 0.80
QTyBERT 24.06 4.40 2.32 3.35 1.39 0.79
Static Avg (Δmax ) 47.77 (3.77) 5.61 (0.94) 3.05 (0.40) 3.52 (0.98) 1.47 (0.42) 0.77 (0.04)
BERT 54.21 6.99 3.25 4.85 1.77 0.82
QTyBERT 51.42 5.87 3.75 4.44 1.47 0.81
TB
TransEnc BiLSTM+WgtAttn LSTM GRU CNN RNN
56.63 (2.76) 4.22 (0.47) 5.40 (0.11) 4.02 (0.14) 2.69 (0.27) 2.51 (0.05)
72.13 7.99 5.76 4.51 2.97 2.70
72.56 7.87 5.36 4.63 2.77 2.70
132.39 (7.40) 8.04 (0.14) 7.08 (0.19) 7.45 (0.54) 2.81 (0.30) 2.60 (0.12)
161.32 15.10 7.72 10.25 3.00 2.84
160.30 14.03 7.25 9.33 2.80 2.78
Spirit
TransEnc BiLSTM+WgtAttn LSTM GRU 1D-CNN RNN
20.89 (3.21) 4.12 (0.43) 1.95 (0.17) 2.86 (0.18) 1.28 (0.08) 1.01 (0.01)
25.00 5.86 2.13 3.27 1.50 1.09
25.21 5.86 2.11 3.30 1.53 1.05
45.69 (3.43) 6.02 (0.13) 2.95 (0.23) 3.86 (0.37) 1.95 (0.10) 1.04 (0.08)
55.13 7.26 3.10 4.11 2.10 1.08
55.86 7.96 3.00 4.17 2.08 1.06
event compared to 8.93–15.59 ms per log event for BERT. These correspond to more than a 94% reduction in embedding generation time on both CPU settings. For example, on TB with over 1.39 million log events, the total log embedding generation time is reduced from more than 10,300 seconds (≈2.9 hours) with BERT to about 500 seconds (≈8 minutes) with QTyBERT on 8 CPU cores, and from over 21,700 seconds (≈6 hours) with BERT to under 900 seconds (≈15 minutes) with QTyBERT on 4 CPU cores. Compared to static embedding methods (FastText, GloVe, and Word2Vec), QTyBERT is still slower, but also achieves sub-millisecond latency per log event across all systems and CPU configurations. Detection latency. As shown in Table 5, the detection latency is highly consistent when using log embeddings from QTyBERT and BERT, with differences of less than 5% in most cases across DL models and systems. This is expected, as QTyBERT preserves the same embedding dimensionality as BERT (Section 4.1.2), leading to similar processing times for downstream DL models. 5.1.3 Trade-Off. Figure 3 plots, for each system and each representation method, the average F1-score across DL models against the average embedding generation time per log event under both CPU settings. Here, “Static” denotes the average results of static word embedding methods (Word2Vec, GloVe, and FastText). As Figure 3 shows, QTyBERT achieves a better trade-off between detection effectiveness and log embedding generation efficiency compared to static word embedding and BERT methods.
5.2
4-core
System
Training Costs
As shown in Table 6, the training cost of QTyBERT consists of two components. First, obtaining SysBE for each target system requires only about 0.05 seconds under both CPU settings. Second, CroSysEh is trained once for all systems. As CroSysEh is lightweight,
optimizing its parameters takes only about 7 seconds. The overall training cost of CroSysEh is approximately 289 seconds (≈ 4.8 minutes), dominated by log embedding generation for Dcro using BERT and TinyBERT. Importantly, this cost is incurred only once. During deployment, QTyBERT reduces embedding generation time by approximately 94% compared to BERT, while maintaining comparable anomaly detection effectiveness for downstream DL models. In production environments where logs are continuously generated at a large scale, the resulting recurring savings in embedding generation will quickly outweigh this one-time training cost. Table 6: Training cost of QTyBERT.
5.3
Component SysBE
Setting BGL, 8/4-core CPU TB, 8/4-core CPU Spirit, 8/4-core CPU
BERT TinyBERT CroSysEh
GPU GPU 80 epochs, GPU
Time (s) 0.05 / 0.47 0.05 / 0.47 0.05 / 0.47 218.41 63.85 ∼7.16 (0.09/epoch)
Ablation study
We conduct an ablation study using RNN as the downstream DL model, as it has the lowest detection latency (Table 5) and exhibits the highest effectiveness gains with QTyBERT-based log embeddings among all DL models (Table 2). Table 7 and Table 8 report the ablation results on detection effectiveness and log embedding generation efficiency, respectively. CroSysEh. Removing CroSysEh (w/o CroSysEh) leads to F1score drops on all systems: BGL (-9.73%), TB (-2.89%), and Spirit (-4.72%). This confirms that CroSysEh improves the effectiveness of
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Wang et al.
log embeddings generated by SysBE for anomaly detection. Meanwhile, CroSysEh adds less than 0.6% to the total embedding generation time across all systems and CPU settings, meaning the effectiveness gains come with only a marginal increase in computational cost. Replacing cross-system training with a single-system variant (w/ sig.CroSysEh) in CroSysEh yields slightly higher F1scores (0.29-2.1%) on all systems, suggesting that single-system training can better fit system-specific patterns. Cross-system training learns from logs of multiple systems, trading a small amount of dataset-specific performance for a shared CroSysEh reusable across systems without retraining. This is more practical for organizations operating multiple systems. SysBE. Removing SysBE (w/o SysBE) causes only minor changes in F1-scores but significantly increases embedding generation time by around 3%-20% across all systems, indicating that SysBE’s quantization substantially improves efficiency while having little impact on downstream anomaly detection effectiveness. However, removing the calibration step (w/o calibration) in SysBE causes dramatic drops in F1-score across all systems: BGL (−19.19%), TB (−4.68%), and Spirit (−14.88%). This confirms that system-specific calibration is essential during quantization for preserving the embedding quality. As shown in Table 9, F1-scores drop notably when fewer than 70 calibration samples are used, indicating that insufficient calibration samples fail to adequately cover the target system’s activation distribution, which in turn degrades quantization quality and detection effectviness. Table 7: Ablation study on effectiveness (F1-score, %). Method
BGL (Δ)
TB (Δ)
Spirit (Δ)
QTyBERT w/o CroSysEh w/ sig.CroSysEh w/o SysBE w/o calibration
89.89 80.16 (-9.73) 90.18 (+0.29) 90.59 (+0.70) 70.70 (-19.19)
95.97 93.08 (-2.89) 96.24 (+0.27) 96.09 (+0.12) 91.29 (-4.68)
95.14 91.20 (-4.72) 97.24 (+2.10) 95.41 (+0.27) 80.26 (-14.88)
Table 8: Ablation study on log embedding generation efficiency (in seconds). Log Embedding Generation Time (s) CPU
QTyBERT
w/o CroSysEh (Δ)
w/o SysBE (Δ)
BGL
8-core 4-core
167.66 297.78
166.63 (-1.03) 296.01 (-1.77)
187.79 (+20.13) 356.56 (+58.78)
TB
8-core 4-core
504.22 897.74
501.24 (-2.98) 892.76 (-4.98)
557.68 (+53.46) 928.21 (+30.47)
Spirit
8-core 4-core
178.20 303.14
177.18 (-1.02) 301.29 (-1.85)
186.78 (+8.58) 320.98 (+17.84)
6
Threats to Validity
A threat to construct validity is that some DL models were originally designed for session-level anomaly detection. By studying prior
Table 9: Effect of calibration sample size (F1-score, %) N of log events
BGL
TB
Spirit
30 50 70 (ours) 100
72.41 70.69 89.89 89.99
91.84 90.26 95.97 94.98
72.89 87.29 95.14 93.50
settings [11, 16, 39], we find that both session-level and event-level detection operate on windowed log sequences and differ only in prediction granularity. Therefore, applying these DL models to our setting primarily requires adapting the prediction target. One threat to internal validity concerns the construction of the calibration dataset. In our experiments, we randomly sample 70 unlabeled log events from each target system, which yields high effectiveness across all three systems. However, prior work has shown that random calibration data selection may introduce performance instability due to activation distribution mismatch [43], and more principled selection strategies may further improve calibration quality. We mitigate this threat by using system-specific log events for calibration, ensuring the calibration data reflects the actual activation distribution of the target system. A potential threat to external validity lies in our evaluation. Our experiments were conducted on publicly available datasets of largescale supercomputing systems. While these real-world datasets are widely used in prior work to ensure fair comparison, production environments of different software systems may introduce additional diversity and complexity, due to the heterogeneous nature of software systems and varied logging practices. Expanding the evaluation to more software systems and incorporating feedback from practitioners would provide complementary insights.
7
Conclusion
This paper makes contributions to semantic log representation for DL-based log event-level anomaly detection. First, we conduct a comprehensive empirical study benchmarking widely used semantic log representation methods across a broad set of DL models under CPU-only deployment settings using publicly available log datasets. We identify a clear trade-off between static word embedding methods and the BERT-based contextual embedding method in detection effectiveness and log embedding generation efficiency. Second, motivated by this finding, we propose QTyBERT, a novel semantic log representation method that better balances this tradeoff. Future work will aim to improve the generalizability and interpretability of QTyBERT. We are seeking opportunities to extend its evaluation using log datasets from our local supercomputing center, which will allow us to study its performance under more diverse operational conditions. We also plan to collaborate with practitioners to assess its practical usage in real-world practices. Their feedback will guide subsequent enhancements to improve the usability.
A Comparative Study of Semantic Log Representations for Software Log-based Anomaly Detection
8
Data Availability
The datasets used in this paper are publicly available and can be accessed from their original sources as cited in the paper. Upon acceptance, we will make this package publicly available.
9
Acknowledgment
This work is funded by the EuroHPC Joint Undertaking and its members, including top-up funding by the Ministry of Education and Culture. The work is supported by the Research Council of Finland (grant id: 359861, the MuFAno project). The authors acknowledge CSC-IT Center for Science, Finland, for providing computational resources.
References [1] Vitor Cerqueira, Luís Torgo, and Igor Mozetič. 2020. Evaluating time series forecasting models: an empirical study on performance estimation methods. Machine Learning 109, 11 (Nov. 2020), 1997–2028. doi:10.1007/s10994-020-05910-7 [2] Jining Chen, Weitu Chong, Siyu Yu, Zhun Xu, Chaohong Tan, and Ningjiang Chen. 2022. TCN-based Lightweight Log Anomaly Detection in Cloud-edge Collaborative Environment. In 2022 Tenth International Conference on Advanced Cloud and Big Data (CBD). 13–18. doi:10.1109/CBD58033.2022.00012 [3] Rui Chen, Shenglin Zhang, Dongwen Li, Yuzhe Zhang, Fangrui Guo, Weibin Meng, Dan Pei, Yuzhi Zhang, Xu Chen, and Yuqing Liu. 2020. LogTransfer: CrossSystem Log Anomaly Detection for Software Systems with Transfer Learning . In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE Computer Society, Los Alamitos, CA, USA, 37–47. doi:10.1109/ ISSRE5003.2020.00013 [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. doi:10.18653/ v1/N19-1423 [5] Ying Fu, Meng Yan, Zhou Xu, Xin Xia, Xiaohong Zhang, and Dan Yang. 2022. An empirical study of the impact of log parsers on the performance of log-based anomaly detection. Empirical Software Engineering 28, 1 (Nov. 2022), 39 pages. doi:10.1007/s10664-022-10214-6 [6] Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett. 2021. Compressing Large-Scale Transformer-Based Models: A Case Study on BERT. Transactions of the Association for Computational Linguistics 9 (2021), 1061–1080. doi:10.1162/ tacl_a_00413 [7] Google Research. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. https://github.com/google-research/bert. Accessed: 2024-03-14. [8] Shayan Hashemi and Mika Mäntylä. 2024. Onelog: towards end-to-end software log anomaly detection. Automated Software Engineering 31, 2 (2024), 37. [9] Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu. 2017. Drain: An Online Log Parsing Approach with Fixed Depth Tree. In 2017 IEEE International Conference on Web Services (ICWS). 33–40. doi:10.1109/ICWS.2017.13 [10] Steven C. Hespeler, Pablo Moriano, Mingyan Li, and Samuel C. Hollifield. 2025. Temporal cross-validation impacts multivariate time series subsequence anomaly detection evaluation. arXiv:2506.12183 [stat.ML] https://arxiv.org/abs/2506.12183 [11] Adha Hrusto, Nauman Bin Ali, Emelie Engström, and Yuqing Wang. 2025. Monitoring data for Anomaly Detection in Cloud-Based Systems: A Systematic Mapping Study. ACM Transactions on Software Engineering and Methodology (June 2025). doi:10.1145/3744556 Just Accepted. [12] Shaohan Huang, Yi Liu, Carol Fung, Rong He, Yining Zhao, Hailong Yang, and Zhongzhi Luan. 2020. HitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log. IEEE Trans. on Netw. and Serv. Manag. 17, 4 (Dec. 2020), 2064–2076. doi:10.1109/TNSM.2020.3034647 [13] Peng Jia, Shaofeng Cai, Beng Chin Ooi, Pinghui Wang, and Yiyuan Xiong. 2023. Robust and Transferable Log-based Anomaly Detection. Proc. ACM Manag. Data 1, 1, Article 64 (May 2023), 26 pages. doi:10.1145/3588918 [14] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for Natural Language Understanding. arXiv:1909.10351 [cs.CL] https://arxiv.org/abs/1909.10351 [15] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomás Mikolov. 2016. FastText.zip: Compressing text classification models. CoRR abs/1612.03651 (2016). arXiv:1612.03651 http://arxiv.org/abs/1612.03651
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
[16] Van-Hoang Le and Hongyu Zhang. 2021. Log-based anomaly detection without log parsing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 492–504. doi:10.1109/ASE51524.2021.9678773 [17] Van-Hoang Le and Hongyu Zhang. 2022. Log-based anomaly detection with deep learning: how far are we?. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 1356–1367. doi:10.1145/3510003. 3510155 [18] Yukyung Lee, Jina Kim, and Pilsung Kang. 2023. LAnoBERT: System log anomaly detection based on BERT masked language model. Applied Soft Computing 146 (2023), 110689. doi:10.1016/j.asoc.2023.110689 [19] Xiaoyun Li, Pengfei Chen, Linxiao Jing, Zilong He, and Guangba Yu. 2023. SwissLog: Robust Anomaly Detection and Localization for Interleaved Unstructured Logs. IEEE Transactions on Dependable and Secure Computing 20, 4 (2023), 2762– 2780. doi:10.1109/TDSC.2022.3162857 [20] Siyang Lu, Xiang Wei, Yandong Li, and Liqiang Wang. 2018. Detecting Anomaly in Big Data System Logs Using Convolutional Neural Network. In 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress(DASC/PiCom/DataCom/CyberSciTech). 151–158. doi:10.1109/DASC/PiCom/DataCom/CyberSciTec.2018.00037 [21] Chuangying Meng and Ningjiang Chen. 2024. TinyLog: Log Anomaly Detection with Lightweight Temporal Convolutional Network for Edge Device. In 2024 International Joint Conference on Neural Networks (IJCNN). 1–8. doi:10.1109/ IJCNN60899.2024.10651312 [22] Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, and Rong Zhou. 2019. Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (Macao, China) (IJCAI’19). AAAI Press, 4739–4745. [23] Mika V. Mäntylä, Yuqing Wang, and Jesse Nyyssölä. 2024. LogLead - Fast and Integrated Log Loader, Enhancer, and Anomaly Detector. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 395–399. doi:10.1109/SANER60148.2024.00046 [24] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or Down? Adaptive Rounding for Post-Training Quantization. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 7197–7206. https://proceedings.mlr.press/v119/nagel20a.html [25] Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016. Integrating Distributional Lexical Contrast into Word Embeddings for Antonym-Synonym Distinction. arXiv:1605.07766 [cs.CL] https://arxiv.org/abs/1605.07766 [26] Adam Oliner and Jon Stearley. 2007. What Supercomputers Say: A Study of Five System Logs. In 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07). 575–584. doi:10.1109/DSN.2007.103 [27] ONNX Project. 2025. ONNX: Open Neural Network Exchange — Introduction. https://onnx.ai/onnx/intro/. Accessed: 2025-09-11. [28] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 1532–1543. doi:10.3115/v1/D14-1162 [29] Riley Peronto. 2024. The State of Log Data: 6 Trends Impacting Observability and Security. Blog post, Chronosphere. https://chronosphere.io/learn/observabilitylog-data-trends/ [30] Emad Ul Haq Qazi, Abdulrazaq Almorjan, and Tanveer Zia. 2022. A OneDimensional Convolutional Neural Network (1D-CNN) Based Deep Learning System for Network Intrusion Detection. Applied Sciences 12, 16 (2022). doi:10.3390/app12167986 [31] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 [cs.CL] https://arxiv.org/abs/1910.01108 [32] Issam Sedki, Abdelwahab Hamou-Lhadj, Otmane Ait-Mohamed, and Naser EzzatiJivan. 2023. Towards a Classification of Log Parsing Errors. In 2023 IEEE/ACM 31st International Conference on Program Comprehension (ICPC). 84–88. doi:10. 1109/ICPC58990.2023.00023 [33] Hudan Studiawan, Ferdous Sohel, and Christian Payne. 2021. Anomaly Detection in Operating System Logs with Deep Learning-Based Sentiment Analysis. IEEE Transactions on Dependable and Secure Computing 18, 5 (2021), 2136–2148. doi:10. 1109/TDSC.2020.3037903 [34] Lei Sun and Xiaolong Xu. 2023. LogPal: A Generic Anomaly Detection Scheme of Heterogeneous Logs for Network Systems. Security and Communication Networks 2023, 1 (2023), 2803139. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1155/2023/2803139 doi:10.1155/2023/2803139 [35] Marek Suppa, Katarína Benešová, and Andrej Švec. 2021. Cost-effective Deployment of BERT Models in Serverless Environment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, Young-bum Kim,
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Yunyao Li, and Owen Rambow (Eds.). Association for Computational Linguistics, Online, 187–195. doi:10.18653/v1/2021.naacl-industry.24 [36] USENIX Association. [n. d.]. The Computer Failure Data Repository (CFDR). https://www.usenix.org/cfdr. Accessed: 2025-09-08. [37] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9, Nov (2008), 2579–2605. http: //www.jmlr.org/papers/v9/vandermaaten08a.html [38] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of PreTrained Transformers. arXiv:2002.10957 [cs.CL] https://arxiv.org/abs/2002.10957 [39] Yuqing Wang, Mika V. Mäntylä, Jesse Nyyssölä, Ke Ping, and Liqiang Wang. 2025. Cross-System Software Log-based Anomaly Detection Using Meta-Learning. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 454–464. doi:10.1109/SANER64311.2025.00049 [40] Zumin Wang, Jiyu Tian, Hui Fang, Liming Chen, and Jing Qin. 2022. LightLog: A lightweight temporal convolutional network for log anomaly detection on the edge. Computer Networks 203 (2022), 108616. doi:10.1016/j.comnet.2021.108616 [41] Xingfang Wu, Heng Li, and Foutse Khomh. 2023. On the effectiveness of log representation for log-based anomaly detection. Empirical Softw. Engg. 28, 6 (Oct.
Wang et al.
2023), 39 pages. doi:10.1007/s10664-023-10364-1 [42] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016). https: //arxiv.org/abs/1609.08144 [43] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. Ternarybert: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009.12812 (2020). [44] Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, Junjie Chen, Xiaoting He, Randolph Yao, Jian-Guang Lou, Murali Chintalapati, Furao Shen, and Dongmei Zhang. 2019. Robust log-based anomaly detection on unstable log data. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 807–817. doi:10.1145/3338906.3338931
Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009