Fast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical Study

Fast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical Study Jesse Nyyssöläa,∗, Mika Mäntyläa , Yuqing Wanga

arXiv:2611.07760v1 [cs.SE] 5 Nov 2026

a Department of Computer Science, University of Helsinki, Helsinki, Finland

Abstract Context: Anomaly detection on software logs ensures system reliability through failure identification, root cause analysis and fault localization. While supervised models show excellent performance, in a practical setting there rarely are labeled training data available. This calls for the use of unsupervised models. Objectives: We study effects of data preprocessing, anomaly detection method, and the dataset. For preprocessing, we examine three options: The use of anomaly-free training data, using randomized as opposed to chronologically selected training data, and the use of a custom loader to extract log message from the log line. Regarding anomaly detection we use four models (KMeans, IsolationForest, RarityModel, Out-Of-Vocabulary Detector) and three representations (Words, trigrams, and parsed events). We evaluate these approaches on seven datasets: BGL, HDFS, Hadoop, Spirit, Thunderbird, Liberty, and LO2Token. Methods: We run each combination of the selected factors and measure performance with AUCROC and F1-score metrics. Then we utilize a generalized linear model to establish relationships between the factors. Furthermore, we address execution times. Results: Our evidence reveals large variability across all datasets, preventing us from making recommendations that apply universally. For instance, log parsing enhances performance on some datasets but reduces it on others. The evidence suggests that using anomaly-free data for training unsupervised models can improve anomaly detection performance, though this requires extra effort to identify such data. Previous work has argued that shuffling data leads to unrealistic model performance (data leakage). Yet, our results show this does not hold for the unsupervised models we tested, while it remains true for supervised models in the same setup. Regarding execution time, the results are clear. For fast execution of the pipeline, use word representation and avoid IsolationForest model. Conclusion: We encourage practitioners to be aware of dataset-specific differences, and study what has worked for similar datasets in the past. Keywords: Anomaly Detection, Log Analysis, Machine Learning, Unsupervised 1. Introduction Software log anomaly detection has two contrasting approaches. First is based on deep-learning, which is emphasized on research publications [1], [2], while the second is based on simple rule-based detectors and often used in industry [3]. Computationally intensive deep-learning approaches, while gaining performance improvements, are practically ∗ Corresponding author

Email addresses: jesse.nyyssola@helsinki.fi (Jesse Nyyssölä), mika.mantyla@helsinki.fi (Mika Mäntylä), yuqing.wang@helsinki.fi (Yuqing Wang)

harder to utilize in practice due to their complexity, heavy computational cost, and reliance on preprocessing [4], [5], [6], [7]. On the other hand, simple rule-based detectors, such as sequence length, have been shown to reach good detection performance on commonly used HDFS dataset in anomaly detection research [8]. In industrial settings, there are often large amounts of data, but little time, which makes simple detectors much more sought after. Additionally, labeled data is rare because the process is labor intensive, so it is better if our detectors can utilize the data in its rawest form. This study addresses

several factors when one has to operate with label free datasets. Conducting anomaly detection on software logs contains various options that a practitioner can choose from. These options include everything from data selection to model utilization. Studies on the topic generally select certain settings and compare only one selected factor, e.g., model selection [1]. This makes it difficult to compare results across studies as the other settings of the studies are different. In this study we, aim to conduct empirical evaluation across various factors to examine the impact that individual factors might introduce. As such, this study is also heavily motivated by challenges in practical settings where the practitioner is presented with various options and does not know what to choose. This study aims to provide insights and guidelines on which approaches work well given the characteristics of the data. Past work in unsupervised log anomaly detection is limited as we only found a handful of papers that give due consideration to preprocessing and data selection. For example, a recent study that aimed to provide a comprehensive view on techniques for log-based anomaly detection admits that the lack of preprocessing optimization can affect detection accuracy and generalizability of results [9]. A second major gap in existing work is on the data representations, that is, in what format it is given to the model. Past work has shown that traditional machine learning (ML) models work better with classical log representation techniques, while deep learning models work better with semantic-based representations [5]. This study will focus on lightweight representations on lightweight models as we are optimizing for speed. A more thorough summary of previous work is given in Section 2. This study extends our previous paper [10]. The previous paper had 4 datasets; here we introduce 3 new ones. We also include two new preprocessing approaches which results in four times more experiments than previously. In total we have 672 experimental setups which are the result of iterating across six factors (dataset, model, representation, and three preprosessing options). Furthermore, this extension of experimental setups enables us to use generalized linear models that make the evidence in this paper far more robust than what we presented previously. These extensions makes this paper the most comprehensive study regarding anomaly detection on software logs with fast unsupervised methods. We formulate the following

research questions. • RQ1: What effect do the model, representation and preprocessing options have on anomaly detection performance on software logs? • RQ2: To what extent is the variability explained by the dataset? • RQ3: How do the computational costs of different models, representations, and preprocessing approaches compare? The distinction between these two research questions is that in RQ1, we address factors that the practitioner might be able to affect, e.g., they could change the model. However, data is what they want to analyze in the first place. Therefore, our datasetspecific analysis in RQ2 will focus on examining the variability and generalizability of the results between datasets in relation to the selection in RQ1. RQ3 aims to address the practical reality of the execution times throughout the anomaly detection pipeline while considering the different factors in this study. This study will proceed by providing an overview of the related work regarding unsupervised anomaly detection on software logs in Section 2. In the Research Methods section (Section 3) we describe the preprocessing, datasets, metrics and models in more detail as well as give an overview of the statistical approaches used in the study. The Results section (Section 4) presents the results by starting from a generic overview and proceeding to more specific findings. The findings along with their implications and key contributions are discussed in the Discussion section (Section 5). Finally, we conclude the study in Section 6. 2. Background In this section, we first introduce an overview taxonomy of prior works in Section 2.1. Then we explore the most notable prior works in more detail in Section 2.2 2.1. Taxonomy overview To position our work, we present a taxonomy of relevant previous work in Table 1. The table is categorized by the factors in our empirical evaluation. For a study to be included in the taxonomy, it must 2

be a primary study that evaluates anomaly detection for software logs. Furthermore, it either needs to include an unsupervised model in the analysis [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], or it needs to do a comprehensive analysis on some of the other studied factors (representations, anomaly free training data, shuffled training data, custom message vs. full row) as in [5], [8], [9], [22]. The total list of representative works as noted by their primary tool or author is as follows: Loglizer [11], LogAnomaly [12], DeepLog [17], AutoLog [13], NeuralLog [14], Wu et al. [5], Le and Chang [15], Landauer et al. [8], Ali et al. [9], ATT-GRU [18], TPLogAD [16], PLELog [19], LogFiT [20], and MLAD [21]. The taxonomy illustrates which datasets, methods, representations and preprocessing approaches used in this study have been studied in the past. We can see which approaches have been popular and which have received less attention. For instance, almost all of the studies use the BGL and HDFS datasets. Similarly, almost all studies use a log parser to extract events from the log data. The Dataset category in the taxonomy contains the datasets used in this study. Some datasets used by other studies are not included here because they use preprocessed IDs which makes using word and trigram representations impossible, e.g., the ADFA dataset [24]. The Liberty dataset is used in anomaly detection research, and is a fitting dataset for us too, but it is not included in the representative works. LO2Token is a new dataset for this study that is extracted from a recently published dataset [25]. The Method category contains the models used in the prior works as categorized under broader terms. For instance, this study uses IsolationForest and KMeans models which would go under Tree-based and Cluster-based models respectively. Statistical methods is a broad category that includes both simpler approaches like analyzing frequency distributions and more complex techniques such as PCA. The key distinction is that the statistical models aim to summarize data through closed-form calculations while ML-based processes typically utilize a cost function through an iterative process. Note that for the method section, we included a study as a representative work if it included the appropriate method either as a part of the evaluation or as a novel approach. The Representation category consists of words, trigrams and parsed event categories. A widely

used technique for log representation is to use a clustering algorithm, also known as log parsing, to find out the static part of a log message called the template and use that (or its ID) to represent the logline [1]. Drain [26] is one such parser commonly used in software log anomaly detection studies. However, recent studies have demonstrated that there necessarily is not a strong correlation between parsing and anomaly detection accuracy [27], and some studies have begun to implement parserless approaches to achieve competitive results, especially when accounting for performance [28], [10], [14]. Typically parserless representations split the log message by whitespace and apply some simple regex or other normalization. The use of large language models (LLMs) for parsing has also been studied as part of semi-supervised anomaly detection pipeline in the SemiRALD approach [29]. It achieved better parsing accuracy than common parsers with competitive execution time despite the complex nature of LLMs. However, LLMs are not currently scalable for large amounts of completely unsupervised data. It should be noted that comparing execution times between studies is difficult due to the vastly different environments. For instance, [29] reported that parsing HDFS with Drain takes over 100 minutes, while another recent study parsed 1 million lines (21%) in under two minutes [30] with the same parser and dataset. The Training data category examines two options for training data. In short, the first considers whether the training data includes anomalies or are they filtered out. If there are anomalies in the training data, an unsupervised model has no way of distinguishing faulty behavior from simply rare behavior. It is still worth studying both options, as some datasets might show good results when effectively equating anomalousness with rarity [10]. The second option for training data considers the impact of using chronological versus random training data selection. Using completely random training data selection leads to some training data points being chronologically later than some test data points. This can be considered as a form of data leakage [15]. However, in a practical setting it can be possible to isolate the anomalous region chronologically, which is why we study the effect of it in this study. The final option examines the effect of using a custom loader for the dataset to only extract the message from the log line (i.e., discarding the metadata like timestamps). We found no studies that explicitly study this with regards to unsupervised 3

Table 1: Taxonomy of Prior Work by Studied Factors

Category

Dataset

Method Representation Training data Custom loader

Value BGL HDFS Hadoop Spirit Thunderbird Liberty LO2Token Tree-based ML Cluster-based ML Statistical Words Trigrams Parsed event Anomaly free Chronological vs. random Extract log message

Representative Works [10], [9], [5], [11], [12], [13], [14], [15], [8], [16], [17], [19], [23], [20], [21] [10], [9], [5], [11], [12], [14], [15], [8], [18], [16], [17], [19], [23], [20], [21] [10], [9], [13], [8] [5], [9], [14], [15], [8], [16] [10], [9], [5], [14], [15], [8], [16], [20], [21] [10], [9], [13] [10], [9], [12], [13], [19] [10], [9], [11], [12], [8], [18], [17], [19], [23] [10], [5], [14] [10] [10], [5], [11], [12], [13], [15], [8], [18], [16], [23] [10], [11] [15], [5] filtering out anomalies from the training data. Although the approach requires training data for only one class (i.e. normal), the models themselves work on purely unsupervised principles. This is also referred to as novelty detection [31]. The assumption in this setup is that the practitioner would have access to clean (i.e., anomaly free) logs which they would use as training data to gain better performance on unsupervised models [10]. Crucially, this paper will improve upon our previous one by including more datasets and studying more preprocessing approaches. One such approach involves whether or not the data is shuffled before being assigned to training and test sets. So far, there have been only a handful of studies assessing the effect of chronological vs. shuffled training data [8], and in many cases the selection is not reported in the study. The third preprocessing option is whether or not to use a custom loader to extract only the log message from the log line before performing any other preprocessing. As explained in the previous section, we found that the research related to using a custom loader for unsupervised anomaly detection is very limited. One of the most prevalent paper in the taxonomy [11] used a tool called Loglizer. When the paper was published in 2021, they reported that Loglizer1 is

anomaly detection, yet it has become the common practice to extract the log message from the log line. We expect that the metadata is irrelevant noise most of the time, but at the same time, if the models are able to handle it gracefully, the metadata can help identify genuine anomalies. Regarding the training data category and custom loader options, we included a study in the taxonomy if it studied as opposed to just utilized a given approach. For instance, if the study uses training data where the anomalies are filtered out, but does not study the effect of filtering, it is not included in the taxonomy. This is done because papers do not often report the preprocessing settings so the results would be arbitrarily determined by the authors decision to report something rather than what they actually did. This also explains why there are only a few studies in the representative works for the preprocessing approaches. 2.2. Representative works As this paper directly follows our work done in [10], it is naturally the most prevalent representative work in the taxonomy. It studied anomaly detection performance of four unsupervised anomaly detection models on five different datasets. It also studied the effect of data representation, which refers to the format in which the data is given to the model. In addition, it studied the impact of

1 https://github.com/logpai/loglizer

4

the only available toolkit for log anomaly analysis. The study that reported results with the Loglizer [11] toolkit used several methods, of which Invariant Mining achieved the best results with an F1 score of 0.91 on both HDFS and BGL datasets. The authors note the challenges of setting and testing the anomaly detection threshold in unsupervised models [11]. The unsupevised models used in that study were also much more time consuming. For example, 1.5 million lines on HDFS took more than 1000 seconds with Invariant Mining. Unlike most other papers, the study also considered the amount of anomalies in the training data by dynamically adjusting it [11]. This is noted as representing the "anomaly free training data" category in the taxonomy. LogAnomaly [12] is an approach that utilizes an unsupervised LSTM model that combines word and template vectors and manages to achieve F1 scores of 0.96 (BGL) and 0.95 (HDFS). The study provided comparison also to lighter unsupervised machine learning and statistical models. These include PCA and clustering models. While the method reached good results, other studies have shown that deep learning-based models are sensitive to hyperparameter tuning and computationally expensive [32]. AutoLog [13] is another paper that presents a new deep learning based approach. It focuses on training the model with anomaly free data and predicting anomalies based on reconstruction error. Their model uses a deep autoencoder and manages to reach F1-scores of 0.95 on BGL and 0.97 on HDFS. Additionally, they compare their model with unsupervised models such as the IsolationForest which reached worse scores of 0.70 on BGL and 0.61 on HDFS. As the authors utilize anomaly free training data, the experimental approach should be similar to the anomaly free training data setup used in this study. A study [5] conducted a thorough analysis on log representations used in anomaly detection. The authors applied several classical and semantic-based representations on traditional and deep-learning models. One of the key findings was that the traditional models performed better with classical representations, which is a major motivation for our work. The study did not include unsupervised methods but it was included in the taxonomy because it provides the most thorough analysis on log representations. Regarding their preprocessing philosophy they explicitly state only using randomly

split training data without consider the chronological order of events [5]. So far, there have been few studies on the impact of using chronologically sorted vs. randomly selected training data. A recent study that critically evaluated common datasets included it as a feature to study [15]. They found that on some scenarios there is a significant improvement in performance when using randomly selected data, as it provides a more diverse set of samples. However, model selection played an important role, as supervised models did not experience a similar performance boost on randomly selected training data. Some recent studies have shown increased performance over classic approaches like PCA and Invariant Mining by using attention, temporal neural networks, and richer log representations. NeuralLog [14] introduces an anomaly detection approach that works without parsing and as such is one of the few studies in the taxonomy that does not rely on parsed events. By doing this, it aims to solve the issue of Out-of-vocabulary (OOV) words and semantic misunderstandings. NeuralLog achieves great results on several datasets, but employs a BERT encoder and transformer-based classification [14], which are relatively time-consuming and resource intensive approaches. ATT-GRU [18] adds attention to a GRU model and beats DeepLog, PCA, and IM with higher accuracy and lower latency, but its complex design and limited implementation details make it hard to judge whether the extra complexity is worthwhile. It depends on labeled historical data, which is often unrealistic in mostly unsupervised log analysis settings. TPLogAD [16] combines BERT-based template and parameter embeddings with an attention-enhanced BiLSTM in a fully unsupervised framework, achieving strong results against several state-of-the-art baselines, but at the cost of a heavy, intricate architecture and unclear procedures that may not scale well to realtime, large production systems. Overall, these works show a consistent pattern. More complex neural models tend to perform better to a certain extent, but they introduce higher computational cost, complex tuning, and reduced transparency and reproducibility. This is despite the fact that the recent studies have highlighted the simplicity of studied public datasets [22] and that classic ML can reach competitive results [4]. The practical concerns regarding the benchmark studies raise questions whether the approaches can provide robust, and cost-effective solutions for large-scale, 5

sequence based datasets have multiple IDs to influence the prediction while event based ones only have one. For the word and trigram representations, the individual items used in the model just become much longer but they work effectively the same way. In general, logs have relatively homogeneous content which implies that a small percentage for model training training is enough. DeepLog demonstrated that for HDFS a less than 1% split for training is sufficient [17]. As was done in the previous paper [10], this study uses 5% of the data (after reduction) for training for every dataset except Hadoop which uses 50% due to a very small number of sequences. For the large event-based dataset, we only utilize a small percentage of data to be able to run them and have their size comparable to BGL. The values are on Table 2. For the sequence based datasets, we utilize the whole data except for LO2Token which selects a random error case along with the correct test. This results in a balanced dataset with 1727 normal and 1727 anomalous sequences, and effectively reduces the data usage to approximately 1.8% of the total data. This is indicated by the asterisk (*) in Table 2. LO2Token is the new dataset created for this study based on the recently published LO2 microservice dataset [25]. The LO2 dataset was created by running 53 negative test setups along with a positive test for complete 1740 runs. Because it is a microservice dataset, each seven services have their own log files. Normalizing anomaly detection scoring between different datatypes has proven to be difficult and requires purpose-built solutions [38]. Such multi-service analysis is not in the scope of this study, so for the LO2Token dataset we only selected the logs from the Token service. After removing runs with issues there was a total of 1727 runs where each have the 53 negative tests with errors and the normal positive test. This results in a total of 93,258 log files which are effectively treated as sequences in this study. For the LO2Token dataset correct sequences are reduced by a factor = 12.744 to make their size not significant as they were ran for a longer time. We chose Token as the included service because the initial dataset paper [25] showed that it appears to be the best service to predict the prevalence of an error with an overall average F1 score of 0.980 when using the supervised Decision Tree model. Initial tests quickly showed that unsupervised models do not reach such performance and the results are more

log anomaly detection in real-world environments. 3. Research methods This section describes the research methods as divided into three sections. In the first section, we explain the details regarding the experimental design. The design is split into subsections that match the structure of the taxonomy used to position our work in Section 2.1 as well as the experimental procedure. Next, we describe the metrics used in anomaly detection and the reasoning for selecting those metrics. Finally, we outline details about the technical implementation of the study. 3.1. Experimental design The four categories of the experimental design that consider the factors being studied are datasets, models, representations, and preprocessing options. This section describes each in detail. Additionally, we explain details about the experimental procedures that we will use to conduct the experiments. 3.1.1. Datasets The datasets used in this study are BGL [33], Thunderbird [33], HDFS [34], Hadoop [35], Spirit [33], Liberty [33], and LO2Token. BGL, HDFS, and Hadoop are accessible via LogHub [36]. Thunderbird, Spirit, and Liberty can be downloaded from the Usenix website 2 . LO2Token dataset is included in the replication package of this study in Zenodo [37]. Table 2 shows the summary of the datasets as split by event-based and sequence-based datasets. The distinction between event- and sequencebased datasets is important as they use a fundamentally different approach for anomaly detection. The event-based datasets have labels on each logline, and as such we can use the anomaly detection models to score each line and assess that accurately. The sequence-based datasets have labels on sequence level which means that normal lines within that sequence would also be labeled as anomalies in the event-based approach. Therefore, the sequencebased approach works by appending the contents of the whole sequence into a single prediction. The most significant difference with the sequence and event based datasets for anomaly detection is that when using event ID as the log representation, the 2 https://www.usenix.org/cfdr-data#hpc4

6

Table 2: Dataset Summaries

Log lines Normal lines (a) Event-based Datasets Anomalous lines Data used Train portion

BGL 4,747,963 4,399,503 348,460 100% 5%

Log lines Normal sequences (b) Sequence-based Datasets Anomalous sequences Data used Train portion in line with the other commonly used datasets in anomaly detection research.

Thunderbird 211,212,174 193,744,715 17,467,459 3% 5%

HDFS 11,175,629 558,223 16,838 100% 5%

Spirit 272,298,895 90,656,198 181,642,697 2% 5%

Hadoop 394,308 11 43 100% 50%

Liberty 265,569,119 168,478,341 97,090,778 2% 5%

LO2Token 191,850,132 1,727 91,531 1.8%* 5%

data points to the closest one. As data is added, the centroids are updated based on the mean of all the data points in that centroid. The trained model should have clearly defined clusters around the centroids. For prediction, we give the anomaly score as the distance to the nearest centroid. We started to design the Out-of-vocabulary Detector (OOVD) as a tool for aiding other models. However, when training the model with only normal data OOV terms are great candidates for being anomalies as the task of anomaly detection turns into novelty detection [31]. As such the OOVD by itself can produce good results when there is clean training data available. It is important to note that the OOVD can not work if there are anomalies in the training data vocabulary. Other models can work with the anomalous data because they determine the anomalousness of a term is by its relation to other terms, but OOVD only checks whether it belongs to a set. Therefore, if the set is created randomly the results will be random as well. Counting OOV terms in millions of lists individually would be a slow operation so we utilize a strategy that takes advantage of the way OOV terms are handled by vectorizers. When transforming the test data with CountVectorizer that was fitted with training data, the OOV terms will automatically get the value 0. Then we can simply subtract the number of in-vocabulary terms calculated by the CountVectorizer from the total number of terms to get the number of OOV terms. Working with total counts as such can be hundreds of times faster than matching the words individually in a for-loop. We implemented a simple class based on nega-

3.1.2. Models For this study we used the same model selection as the previous work [10] . During our initial testing, we evaluated far wider range of unsupervised models, but some popular unsupervised approaches—such as Local Outlier Factor and Oneclass SVM—proved computationally too slow for our needs and unlikely to therefore be used with large industrial data sets. The models include two established models in the Sklearn library (Isolation Forest and KMeans) and two simpler custom models we developed to have more performance efficient alternatives (Outof-vocabulary Detector and Rarity Model). All of the models work by producing a numeric value that correspond to how anomalous a given event or sequence is. As the models work identically with all three log representations (words, trigrams, event IDs), in this section we refer to them all collectively as "terms". One model implemented with Sklearn [39] is Isolation Forest (IF). It is based on random isolation of features and splitting between maximum and minimum values of selected feature [39]. Due to their nature, anomalies are easier to separate than normal data. Based on that, it can be determined that the number of partitions it takes to isolate a data point corresponds with how likely it is to be normal. The second model implemented with Sklearn is the KMeans clustering algorithm [39]. It works by starting with random centroids and assigning 7

tive logarithm that we called the Rarity Model (RM). In essence, negative logarithm measures the relative infrequency of individual terms in the data that increases based on their rarity. While utilizing negative logarithm in such a way is not novel, our aim was to create a very efficient baseline model. For implementation, the rarity scores are first precalculated for each term in the training data and assigned into a vector. Next, the dot product of the precalculated vector and the sparse matrix generated by the Tf-idf vectorizer from the test data is calculated. The total rarity score for the log message will be the dot product divided by the total number of terms in the message. Given its simple design and reliance on pre-built matrix multiplication routines, the RM functions with remarkable speed [10].

setup, anomalousness is instead based on the deviation from normal. The second option is Shuffle which shuffles the data before assigning training and test sets. If the data is not shuffled, the data will be in chronological order so that all training data is earlier than all test data. The Custom option refers to whether or not we use a custom loader for the data that retrieves only the message part from the log line. In contrast, a generic loader uses the whole row. For the eventbased datasets, we use a generic "Raw Loader" which simply retrieves each line as their own event. Although, if it contains labels, we remove it afterwards. For the sequence based datasets we can not use a completely generic loader as we need to create the sequences. In addition, multi-line entries (e.g., stacktraces) are combined for practical purposes. Within a logline, however, there is no custom logic. Table 4 shows a sample from the HDFS dataset as split into separate fields. When using a custom loader, only the log message shown in cursive is selected for analysis. The rationale for studying the impact of using a custom loader is that it takes effort to create it while the benefit remains unclear. On one hand, the general idea is that the anomalous behavior should appear in the log message, so that would be the only relevant field in the log line. Furthermore, log parsers typically only utilize the log message part. On the other hand, excluding the other fields removes information that might possibly be better indicators of the labeled anomalies than the log message itself. We will study all eight combinations of the three preprocessing options. Table 5 provides textual descriptions on each setup, but further in the paper we will refer to them with abbreviations. Specifically, F for Filtered (or Anomaly Free), S for Shuffle, and C for Custom Loader. For the eight combinations, we use a string identifier that denotes whether specific preprocessing option is on or off.

3.1.3. Representations In this study, creating representations means that the normalized logs are transformed into event IDs or lists consisting of trigrams or words. Words are extracted simply by using whitespace as the separator. Event IDs are created with the Drain parser [26]. By "trigrams", we refer to character trigrams, that is, three consecutive characters in the log message. For example, the first three trigrams of the message "Stopping..." would be "Sto", "top" and "opp". An example of these representations along with their raw and normalized log messages is shown in Table 3. 3.1.4. Preprocessing options In the study, we use three preprocessing options that can be set to either true or false. First one is the Filter option which, when set to true, will remove the anomalous entries from the training data. We acknowledge that in an industrial setting, it is not feasible to filter out anomalies. However, the industry can separate logs that are gathered during the system’s normal operation that should work as the training data. Hence, the anomaly free setup does require a certain level of curation from the log owner but at the a very high level. Using the anomaly data changes fundamentally what is being examined. When there are anomalies in the training data but no labels, it is impossible to identify which events are true anomalies and which are not. Then the anomaly detection essentially relies on the assumption that rarity equates anomalousness. In contrast, with the anomaly free

3.1.5. Experimental Procedure and Statistical Evaluation To conduct the study we will run all the experimental combinations five times similar to [11]. The combinations include 7 datasets, 4 models, 3 representations, 8 preprocessing option setups, and 5 repeats which result in a total of 3360 individual data points for both metrics. We will use statistical approaches to evaluate these results after con8

Table 3: Example message from the preprocessed BGL dataset

Raw log message ’4 ddr error(s) detected and corrected on rank 0, symbol 11 over 20609 seconds’

Normalized message ’0 ddr error(s) detected and corrected on rank 0, symbol 0 over 0 seconds’

Representations Trigrams

Words

[ ’0’, ’ddr’, ’error(s)’, ’detected’, ’and’, ’corrected’, ’on’, ’rank’, ’0’, ’symbol’, ’0’, ’over’, ’0’, ’seconds’]

[’0 d’, ’dd’, ’ddr’, ’dr ’, ’r e’, ’er ’, ’err’, ’rro’, ’ror’, ’or(’, ’(r(s’, ’(s)’, ’s) ’, ’) d’, ’ d ’, ’de’, ’det’, ’ete’, ’tec’, ’ect’, ...]

Event ID ’e10’

Table 4: Example of field extraction from HDFS logs using custom loader

Timestamp

Level

Component

081110 002221 4855 081110 002223 4824 081110 002253 34

INFO

dfs.DataNode$ PacketResponder dfs.DataNode$ PacketResponder dfs.FSNamesystem

081110 002337 13

INFO

INFO INFO

Message

PacketResponder 0 for block blk_3259872359515191972 terminating PacketResponder 1 for block blk_5963816136267066159 terminating BLOCK* NameSystem.allocateBlock: /user/root/sortrand/_temporary/part-00138. blk_-210021574616486609 dfs.DataBlockScanner Verification succeeded for blk_1547954353065580372 these experiments depend on the type of finding and are elaborated at the end of the Result section (4.5).

ducting all experiments. In particular, we utilize a generalized linear model (GLM). We will first fit the model by using the variables independently to get a general overview of the coefficients. After that we will apply the dataset as an interaction term on each different variable to be effectively be able to assess the impact of the dataset on that variable. E.g., if some anomaly detection model works well on some dataset. Due to the limitations of using F1-score (as explained in the next section, 3.2) and the scope of the paper, we will conduct the GLM analysis only using the AUCROC metric. After the GLM with interaction terms is fitted, we aim to do further analysis on the residuals which means the difference of the actual value and the value predicted by the GLM. The residuals can highlight scenarios where the model does not perform well, which in practice means that there are some results that are not explained by the individual variables or their interactions with the dataset. These could be, for example, interactions between the model and representation selection. Due to the exploratory nature of the study, we aim to conduct additional experiments after the regression analysis to gain further insights regarding the most important findings. The exact details of

3.2. Anomaly Detection Metrics In the study we will analyze model performance primarily with AUC-ROC, but we will provide an overview of the F1-scores too. AUC-ROC [40] is a combination of the abbreviations Area Under Curve (AUC) and Receiver Operating Characteristic (ROC). ROC curve is a plot of the true positive rate against the false positive rate across a range of thresholds. With AUC we refer to the area under this curve. This area goes from 0 to 1 and in practice it means the probability that a randomly selected positive sample is ranked higher than a negative one [40]. F1-score is the most common metric in anomaly detection that measures the harmonic mean of precision and recall. The main reason for emphasizing AUC-ROC in this study is that it is not dependent on a threshold. With supervised models you could assess the model specific anomaly scores in relation to the labels and adjust the threshold accordingly. In unsupervised models this is not possible which leads to subjective threshold selection. 9

Table 5: Comparison of different experimental setups and their effects on data processing

Setup name f0_s0_c0

F

f1_s0_c0

✓

f0_s1_c0

S

✓

f0_s0_c1

✓

f1_s1_c0

✓

f1_s0_c1

✓

f0_s1_c1 f1_s1_c1

C

✓

✓ ✓ ✓

✓

✓

✓

Description Uses full log rows. Data split chronologically into train/test. Anomalies retained in training data. Uses full log rows. Data split chronologically into train/test. Anomalies removed from training data. Uses full log rows. Data shuffled before train/test split. Anomalies retained in training data. Loader selects log message only. Data split chronologically into train/test. Anomalies retained in training data. Uses full log rows. Data shuffled before train/test split. Anomalies removed from training data. Loader selects log message only. Data split chronologically into train/test. Anomalies removed from training data. Loader selects log message only. Data shuffled before train/test split. Anomalies retained in training data. Loader selects log message only. Data shuffled before train/test split. Anomalies removed from training data.

As realistic threshold selection (i.e., without labels) for the F1-score is out of scope of this paper, we will utilize labels in a Bayesian optimization loop that finds the best threshold over a maximum of 20 iterations. We emphasize that this is not possible without labels, and as such, the F1-scores serve as an example of what could be possible.

process as in the previous study [10]. First, the data is loaded directly from raw log files to Polars [41] dataframes. Next, the logs are normalized to reduce the effect of parameters (non-static parts of the message). However, due to placing strong emphasis on performance time we only perform three operations with fast Polars-native string operations:

3.3. Implementation 3.3.1. Computing environment All the tests are run on a PC with Intel Core i713700K, which has 16 cores at 4.4 GHz for P-Cores and 3.4 GHz for E-Cores. Regarding the computational performance, all experimental setups are limited by the CPU only with some operations using only a single core, while others can be distributed to multiple cores. Some of the setups require a large amount of memory. Our PC has 128 GB of physical RAM and uses 128 GB of SSD storage as virtual memory. For running the experiments, we utilize a modified version of the LogLead3 tool that was designed to benchmark anomaly detection algorithms and log representations. The replication package is publicly available on Zenodo4 .

1. Convert all letters to lowercase, 2. Replace every numeral with the digit ’0’, 3. Reduce any consecutive series of zeros to a single ’0’. For example, the string "Time 12:34:56" would be converted to "time 0:0:0". When it comes to words and trigrams, normalization will make the resulting vocabulary smaller. Although the primary reason for this is to improve the accuracy due to minimizing the effect of parameters, a smaller vocabulary also makes the model faster. In the next step, representations are created as explained in the section 3.1.3. The representations are then vectorized so that they can be used with the machine learning models. For most models, we use the TfidfVectorizer [39], which is a common technique for feature extraction that adds weight to the most important tokens. However, for the Outof-vocabulary Detector we utilize absolute counts so we use the CountVectorizer. After preprocessing, the data is split into training and test sets as defined in section 3.1.1. Finally, actual training and prediction is performed as

3.3.2. The anomaly detection pipeline The anomaly detection pipeline starts from preprocessing steps that are common for all experimental setups. For these we follow exactly the same 3 https://github.com/EvoTestOps/LogLead 4 Ready but not published.

10

defined in the Sklearn library for KMeans and IF, while OOVD and RM use similar custom functions. In addition to binary prediction, all the models are also able to produce a score that corresponds to the perceived anomalousness. In fact, the models work by introducing a threshold on those scores. For the analysis of the results, the scores are crucial as it enables the AUC-ROC analysis.

plot across various setups, consider them in groups. The group of four on the left correspond to anomalous training data while the group of four on the right use anomaly free training data. Groups of two plots alternate between using and not using shuffled data for training. Each plot alternates regarding the use of a custom loader. The group of datapoints exactly at AUCROC = 0.5 for OOVD is expected to occur when filtering is off. This is due to the nature of OOVD as explained in the OOVD part of section 3.1.2. Furthermore, regarding filtering, when the ratio of anomalous lines to total lines in a dataset is high, we expect the results with anomalous training data to be significantly lower, as the unsupervised models cannot differentiate between common messages that are and are not labeled as anomalies. This raises the question that should they be called anomalies in the first place. Next we present the key findings for each dataset. BGL is an average-level dataset in terms of difficulty. The deviation across setups is relatively small. The setup f1_s1_c1 provides the best overall results but also the largest standard deviation with some outliers getting a score of 0. We investigated these outliers and found that they occur with the event ID representation, suggesting that there is an issue with the interaction of the loader and the parser. HDFS is the easiest dataset set where the rarity of the log content strongly corresponds to the labeled anomalousness. This can be determined from the fact that almost all models perform well even when there are anomalies in the training data. OOVD performs worse than the other models. On f1_s0_c1 the performance of IsolationForest and RarityModel deteriorate while OOVD improves. This suggests that the number of OOV terms increases and the other models are unable to handle them. It is a reasonable assumption as filtering, not shuffling, and using a custom loader all lead to a smaller vocabulary versus their counterparts. Hadoop is the smallest dataset which shows a large variation even on exactly the same setup. The best performance requires anomaly free data and a KMeans or IF model. LO2Token scores have the smallest standard deviation across all studied datasets. KMeans is generally the best model, while OOVD has bad performance regardless of other factors. We examined the data and found that the log lines that are characteristic to a sequence being labeled as an anomaly

4. Results This section begins by giving a high level overview of the results. This is followed by regression analysis for anomaly detection performance. Then, we present the results regarding execution time. Finally, we conduct a supplementary experiment to gain insight on the impact of shuffling training data by comparing the unsupervised models to a supervised one. 4.1. Overview To start, we provide an easy summary of each variable by showing the mean AUCROC and F1score in Table 6. To calculate the means and standard deviations, we include a single value within a category, but every value from other categories in the pool. The total number is denoted as n in the table. For example, the AUCROC mean of 0.658 for BGL includes the results of all models, representations, and Filter/Shuffle/Custom options, as long as the experiment used BGL as the dataset. The table suggests that, as expected, the most impactful individual category is the dataset because it shows the widest range means. For instance, the F1 score range for datasets is from 0.276 to 0.884 while the second largest range is in Filtering which improves the F1 score from 0.557 to 0.653. Because the standard deviations are so large and the means are relatively consistent, this table only gives a crude overview. In other words, there are good and bad situations for each parameter, but they cannot be distinguished from such a broad generalization. Figure 1 shows an overview of all the AUC-ROC results when separated by dataset. There are seven subplots, each representing a dataset, and each plot has eight setups regarding the preprocessing options. Furthermore, individual runs are overlayed on the boxplots with color codes corresponding to model selection. Representation selection can not be distinguished from the plot. To help reading the 11

Model KMeans

IsolationForest

RarityModel

OOV_detector

BGL 1.0

AUCROC

0.8 0.6 0.4 0.2 0.0 HDFS 1.0

AUCROC

0.8 0.6 0.4 0.2 0.0 Hadoop 1.0

AUCROC

0.8 0.6 0.4 0.2 0.0 LO2Token 1.0

AUCROC

0.8 0.6 0.4 0.2 0.0 Liberty 1.0

AUCROC

0.8 0.6 0.4 0.2 0.0 Spirit 1.0

AUCROC

0.8 0.6 0.4 0.2 0.0 Thunderbird 1.0

AUCROC

0.8 0.6 0.4 0.2 0.0

f0_s0_c1

f0_s0_c0

f0_s1_c1

f0_s1_c0

f1_s0_c1

FSC Combinations

Figure 1: Boxplots by dataset

12

f1_s0_c0

f1_s1_c1

f1_s1_c0

Table 6: Result Summary

Category

Dataset

Model

Representation Anomaly Free Shuffle Custom

Value

n

BGL Hadoop HDFS Liberty LO2Token Spirit Thunderbird IsolationForest KMeans RarityModel OOV Detector Words Trigrams Event ID 0 1 0 1 0 1

96 96 96 96 96 96 96 168 168 168 168 224 224 224 336 336 336 336 336 336

AUCROC Mean Std 0.658 0.169 0.645 0.179 0.845 0.182 0.571 0.290 0.616 0.168 0.466 0.363 0.538 0.288 0.573 0.273 0.649 0.304 0.598 0.281 0.662 0.194 0.656 0.272 0.656 0.248 0.549 0.272 0.538 0.247 0.702 0.264 0.625 0.255 0.617 0.282 0.623 0.274 0.618 0.263

F1 Score Mean Std 0.276 0.173 0.884 0.120 0.569 0.267 0.661 0.219 0.698 0.071 0.809 0.207 0.329 0.257 0.589 0.263 0.616 0.281 0.642 0.288 0.573 0.324 0.630 0.281 0.625 0.277 0.560 0.309 0.557 0.291 0.653 0.283 0.598 0.289 0.611 0.293 0.607 0.291 0.603 0.291

dent on specific details of the data rather than a generalizable result. Thunderbird is similar to the other supercomputer datasets (BGL, Liberty, Spirit, Thunderbird). In terms of difficulty and deviation, it could be characterized as being between BGL and Spirit. Unlike on Spirit, for Thunderbird IsolationForest performs well on exactly the opposite anomaly data.

also appear in the normal runs, but in much smaller numbers. Regardless, it means that those lines are not considered anomalous by OOVD. The results with the custom loader are worse, which implies that there is relevant information in the discarded part of the logline. Liberty is completely dependent on filtering, but even then, model selection matters. The low scores of IsolationForest where OOVD gets near perfect AUCROC suggest, again, that there are many OOV terms that the model is unable to handle. Interestingly, f1_s0_c1 has an uniquely low average score among other setups with filtering. In practice, this means that models require either information from the log line (not using custom loader) or in other parts of the dataset (shuffling) to get great performance, but if they get neither, they do not work well.

4.2. GLM analysis To evaluate the effects of each parameter and setting, we employed the generalized linear model (GLM). Linear models require a reference variable which affects the coefficients but not the model itself in terms of predictive capabilities. Based on previous results and ease of interpretation we chose the following reference variables: • Dataset: BGL

Spirit has results quite similar to Liberty. Across all datasets, Spirit has the worst results without filtering due to the high number of anomalous log lines. Essentially, because there are more events labeled as anomalies than normal, the models generally get a worse than random result when using anomalous training data. IsolationForest on f1_s0_c1 is an exception, but that is likely depen-

• Model: KMeans • Representation: Trigrams • Filter: False • Shuffle: False 13

Intercept Hadoop HDFS Liberty LO2Token

Variable

Spirit Thunderbird IsolationForest RarityModel OOV_detector Words Event_ID Filter Shuffle Custom

−0.2

0.0

0.2 Coefficient Value

0.4

0.6

Figure 2: GLM with independent coefficients

• Custom: False

dataset as the interaction term. It also shows the confidence interval as a line on top of the bar to illustrate whether the association is statistically significant. The model has the pseudo-R-squared value of 0.70 which shows notable improvement from the previous model. For more detailed statistics on the GLM with interaction terms, see Table 7. It includes the coefficients, standard error, zstatistic, and the p-value. The most interesting findings relate to the coefficients that clearly contradict the findings related to the independent coefficients, for instance, the independent result suggesting that parsing (EventID in the figure) degrades performance. With the interaction model, this effect appears to be datasetspecific. In the first block of Figure 3 , the use of parsed EventIDs in the Spirit, Thunderbird, and Thunderbird-in datasets results in a decline in performance on average. Conversely, in the LO2 token and HDFS datasets, utilizing parsing yields a performance improvement. Thus, parsing exhibits the most significant impact, both positively and negatively.

Figure 2 shows the coefficients with GLM using the variables independently. The Cox and Snell pseudo-R-squared for the model is 0.37 which means that the model explains 37 % of the variability. The purpose of GLM in our study is to give a high level overview of each variable when all the other variables are controlled. The independent variables suggest the following: HDFS is the easiest dataset, followed by BGL and Hadoop that are very close to each other. Sprit is the most difficult dataset. KMeans and OOVD are the best performing models while Isolation Forrest is the worst. Trigrams and Word representations beat the parsed Event IDs. Filtering anomalies from training data yields great benefits, whereas shuffling and using a custom loader does not have a significant impact. These findings are in line with the previous boxplots. The model with interaction terms gives a much more nuanced view of the data. Figure 3 shows the GLM coefficients when each other variable uses 14

Interaction Terms Only Dataset × Binary param Dataset × Model Dataset × Representation

Spirit * Event_ID Thunderbird * Event_ID Liberty * Event_ID Spirit * Words Hadoop * Words Liberty * Words Hadoop * Event_ID HDFS * Words Thunderbird * Words HDFS * Event_ID LO2Token * Words LO2Token * Event_ID HDFS * OOV_detector LO2Token * OOV_detector Hadoop * RarityModel LO2Token * RarityModel LO2Token * IsolationForest Liberty * IsolationForest Spirit * IsolationForest HDFS * RarityModel Hadoop * IsolationForest Liberty * RarityModel HDFS * IsolationForest Thunderbird * RarityModel Hadoop * OOV_detector Spirit * RarityModel Thunderbird * IsolationForest Liberty * OOV_detector Thunderbird * OOV_detector Spirit * OOV_detector HDFS * Filter Thunderbird * Shuffle Spirit * Shuffle Hadoop * Shuffle LO2Token * Shuffle LO2Token * Custom HDFS * Custom Liberty * Shuffle LO2Token * Filter Liberty * Custom HDFS * Shuffle Hadoop * Custom Spirit * Custom Thunderbird * Custom Hadoop * Filter Thunderbird * Filter Liberty * Filter Spirit * Filter

−0.4

−0.3

−0.2

−0.1

0.0

0.1

Coefficient Value Figure 3: Coefficients of GLM with interaction terms

15

0.2

0.3

0.4

Table 7: GLM Summary (with Interaction Terms)

Variable

Coef

Std Err

z

p-value

Interaction: Dataset × Model Hadoop × Isolation Forest HDFS × Isolation Forest Liberty × Isolation Forest LO2Token × Isolation Forest Spirit × Isolation Forest Thunderbird × Isolation Forest Hadoop × Rarity Model HDFS × Rarity Model Liberty × Rarity Model LO2Token × Rarity Model Spirit × Rarity Model Thunderbird × Rarity Model Hadoop × OOV Detector HDFS × OOV Detector Liberty × OOV Detector LO2Token × OOV Detector Spirit × OOV Detector Thunderbird × OOV Detector

-0.0182 -0.0018 -0.0692 -0.1361 -0.0635 0.0762 -0.2262 -0.0192 -0.0054 -0.2220 0.0544 0.0319 0.0390 -0.3302 0.1494 -0.2938 0.3694 0.3116

0.026 0.026 0.042 0.025 0.043 0.039 0.023 0.020 0.036 0.024 0.036 0.031 0.022 0.021 0.037 0.021 0.030 0.028

-0.693 -0.070 -1.650 -5.364 -1.487 1.934 -10.019 -0.946 -0.152 -9.380 1.497 1.016 1.788 -15.404 4.047 -13.913 12.167 10.943

0.489 0.944 0.099 < 0.001 0.137 0.053 < 0.001 0.344 0.879 < 0.001 0.134 0.309 0.074 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001

Interaction: Dataset × Representation Hadoop × Words HDFS × Words Liberty × Words LO2Token × Words Spirit × Words Thunderbird × Words Hadoop × Event ID HDFS × Event ID Liberty × Event ID LO2Token × Event ID Spirit × Event ID Thunderbird × Event ID

-0.0070 0.0300 -0.0069 0.0939 -0.0082 0.0348 -0.0044 0.0878 -0.0146 0.1701 -0.0969 -0.0947

0.017 0.018 0.032 0.018 0.030 0.029 0.022 0.022 0.032 0.021 0.033 0.029

-0.402 1.629 -0.219 5.279 -0.276 1.198 -0.201 4.050 -0.450 7.967 -2.966 -3.283

0.687 0.103 0.826 < 0.001 0.782 0.231 0.841 < 0.001 0.653 < 0.001 0.003 0.001

Interaction: Dataset × Preprocessing option Hadoop × Anomaly Free HDFS × Anomaly Free Liberty × Anomaly Free LO2Token × Anomaly Free Spirit × Anomaly Free Thunderbird × Anomaly Free Hadoop × Shuffled HDFS × Shuffled Liberty × Shuffled LO2Token × Shuffled Spirit × Shuffled Thunderbird × Shuffled Hadoop × Custom HDFS × Custom Liberty × Custom LO2Token × Custom Spirit × Custom Thunderbird × Custom

0.0635 -0.1065 0.1227 -0.0093 0.2861 0.0942 -0.0365 0.0089 -0.0121 -0.0313 -0.0502 -0.0730 0.0123 -0.0152 0.0076 -0.0253 0.0370 0.0485

0.017 0.017 0.026 0.017 0.026 0.024 0.017 0.017 0.026 0.017 0.026 0.024 0.017 0.017 0.026 0.017 0.026 0.024

3.780 -6.280 4.654 -0.545 10.942 3.881 -2.170 0.524 -0.457 -1.842 -1.921 -3.006 0.731 -0.897 0.287 -1.491 1.415 2.000

< 0.001 < 0.001 < 0.001 0.585 < 0.001 < 0.001 0.030 0.600 0.648 0.065 0.055 0.003 0.465 0.370 0.774 0.136 0.157 0.046

16

Looking at the anomaly detection model section of the figure, represented in blue, a similar pattern emerges with the OOVD detector. OOVD performs exceptionally well in the Liberty, Thunderbird, and Spirit datasets, delivering substantial improvements. The high coefficients of OOVD for the supercomputer datasets highlight how dependent the model is on finding new, different log lines in the test data for those datasets. However, it significantly degrades performance in the LO2 token and HDFS datasets. Thus, OOVD proves to be both the most effective and the least effective approach, depending on the dataset. Finally, examining the red section of the figure, data filtering to be anomaly free appears to enhance performance across several datasets specifically Hadoop, Thunderbird, Liberty, and Spirit—while it negatively impacts the HDFS dataset. Regarding the representation, the datasetspecific differences remain modest between Trigrams and Words. However, unlike with the independent variables, Event ID has a higher coefficient as an interaction term with HDFS and LO2Token datasets. It intuitively makes sense that the event ID would work better with sequence-based datasets, as its very purpose is to condense the log line into a single identifier. These results do not yet conclusively indicate that Event IDs should always be used with sequence-based datasets. Notably, the supercomputer datasets (Spirit, Thunderbird, and Liberty) exhibit consistent behavior, improving or degrading performance collectively. This suggests that these supercomputer datasets share similar properties which was to be expected due to their common source.

suggesting a systematic problem in the model. Because it trends downwards, it means that the low predictions were actually higher and the high predictions were actually lower. Although we could adjust the model by increasing interaction terms, for this study we did not do that because we want to keep the results interpretable. For practitioners, the residual analysis also suggests that the conclusions derived from the regression model are not necessarily generalizable to instances with a high number of outliers. For instance, the setup f1_s0_c1 has 0 positive outliers and 31 negative ones, which means that in 31 cases the model predicted a much better result than was actually the case. The prior analysis on the boxplots (Figure 1) suggests that these outliers originate from Liberty, where this particular setup did not work well. Overall, the variable with the most outliers was Isolation Forest with 44 positive outliers and 65 negative ones. For example, the positive outliers are very apparent in the Thunderbird boxplots as the orange dots. 4.4. Execution time To address the computational efficiency of different models and representations, we measured the execution time. The results are divided into three tables. Table 9 shows the model training and test times for each sequence-based dataset as paired with each representation and model. Table 10 shows the same information for event-based datasets. We use the term "Test time" rather than "Inference time", because we measure the time to do both, the anomaly scoring and the binary prediction. This, combined with the fact that our training data is only 5% of the total data (except for Hadoop which is 50%), results in test times being generally much larger than training times. The results show that OOVD, KMeans and RM are very fast in each scenario. The difference between these models is often under a second, and even with the largest datasets, under 3 seconds. With those models, the slowest combination is using KMeans on Thunderbird with trigram representation, but even that is only about 10 seconds in total. With IF, the slowest model in this study, the same combination takes over 3.5 minutes. Table 11 contains the preprocessing times per dataset. Each setup goes through the normalization step, but only one of the representations is selected. After that, there is the "Split and Vectorize" step which has an execution time heavily im-

4.3. GLM predictions analyzed Figure 4 plots the actual AUCROC values in the data with the values predicted by the GLM model, effectively illustrating the residuals of the predictions. When the residuals are plotted against the predicted AUCROC, we can determine the upper and lower thresholds for the outliers, which are defined as being two standard deviations from the mean. Outliers are further classified in Table 8 to show instances where the prediction of the model fails. Examining these outliers is important because it helps identify the variability in the data that the regression model itself cannot capture. In particular, in the Residuals vs. Predicted AUCROC plot in Figure 4 we can observe a diagonal trend 17

Predicted vs. Actual AUCROC

1.0

Residuals (Actual - Predicted)

Predicted AUCROC

1.0 0.8 0.6 0.4 0.2 0.0

Residuals vs. Predicted AUCROC

0.8 0.6 0.4 0.2 0.0 −0.2 −0.4 −0.6

0.0

0.2

0.4

0.6

0.8

1.0

0.0

Actual AUCROC BGL HDFS

Hadoop LO2Token

0.2

0.4

0.6

Liberty Spirit

Thunderbird Perfect Prediction

Setup

Model Representation

1.0

Outlier Thresholds (±2σ)

Figure 4: GLM predicted vs. actual, and residuals

Category Dataset

0.8

Predicted AUCROC

Value Liberty Thunderbird Spirit BGL HDFS f0_s0_c1 f1_s1_c1 f1_s0_c1 f0_s1_c0 f1_s0_c0 f1_s1_c0 f0_s1_c1 f0_s0_c0 IsolationForest KMeans OOV_detector Event_ID Words Trigrams

Positive Outliers (n=60) 20 21 14 5 0 31 5 0 13 0 0 5 6 44 6 10 22 15 23

Table 8: Outliers by category

18

Negative Outliers (n=100) 37 19 24 10 10 0 28 29 5 17 14 5 2 65 30 5 43 40 17

Total Outliers 57 40 38 15 10 31 33 29 18 17 14 10 8 109 36 15 65 55 40

pacted by the chosen representation. In the table, this is indicated by a letter: words (W), trigrams (T), and parsed event IDs (P). Overall, preprocessing is computationally demanding compared to the efficient anomaly detection models selected for this study. For example, if you were to select the fastest way to analyze BGL, the preprocessing would take around 47 seconds, while the model training and test would be under 4 seconds. Similar pattern applies to other datasets as well. The time to split and vectorize is dependent on the number of items in the data. Therefore, if we condense the logline into a single ID, the time is much lower than for the word representation. Conversely, if we split the line into much smaller trigrams, the time increases. However, parsing the data is much slower than splitting for words or trigrams. When combining both the operation to generate the representation, and to split and vectorize it, the results are clear. Word representation is by far the fastest with the average time of 30.92 seconds. Parsing is second with the time 104.33 seconds. Trigrams is the slowest with the time 185.89 seconds.

ations between s = 0 and s = 1 for DT, other models, and other models when comparing only runs with anomalous training data (f = 0). These are presented in table 12. Because DT requires samples from both classes (normal and anomaly), we included the f = 0 row for other models as well. The results are clear; supervised models really benefit from the more diverse training data while for unsupervised models it is mostly negligible. For unsupervised models, these results suggest that even with a small amount of training data from the beginning, the models can roughly learn what normal behavior looks like without needing exact samples. 5. Discussion RQ1: What effect do the model, representation and preprocessing options have on anomaly detection performance on software logs? Similarly to previous work [10], our results indicated that there is no single best option. For example, from the models, OOVD reached the best average AUCROC score, but it simply does not work unless you have clean training data. Even the question of whether to use the clean training data depends on the model and dataset. We suspect that this is due to the fact that the models cannot inherently handle out-of-vocabulary terms, which become much more prevalent if the anomalies are filtered from the training data. To be clear, there is a meaningful effect in the selection of the approach, but based on the results, it cannot be generalized. RQ2: To what extent is the variability explained by the dataset? As expected, the effect of the dataset was so large that it is hard to generalize the findings regarding the other variables. As such, it is one of the main contributions of this study to present the characteristics of each dataset. For example, the eventlevel supercomputer logs are completely dependent on finding new log messages, as illustrated by the performance of the OOVD model on anomaly free data. For the sequence-based models this was not the case. Especially, the new LO2Token does not work with OOVD at all, because it has a small number of anomalous log messages in normal sequences. We expect the same to be true for HDFS and Hadoop, to some extent.

4.5. Supplementary analysis: Shuffling on a supervised model The most surprising finding was the small impact of the Shuffle preprocessing option. Intuitively, shuffling the data before assigning training and test sets should always have a positive effect, as it provides more diverse training data and allows a loop hole to see to the future. To address this issue further, we ran the experimental setups with a supervised Decision Tree (DT) classifier. The assumption is that supervised models are inherently more dependent on diverse training data due to the use of labels. The plot 5 shows the scores when plotting without shuffling (s = 0) against with shuffling (s = 1) for runs with otherwise the same parameters. This effectively shows that dots above the black line perform better with shuffling. For DT we can observe that most datapoints overlap at perfect accuracy but for the ones that do not, they are generally above the line. The same does not apply for other models. There are some instances where shuffling clearly improves performance, such as the green dots representing Liberty. These correspond to the same findings we have presented in the boxplot and in the residual analysis. To get a better idea of the general impact of shuffling we calculated the averages and standard devi19

Table 9: Training and testing time for sequence datasets (seconds)

Log rep.

Model

Words

OOVD IF KMeans RM OOVD IF KMeans RM OOVD IF KMeans RM

Trigrams

Events

Hadoop Train Test <0.01 <0.01 0.06 0.04 0.08 <0.01 <0.01 <0.01 <0.01 <0.01 0.07 0.02 <0.01 <0.01 <0.01 <0.01 <0.01 <0.01 0.08 <0.01 <0.01 <0.01 <0.01 <0.01

HDFS Train Test <0.01 0.41 0.38 6.55 0.11 0.55 <0.01 0.43 <0.01 0.49 2.74 26.10 0.14 1.19 0.01 0.50 <0.01 0.40 0.53 3.67 0.01 0.48 <0.01 0.40

LO2Token Train Test <0.01 <0.01 0.07 0.08 0.09 0.01 <0.01 <0.01 <0.01 <0.01 0.10 3.11 0.02 0.06 <0.01 0.01 <0.01 <0.01 0.24 0.03 <0.01 <0.01 <0.01 <0.01

Table 10: Training and testing time for event datasets (seconds)

Log rep.

Model

Words

OOVD IF KMeans RM OOVD IF KMeans RM OOVD IF KMeans RM

Trigrams

Events

BGL Train Test <0.01 3.63 0.64 24.98 0.16 3.96 <0.01 3.63 <0.01 3.75 2.29 59.41 0.23 4.91 0.02 3.74 <0.01 3.50 0.46 22.64 0.05 3.75 <0.01 3.52

Tb Train Test <0.01 5.55 1.08 49.07 0.26 6.48 0.01 5.89 <0.01 6.96 4.67 212.76 0.84 9.40 0.05 6.53 <0.01 5.96 0.93 30.80 0.07 6.54 <0.01 6.29

Liberty Train Test <0.01 4.46 0.92 30.85 0.19 4.86 <0.01 4.40 <0.01 4.54 3.44 71.76 0.29 5.97 0.02 4.63 <0.01 4.30 0.55 25.17 0.05 4.58 <0.01 4.30

Spirit Train Test <0.01 4.41 1.19 37.80 0.22 5.26 <0.01 4.43 <0.01 4.82 4.63 96.91 0.41 7.17 0.03 5.14 <0.01 4.55 0.66 27.03 0.06 4.89 <0.01 4.56

Table 11: Pre-processing times (seconds)

Time to... Normalize Create trigrams Create words Parse event IDs Split & Vec. (W) Split & Vec. (T) Split & Vec. (P)

Hadoop 1.16 0.63 0.04 1.67 0.45 4.98 0.16

BGL 16.58 10.12 0.71 34.58 29.44 149.64 14.73

HDFS 25.20 13.16 0.80 70.43 30.85 251.89 7.36

20

Tb 27.94 21.67 1.70 265.89 59.80 353.65 27.95

Liberty 15.36 10.62 0.82 89.34 34.75 158.24 16.71

Spirit 21.37 13.91 1.20 155.65 45.35 208.70 18.06

LO2T. 13.01 9.61 0.62 25.24 9.91 94.43 2.52

Avg 17.23 11.39 0.84 91.83 30.08 174.50 12.50

DecisionTree: AUCROC (s=0 vs s=1)

0.9

1.0

AUCROC with s=1

AUCROC with s=1

1.0

0.8 0.7 0.6

Other Models: AUCROC (s=0 vs s=1)

0.8 0.6 0.4 0.2 0.0

0.6 BGL HDFS

0.7

0.8

0.9

1.0

AUCROC with s=0 Hadoop LO2Token

0.0

Liberty Spirit

0.2

0.4

0.6

AUCROC with s=0

Thunderbird

Equal Performance Line

Figure 5: Metrics plotted with s = 0 against s = 1

Table 12: Impact of Shuffling on Model Performance

Model Type

Shuffle

Decision Tree

AUCROC

F1 Score

Mean

Std

Mean

Std

s=0 s=1

0.8667 0.9500

0.1403 0.1071

0.7893 0.9626

0.2674 0.0680

Other Models (All)

s=0 s=1

0.6246 0.6166

0.2550 0.2816

0.5985 0.6115

0.2886 0.2932

Other Models (f = 0 only)

s=0 s=1

0.5735 0.5523

0.2465 0.2686

0.5830 0.5731

0.2920 0.3018

21

0.8

1.0

a high percentage of all unique events relatively quickly. This indicates that the sequences themselves have a lot of common log messages which should make the anomaly detection process easier. The focus of this study was on the examination of different combinations of experimental variables rather than proposing a new approach. Therefore, we do not want to make direct comparisons with the approaches used in other papers. It would be very challenging if not impossible to create fair comparisons, as most papers do not report all the necessary details regarding preprocessing that were crucial for our study. In general, the results presented in this study align with the findings of other studies. This study opens various directions for future research. For instance, one could study qualitatively what kind of log messages cause a certain model not to perform well. We did it for LO2Token and OOVD as mentioned in Section 4.2 because we knew the dataset well. However, it was not within the scope of the paper to include similar analysis for all datasets. In the previous study [10], we recommended developing an algorithm to combine OOVD with other models as they do not inherently handle OOV terms well. This requires a technical solution that was not in the scope of this paper. While one of the goals of this paper was to improve our understanding of how different factors affect the anomaly detection performance, it is important to note that our research method does not address this explicitly. To be precise, the research question asked what is the effect of the factors rather than how do the factors affect each other. Empirical experiments are rooted in positivist or post-positivist traditions that aim to establish that there exist a causal connection between two factors but it cannot address why or how it exists. As such our interpretations of why certain settings work well in a given scenario are only interpretations.

RQ3: How do the computational costs of different models, representations, and preprocessing approaches compare? For practical purposes, the results show that word representation with OOVD, KMeans or RM is the fastest option. To improve the performance further, emphasis should be placed on improving the preprocessing practices rather than the models. The results are mostly in-line with previous work [10]. The absolute values differ between these studies because of a different execution environment, but the relative differences remain similar. Compared to the previous study, there was a major improvement on Trigram generation time. This was due to a new function we implemented that generates the trigrams by splitting on three positions rather than iteratively. It still remains the slowest option due to the amount of items to be vectorized. While comparing execution times between studies is difficult, overall the results for execution time presented in this paper remain much lower than in previous papers for both preprocessing [30] and anomaly detection [11]. The use of random instead of chronological random training data selection should have improved the anomaly detection performance [15], yet our experiments showed negative or no effect. We suspect that the difference between our results and the previous results is that we used a much smaller portion of the data for training. Assuming you reach good coverage of the different log messages, then it is better to find them throughout the data, but if the number of messages to be included is low, then it might yield better results to use the chronologically early ones. Figure 6 illustrates the cumulative sum of unique events (parsed with Drain), anomalous events, and unique anomalous events in the datasets. As we can see, many datasets have a surge of new unique events in the beginning. If we use chronological selection, all events after a given point will go to the test set and are therefore “new” as the model encounters them. To perform well, the model needs to find similarities between the new normal events and the known ones and assign them correctly as normal. The results suggest that the unsupervised models are able to do this reasonably well. Such figures could be used by practitioners to highlight relevant parts of execution where there is sudden surge of new activity, and then make informed decisions on whether or not they should be included in the training data. The plots also show that the sequence-based datasets generally reach

5.1. Threats to validity This section describes the threats to validity as divided into internal validity, external validity, and construct validity. 5.1.1. Internal validity Internal validity considers to what extent does study design and methodology allow us to draw accurate conclusions about the factors being studied. A key limitation is that we did not optimize the model-specific parameters. This means that 22

Figure 6: Cumulative sum of events by dataset

23

the performance metrics that we report may not reflect the best possible performance for each approach. Furthermore, it likely influences the possibility to conduct fair comparative analysis between methods. We chose not to include hyperparameter optimization because the number of factors being studied was already so large and hyperparameter optimization would increase that number exponentially.

the effects of four models (KMeans, IsolationForest, RarityModel, and Out-Of-Vocabulary Detector), three representations (words, trigrams, and parsed events), and three preprocessing options (anomaly-free training data, shuffled vs. chronological data selection, and custom vs. generic log message extraction), across seven datasets. This analysis, with 672 experimental setups, offers practitioners and researchers a detailed understanding of how these factors interact and influence performance.

5.1.2. External validity External validity addresses the generalizability of our findings in other contexts and practical applications. One limitation is the applicability of our custom loader implementations. We chose to use the loaders to extract only the log message for two reasons. First, this approach provides a uniform and fair baseline for comparing the datasets, since the fields in the log line may differ between datasets, but all include the log message. Second, log parsing as a discipline focuses predominantly on working with just the log messages. However, we acknowledge that using only the log message likely does not yield the best results for all datasets. In practical applications, practitioners might implement loaders tailored to each case to retain other relevant data while discarding irrelevant fields. For our setup, this was not feasible, as the fairness of the evaluation would have been jeopardized. Nevertheless, we suspect that the results regarding the use of custom loaders would have been better under such configurations.

Second, our findings highlight the critical role of dataset characteristics in determining the effectiveness of anomaly detection methods. Our statistical analysis, using generalized linear models, quantifies the interactions between factors and datasets, revealing that the dataset itself explains the majority of performance variability. While no single approach universally outperforms others, we identify patterns that generalize across similar datasets. For example, log parsing improves performance on sequence-based datasets but can degrade it on event-based ones. Similarly, the Out-Of-Vocabulary Detector excels on supercomputer datasets with anomaly-free training data but fails on sequence-based datasets like HDFS and LO2Token. These insights underscore the importance of tailoring the choice of model, representation, and preprocessing to the specific properties of the dataset at hand. Third, we challenge the assumption that shuffling training data—a form of data leakage—leads to unrealistic performance improvements. Our results show that while shuffling benefits supervised models, its impact on unsupervised models is negligible.

5.1.3. Construct validity Construct validity concerns whether we are measuring what we are supposed to. A limitation in our study is that using ground-truth labels to compute the F1-score is not realistic for a practical setting, because such labels for anomaly detection are typically unavailable in the industry. To overcome this we used AUC-ROC as our primary The differences in evaluation strategy also means that the results are not directly aligned with those used in comparable prior work [11], [12].

Fourth, we provide evidence that using anomalyfree training data improves performance for unsupervised models. While getting anomaly-free data requires additional effort, in practice, this is not as difficult as one might assume. For example, anomaly-free data can be obtained from periods of normal operation—such as days when site reliability engineers confirm that system performance was stable—which can then be used for model training and deployed in production. For new datasets, we recommend first identifying similar datasets in the literature and adopting the methods that have proven effective in those contexts. When no comparable datasets exist, empirical experimentation remains necessary.

6. Conclusion First, we present one of the most comprehensive empirical evaluations of unsupervised anomaly detection approaches on software logs. We study 24

[4] B. Yu, J. Yao, Q. Fu, Z. Zhong, H. Xie, Y. Wu, Y. Ma, P. He, Deep learning or classical machine learning? an empirical study on logbased anomaly detection, in: 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), IEEE Computer Society, 2023, pp. 392–404.

CRediT authorship contribution statement Jesse Nyyssölä: Writing – review & editing, Writing – original draft, Methodology, Data curation, Conceptualization, Software, Formal analysis. Mika Mäntylä: Writing – review & editing, Writing – original draft, Methodology, Conceptualization, Supervision, Validation. Yuqing Wang: Writing – review & editing, Validation.

[5] X. Wu, H. Li, F. Khomh, On the effectiveness of log representation for log-based anomaly detection, Empirical Software Engineering 28 (6) (oct 2023). doi:10.1007/ s10664-023-10364-1.

Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

[6] M. Landauer, S. Onder, F. Skopik, M. Wurzenberger, Deep learning for anomaly detection in log data: A survey, Machine Learning with Applications 12 (2023) 100470.

Acknowledgment

[7] M. Mäntylä, M. Varela, S. Hashemi, Pinpointing anomaly events in logs from stability testing – n-grams vs. deep-learning, 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (2022) 285–292. URL https://api.semanticscholar.org/ CorpusID:246996931

Funded by the Academy of Finland (grant ID 359861). Data availability Scripts for replication, parsed pre-run results, and the LO2Token dataset are available on Zenodo: https://doi.org/10.5281/zenodo.17787039

[8] M. Landauer, F. Skopik, M. Wurzenberger, A critical review of common log data sets used for evaluation of sequence-based anomaly detection techniques, Proc. ACM Softw. Eng. 1 (FSE) (Jul. 2024). doi:10.1145/3660768. URL https://doi.org/10.1145/3660768

References [1] D. A. Bhanage, A. V. Pawar, K. Kotecha, It infrastructure anomaly detection and failure handling: A systematic literature review focusing on datasets, log preprocessing, machine & deep learning approaches and automated tool, IEEE Access 9 (2021) 156392–156421. doi:10.1109/ACCESS.2021.3128283.

[9] S. Ali, C. Boufaied, D. Bianculli, P. Branco, L. Briand, A comprehensive study of machine learning techniques for log-based anomaly detection, Empirical Software Engineering 30 (5) (2025) 129.

[2] R. B. Yadav, P. S. Kumar, S. V. Dhavale, A survey on log anomaly detection using deep learning, in: 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), 2020, pp. 1215–1220. doi:10. 1109/ICRITO48877.2020.9197818.

[10] J. Nyyssölä, M. Mäntylä, Speed and performance of parserless and unsupervised anomaly detection methods on software logs, in: 2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS), 2024, pp. 657–666. doi:10.1109/QRS62785. 2024.00071.

[3] A. Hrusto, P. Runeson, E. Engström, M. C. Ohlsson, Advancing software monitoring: An industry survey on ml-driven alert management strategies, in: 2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), IEEE, 2024, pp. 435– 442.

[11] Z. Chen, J. Liu, W. Gu, Y. Su, M. R. Lyu, Experience report: deep learning-based system log analysis for anomaly detection, arXiv preprint arXiv:2107.05908 (2021). 25

[12] W. Meng, Y. Liu, Y. Zhu, S. Zhang, D. Pei, Y. Liu, Y. Chen, R. Zhang, S. Tao, P. Sun, R. Zhou, Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs, in: Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI’19, AAAI Press, 2019, p. 4739–4745.

[19] L. Yang, J. Chen, Z. Wang, W. Wang, J. Jiang, X. Dong, W. Zhang, Plelog: Semi-supervised log-based anomaly detection via probabilistic label estimation, in: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSECompanion), IEEE, 2021, pp. 230–231. [20] C. Almodovar, F. Sabrina, S. Karimi, S. Azad, Logfit: Log anomaly detection using fine-tuned language models, IEEE Transactions on Network and Service Management 21 (2) (2024) 1715–1723.

[13] M. Catillo, A. Pecchia, U. Villano, Autolog: Anomaly detection by deep autoencoding of system logs, Expert Systems with Applications 191 (2022) 116263. doi:https: //doi.org/10.1016/j.eswa.2021.116263. URL https://www.sciencedirect.com/ science/article/pii/S0957417421015724

[21] R. Zang, H. Guo, J. Yang, J. Liu, Z. Li, T. Zheng, X. Shi, L. Zheng, B. Zhang, Mlad: A unified model for multi-system log anomaly detection (2024). arXiv:2401.07655. URL https://arxiv.org/abs/2401.07655

[14] V. Le, H. Zhang, Log-based anomaly detection without log parsing, CoRR abs/2108.01955 (2021). arXiv:2108.01955. URL https://arxiv.org/abs/2108.01955

[22] X. Wu, H. Li, F. Khomh, What information contributes to log-based anomaly detection? insights from a configurable transformer-based approach, Automated Software Engineering 32 (2), cited by: 0 (2025). doi:10.1007/s10515-025-00527-3. URL https://www.scopus.com/ inward/record.uri?eid=2-s2. 0-105007135246&doi=10.1007% 2Fs10515-025-00527-3&partnerID=40& md5=8ac64d486ea73d51614021a0337430d9

[15] V.-H. Le, H. Zhang, Log-based anomaly detection with deep learning: How far are we?, in: Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 1356–1367. doi:10.1145/3510003.3510155. URL https://doi.org/10.1145/3510003. 3510155

[23] H. Guo, S. Yuan, X. Wu, Logbert: Log anomaly detection via bert (2021). doi:10. 48550/ARXIV.2103.04475. URL https://arxiv.org/abs/2103.04475

[16] J. Lu, C. Wu, Tplogad: Unsupervised log anomaly detection based on event templates and key parameters (2024). arXiv:2411. 15250. URL https://arxiv.org/abs/2411.15250

[24] G. Creech, J. Hu, Generation of a new ids test dataset: Time to retire the kdd collection, in: 2013 IEEE Wireless Communications and Networking Conference (WCNC), 2013, pp. 4487– 4492. doi:10.1109/WCNC.2013.6555301.

[17] M. Du, F. Li, G. Zheng, V. Srikumar, Deeplog: Anomaly detection and diagnosis from system logs through deep learning, in: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 1285–1298. doi:10.1145/3133956.3134015. URL https://doi.org/10.1145/3133956. 3134015

[25] A. Bakhtin, J. Nyyssölä, Y. Wang, N. Ahmad, K. Ping, M. Esposito, M. Mäntylä, D. Taibi, Lo2: Microservice api anomaly dataset of logs and metrics (2025). arXiv:2504.12067. URL https://arxiv.org/abs/2504.12067 [26] P. He, J. Zhu, Z. Zheng, M. R. Lyu, Drain: An online log parsing approach with fixed depth tree, in: 2017 IEEE International Conference on Web Services (ICWS), 2017, pp. 33–40. doi:10.1109/ICWS.2017.13.

[18] Y. Xie, L. Ji, X. Cheng, An attention-based gru network for anomaly detection from system logs, IEICE TRANSACTIONS on Information and Systems 103 (8) (2020) 1916–1919. 26

[27] Y. Fu, M. Yan, Z. Xu, X. Xia, X. Zhang, D. Yang, An empirical study of the impact of log parsers on the performance of log-based anomaly detection, Empirical Softw. Engg. 28 (1) (nov 2022). doi:10.1007/s10664-022-10214-6. URL https://doi.org/10.1007/ s10664-022-10214-6

[34] W. Xu, L. Huang, A. Fox, D. Patterson, M. I. Jordan, Detecting large-scale system problems by mining console logs, in: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, Association for Computing Machinery, New York, NY, USA, 2009, p. 117–132. doi:10.1145/1629575.1629587. URL https://doi.org/10.1145/1629575. 1629587

[28] Y. Sun, J. W. Keung, Z. Yang, S. Liu, H. Yu, Improving anomaly detection in software logs through hybrid language modeling and reduced reliance on parser, Automated Software Engineering 33 (1), cited by: 0 (2026). doi:10.1007/s10515-025-00548-y. URL https://www.scopus.com/ inward/record.uri?eid=2-s2. 0-105017587939&doi=10.1007% 2Fs10515-025-00548-y&partnerID=40& md5=8acec71703c77751841c8d44593ed335

[35] Q. Lin, H. Zhang, J.-G. Lou, Y. Zhang, X. Chen, Log clustering based problem identification for online service systems, in: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), IEEE, 2016, pp. 102–111. [36] S. He, J. Zhu, P. He, M. R. Lyu, Loghub: a large collection of system log datasets towards automated log analytics, arXiv preprint arXiv:2008.06448 (2020).

[29] Y. Sun, J. W. Keung, Z. Yang, S. Liu, H. K. Yu, Semirald: A semi-supervised hybrid language model for robust anomalous log detection, Information and Software Technology 183 (2025) 107743. doi:https://doi.org/ 10.1016/j.infsof.2025.107743. URL https://www.sciencedirect.com/ science/article/pii/S0950584925000825

[37] J. Nyyssölä, M. Mäntylä, Y. Wang, Fast unsupervised anomaly detection methods on software logs - an empirical study (2025). doi:10.5281/zenodo.17787039. URL https://doi.org/10.5281/zenodo. 17787039 [38] J. Mylläri, T. Aalto, J. K. Nurminen, Ladle: a method for unsupervised anomaly detection across log types, Automated Software Engineering 32 (2) (2025) 34.

[30] C. Liu, Y. Tian, S. Yu, D. Gao, Y. Wu, S. Huang, X. Hu, N. Chen, Xdrain: Effective log parsing in log streams using fixed-depth forest, Information and Software Technology 176 (2024) 107546. doi:https://doi.org/ 10.1016/j.infsof.2024.107546. URL https://www.sciencedirect.com/ science/article/pii/S0950584924001514

[39] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikitlearn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825– 2830.

[31] Q. Fournier, D. Aloise, L. R. Costa, Language models for novelty detection in system call traces (2023). arXiv:2309.02206.

[40] A. P. Bradley, The use of the area under the roc curve in the evaluation of machine learning algorithms, Pattern Recognition 30 (7) (1997) 1145–1159. doi:https://doi.org/10.1016/ S0031-3203(96)00142-2. URL https://www.sciencedirect.com/ science/article/pii/S0031320396001422

[32] S. Ali, C. Boufaied, D. Bianculli, P. Branco, L. Briand, N. Aschbacher, An empirical study on log-based anomaly detection using machine learning (2023). arXiv:2307.16714. [33] A. Oliner, J. Stearley, What supercomputers say: A study of five system logs, in: 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07), 2007, pp. 575–584. doi:10.1109/DSN.2007. 103.

[41] R. Vink, Polars: A DataFrame library in Rust, https://github.com/pola-rs/polars, accessed: 2023-11-10 (2023). 27

Related documents

• LogLLM: Log-based Anomaly Detection Using Large Language Models#1186310
• Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection#1076938
• Log-based vs Graph-based Approaches to Fault Diagnosis#14085
• FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection#216814
• Graph neural networks based log anomaly detection and explanation.#342574