arXiv:2606.17094v1 [cs.SE] 13 Jun 2026
LogCopilot: Automating Log Aggregation Analysis through Large Language Models SENYU XIE, Fudan University, China CHENXI ZHANG∗ , Xidian University, China TONG ZHOU, Fudan University, China JIACHENG LIU, Fudan University, China XIAOYU HONG, Fudan University, China QINGSHAN LI, Xidian University, China XIN PENG, Fudan University, China Logs record the runtime behavior of software and are widely used in various tasks such as debugging, testing, and fault diagnosis. With the increase in system size and complexity, log analysis has gradually become a challenging task. Current industrial systems typically use log aggregation systems such as Grafana Loki and ELK to simplify the log collection and analysis process. Engineers write queries using the DSL query language provided by these systems can complete a variety of log analysis tasks. However, writing these queries is often time-consuming and labor-intensive, as it requires engineers to have a thorough understanding of the DSL syntax and the detailed information contained in the logs. To address these challenges, this paper proposes LogCopilot, an automated log aggregation analysis framework based on large language models (LLMs). LogCopilot accepts natural language log analysis instructions and accomplishes automated log analysis through knowledge retrieval and tool calling. LogCopilot constructs a hierarchical knowledge base to represent and provide key knowledge in logs. And it achieves automated log aggregation analysis by generating and executing LogQL queries. The evaluation based on four log datasets confirm the effectiveness of LogCopilot, which achieves an average accuracy of 76.8% and outperforms baseline approaches. Moreover, experiment results shows that LogCopilot is effective in LogQL query generation. CCS Concepts: • Software and its engineering → Software notations and tools; • Computing methodologies → Artificial intelligence. ACM Reference Format: Senyu Xie, Chenxi Zhang, Tong Zhou, Jiacheng Liu, Xiaoyu Hong, Qingshan Li, and Xin Peng. 2018. LogCopilot: Automating Log Aggregation Analysis through Large Language Models. J. ACM 37, 4, Article 111 (August 2018), 22 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Software logs are semi-structured text that are continuously generated during the runtime of software systems. Logs record the runtime behaviors of software systems, and are widely used in ∗ Chenxi Zhang is the corresponding author.
Authors’ Contact Information: Senyu Xie, Fudan University, Shanghai, China, [email protected]; Chenxi Zhang, Xidian University, Xi’an, China, [email protected]; Tong Zhou, Fudan University, Shanghai, China, [email protected]; Jiacheng Liu, Fudan University, Shanghai, China, [email protected]; Xiaoyu Hong, Fudan University, Shanghai, China, [email protected]; Qingshan Li, Xidian University, Xi’an, China, [email protected]; Xin Peng, Fudan University, Shanghai, China, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-735X/2018/8-ART111 https://doi.org/XXXXXXX.XXXXXXX J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:2
Trovato et al.
various tasks, such as debugging, testing, and fault diagnosis [11]. With the increasing scale of modern software systems and the application of new distributed architectures such as microservices, the collection, storage, and analysis of logs has gradually become challenging tasks [12]. Therefore, many log aggregation systems have emerged and are widely used in industrial software systems. Widely used log aggregation systems such as Grafana Loki [7], ELK [46], Datadog [4], etc., support the collection, standardization, and integration of log data from the entire software system. Log aggregation systems typically provide DSL-based interfaces to simplify log analysis process, allowing engineers to query or aggregate logs by writing DSL. For example, Grafana Loki implements the log query language LogQL [8]. Engineers can write LogQL queries to query specific logs or calculate log-related metrics based on the predefined operators or functions such as filtering and grouping. However, writing the DSL queries also places high professional requirements on engineers. Engineers need to understand the syntax of DSLs and need to understand the details of logs, such as format, level, variables, etc. For example, the LogQL query shown in Figure 2 aims to count the number of times the remaining tickets for each train were queried in the past 30 minutes. To write this query, engineers need to know the log format (the red segment in the figure) and the position of the train number variable in the log (the green segment in the figure). In modern large-scale software systems, these requirements impose a significant learning cost on engineers. Recently, researchers have also been exploring various technologies to assist with log analysis [3, 26, 37, 40]. LogAssist [26] and LogNotion [3] help engineers quickly understand the various concepts and knowledge in logs by summarizing the logs. However, they still rely on engineers to write the analysis statements themselves. LogSay [37] achieves natural language-driven log numerical analysis by training a retriever and an inferencer. LogQLLM [40] achieves natural language-toLogQL translation by fine-tuning the LLMs. However, both of the above approaches require the construction of large-scale labeled datasets, which entails significant manual labor costs. Moreover, these models often have generalization issues, and in the rapidly evolving modern software systems, the models need to be retrained frequently. To address the above challenges, this paper proposes LogCopilot, an automated log aggregation analysis framework based on large language models. LogCopilot accepts natural language log analysis instructions and accomplishes automated log aggregation analysis through LLMdriven knowledge retrieval and tool calling. Specifically, LogCopilot first performs log analysis and workflow recognition to identify key knowledge in the logs, including variables, templates, and workflows. Then, LogCopilot utilizes LLMs to hierarchically summarize the knowledge and construct a hierarchical knowledge base. During the log analysis phase, LogCopilot supports two tasks: knowledge base question-answering (Q&A) and log aggregation analysis. Given a log analysis instruction, LogCopilot first understands the intent, then retrieves relevant knowledge from the knowledge base as needed. For log aggregation analysis tasks, it generates LogQL queries to call Grafana Loki for log aggregation analysis. Finally, it generates an analysis report based on the knowledge, aggregation analysis results, etc. We evaluate LogCopilot on four log datasets from previous studies [37, 40, 59, 63]. Build upon existing works [37, 40], we manually constructed a natural language log analysis question-answering benchmark based on these four datasets. The experimental results prove that LogCopilot can effectively perform log analysis and outperforms baseline approaches. When using GPT-4o, its average answer accuracy can achieve 76.8%. Furthermore, experiments show that LogCopilot is effective in LogQL query generation. In summary, the main contributions of this work are:
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:3
Fig. 1. Example of the Log Variable, Log Template, and Log Sequence
• We introduce LogCopilot, a novel automated log aggregation analysis framework based on large language models. • We conduct a series of experiments to demonstrate the effectiveness of LogCopilot in automated log aggregation analysis. • The code and dataset used in this paper are publicly available in the replication package [27]. 2 2.1
Background Log Terminology
Software logs, which are generated from various logging statements in the source code and are collected in an interleaving manner during the software execution, play a crucial role in the maintenance of industrial software systems. Logs record the system’s runtime information and can be used to perform log aggregation analysis, extracting useful insights from massive logs. Figure 1 shows an example of logs from a public dataset of HDFS [63]. Each line in the upper part of Figure 1 is a log message, which is a raw semi-structured sentence. The log message consists of two elements: 1) log template: the fixed text written by the developers to describe a system event; 2) log variables: the values of the system attributes which carry dynamic runtime information, such as URL, file name, IP address, etc. The goal of log parsing is to distinguish the static and dynamic components from a log message. The output of a log parser is a structured log message, containing a log template and the key variables. A series of log messages form a log message sequence. A log sequence is a series of log templates that record an execution flow of a specific task, which share the same task ID. The lower part of Figure 1 shows an example of log sequence with the task ID blk-370867. 2.2
Grafana Loki and LogQL
To supprt efficient log analysis and monitoring in large-scale systems, log aggregation and management systems have become a significant infrastructure for efficient monitoring, debugging, and analytics. In recent years, Grafana Loki [7], an open-source log aggregation system, has been developed inspired by Prometheus [36]. It offers a scalable and cost-effective solution for managing structured logs and is designed to be easy to operate. Instead of indexing the contents of the logs, but rather than a set of labels for each log stream. Inspired by Prometheus, Grafana Loki also provides a multi-dimensional log data model, an easy-to-use log query language, i.e., LogQL [8]. To help engineers manipulate log data, log aggregation systems usually provide domain-specific languages (DSLs) offering interface to interact with log data. LogQL [8] is Grafana Loki’s log J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:4
Trovato et al.
Fig. 2. Example of the LogQL Query
querying language, which is inspired by Prometheus’ metric querying language called PromQL. LogQL queries act as if they are a distributed grep to aggregate log sources. There are two types of LogQL queries: log queries and metric queries. Log queries return the contents of log lines. For example, {container=“query-frontend”} |= “metrics.go” selects log lines sourced from specific container and apply keyword filtering. Metric queries extend log queries by applying a function to log query results, creating metrics from logs. For example, count_over_time({job=“mysql”}[5m]) counts all the log lines within the last five minutes for the MySQL job. Figure 2 shows a typical LogQL query aiming at counting how many times the remaining ticket availability was queried for each train in the past 30 minutes. In this example, {application=“trainticket”, service=“ts-seat-service”, level=“INFO”, logger=“seat.service.SeatServiceImpl”} is used to filter log streams based on the specified labels. The next two lines use regex expressions to filter the log lines and use regex capturing groups to extract labels from log lines respectively. Then the query uses a log range aggregation function to count the number of logs of each log stream during the specified time window [30m]. Finally, an instant vector aggregation grouping by train number is applied to calculate the total occurrences for each train. Readers can refer to [8] for more details of the LogQL. 3
Approach Overview
LogCopilot is a large language models-based log aggregation analysis framework. It accepts natural language log analysis instructions and accomplishes automated log analysis through LLM-driven knowledge retrieval and tool calling. Figure 3 shows an overview of LogCopilot, which consists of the following two phases. Phase-1: Offline Log Knowledge Base Construction: In order to provide LLM with knowledge related to log analysis, LogCopilot first constructs a log knowledge base by extracting and summarizing the variables, templates, and workflows in the raw logs. Phase-2: Online Log Analysis: For a given log analysis instruction, LogCopilot first retrieves the required knowledge from the knowledge base based on the intent of the instruction. And generates LogQL to call Grafana Loki to analyze the raw logs when required. Finally, the analysis report is generated based on the retrieved knowledge and the raw log analysis result. 4
Knowledge Base Construction
To support different granularity log aggregation analysis tasks, we focus on log knowledge in three dimensions: variables, templates, and workflow. These three dimensions of knowledge contain information about log messages and log sequences. And they have a hierarchical association relationship, which is that a variable may be used by different templates and a template may be included in different workflows. To construct the knowledge base, we first perform log parsing to obtain log templates. Then we group the raw logs into log sequences and recognize workflows from them. Finally, we summarize the semantic information of variables, templates, and workflows respectively using LLMs, and store them associatively as the final knowledge base. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:5
Fig. 3. Overview of LogCopilot
4.1
Log Parsing
Following existing researches on log analysis [5, 11, 26], we obtain log templates through log parsing. We adopt a state-of-the-art LLM-based log parsing approach, Lilac [17], which parses logs based on in-context learning and uses adaptive caching to avoid duplicate parsing. We parse each raw log message into a log template and a list of variable values. The log template contains the static part of a log, while variables in the log are replaced by placeholders, and each template is assigned an id (e.g., E1, E2). 4.2
Workflow Recognition
A sequence of log messages records the execution process of a certain task in the system. Workflow is often used to represent a set of log sequences with similar behavior or corresponds to the same task, and has been widely used in existing researches [26, 58]. Therefore, we recognize workflows from logs to represent knowledge related to log sequences. Specifically, we follow the workflow construction method in LogAssist [26]. We first group the raw logs based on grouping ids (e.g., task id, trace id, etc.) to obtain the log sequence. For logs without grouping ids, we group them by time window. Then, we represent each log sequence as a sequence consisting of log templates. Finally, for repeating patterns (i.e., consecutive reoccurring templates or template combinations) in the sequence, we collapse them using the n-gram. In this way, we represent each workflow as a concise sequence consisting of log templates. 4.3
Knowledge Summarization
The original variables, templates, and workflows lack high-level semantic information and are insufficient to support natural language-driven log analysis. Therefore, in this step, LogCopilot uses LLM to summarizes the semantic information of variables, templates, and workflows. The summaries of each knowledge is represented as a paragraph of natural language and are saved as the knowledge base. Because these three types of knowledge have a hierarchical relationship, we conduct a three-step hierarchical summarization process. Variable Summarization. We use a two-stage method to recognize the semantics of variables and generate summaries for each variable. First, we use LLMs to recognize the semantics of the variables contained in each template. For each template, we find the workflows that contain that template and cluster those workflows using TF-IDF [1] and k-means [28] (k is 3 in this paper). We then select the most representative workflows within each cluster, and for each workflow we randomly select one of its corresponding raw log sequences. We include the template and these raw log sequences in a prompt, and prompt LLM to label the name of the variables in the template J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:6
Trovato et al.
Fig. 4. Example of the Log Knowledge Base
and generate a description for each variable. A variable may appear in more than one log template. Therefore, we feed variables with the same name along with their descriptions into the LLM in the second step. And we prompt LLM to integrate these descriptions to generate the final summary of the variable. Template Summarization. Based on the obtained variable summaries, we summarize the templates. We first use the same clustering method as in the variable summarization step to select representative log sequences for each template. Then we include the template, the summaries of the variables in the template, and representative log sequences of this template in a designed prompt. Finally, we prompt LLM to integrate this information to generate the summary of the template. Workflow Summarization. Similarly, we summarize the workflow based on the obtained template summaries. We include the workflow, the summaries of the templates in the workflow, and several log sequences corresponding to this workflow in a designed prompt. Then we prompt LLM to generate a summary of the workflow. After the above steps, the final output is the three types of knowledge of the log in the system. These three types of knowledge form a hierarchical relationship, as shown in the example in Figure 4. It can be seen that each type of knowledge has a summary, and the information it contains, such as services and levels, can also be used for log analysis. We use Elasticsearch [45] to store the knowledge and the relationship between them. And we use an embedding model (text-embedding-3-large [32] in this paper) to create embeddings for the summaries of variables, templates, and workflows, respectively. All the summaries are embedded into 3072-dimensional vectors, and all the vectors are stored to support online retrieval. 5
Log Analysis
Based on the constructed log knowledge base and the log aggregation system Grafana Loki [7], LogCopilot supports natural language instruction-driven automated log analysis. LogCopilot supports J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:7
two types of log analysis tasks: log knowledge base Q&A tasks, and raw log analysis tasks. For a given log analysis instruction, LogCopilot conducts log analysis in a four-step process: intention understanding, knowledge retrieval, log aggregation, and report generation. 5.1
Intention Understanding
Given a log analysis instruction, LogCopilot first understands the intent of the instruction to clarify which log analysis task it corresponds to. LogCopilot designs a prompt that contains the description of the knowledge base, the functionality of Grafana Loki, and some representative log analysis tasks and their analysis process. Then, we prompt LLM to identify which type of log analysis task the input instruction corresponds to. For instructions that are difficult to understand, LogCopilot returns the reasons to help users re-clarify the instructions. 5.2
Knowledge Retrieval
Based on the user intent, LogCopilot retrieves the knowledge related to the log analysis task from the log knowledge base. Since Grafana Loki mainly supports aggregated analysis of templates and variables, we retrieve different knowledge for the two types of tasks. For the raw log analysis tasks, we retrieve template knowledge and variable knowledge. LogCopilot uses the same embedding model as in the knowledge base construction step to generate the embedding of the log analysis instruction. We retrieve the top-𝐾 similar templates using the cosine similarity between the instruction and summary of the template. Then, LogCopilot retrieves the knowledge of the variables that are included in these templates from the knowledge base. For the knowledge base Q&A tasks, we retrieve all types of knowledge. We generate the log analysis instruction embeddings in the same way. However, for this type of task, we retrieve top-𝐾 similar variables, templates, and workflows separately using cosine similarity. Moreover, we follow the global search method in GraphRAG [6] to further score and rank the retrieved knowledge. For each type of retrieved knowledge, we input it into LLM and let LLM determine which knowledge is helpful in answering the question, and generate key points and scores for each knowledge. Finally, we rank each type of knowledge based on the generated scores as the final retrieval results. 5.3
Aggregation Analysis
LogCopilot implements aggregated analysis of raw logs by generating LogQL queries and calling Grafana Loki [7]. Based on the retrieved knowledge, LogCopilot designs a prompt based on the paradigms of CoT and in-context learning. Figure 5 shows the structure of the prompt, the prompt has been simplified for clarity, while the complete version is available in our replication package. Note that LogQL supports two aggregation modes, the first returns specific log messages, the second returns metrics calculated based on the raw logs. Therefore, we describe these two aggregation modes in the prompt and include typical examples, thus enabling the model to generate LogQL queries as required. And we include the retrieved template and variable knowledge in the prompt. After generating LogQL query, LogCopilot calls Grafana Loki to execute it. The generated LogQL queries may have syntax or parameter errors that will cause execution failures, thus LogCopilot designs a rectification process which is inspired by Reflexion [43]. When the execution fails, LogCopilot adds the error message and the generated LogQL query to the prompt shown in Figure 5 and prompts LLM to attempt repairing the LogQL query. Then the repaired LogQL query will be executed again, we repeat the process at most three times to avoid getting into an infinite loop. Finally, we will extract the log aggregation analysis results obtained from the execution as the final result. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:8
Trovato et al.
Fig. 5. Prompt for LogQL Generation
5.4
Report Generation
Based on the retrieved knowledge and aggregation analysis results, LogCopilot leverages LLMs to generate the final analysis report. We designed a prompt to allow LLM to output a structured analysis report. For the knowledge base Q&A tasks, we include all the retrieved knowledge in the prompt. For the raw log analysis tasks, besides the retrieved knowledge, we also include the generated LogQL and the log aggregation analysis result in the prompt. As shown in Figure 6, the analysis report is represented in natural language and consists of three fixed parts and one optional part. Summary describes the log analysis task using a concise description. Explanation describes and explains the entire analysis process in detail. LogQL shows the generated LogQL queries, and this part is only used in raw log analysis tasks. 6
Evaluation
We conduct a series of experimental studies to answer the following research questions. • RQ1: How effective is LogCopilot in handling natural language log analysis tasks? • RQ2: How much does knowledge summarization contribute to the effectiveness of LogCopilot? • RQ3: How does rectification affect the performance of LogQL queries generation? • RQ4: How well does LogCopilot perform in log knowledge retrieval? J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:9
Fig. 6. An Example of the Log Analysis Report Table 1. Details of the Benchmark Dataset
System
6.1
Time Span
# Messages
# Templates
# Tasks
HDFS
38.7h
11,167,740
46
100
OpenSSH
682.4h
638,947
38
100
OpenStack
64.4h
207,632
48
100
TrainTicket
652.1h
1,644,848
180
100
Experiment Setup
6.1.1 Datasets. To evaluate the effectiveness of LogCopilot, we construct a new log analysis benchmark dataset based on logs from four systems: HDFS, OpenSSH, OpenStack, and TrainTicket [61]. These four systems encompass various types, including microservices, cloud management platforms, big data platforms, etc., which can represent different log analysis scenarios. On the basis of existing studies [37, 40], we construct 400 log analysis tasks (100 tasks per system), each task consisting of a natural language question and the corresponding answer (including the analysis result and LogQL). Detailed information of the benchmark dataset is listed in Table 1. Log Source. The HDFS, OpenSSH, and OpenStack logs are sourced from Loghub-2.0 [18], which is a widely used benchmark dataset for log parsing tasks. We use all the log messages it provides. TrainTicket is a medium-scale open source microservice system for railway ticketing and has been widely used in research on log analysis [13, 16, 56], trace analysis [2, 9, 33, 50, 62], and other AIOps tasks [38, 53, 54, 57]. The log data we use comes from the dataset in DeepTraLog [59], which collected 7,705,050 logs from TrainTicket by simulating user requests. Our experiment uses 1,644,848 normal logs in this dataset. Furthermore, these logs contain trace IDs, enabling us to achieve cross service log correlation and analysis in microservice systems. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:10
Trovato et al.
Dataset Construction. Although previous studies [37, 40] have constructed log analysis datasets based on HDFS, OpenSSH, and OpenStack, these datasets all suffer from some shortcomings. LogSay [37] constructed a log numerical reasoning QA dataset with 1,972 question-answer pairs from three different system logs [63]: HDFS, Spark, and OpenSSH. However, the logs used by LogSay come from LogHub 1.0 [63], where each system has only 2,000 log messages. This limitation prevents the numerical reasoning tasks in LogSay from accurately reflecting the scenario of performing aggregate analysis on large volumes of logs in actual systems. LogQLLM [40] built a text-to-LogQL dataset containing 424 cases using logs from three systems (HDFS, OpenSSH, and OpenStack) from Loghub-2.0 [63]. Although the dataset in LogQLLM contains a large number of log messages and a lengthy collection time span, it focuses solely on the text-to-LogQL task, thereby limiting its applicability to evaluating only a subset of log analysis scenarios. Moreover, both datasets suffer from certain quality issues. First, the analysis tasks in them both contain significant duplication. For example, in LogSay, a single task may appear over ten times, differing only in the parameters under examination. Second, many problem descriptions are imprecise and exhibit considerable ambiguity. Building upon these foundational efforts while addressing their limitations, we manually construct a log analysis benchmark dataset using the logs of four systems, as shown in Table 1. Three Ph.D. students have participated in the dataset construction process. First, two Ph.D. students analysis the LogSay and LogQLLM datasets and selected high-quality analysis tasks. Specifically, each student is asked to thoroughly understand and test each log analysis task, identifying cases where both task descriptions and answers are accurate and reasonable. Finally, the two students merged their analysis results, discussed divergent cases, and deduplicated overlapping tasks. Through this process, we get 33 tasks from the LogSay dataset and 119 tasks from the LogQLLM dataset. These tasks are used as initial seeds, and the three PhD students are asked to instantiate them in the logs of different systems. Subsequently, we reference log analysis tasks mentioned in the open-source community and existing research, and further supplemented new log analysis tasks based on the characteristics of each system’s logs. Finally, we let the three Ph.D. students cross-review the constructed tasks and invited a faculty member with industry experience to review all tasks. For tasks with disputes, all participants discussed and modified them together. The final benchmark comprises 100 curated log analysis tasks per system, yielding 400 executable tasks across four datasets, encompassing a wide range of use cases. Each log analysis task in the benchmark includes a natural language question and its corresponding analysis result. For tasks involving log aggregation analysis, we also compose the corresponding LogQL query. Furthermore, we verify the correctness of all answers and LogQL queries through actual execution or manual analysis. Additionally, we import all logs into Grafana Loki and package them as Docker images to ensure consistency across environments. Researchers can use these images to rapidly build test environments and evaluate log analysis tools. We believe this benchmark dataset will facilitate future research in log analysis. 6.1.2 Baselines. To evaluate the effectiveness, we compare our method with the following three baselines. • Few-shot Learning: It is a similar case retrieval-based approach. We first implement a Chainof-Thought (CoT) [48] prompting strategy for text-to-LogQL. Then for each test question, we retrieve top-𝑘 semantically similar questions from historical question-answer pairs, injecting them as in-context demonstrations. The retrieval results of the generated LogQL queries are used as final answers. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:11
• LogSay [37]: It implemented a multi-step “Retriever-Reader” system for automated answering of log-related questions. It adopts a Siamese network [47] and keyword-based filtering to retrieve relevant logs. QANet [55] architecture is used to perform numerical reasoning, which yields the final answer. • LogQLLM [40]: This approach uses LoRA [14] to fine-tune the generic LLMs to obtain the specialized text-to-LogQL model. In our experiment, we fine-tuned the GPT-4o model using the fine-tuning method and the 424 training cases provided in their paper. Similar to Few-shot Learning, the generated LogQL queries are executed to yield the final answers. 6.1.3 Metrics. We adopt the following three metrics to measure the effectiveness of LogCopilot and baseline approaches. • Accuracy: The percentage of correctly handled tasks out of all tasks. A task will be considered correctly handled only when the final answer output by the approach is the same as the ground-truth answer. Note that since the final answer is generated by LLMs and it may have multiple equivalent forms, we manually label the results and consider all equivalent forms as correct. • SyntaxAcc: The percentage of syntactically correct LogQL queries out of all generated LogQL queries. A LogQL query is considered syntactically correct only if it passes Grafana Loki’s syntax checking. • Recall: To evaluate the effectiveness of LogCopilot in log knowledge retrieval, we calculate Recall for each log analysis task in test dataset as follows: |D𝑟𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑 ∩ D𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 | (1) |D𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 | D𝑟𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑 is the retrieved documents from log knowledge base. D𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 is the ground-truth relevant documents for the task. This metric quantifies the ability of retrieval component to capture all knowledge prerequisites for accurate log analysis. Higher 𝑅𝑒𝑐𝑎𝑙𝑙 indicates fewer information gaps in downstream analyzing. 𝑅𝑒𝑐𝑎𝑙𝑙 =
6.1.4 Implementation. We implement the prototype of LogCopilot with Python 3.10 in approximately 2,210 lines of code. All experiments are conducted on a Ubuntu 22.04LTS server with 8 × Intel(R) Xeon(R) Platinum 8269CY 2.5GHz CPU and 32GB RAM. We select two LLMs in our experiments, i.e., GPT-4o [30] and GPT-4o-LogQL. GPT-4o demonstrates impressive performance in reasoning and coding, especially for complex queries. GPT4o-LogQL, the top-performing variant in LogQLLM’s original experiments, is derived through fine-tuning of GPT-4o on the LogQLLM benchmark. Following standard practice, we partition the dataset into 80% training and 20% validation splits. We incorporate both generic (GPT-4o) and LogQL-specialized (GPT-4o-LogQL) LLMs to provide a more comprehensive view of the effectiveness of LogCopilot and baseline approaches. We set the temperature of all LLMs as zero to ensure the reproduction. 6.2
RQ1: Effectiveness of LogCopilot
In this experiment, we evaluate the effectiveness of LogCopilot in handling natural language log analysis tasks. LogCopilot employs both GPT-4o and GPT-4o-LogQL as the backbone model. We only adopt GPT-4o-LogQL as the backbone model of LogQLLM because its performance significantly exceeds that of the other two fine-tuned models. We adopt GPT-4o as the backbone model of Few-shot Learning baseline. The results of this evaluation are detailed in Table 2, which provides a comparative analysis of five methods across four datasets in terms of Accuracy. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:12
Trovato et al.
Table 2. Comparision of Accuracy of LogCopilot and baselines
LogCopilot System
LogSay
Few-shot
LogQLLM GPT-4o
GPT-4o-LogQL
HDFS
0.100
0.180
0.060
0.850
0.850
OpenSSH
0.070
0.480
0.220
0.730
0.700
OpenStack
0.010
0.530
0.080
0.750
0.790
TrainTicket
0.030
0.080
0.000
0.740
0.820
Average
0.053
0.318
0.090
0.768
0.790
The experimental results shown in Table 2 demonstrate that LogCopilot consistently outperforms all baseline methods, achieving the highest Accuracy across the HDFS, OpenSSH, OpenStack, and TrainTicket systems. Notably, both variants of LogCopilot, using the generic model (GPT-4o) and the LogQL-specialized model (GPT-4o-LogQL), achieve strong average Accuracy exceeding 0.750. These results highlight the effectiveness of our approach in comprehending user intent and generating correct LogQL queries across diverse systems. Among all the baseline methods, LogSay shows the poorest performance, with a maximum Accuracy of only 0.100 on the HDFS dataset and a minimum of just 0.010 on OpenStack. This is primarily due to the essence of LogSay. It integrates different neural networks in different phases, each requiring large amount of training data. In our experiments, we synthesize approximately 396 task instances using GPT-4o based on our benchmark to train LogSay, but the result remains unsatisfactory. LogSay fails to generalize effectively with limited resource, suggesting its low practicality for real-world natural language log analysis. The Few-shot Learning baseline, which is based on GPT-4o with 5 historical examples, exhibits moderate performance on benchmark. It performs relatively well on OpenSSH and OpenStack. These results indicate that the guidance and knowledge provided by historical examples can help improve the performance of log analysis. However, its performance is still worse than LogCopilot, since that few-shot prompting alone cannot cover all the knowledge required for complex tasks. Moreover, this method heavily relies on curated examples and lacks robustness when facing diverse systems. LogQLLM refers to GPT-4o-LogQL and is expected to be specialized in LogQL syntax. However, its actual performance on our benchmark is poor, with an average Accuracy of only 0.090. It even underperforms the Few-shot Learning baseline, which learns not only log knowledge but also LogQL syntax from historical examples, across all the systems. This implies that only mastering the LogQL syntax without integrating semantic understanding or contextual knowledge significantly limits the effectiveness of the model. In contrast, LogCopilot using the same GPT-4o-LogQL achieves the highest average Accuracy of 0.790. This evidences the significance of log template knowledge in improving semantic understanding and generating correct LogQL queries, which may contain regex expressions and pattern filtering. To further investigate the impact of backbone models, we compare the two variants of LogCopilot: one powered by GPT-4o and the other powered by GPT-4o-LogQL. The results show that LogCopilot using GPT-4o-LogQL achieves a higher average Accuracy and outperforms LogCopilot using GPT-4o on OpenStack and TrainTicket. This may be due to the better LogQL generation capability of GPT-4o-LogQL. However, GPT-4o-LogQL slightly underperforms GPT-4o J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
(a) HDFS
111:13
(b) OpenSSH
Fig. 7. Distribution of Questions Successfully Handled by LogCopilot and Baselines Table 3. Accuracy of LogCopilot without Knowledge Summarization
Approach
HDFS
OpenSSH
OpenStack
TrainTicket
w/ KS
0.850
0.730
0.750
0.740
w/o KS
0.660(↓22.4%)
0.640(↓12.3%)
0.650(↓13.3%)
0.670(↓9.5%)
on the OpenSSH dataset (0.030 absolute drop). We believe that this is caused by the randomness of LLMs even though we have set the temperature as zero. Figure 7 presents two Venn diagrams, demonstrating the unique questions successfully handled by each approach on HDFS and OpenSSH systems. These two diagrams highlight the complementarity and overlap among the approaches. Through Figure 7a, we can find that LogCopilot handles the most questions by understanding user intent and applying correct log aggregation analysis. Notably, LogCopilot correctly answers 56 unique questions that fail in all baseline approaches. Only 2 questions in HDFS are covered by LogQLLM, Few-shot Learning, and LogCopilot at the same time. This result underscores LogCopilot’s superior generalization and semantic understanding. Figure 7b indicates that LogCopilot could solve 27 tasks that are beyond the capabilities of baseline approaches. Although the intersection of Few-shot Learning and LogCopilot, with a size of 24, shows the basic capability of Few-shot Learning, 30 questions still remain unsolved by Few-shot Learning. The results emphasize limited capability of baseline approaches, suggesting that LogCopilot can handle diverse cases that are typically challenging. In conclusion, LogCopilot leverages the semantic understanding of LLMs and retrieved log knowledge to perform log aggregation analysis, which is effective in answering log analysis questions and significantly outperforms baseline approaches. Compared to baseline approaches, LogCopilot achieves higher Accuracy and covers more questions. 6.3
RQ2: Contribution of knowledge summarization
In Section 4.3, we have discussed knowledge summarization, which attaches high-level semantic information to variables, templates, and workflows. In this part, we conduct an ablation experiment to explore the contribution of log knowledge summaries generated by LLMs. To this end, we run LogCopilot without knowledge summarization and compare the Accuracy with the full version. Instead of embedding the summaries of variables, templates, and workflows, LogCopilot w/o KS creates embeddings based on the original content of variables, templates, and workflows. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:14
Trovato et al.
Table 4. Evaluation of the Contribution of Rectification
Metric SyntaxAcc
Accuracy
Approach
HDFS
OpenSSH
OpenStack
TrainTicket
w/o RE
0.888
0.850
0.775
0.650
w/ RE
1.000(↑12.6%)
0.938(↑10.4%)
0.950(↑22.6%)
0.850(↑30.8%)
w/o RE
0.800
0.650
0.613
0.575
w/ RE
0.850(↑6.3%)
0.713(↑9.6%)
0.750(↑22.4%)
0.725(↑26.1%)
Table 3 demonstrates the experimental results. Our observations indicate that the overall performance of LogCopilot declines when knowledge summaries are lacking. Notably, HDFS suffers the most severe Accuracy degradation (22.4%). This is because the semantics of individual templates in HDFS tend to be more low-level and system-centric, often reflecting internal operations such as data block management and node communication. As a result, there exists a semantic gap between these fine-grained log events and the high-level natural language questions. On the other three datasets, the Accuracy reduction ranges from 9.5% to 13.3%. This ablation study confirms the critical contribution of log knowledge summarization to LogCopilot’s effectiveness, particularly for systems with fine-grained log events. 6.4
RQ3: Contribution of rectification
In this part, we explore the effect of LogCopilot’s rectification mechanism mentioned in Section 5.3. It appends pending LogQL query with error message from Grafana Loki to context, and prompts LLM to repair the LogQL query by regeneration. Through controlled rectification experiments, we quantitatively evaluate LogCopilot’s performance improvement by comparing pre- and postrectification results. We use w/o RE to indicate the results without rectification, and w/ RE to indicate the final results after at most three turns of rectification. Note that tasks not involving LogQL are not counted. The experimental results are depicted in Table 4. The results indicate that the rectification mechanism consistently improves SyntaxAcc and Accuracy across all datasets. The SyntaxAcc on HDFS increases from 0.888 to a remarkable 1.000 after rectification. We observe the highest SyntaxAcc improvement (30.8%) on TrainTicket. On the same system, the Accuracy improves from 0.575 to 0.725, corresponding to a relative gain of 26.1%. Moreover, the Accuracy of all four systems exceeds 0.7. These observations demonstrate the mechanism’s capability to largely eliminate syntax errors in generated queries for complex log analysis tasks. In terms of execution error type, one of the most common errors prior to rectification is the improper or missing use of aggregation functions. This frequently results in runtime errors such as “maximum of series (500) reached for a single query", which occurs when LogQL query attempts to process too many time series without aggregating them appropriately. 6.5
RQ4: Recall of knowledge retrieval
Recall of log knowledge retrieval is crucial to the effectiveness of LogCopilot. Therefore, in this part, we evaluate the recall of LogCopilot in retrieving log knowledge. We first label the ground truth of all the variables, templates and workflows that are essential in the LogQL query generation phase for each question. Then we calculate the recall rate for each question, based on Equation 1. To simplify the evaluation, we do not distinguish between the three types of knowledge documents in recall calculation. Instead, we treat all retrieved knowledge pieces as belonging to a unified set J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:15
Table 5. Recall of Log Knowledge Retrieval
System
Average Recall
HDFS
0.978
OpenSSH
0.958
OpenStack
0.994
TrainTicket
0.958
Fig. 8. A Successful Case
and compute recall based on whether the ground-truth items, regardless of their document type, appear in the top retrieved results. Finally, for each dataset, we average the results from all the questions to obtain the average recall. Table 5 presents the average recall of LogCopilot in knowledge retrieval. It can be seen that all of the average recall rates exceed 0.95, and LogCopilot reaches the highest average recall of 0.994 on OpenStack. This indicates that LogCopilot can effectively understand the intent of the question and retrieve the required variables, templates, and workflows. 7 7.1
Discussion Case Study
To understand the capabilities and limitations of LogCopilot, we further manually analyze a successful case and a failed case in this section. The two cases are shown in Figure 8 and Figure 9, where the green part is the relevant log knowledge retrieved by LogCopilot. These cases are obtained based on LogCopilot powered by GPT-4o-LogQL. 7.1.1 Successful Case. In Figure 8, we present a successful case where LogCopilot understands user intent, performs correct log aggregation analysis using LogQL query, and generates answer J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:16
Trovato et al.
Fig. 9. A Failed Case
matching the ground truth. In this case, the user wants to query the train numbers starting with “G” or “D” used in the “getLeftTicketOfInterval” operation. LogCopilot performs intention understanding and generates LogQL query to aggregate logs. As shown in the yellow part of the figure, LogCopilot first finds the semantically relevant log templates, e.g. “[getLeftTicketOfInterval] TrainNumber start with <*>”, then finds the log variables corresponding to these log templates. As shown in the purple part of the figure, LogCopilot selects the stream labels based on the retrieved log templates. Leveraging the above knowledge, LogCopilot successfully generated the correct LogQL query. In summary, LogCopilot is able to effectively retrieve the knowledge related to the question and leverage the capabilities of LLMs to perform log aggregation analysis, yielding a natural language answer to the questioner. 7.1.2 Failed Case. In Figure 9, we present a failed case where LogCopilot fails to generate the correct LogQL query, yielding a wrong final answer. In this case, LogQL query generated by LogCopilot is missing a grouping clause and a final verctor aggregation function. We highlight the ground-truth aggregation and the wrong aggregation by green and red respectively. However, we find that the relevant log templates and log variables have been retrieved by LogCopilot, which is the red part of the knowledge in Figure 9. It indicates that the retrieval method proposed by LogCopilot is effective, but the randomness or limitations of the capabilities of LLMs can lead to incorrect results. On one hand, the LLMs often struggle to accurately capture the user’s true intent due to the inherent vagueness of natural language. On the other hand, the LLMs may lack a deep understanding of the underlying data model, merely mimicking the syntax without fully grasping the semantics of the operations involved. In summary, sometimes the LogQL generated by LogCopilot is not precise. It is because the LLM does not really understand the underlying data model of Grafana Loki and the impacts of aggregation functions, even though the LLM has been fine-tuned to learn LogQL syntax. In our J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:17
future work, we can use log aggregation tasks integrated with data model view to fine-tune LLMs, enhancing the LLMs’ capabilities in both generating syntactically correct query and manipulating log streams as user’s intent. 7.2
Threats to Validity
7.2.1 Internal Threat. The threat to internal validity mainly lies in the implementation and configuration of baseline approaches. LogSay provides an open-source implementation but the code cannot be used on our benchmark directly. So we manually convert the format of samples in our benchmark and synthesize approximately 396 task instances as the training set using GPT-4o based on the transformed benchmark. LogQLLM has also provided publicly available fine-tuning source code and datasets. Due to the closed-source and proprietary of GPT-4o, we cannot access the GPT-4o fine-tuned by LogQLLM researchers. To alleviate the threat, we carefully follow the paper and use exactly the same script to fine-tune GPT-4o using OpenAI platform. There is no directly available open-source implementation of the few-shot learning-based approach. Therefore, we implement the approach by referring to the example of question answering using embeddings-based search provided in the OpenAI cookbook [31]. To control the influences of the configurations of baseline approaches, we conduct experiments for all baseline approaches and select the best configuration for them. 7.2.2 External Threat. The external threats in our studies are mainly from the subject datasets and noises in labeling. Subject Datasets: In this work, we have only evaluated LogCopilot and three baseline approaches on four datasets. These log data comes from the distributed system (i.e., HDFS and OpenStack), server application (i.e., OpenSSH), and microservice system (i.e., TrainTicket). Although HDFS, OpenSSH, and OpenStack all come from real-world systems and contain millions of log messages, the number of subject systems is still limited and does not cover all the domains. Though TrainTicket is one of the largest open-source microservices systems and has been widely used in existing studies [21, 44, 49, 59], there is still a gap between its scale and industrial large-scale microservices systems. Moreover, although we curate more diverse log analysis tasks than existing studies [37, 40], these log analysis tasks cannot cover all situations. Therefore, the results of our experimental studies may not be generalized to larger or more complex datasets. In the future, we will evaluate our approach on more datasets. Noises in Labeling: Our experiments are based on four log datasets that are widely used by related work [5, 15, 41, 52]. The log analysis questions and corresponding answers are manually synthesized, as there are few well-constructed datasets for natural language log analysis. To evaluate the effectiveness of our approach, manual annotation is required to provide the ground-truth analysis result and LogQL queries for each task. Though this benchmark is manually constructed and carefully checked by PhD students and faculty, data noise may still be introduced during the manual labeling process. Although we believe the amount of noise is small, we will investigate the quantity issue in our future work. 8
Related Work
Natural Language to DSL: Domain-Specific Languages (DSLs) provide tailored syntactic expressions for particular problem spaces, or domains (e.g., SQL for database). In recent years, machine learning and deep learning techniques have been widely used in text-to-DSL tasks, such as text-toSQL [19, 20] and text-to-GraphQL [29]. These studies aim to automatically map natural language instructions to DSL, enabling users to interact with domain-specific tools or data through natural language. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:18
Trovato et al.
Among these studies, text-to-SQL has emerged as a prominent area of study, due to the widespread adoption of SQL for structured data querying and database management tasks. With the advancement of Large Language Models (LLMs), a growing number of researchers have investigated the application of LLMs to text-to-SQL tasks [22–24, 42]. Pourreza and Rafiei [35] propose a decomposed in-context learning framework for text-to-SQL, achieving substantial improvements in execution accuracy. Pourreza et al. [34] introduce a multi-path reasoning framework that enhances text-to-SQL performance by combining diverse candidate generation strategies. Ren et al. [39] present PURPLE, a retrieval-augmented prompting framework that enhances LLM performance in NL2SQL tasks by selecting demonstrations containing relevant logical operator compositions. While extensive research has been conducted on text-to-SQL, much less attention has been paid to translating natural language into LogQL. To the best of our knowledge, there exists only one prior work that explicitly explores the task of text-to-LogQL. Seshagiri et al. [40] fine-tune LLMs using manually curated text-to-LogQL datasets, gaining substantial improvements in LogQL generation. Log Analysis: Logs, which record runtime information and key events, are playing an important role in system monitoring. Therefore, how to reasonably utilize log data has become a key area of research. Many efforts have been dedicated to log analysis, including log parsing, anomaly detection, root cause analysis, etc. Before performing log analysis, it is often essential to apply log parsing techniques to structure the raw log messages into a more analyzable form. Various automated log parsers have been proposed to extract templates or event types from unstructured logs, such as Drain [10], UniParser [25], and SemParser [15]. Some prior studies try to assist practitioners in comprehending system workflows, to facilitate debugging and test case design. Locke et al. [26] propose LogAssist, which seeks to summarize logs into workflows to facilitate log analysis tasks. Logs can be utilized for anomaly detection. Du et al. [5] leverage a deep neural network model to detect anomalies based on recognized log sequences. Zhang et al. [60] implement LogRobust, leveraging an attention-based Bi-LSTM model to detect anomalies on unstable log data. Some prior studies try to help users answer log analysis questions. Qi et al. [37] propose LogSay, a multistep “Retriever-Reader” system for automated answering of log-related questions, adopt deep learning models to perform log numerical reasoning. Approaches in previous research often relies on manually curated rules or are trained on limited datasets. Due to their limited generalizability, these methods often require substantial effort to adapt to new datasets or systems. Otherwise, their performance degrades significantly. LLMs have shown great capabilities in various domains, indicating their potential in log analysis. Some prior studies have explored the integration of LLMs with log analysis. Jiang et al. [17] and Xu et al. [52] prompt LLMs to parse logs using few-shot learning, achieving promising performance. Shan et al. [41] propose a LLM-based strategy for localizing configuration errors through logs. Xu et al. [51] use LLMs to automatically decide where and what to log based on the in-context learning paradigm. In this paper, we propose a RAG-based log analysis framework to directly answer natural language questions, which is user-friendly compared to existing log analysis works. 9
Conclusion
In this paper, we propose LogCopilot, an automated log aggregation analysis framework based on large language models. LogCopilot accepts natural language log analysis instructions and accomplishes automated log analysis through LLM-driven knowledge retrieval and tool calling. Specifically, LogCopilot first performs log analysis and workflow recognition to identify key knowledge in the logs. Then, it hierarchically summarize the key knowledge and construct a hierarchical knowledge base. In the log analysis phase, LogCopilot supports both knowledge J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:19
based Q&A and log aggregation analysis based on the hierarchical knowledge base. To evaluate LogCopilot, we construct a log aggregation analysis benchmark dataset based on log data from four different types of systems. Experimental results demonstrate that LogCopilot can perform log aggregation analysis effectively and outperform all the baseline approaches. 10
Data Availability
All the data and code of the work can be found in our replication package [27]. References [1] Prafulla Bafna, Dhanya Pramod, and Anagha Vaidya. 2016. Document clustering: TF-IDF approach. In 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT). IEEE, 61–66. [2] Zhuangbin Chen, Junsong Pu, and Zibin Zheng. 2025. Tracezip: Efficient Distributed Tracing via Trace Compression. Proc. ACM Softw. Eng. 2, ISSTA (2025), 411–433. doi:10.1145/3728888 [3] Guojun Chu, Jingyu Wang, Tao Sun, Qi Qi, Haifeng Sun, Zirui Zhuang, and Jianxin Liao. 2025. LogNotion: Highlighting Massive Logs to Assist Human Reading and Decision Making. IEEE Trans. Serv. Comput. 18, 2 (2025), 940–953. doi:10.1109/TSC.2025.3528327 [4] Datadog. 2025. Datadog. Retrieved May 26, 2025. https://www.datadoghq.com/ [5] Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017. DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 - November 03, 2017. ACM, 1285–1298. doi:10.1145/3133956.3134015 [6] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. CoRR abs/2404.16130 (2024). doi:10.48550/ARXIV.2404.16130 [7] Grafana. 2025. Grafana Loki. Retrieved May 26, 2025. https://grafana.com/oss/loki/ [8] Grafana. 2025. LogQL. Retrieved May 26, 2025. https://grafana.com/docs/loki/latest/query/ [9] Xiaofeng Guo, Xin Peng, Hanzhang Wang, Wanxue Li, Huai Jiang, Dan Ding, Tao Xie, and Liangfei Su. 2020. Graphbased trace analysis for microservice architecture understanding and problem diagnosis. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020. ACM, 1387–1397. doi:10.1145/3368089.3417066 [10] Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu. 2017. Drain: An Online Log Parsing Approach with Fixed Depth Tree. In 2017 IEEE International Conference on Web Services, ICWS 2017. IEEE, 33–40. doi:10.1109/ICWS.2017.13 [11] Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R. Lyu. 2022. A Survey on Automated Log Analysis for Reliability Engineering. ACM Comput. Surv. 54, 6 (2022), 130:1–130:37. doi:10.1145/3460345 [12] Shilin He, Xu Zhang, Pinjia He, Yong Xu, Liqun Li, Yu Kang, Minghua Ma, Yining Wei, Yingnong Dang, Saravanakumar Rajmohan, and Qingwei Lin. [n. d.]. An empirical study of log analysis at Microsoft. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022. 1465–1476. doi:10.1145/3540250.3558963 [13] Shilin He, Xu Zhang, Pinjia He, Yong Xu, Liqun Li, Yu Kang, Minghua Ma, Yining Wei, Yingnong Dang, Saravanakumar Rajmohan, and Qingwei Lin. 2022. An empirical study of log analysis at Microsoft. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022. ACM, 1465–1476. doi:10.1145/3540250.3558963 [14] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [n. d.]. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022. https://openreview.net/forum?id=nZeVKeeFYf9 [15] Yintong Huo, Yuxin Su, Cheryl Lee, and Michael R. Lyu. 2023. SemParser: A Semantic Parser for Log Analytics. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023. IEEE, 881–893. doi:10.1109/ICSE48619.2023.00082 [16] Xinrui Jiang, Yicheng Pan, Meng Ma, and Ping Wang. 2023. Look Deep into the Microservice System Anomaly through Very Sparse Logs. In Proceedings of the ACM Web Conference 2023, WWW 2023. ACM, 2970–2978. doi:10.1145/3543507. 3583338 [17] Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, and Michael R. Lyu. 2024. LILAC: Log Parsing using LLMs with Adaptive Parsing Cache. Proc. ACM Softw. Eng. 1, FSE (2024), 137–160. doi:10.1145/3643733 [18] Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Jieming Zhu, and Michael R. Lyu. 2024. A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024. ACM, 223–234. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:20
Trovato et al.
doi:10.1145/3650212.3652123 [19] George Katsogiannis-Meimarakis and Georgia Koutrika. 2023. A survey on deep learning approaches for text-to-SQL. VLDB J. 32, 4 (2023), 905–936. doi:10.1007/S00778-022-00776-8 [20] Hyeonji Kim, Byeong-Hoon So, Wook-Shin Han, and Hongrae Lee. 2020. Natural language to SQL: Where are we today? Proc. VLDB Endow. 13, 10 (2020), 1737–1750. doi:10.14778/3401960.3401970 [21] Cheryl Lee, Tianyi Yang, Zhuangbin Chen, Yuxin Su, and Michael R. Lyu. 2023. Eadro: An End-to-End Troubleshooting Framework for Microservices on Multi-source Data. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023. IEEE, 1750–1762. doi:10.1109/ICSE48619.2023.00150 [22] Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024. The Dawn of Natural Language to SQL: Are We Fully Ready? [Experiment, Analysis & Benchmark ]. Proc. VLDB Endow. 17, 11 (2024), 3318–3331. doi:10.14778/ 3681954.3682003 [23] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen-Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. [n. d.]. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. In Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023. [24] Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuyu Luo, Yuxin Zhang, Ju Fan, Guoliang Li, and Nan Tang. 2024. A Survey of NL2SQL with Large Language Models: Where are we, and where are we going? CoRR abs/2408.05109 (2024). doi:10.48550/ARXIV.2408.05109 [25] Yudong Liu, Xu Zhang, Shilin He, Hongyu Zhang, Liqun Li, Yu Kang, Yong Xu, Minghua Ma, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2022. UniParser: A Unified Log Parser for Heterogeneous Log Data. In WWW ’22: The ACM Web Conference 2022. ACM, 1893–1901. doi:10.1145/3485447.3511993 [26] Steven Locke, Heng Li, Tse-Hsun Peter Chen, Weiyi Shang, and Wei Liu. 2022. LogAssist: Assisting Log Analysis Through Log Summarization. IEEE Trans. Software Eng. 48, 9 (2022), 3227–3241. doi:10.1109/TSE.2021.3083715 [27] LogCopilot. 2026. LogCopilot. Retrieved May 26, 2026. https://github.com/FudanSELab/LogCopilot [28] James MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, Vol. 5. University of California press, 281–298. [29] Pin Ni, Ramin Okhrati, Steven Guan, and Victor Chang. 2024. Knowledge Graph and Deep Learning-based Text-toGraphQL Model for Intelligent Medical Consultation Chatbot. Inf. Syst. Frontiers 26, 1 (2024), 137–156. doi:10.1007/ S10796-022-10295-0 [30] OpenAI. 2024. GPT-4o. Retrieved May 26, 2025. https://platform.openai.com/docs/models/gpt-4o [31] OpenAI. 2024. OpenAI Cookbook. Retrieved May 26, 2025. https://cookbook.openai.com/ [32] OpenAI. 2025. text-embedding-3-large. Retrieved May 26, 2025. https://platform.openai.com/docs/models/textembedding-3-large [33] Xin Peng, Chenxi Zhang, Zhongyuan Zhao, Akasaka Isami, Xiaofeng Guo, and Yunna Cui. 2022. Trace analysis based microservice architecture measurement. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022. ACM, 1589–1599. doi:10.1145/3540250. 3558951 [34] Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan Ö. Arik. 2025. CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL. In The Thirteenth International Conference on Learning Representations, ICLR 2025. https://openreview.net/forum?id=CvGqMD5OtX [35] Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/ 72223cc66f63ca1aa59edaec1b3670e6-Abstract-Conference.html [36] Prometheus. 2025. Prometheus. Retrieved May 26, 2025. https://prometheus.io/ [37] Jiaxing Qi, Zhongzhi Luan, Shaohan Huang, Carol J. Fung, and Hailong Yang. 2024. LogSay: An Efficient Comprehension System for Log Numerical Reasoning. IEEE Trans. Computers 73, 7 (2024), 1809–1821. doi:10.1109/TC.2024.3386068 [38] Rui Ren, Jingbang Yang, Linxiao Yang, Xinyue Gu, and Liang Sun. 2024. SLIM: a Scalable and Interpretable Light-weight Fault Localization Algorithm for Imbalanced Data in Microservice. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024. ACM, 27–39. doi:10.1145/3691620.3694984 [39] Tonghui Ren, Yuankai Fan, Zhenying He, Ren Huang, Jiaqi Dai, Can Huang, Yinan Jing, Kai Zhang, Yifan Yang, and X. Sean Wang. 2024. PURPLE: Making a Large Language Model a Better SQL Writer. In 40th IEEE International Conference on Data Engineering, ICDE 2024. IEEE, 15–28. doi:10.1109/ICDE60146.2024.00009
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
LogCopilot: Automating Log Aggregation Analysis through Large Language Models
111:21
[40] Vishwanath Seshagiri, Siddharth Balyan, Vaastav Anand, Kaustubh D. Dhole, Ishan Sharma, Avani Wildani, José Cambronero, and Andreas Züfle. 2024. Chatting with Logs: An exploratory study on Finetuning LLMs for LogQL. CoRR abs/2412.03612 (2024). arXiv:2412.03612 doi:10.48550/ARXIV.2412.03612 [41] Shiwen Shan, Yintong Huo, Yuxin Su, Yichen Li, Dan Li, and Zibin Zheng. 2024. Face It Yourselves: An LLM-Based Two-Stage Strategy to Localize Configuration Errors via Logs. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024. ACM, 13–25. doi:10.1145/3650212.3652106 [42] Liang Shi, Zhengju Tang, and Zhi Yang. 2024. A Survey on Employing Large Language Models for Text-to-SQL Tasks. CoRR abs/2407.15186 (2024). arXiv:2407.15186 doi:10.48550/ARXIV.2407.15186 [43] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. [n. d.]. Reflexion: language agents with verbal reinforcement learning. In Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023. [44] Jacopo Soldani and Antonio Brogi. 2023. Anomaly Detection and Failure Root Cause Analysis in (Micro) Service-Based Cloud Applications: A Survey. ACM Comput. Surv. 55, 3 (2023), 59:1–59:39. doi:10.1145/3501297 [45] Elastic Stack. 2025. Elasticsearch. Retrieved May 26, 2025. https://www.elastic.co/cn/elasticsearch [46] Elastic Stack. 2025. ELK. Retrieved May 26, 2025. https://www.elastic.co/cn/elastic-stack [47] Ziheng Wang, Farzad Niknia, Shanshan Liu, Pedro Reviriego, Paolo Montuschi, and Fabrizio Lombardi. 2023. Tolerance of Siamese Networks (SNs) to Memory Errors: Analysis and Design. IEEE Trans. Computers 72, 4 (2023), 1136–1149. doi:10.1109/TC.2022.3186628 [48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. [n. d.]. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. ([n. d.]). [49] Xiaoxue Wu, Wei Zheng, Xin Xia, and David Lo. 2022. Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction. IEEE Trans. Software Eng. 48, 7 (2022), 2541–2556. doi:10.1109/TSE.2021.3063727 [50] Shuaiyu Xie, Jian Wang, Maodong Li, Peiran Chen, Jifeng Xuan, and Bing Li. 2025. TracePicker: Optimization-Based Trace Sampling for Microservice-Based Systems. Proc. ACM Softw. Eng. 2, FSE (2025), 1802–1823. doi:10.1145/3729351 [51] Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2024. UniLog: Automatic Logging via LLM and In-Context Learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024. ACM, 14:1–14:12. doi:10.1145/3597503.3623326 [52] Junjielong Xu, Ruichun Yang, Yintong Huo, Chengyu Zhang, and Pinjia He. 2024. DivLog: Log Parsing with Prompt Enhanced In-Context Learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024. ACM, 199:1–199:12. doi:10.1145/3597503.3639155 [53] Ke Xv, Shikai Guo, Hui Li, Chenchen Li, Rong Chen, Xiaochen Li, and He Jiang. 2025. Making Fault Localization in Online Service Systems More Actionable and Interpretable. ACM Trans. Softw. Eng. Methodol. 34, 6 (2025), 165:1–165:26. [54] Jian Yang, Zian Wang, Shuangwu Chen, Huasen He, Yunpeng Hou, and Xiaofeng Jiang. 2025. HG-PAD: Heterogeneous Graph Structure Learning Aided Performance Anomaly Diagnosis in Microservice Systems. IEEE Transactions on Services Computing (2025). [55] Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In International conference on learning representations, Vol. 2. [56] Boxi Yu, Jiayi Yao, Qiuai Fu, Zhiqing Zhong, Haotian Xie, Yaoliang Wu, Yuchi Ma, and Pinjia He. 2024. Deep Learning or Classical Machine Learning? An Empirical Study on Log-Based Anomaly Detection. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024. ACM, 35:1–35:13. doi:10.1145/3597503.3623308 [57] Guangba Yu, Pengfei Chen, Yufeng Li, Hongyang Chen, Xiaoyun Li, and Zibin Zheng. 2023. Nezha: Interpretable Fine-Grained Root Causes Analysis for Microservices on Multi-modal Observability Data. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023. ACM, 553–565. doi:10.1145/3611643.3616249 [58] Xiao Yu, Pallavi Joshi, Jianwu Xu, Guoliang Jin, Hui Zhang, and Guofei Jiang. [n. d.]. CloudSeer: Workflow Monitoring of Cloud Infrastructures via Interleaved Logs. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2016. 489–502. doi:10.1145/2872362.2872407 [59] Chenxi Zhang, Xin Peng, Chaofeng Sha, Ke Zhang, Zhenqing Fu, Xiya Wu, Qingwei Lin, and Dongmei Zhang. 2022. DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022. ACM, 623–634. doi:10.1145/3510003.3510180 [60] Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, Junjie Chen, Xiaoting He, Randolph Yao, Jian-Guang Lou, Murali Chintalapati, Furao Shen, and Dongmei Zhang. 2019. Robust log-based anomaly detection on unstable log data. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019. ACM, 807–817. doi:10.1145/3338906.3338931
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.
111:22
Trovato et al.
[61] Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. 2021. Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study. IEEE Trans. Software Eng. 47, 2 (2021), 243–260. doi:10.1109/TSE.2018.2887384 [62] Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Dewei Liu, Qilin Xiang, and Chuan He. 2019. Latent error prediction and fault localization for microservice applications by learning from system trace logs. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019. ACM, 683–694. doi:10.1145/3338906.3338961 [63] Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, and Michael R. Lyu. 2023. Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics. In 34th IEEE International Symposium on Software Reliability Engineering, ISSRE 2023. IEEE, 355–366. doi:10.1109/ISSRE59848.2023.00071
Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009
J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018.