Towards LLM Accelerated Rapid Reviews for Software Tool Discovery - Case for Log Anomaly Detection
arXiv:2606.16839v1 [cs.SE] 15 Jun 2026
Jesse Nyyssölä1 [0009-0006-7276-5696] , Hamza Bin Mazhar1 [0009-0009-6352-9810] , Alexander Bakhtin2 [0000-0003-3513-7253] , Matteo Esposito2 [0000-0002-8451-3668] , Nana Reinikainen1 [0009-0002-3077-3939] , Yuqing Wang1 [0000-0003-0175-005X] , Ying Song1 [0000-0001-9791-1879] , Davide Taibi3,2 [0000-0002-3210-3990] , and Mika Mäntylä1 [0000-0002-2841-5879] University of Helsinki, Fabianinkatu 33, Helsinki, Finland {jesse.nyyssola,hamza.mazhar,nana.reinikainen,yuqing.wang, ying.song,mika.mantyla}@helsinki.fi 2 University of Oulu, Pentti Kaiteran katu 1, Oulu, Finland {alexander.bakhtin,matteo.esposito}@oulu.fi 3 University of Southern Denmark, Spinderigade 24, Vejle, Denmark [email protected] 1
Abstract. In software engineering research, the primary outcome is frequently a tool. However, for practitioners and academics alike, it is hard to tell which tools are maintained and do they work out of the box. In this paper, we propose a pipeline to identify relevant studies with LLM screening, extract the tools presented in them, and run them with LLM-based coding agent. To evaluate the feasibility of our approach we focus on software log anomaly detection tools. We begin the study by designing a broad search string that yields 3233 hits from Scopus. We request two LLMs to provide an inclusion probability for each titleabstract pair according to the inclusion and exclusion criteria. From the 3233 exported abstracts, this screening reduced the number of included papers to 569, out of which we could download 470. These papers included 206 unique links and after manual evaluation we determined 83 to be tools. Finally, we ran the LLM-based coding agent on these 83 links, and got 24 successfully running tools. As replicating our approach would require roughly only 4 hours of human effort, of which 3 hours were manual PDF downloading, and 12 hours of LLM running time, this demonstrates promising efficiency when utilizing LLMs in rapid reviews. Because practitioner-built tools often lack academic papers, in the future we aim to expand our analysis to tool-hosting platforms such as GitHub and PyPI. In the future, we plan to formalize our workflow as LLM Agent Skills to make our approach easier to adopt.
2
J. Nyyssölä et al. Keywords: Rapid review · Large Language Models · Software logs · Anomaly detection · Tool discovery.
1
Introduction
In software engineering research, the primary outcome is frequently a tool [15]. However, utilizing these tools requires practitioners and academics to search, locate and execute the tools, which is a challenging task given the vast volume of academic literature. This paper introduces an LLM accelerated rapid review methodology to systematically identify and retrieve functional tools from academic publications. To understand the feasibility of such a methodology, we test our approach on the area of software log anomaly detection. To be crystal clear, we could have selected any software engineering area where tools are prominent. We chose software log anomaly detection because we possess strong expertise in this topic, while also seeking to ensure that our view of the tools in the area is comprehensive. On a more general level, we further motivate our topic selection as follows. Software logs contain information about the execution of the system which is used for monitoring, debugging, and maintenance. However, logs can contain millions of lines that has motivated hundreds of academic studies and tool proposals. Yet, practitioners have emphasized their preference for tools that can be installed in an hour, are easy to use, and can detect anomalies in five seconds [24]. Finding such tools manually can be difficult, which calls for an approach that assesses the tools in a timely manner. Rapid reviews in software engineering have been endorsed for narrow scoped problem and they are ideal for technology transfer [2]. It has been shown that practitioners are able to apply the outcomes of such reviews [33]. Thus, using rapid reviews to identify tools for log-analysis seems like a perfect fit. Recently, the use of LLMs has been investigated in systematic reviews in software engineering [13, 9, 12, 31, 36] and in other fields [21]. The consensus is that they make many correct decisions, for example in title-abstract screening [12, 21], although reliability issues remain and their performance has been found similar to that of master’s students as early as 2024 [13]. Yet, the time reduction provided by LLM-based automated paper screening makes it a promising technique for accelerating rapid reviews. Currently, it is not only humans who may use software engineering tools but also agentic AI. This further motivates our focus on tool discovery. For example, Esposito et al. [8] propose a MAPE-K and agentic AI framework that relies heavily on the continuous ingestion and evaluation of execution logs, making robust, highly scalable tooling a foundational requirement for such agentic architectures. Thus, to realize this vision, we conduct this rapid review of log anomaly detection tools to address the data processing challenges required for successfully automating microservice anomaly detection. Based on our research objectives, we formulate the following research questions:
Towards LLM Accelerated Rapid Reviews
3
– RQ1: To what extent can LLMs be used in the rapid review process for software tool discovery? – RQ2: How many tools can we identify and successfully run with our process? The first research question focuses on the process that we are trying to refine, that is, LLM accelerated rapid reviews. By addressing possible strengths or challenges, we provide a substantial academic contribution to future research using similar approaches. The second research question concerns more the output of the pipeline. The set of tools identified by the study is a contribution on its own. To disambiguate, when we use the term "tool" we refer to software repositories that implement a solution for software log anomaly detection, including replication packages of academic studies. We intentionally do not differentiate between named tools and unnamed replication packages, as the latter may contain functionally equivalent approaches that simply lack a name. In a similar way, when we refer to a repository as successfully running our focus is that it can be installed and executed to produce an output, rather than on its functional correctness or practitioner-readiness (e.g., documentation, containerization).
2
Methodology
To conduct the study, we designed a pipeline from creating the search string all the way to running the tools with an LLM. This section outlines the methodology and reasoning for our choices. To get an overview of our approach including the results found see Figure 1 and for the LLM selection in the different steps see Table 1. Further details are available in the replication package [28]. Table 1. LLM Pipeline Configuration Stage
Models Used
LLM Screen- Gemini-2.5-Flash, ing OpenAI/GPT-4.1-Mini
Rationale Past work shows an excellent balance of accuracy, speed, and token cost [12].
LLM Reposi- Agent Claude Code (Opus Currently the top-performing coding model tory Runner 4.6) for execution tasks [38].
2.1
Search string generation
In our search, we aim to use a very broad search string that yields a lot of hits in order to ensure we do not miss any studies in key word search phase. Consequently, we formulate the following search string that only focuses on software log anomaly detection. ("log anomaly" OR "log-based anomaly" OR "anomaly detection in log*" OR "log analysis") AND ("software" OR "system")
4
J. Nyyssölä et al.
Search string results n = 3233 LLM Screening (20 min)
Excluded by LLM screening n = 2754
After LLM screening n = 569 Download PDFs (3 hours)
Not downloadable n = 99
Full-text PDFs downloaded n = 470
Extract links (instant script)
This step changes the item of analysis from papers to links, so there is no exclusions.
Unique links extracted n = 206
Manual check (1 hour)
Not a suitable tool n = 123
Distinct tools identified n = 83
LLM Repository Runner
Major issues in running the tool n = 59
Successfully running tools n = 24
Fig. 1. PRISMA-style flow diagram
The search string resulted in 3233 papers as fetched on January 14th 2026 from Scopus, which is one of the largest academic databases containing titles and abstracts from every major publisher like, IEEE, ACM, Springer, Wiley, etc. 2.2
LLM abstract screening
To facilitate abstract screening with LLMs, we use the AISysRev tool [14]. It is designed to take a Scopus export as a CSV and perform screening with the given inclusion and exclusion criteria. As an output, the tool gives an estimated probability for inclusion as assessed by the chosen model(s). We used two models gemini-2.5-flash and openai/gpt-4.1-mini. As compact models, they are expected to perform with an excellent balance of accuracy, speed, and token cost. Furthermore, gpt-4.1-mini was the top performing model in large-scale study of title-abstract screening [12]. We formulate the following inclusion and exclusion criteria.
Towards LLM Accelerated Rapid Reviews
5
– IC1 Studies on "Software log analysis" – IC2 Within scope "Anomaly detection on software logs" – EC1 Exclude papers if the target is other than "Anomalies in reliability and testing" The criteria assess the title and the abstract on three key factors: Topic, scope, and target. These criteria only focus on the content of the expected paper rather than the metadata that is handled by Scopus export. We did not want to exclude secondary studies or datasets, because there is a possibility that they present updated scripts or dependencies. Following the initial LLM screening, we manually validate a stratified sample of 386 abstracts, selected equally from both the included and excluded pools. This sample size, determined using Cochran’s formula [5], provides a 95% confidence level for evaluating the accuracy of the LLM screening. Two authors evaluate each abstract, with a third author resolving any disagreements. 2.3
Tool link extraction from full papers
After title-abstract screening we proceeded to downloading the full papers. Downloading was the most labor intensive part of the whole pipeline as scraping the web page for the download link is generally forbidden in the terms of service of the providers. Our initial test with OpenAlex 4 and Unpaywall 5 APIs could only access roughly 5% of the papers from our sample pool. This forced us to manually download all papers. We assumed that any important tools mentioned in the papers would accompany a link to the tool home page or repository. We recognize that this may not always be the case; however, for this emerging work, it is a practical design limitation we imposed. For extracting the links from the PDFs, we chose not to leverage LLMs as simple pattern matching for links is very efficient and accurate with regular expressions. Specifically, we created regexes to capture links from GitHub, GitLab, Bitbucket, SourceForge, Zenodo, Figshare, PyPI, and HuggingFace. 2.4
LLM repository runner
Figure 2 shows the pipeline for the LLM repository runner. The first step of the pipeline is to clone the project, assuming it is a Git link. In case it is other link, such as Zenodo or Huggingface, the LLM agent can inherently handle the necessary downloads. In the second step, the agent scans the repository for relevant files for installation such as requirements.txt, pyproject.toml and other setup files. In case there are no installation instructions, the agent runs a grep command to find import statements. Based on the findings, the agent formulates an action plan that addresses four factors. 4 5
https://openalex.org/ https://unpaywall.org/
6
J. Nyyssölä et al. No Yes
No
Runnable? Clone repository
Scan files & determine action plan
Yes No
Create venv & install dependencies
Attempt a fix and retry
Dependency error?
Attempt to run a script
Yes
Assign grade
Success?
Fig. 2. LLM repository runner pipeline
(1) Is it runnable? To determine if the tool is runnable, the LLM agent has access to all the files in the repository. We do not define explicit rules for determining if the repository is runnable, it depends on the discretion of the LLM. This filter is intended to address obvious but perhaps unexpected issues, such as no script to run in the first place. In these cases, the repository is simply determined not runnable and the agent ends the process. (2) Data feasibility. The data on which to run the script can be bundled with the repository, the README might have a link to download it, it could be a common dataset without a link, it can be an unknown dataset without a link, or the data is not mentioned at all. The agent is instructed to handle the three first options, but the last two are left to the discretion of the agent. (3) Python version. Python is by far the most popular language used in software log anomaly detection. As such, the prompt focuses on Python related instructions, but at the same time it does not explicitly forbid trying to run nonPython tools. To determine the correct Python version, the LLM agent utilizes primarily setup or requirement files if they are available. In case the correct Python version is not mentioned, the agent attempts to deduce a suitable version based on the dependencies. (4) Requirements. Similar to the Python version, the LLM agent uses available files and information to determine how to install the dependencies. Once the action plan is created and the repository determined to be runnable, the pipeline proceeds to creation of a Python virtual environment and installing dependencies. This is done according to the action plan. Once installed, the LLM agent attempts to run an included script. In case it fails due to a dependency issue, the prompt allows a single attempt to fix it as shown in the "Attempt a fix and retry" box of Figure 2. Retrying in this means only attempting to run the script, not the whole pipeline. After that, or due to any other error, the agent is instructed to stop trying to fix the repository. After trying to run the repository script, the LLM agent is instructed to produce a result file of the outcome. It includes status of the run (success vs. failure), a grading for the installation, and a short assessment. We use the following assessment criteria: – 3 = The tool itself and instructed dependencies worked out of the box, no issues – 2 = Minor issues like dependencies not explicitly given or there was a small issue with them
Towards LLM Accelerated Rapid Reviews
7
– 1 = Significant issues (e.g., install succeeded but run failed) – 0 = Could not run at all (not runnable, fatal errors, or missing data with no fallback) We utilize Claude Code agent with Opus 4.6 model due to its performance on coding and data analysis related benchmarks [38]. It reads a pre-written prompt file that instructs all the steps outlined above. To run the setup, we created a dedicated Linux virtual machine with 32 GB of RAM and using 6 cores from Intel Core i7-13700K running on a local computer. This setup introduces limitations on what we can successfully run, specifically on GPU operations as it lacks CUDA support. However, we argue that the environment is justified given that it reflects a common use case. Furthermore, there are security considerations when installing unknown repositories and giving the LLM free rein in the system. These factors favor the use of an isolated environment.
3
Results
In the results section we outline the key findings of each step in our proposed pipeline. Figure 1 shows how the number of included items reduces drastically on each step. Additionally, it contains a rough estimate of how long each step takes to run or perform when assuming that all the data and tools are ready making it essentially reflect the time-cost of re-running our pipeline.
3.1
LLM title-abstract screening outcomes
Figure 3 shows the average probability of the two models when the papers are sorted by the average probability. This illustrates well the "elbow" [34] where the probability starts rapidly declining. This is a good indicator of a cut-off threshold for inclusion. We selected the start of the elbow at 0.90 probability as the threshold for inclusion which resulted in 569 included papers out of 3233. Human evaluation validates the LLM screening and our "elbow" cutoff point selection. The Cohen’s Kappa was 0.839 between the human consensus and the LLM prediction. This signifies almost perfect agreement on Landis & Koch scale [18]. When considering the human consensus as the ground-truth, the LLM screening reached precision of 0.933, recall of 0.909 and F1-score of 0.921. For our purpose in conducting a rapid review for tool discovery, these are excellent results that clearly meet our expectations as the results are in the top 5% when compared against SESR-eval results [12]. We think that one reason for such strong results is the data and the criteria, which together form a highly separable set of papers. Many papers are clear includes, even more are clear excludes, and relatively few fall into the ambiguous middle ground. This is evident in Figure 3, where only about 400 papers are in the middle with probabilities between 0.25 and 0.9.
8
J. Nyyssölä et al. 1.0
Average Probability
0.8
0.6
0.4
0.2
0.0 0
500
1000
1500
2000
2500
3000
Papers (sorted by avg. probability)
Fig. 3. Average inclusion probability of the papers
3.2
Extracted tools from full papers
From the 569 papers that passed the LLM screening we could download 470. They included 315 links that matched our regexes. There were 206 unique links and 109 duplicates. The most common links in the studies were the repositories for Loghub [42] (33 hits), Loglizer [10] (12 hits), and Logdeep 6 (10 hits). We estimated 206 links to be a feasible number for manual assessment to determine if the link contains a suitable tool for our study. What we mean by suitability is that, the link actually leads to a tool for software log anomaly detection. In other words, in this step we exclude all non-relevant tools as well as supplementary links such as datasets without tools. Additionally, we excluded forks. The final number of unique links to suitable repositories was 83. 3.3
LLM repository runner execution
Finally, we ran the LLM repository runner as described in Section 2.4. This section starts by introducing performance of the installation process, i.e., how successful it is. In addition, we provide an overview on the execution time distribution across the repositories. Install performance The number of repositories of each grade is summarized in Table 2. As shown in the table, the grades 0, 1 and 2 are almost evenly split with most issues caused by dependencies. Unless there are other problems, the extent of the dependency issues appeared to explain the grade to a large degree. 6
https://github.com/d0ng1ee/logdeep The Logdeep repository does not have an associated paper; rather, it is a repository for running some popular approaches.
Towards LLM Accelerated Rapid Reviews
9
Table 2. Repository install grade distribution and common issues per grade Grade # of repos Grade 3 1 Grade 2 23 Grade 1 22 Grade 0 22 Timed out 15 Total 83
Common issues No issues Minor dependency issue Missing dependencies, bugs, API incompatibilities Repository not available, unresolvable dependencies CPU-bound training, download or processing pipelines
The grade 3 repository was LogLead [26]. There were 23 repositories that reached grade 2: aecid-alert-aggregation [17], anomaly-detection-log-datasets [16], BGL_AnoDet [29], BigLog [35], CAT [39], deep-loglizer [3], DeepLog [7], DistilLog [27], FedLAD [20], KnowLog [23], LL-mod-unsupervised [30], LogDeep, LogDLR [41], LogEval [6], LoFI [11], LogGeneratorForAnomalyDetection [37], Loglizer [10], LogRAG [40], Lograph [4], MoLFI [25], NeuralLog [19], OnlineADPipeline [22], and TransSentLog [32]. Note that while the link extraction does not discriminate between primary and secondary studies, this list with its citations was manually collected to try to credit the original authors. Table 2 shows that 15 repositories exhausted the 20-minutes execution budget. Most timeouts were attributed to CPU-bound training that progressed too slowly to finish within the budget. Additional timeouts occurred due to prolonged dependency resolutions, heavy dataset downloads, or preprocessing pipelines. In few repositories, the agent could not resolve issues related to external datasets. All repositories in grade 0 did not execute at all. Most of the repositories were not available, they had either been deleted, were private, or were completely empty. Some had hard unresolvable dependencies like requiring CUDA or external API calls with no alternatives. Some had compatibility issues for libraries and Python versions, while a very few had unresolvable dataset availability issues like missing or proprietary datasets, or broken links. In the 83 repositories, there was only one non-Python project. It depended on Java and got grade 0 because the LLM agent could not install Java. All repositories in grade 1 showed issues that led to partial or failed executions despite heavy agent intervention. Nearly half of the repositories in this grade lacked any dependency specifications, while the other half had outdated or incomplete specifications. A majority of the repositories also contained code bugs, API incompatibilities, or broken imports. In several cases there were hardcoded CUDA calls with no CPU fallbacks. There were also issues with datasets and preprocessing scripts. All repositories in grade 2 required minor agent intervention to become runnable. A vast majority of repositories in this grade had incomplete or incorrect dependency specifications, with some repositories completely missing the specifications. For the missing dependencies, the agent had to infer these from import statements within the code. Some repositories required external data or downloading pre-trained model weights adding to the setup overhead. A couple
10
J. Nyyssölä et al.
of repositories in this grade had hardcoded CUDA device references which had to be changed to CPU to allow execution. Despite these issues, all repositories produced meaningful output after low-effort troubleshooting from the agent. LogLead was the only repository that installed and ran out of the box. One reason that explains LogLead’s performance is that it has been maintained after publication to support log analysis research and students who use it in a course. Execution times Running the script for all repositories took approximately 12 hours. Figure 4 shows the distribution of execution times as color-coded by the grade of the repository. As expected, we can observe that the majority grade 0 repositories fail in two minutes. This is due to the LLM agent determining early, as instructed in the prompt, that the repository is unrunnable, which ends the process immediately. If running the repository succeeds in five minutes, most of them have grade 2. Conversely, if it succeeds while taking more than five minutes, the majority switch to grade 1. This is logical, as the LLM agent tries to solve issues within the repository, which both takes time and reduces the grading.
Repositories
20
Grade 3 Grade 2 Grade 1
Grade 0 Timed out
15 20
20 (timeout)
15 10 5 0
<2
25
5 10
10 15
Execution time (min)
Fig. 4. Distribution of the execution times of the repositories
4
Discussion
The LLM worked surprisingly well in various parts of our pipeline. The result of the abstract screening with LLMs was excellent. Trying to get unknown repositories to run by humans is very time-intensive and exhausting, to the point that studies do not do it to this degree. As such, utilizing the LLM can be seen as improving the quality of the review process as a whole. To address RQ1, we argue that the use of LLMs improved both the quality and efficiency of the rapid review process for software tool discovery. For RQ2, we presented a list of 24 repositories that received an installation grade of 2 or 3 by the LLM. Although
Towards LLM Accelerated Rapid Reviews
11
any qualitative assessment of the tools was outside the scope of this paper, the running tools offer a good starting point for anyone who wants to further assess them or integrate them into agentic frameworks such as MAPE-K [8]. One of the main contributions of the LLM-accelerated rapid review pipeline that we propose is the amount of human effort that is automated. A traditional rapid review of a similar scope would take from days to weeks, but our approach took only around four hours of human effort and 12 hours of running LLMs. Furthermore, majority of the human effort consists of manually downloading PDFs of the papers. While many major publishers provide Text and Data Mining (TDM) API keys for non-commercial research, integrating these into the pipeline would require managing multiple publisher-specific authentication workflows, which would reduce the portability and reproducibility of our approach. Manual downloading is labour-intensive, but it keeps the pipeline self-contained and the results consistent. For scaling this approach to larger reviews, TDM agreements would become justified. Regarding the LLM run time, it is important to note that it took 12 hours to run the 83 repositories sequentially, but outside of rate limits, there is nothing stopping from running several LLM agents in parallel. The 20minute timeout was chosen both to reflect a realistic practitioner installation scenario [24] and to prevent the agent from entering unproductive loops wasting tokens. For further insight for this study, we ran the 15 repositories that timed out again without a time limit and four of them ran well (grade 2), which is a similar ratio as in those that did not time out. One of our primary metrics for evaluating a tool is whether it executes successfully. This approach may exclude good methods that have poor replication packages. However, given that replicability is a hallmark of science, ensuring the tool runs remains the responsibility of the original authors, not ours. To characterize what predicts successful execution, we collected GitHub metadata for the 49 CPU-compatible repositories and tested associations with grade outcome. Due to the number of repositories, there were no statistically significant results, but some of them were practically meaningful. For example, repositories that included a dedicated requirements file ran successfully at a higher rate (61%) than those without one (38%). Some factors (README length, number of stars, and number of forks), showed a weak positive trend toward successful execution. Repository age and days since last commit did not predict success.
5
Threats to Validity
One notable threat to Internal validity is that it is not always clear whether a failure by the LLM runner is caused by the repository or the agent/prompt. Hence, we risk conflating two different things: How good is the repository and how good is the LLM-agent. Another threat is that we conducted the manual assessment of the links only by a single author which is inconsistent with the paper screening validation, but we reason that it is a much more straightforward process with no controversy.
12
J. Nyyssölä et al.
The most obvious threat for External validity is that the pipeline was only tested on log anomaly detection tools, and generalization to other SE tool areas is unconfirmed. Another limiting factor is the CPU-only environment where we ran the experiments, and as such we know for certain that the results do not generalize for GPU environments. For better generalizability, using LLM-agents from other providers would also be beneficial. A third threat to external validity, is the potential for evidence loss, i.e., the tools we should have identified. Each instance where a relevant tool is discovered outside our initial search should prompt a critical examination of the reasons, context, and process failures that led to its omission. Another limiting factor is that our pipeline was designed around Python repositories with local dependencies. This does not account for tools which may require Docker containers or cloud infrastructure. However, this is less problematic because research prototypes are often distributed as self contained repositories not utilizing cloud infrastructure, while industrial sources that utilize such infrastructure often do not make their implementation public. For Construct validity, the main threat is that we assume the tool to be "usable" if the LLM-agent manages to run it. In reality, the usability for humans and LLM-agents differ significantly. For example, for a human adjusting vast configuration files would be very laborious while an LLM could manage it in an instant. Conversely, graphical user interfaces are designed to be intuitive for humans, but an LLM-agent might not be able to run such tools at all. There exist threats to Conclusion validity with regards to how the results of this study should be interpreted. They should not be read as saying that only 24 out of 569 log anomaly detection papers provide a good replication package. There are studies in the set that fall outside the scope of tool-oriented research. Additionally, many deep learning based approaches are hardware or time intensive by design. Rather, the 24 tools in 569 papers showcase that in the context of trying to search an easy and fast tool, there are not that many options. However, in this study, we can not guarantee that the running tools produced the correct, expected output. Hence, in terms of output quality of the tools, this study does not intend to place one tool above another.
6
Future Work
As the work in this study is highly novel, we will outline future work as an explicit outcome that relates to the limitations of this study which are summarized in Table 3. The limitations themselves are motivated by the threats to validity presented in the previous section. Limitation #1: Many significant log analysis tools, such as Angle Grinder7 and Logfile Navigator8 , are not covered in academic literature and thus fell outside the scope of our rapid review search. Future work: Extend our approach beyond academic literature to include public hosting platforms (e.g., GitHub) 7 8
https://github.com/rcoh/angle-grinder https://lnav.org
Towards LLM Accelerated Rapid Reviews
13
Table 3. Summary of future work by limitation # Limitation
Future Work
1
Tools absent from academic literature Include additional sources, such as (e.g., Angle Grinder) not found. GitHub.
2
Generalizability: Only evaluated on Expand to metrics (e.g., CPU, memlog anomaly detection tools. ory) and tracing tools.
3
Single OS, limited GPU support, and Aim for more robust environments, a 20-minute execution timeout. develop dynamic timeout.
4
Issue location: Repository or LLM Study LLM runner performance on agent? dataset with groundtruth (e.g., artifact badges).
and grey literature. Combining different sources can help produce a more comprehensive picture than running them individually, and the expected overlap between sources serves as a natural cross-validation of the findings. Limitation #2: At this stage we have tested our approach only on tools for software log anomaly detection. Whether our work generalizes to all software tools requires further study. Future work: We aim to expand the method to other software monitoring tool areas, such as those that analyze metrics (e.g., CPU and memory usage) and those that analyze execution traces from modern microservice systems. Furthermore, we aim to increase the generalizability of the repository runner by formalizing the workflow as Agent Skills. Limitation #3: Limited running environment with one operating system, limited GPU support, and the 20-minute execution timeout. Future work: We will explore options to run the repositories in a more robust environment without compromising on information security. Furthermore, the timeout window could be made dynamic if we can identify when the extra time is really necessary (running the tool) as opposed to the LLM agent getting stuck. However, given that practitioners value both ease of use and timely installation [24], the constraints in this study reflect a realistic practitioner scenario, too. Limitation #4: While we outlined the several reasons for the LLM runner to fail, it is not always clear if the fundamental issue is in the repository or the agent/prompt. Future work: Gather a dataset of studies that have artifact review badges. These can serve as a ground truth so that we can expect the artifact to be runnable. If the LLM still fails, we have a good reason to believe the pipeline is not performing adequately.
7
Conclusion
This paper presents an LLM accelerated pipeline for conducting rapid reviews for software tool discovery. We tested our approach on software log anomaly detection tools but we could have chosen any other software engineering area
14
J. Nyyssölä et al.
where tools are a typical outcome. We started with a key-word search finding 3233 papers. With the help of LLM abstract screening, regex based link extraction, and finally manual evaluation, we got a list of 83 software tools for log anomaly detection. With the prompt we designed, LLM-based coding agent could eventually run 24 of those with little to no issues. In the future, we plan to apply our approach to investigate other software monitoring tools. This will provide more evidence of our approach and help make the pipeline more robust. Furthermore, we will extend our approach beyond academic literature for a more comprehensive set of tools. We also plan to formalize our approach as Agent Skills which is an emerging standard for packaging reusable capabilities for LLM agents, as specified in a standard originally developed by Anthropic [1]. Data Availability. Data is available via our replication package [28]. Acknowledgments. This work has been funded by the Research Council of Finland (grants n. 359861 and 349488 – MuFAno), and by the Finnish Software Engineering Doctoral Research Network, funded by the Ministry of Education and Culture, Finland. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.
References 1. Anthropic, Agent Skills Contributors: Agent skills: A standardized way to give AI agents new capabilities and expertise. https://agentskills.io (2025) 2. Cartaxo, B., Pinto, G., Soares, S.: Rapid Reviews in Software Engineering, pp. 357– 384. Springer International Publishing, Cham (2020). https://doi.org/10.1007/9783-030-32489-6_13 3. Chen, Z., Liu, J., Gu, W., Su, Y., Lyu, M.R.: Experience report: Deep learningbased system log analysis for anomaly detection (2022), https://arxiv.org/abs/ 2107.05908 4. Chu, G., Wang, J., Qi, Q., Sun, H., Zhuang, Z., He, B., Jing, Y., Zhang, L., Liao, J.: Anomaly detection on interleaved log data with semantic association mining on log-entity graph. IEEE Transactions on Software Engineering 51(2), 581–594 (2025). https://doi.org/10.1109/TSE.2025.3527856 5. Cochran, W.G.: Sampling Techniques. John Wiley & Sons, New York, 3rd edn. (1977) 6. Cui, T., Ma, S., Chen, Z., Xiao, T., Zhao, C., Tao, S., Liu, Y., Zhang, S., Lin, D., Liu, C., Cai, Y., Meng, W., Sun, Y., Pei, D.: LogEval: A comprehensive benchmark suite for LLMs in log analysis. Empirical Softw. Engg. 30(6) (Oct 2025). https://doi.org/10.1007/s10664-025-10701-6 7. Du, M., Li, F., Zheng, G., Srikumar, V.: DeepLog: Anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. p. 1285–1298. CCS ’17, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3133956.3134015
Towards LLM Accelerated Rapid Reviews
15
8. Esposito, M., Bakhtin, A., Ahmad, N., Robredo, M., Su, R., Lenarduzzi, V., Taibi, D.: Autonomic microservice management via agentic AI and MAPE-K integration. In: Bianculli, D., Sartaj, H., Andrikopoulos, V., Pautasso, C., Mikkonen, T., Perez, J., Bureš, T., De Sanctis, M., Muccini, H., Navarro, E., Soliman, M., Zdun, U. (eds.) Software Architecture. ECSA 2025 Tracks and Workshops. pp. 105–118. Springer Nature Switzerland, Cham (2026). https://doi.org/10.1007/978-3-032-04403-7_11 9. Felizardo, K.R., Lima, M.S., Deizepe, A., Conte, T.U., Steinmacher, I.: ChatGPT application in Systematic Literature Reviews in Software Engineering: An evaluation of its accuracy to support the selection activity. In: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. pp. 25–36. ESEM ’24, Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3674805.3686666 10. He, S., Zhu, J., He, P., Lyu, M.R.: Experience report: System log analysis for anomaly detection. In: 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). pp. 207–218 (2016). https://doi.org/10.1109/ISSRE.2016.21 11. Huang, J., Jiang, Z., Liu, J., Huo, Y., Gu, J., Chen, Z., Feng, C., Dong, H., Yang, Z., Lyu, M.R.: Demystifying and extracting fault-indicating information from logs for failure diagnosis. In: 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE). pp. 511–522 (2024). https://doi.org/10.1109/ISSRE62328.2024.00055 12. Huotala, A., Kuutila, M., Mäntylä, M.: SESR-Eval: Dataset for evaluating LLMs in the title-abstract screening of systematic reviews. In: 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). pp. 01–12 (2025). https://doi.org/10.1109/ESEM64174.2025.00053 13. Huotala, A., Kuutila, M., Ralph, P., Mäntylä, M.: The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. pp. 262–271. EASE ’24, Association for Computing Machinery, New York, NY, USA (July 18, 2024). https://doi.org/10.1145/3661167.3661172 14. Huotala, A., Kuutila, M., Turtio, O.P., Sipilä, S., Mäntylä, M.: AISysRev – LLMbased tool for title-abstract screening. 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 05–09, 2026, Montreal, QC, Canada pp. 1–5 (2026), https://arxiv.org/abs/2510.06708 15. Kotti, Z., Gousios, G., Spinellis, D.: Impact of software engineering research in practice: A patent and author survey analysis. IEEE Transactions on Software Engineering 49(4), 2020–2038 (2023). https://doi.org/10.1109/TSE.2022.3208210 16. Landauer, M., Skopik, F., Wurzenberger, M.: A critical review of common log data sets used for evaluation of sequence-based anomaly detection techniques. Proc. ACM Softw. Eng. 1(FSE) (Jul 2024). https://doi.org/10.1145/3660768 17. Landauer, M., Wurzenberger, M., Skopik, F., Hotwagner, W., Höld, G.: AMiner: A modular log data analysis pipeline for anomaly-based intrusion detection. Digital Threats 4(1) (Mar 2023). https://doi.org/10.1145/3567675 18. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977), http://www.jstor.org/stable/2529310 19. Le, V.H., Zhang, H.: Log-based anomaly detection without log parsing. In: Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering. p. 492–504. ASE ’21, IEEE Press (2022). https://doi.org/10.1109/ASE51524.2021.9678773
16
J. Nyyssölä et al.
20. Liao, Y., Keung, J., Mao, Z., Zhang, J., Li, J.: FedLAD: A modular and adaptive testbed for federated log anomaly detection. In: 2025 IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOSC). pp. 227–232. IEEE Computer Society, Los Alamitos, CA, USA (Oct 2025). https://doi.org/10.1109/ACSOS-C66519.2025.00066 21. Lieberum, J.L., Toews, M., Metzendorf, M.I., Heilmeyer, F., Siemens, W., Haverkamp, C., Böhringer, D., Meerpohl, J.J., Eisele-Metzger, A.: Large language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review. Journal of Clinical Epidemiology 181 (2025). https://doi.org/10.1016/j.jclinepi.2025.111746 22. Lupton, S., Yu, L., Washizaki, H., Yoshioka, N., Fukazawa, Y.: Assessment of real-world incident detection through a component-based online log anomaly detection pipeline framework. In: 2023 10th International Conference on Dependable Systems and Their Applications (DSA). pp. 477–478 (2023). https://doi.org/10.1109/DSA59317.2023.00066 23. Ma, L., Yang, W., Xu, B., Jiang, S., Fei, B., Liang, J., Zhou, M., Xiao, Y.: KnowLog: Knowledge enhanced pre-trained language model for log understanding. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. ICSE ’24, Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3597503.3623304 24. Ma, X., Li, Y., Keung, J., Yu, X., Zou, H., Yang, Z., Sarro, F., Barr, E.T.: Practitioners’ expectations on log anomaly detection. IEEE Transactions on Software Engineering 51(9), 2455–2471 (2025). https://doi.org/10.1109/TSE.2025.3586700 25. Messaoudi, S., Panichella, A., Bianculli, D., Briand, L., Sasnauskas, R.: A search-based approach for accurate identification of log message formats. In: Proceedings of the 26th Conference on Program Comprehension. p. 167–177. ICPC ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3196321.3196340 26. Mäntylä, M.V., Wang, Y., Nyyssölä, J.: LogLead - fast and integrated log loader, enhancer, and anomaly detector. In: 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). pp. 395–399 (2024). https://doi.org/10.1109/SANER60148.2024.00046 27. Nguyen, H.T., Nguyen, L.V., Le, V.H., Zhang, H., Le, M.T.: Efficient log-based anomaly detection with knowledge distillation. In: 2024 IEEE International Conference on Web Services (ICWS). pp. 578–589 (2024). https://doi.org/10.1109/ICWS62655.2024.00078 28. Nyyssölä, J., Mazhar, H.B., Bakhtin, A., Esposito, M., Reinikainen, N., Wang, Y., Song, Y., Taibi, D., Mäntylä, M.: Replication package for "Towards LLM accelerated rapid reviews for software tool discovery - case for log anomaly detection" (Apr 2026). https://doi.org/10.5281/zenodo.19878559 29. Nyyssölä, J., Mäntylä, M.: Event-level anomaly detection on software logs: Role of algorithm, threshold, and window size. In: 2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS). pp. 649–656 (2024). https://doi.org/10.1109/QRS62785.2024.00070 30. Nyyssölä, J., Mäntylä, M.: Speed and performance of parserless and unsupervised anomaly detection methods on software logs. In: 2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS). pp. 657–666 (2024). https://doi.org/10.1109/QRS62785.2024.00071 31. Petersen, K., Gerken, J.M.: On the road to interactive LLM-based systematic mapping studies. Information and Software Technology 178, 107611 (2025). https://doi.org/10.1016/j.infsof.2024.107611
Towards LLM Accelerated Rapid Reviews
17
32. Pham, T.A., Lee, J.H.: TransSentLog: Interpretable anomaly detection using transformer and sentiment analysis on individual log event. IEEE Access PP, 1–1 (01 2023). https://doi.org/10.1109/ACCESS.2023.3311146 33. Pizard, S., Lezama, J., García, R., Vallespir, D., Kitchenham, B.: Using rapid reviews to support software engineering practice: a systematic review and a replication study. Empirical Softw. Engg. 30(1) (Oct 2024). https://doi.org/10.1007/s10664-024-10545-6 34. Syakur, M.A., Khotimah, B.K., Rochman, E.M.S., Satoto, B.D.: Integration kmeans clustering method and elbow method for identification of the best customer profile cluster. IOP Conference Series: Materials Science and Engineering 336(1), 012017 (apr 2018). https://doi.org/10.1088/1757-899X/336/1/012017 35. Tao, S., Liu, Y., Meng, W., Ren, Z., Yang, H., Chen, X., Zhang, L., Xie, Y., Su, C., Oiao, X., Tian, W., Zhu, Y., Han, T., Qin, Y., Li, Y.: Biglog: Unsupervised large-scale pre-training for a unified log representation. In: 2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS). pp. 1–11 (2023). https://doi.org/10.1109/IWQoS57198.2023.10188759 36. Thode, L., Iftikhar, U., Mendez, D.: Exploring the use of LLMs for the selection phase in systematic literature studies. Information and Software Technology p. 107757 (2025). https://doi.org/10.1016/j.infsof.2025.107757 37. Uchida, H., Tominaga, K., Itai, H., Li, Y., Nakatoh, Y.: Multi-parameter log anomaly detection with an unsupervised learning approach. In: 2024 International Symposium on Parallel Computing and Distributed Systems (PCDS). pp. 1–5 (2024). https://doi.org/10.1109/PCDS61776.2024.10743635 38. White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., Shubh-Agrawal, Sandha, S.S., Naidu, S.V., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W., Goldblum, M.: LiveBench: A challenging, contamination-free LLM benchmark. In: The Thirteenth International Conference on Learning Representations (2025) 39. Zhang, S., Liu, Y., Zhang, X., Cheng, W., Chen, H., Xiong, H.: CAT: Beyond efficient transformer for content-aware anomaly detection in event sequences. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. p. 4541–4550. KDD ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3534678.3539155 40. Zhang, W., Zhang, Q., Yu, E., Ren, Y., Meng, Y., Qiu, M., Wang, J.: Leveraging RAG-Enhanced large language model for semi-supervised log anomaly detection. In: 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE). pp. 168–179 (2024). https://doi.org/10.1109/ISSRE62328.2024.00026 41. Zhou, J., Ying, S., Wang, S., Zhao, D., Xiang, J., Liang, K., Liu, P.: LogDLR: Unsupervised cross-system log anomaly detection through domain-invariant latent representation. IEEE Transactions on Dependable and Secure Computing 22(4), 4456–4471 (2025). https://doi.org/10.1109/TDSC.2025.3548050 42. Zhu, J., He, S., He, P., Liu, J., Lyu, M.R.: Loghub: A large collection of system log datasets for AI-driven log analytics. In: 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). pp. 355–366 (2023). https://doi.org/10.1109/ISSRE59848.2023.00071