Assessing REST API Test Generation Strategies with Log Coverage Nana Reinikainen
Mika Mäntylä
Yuqing Wang
Department of Computer Science University of Helsinki Helsinki, Finland [email protected]
Department of Computer Science University of Helsinki Helsinki, Finland [email protected]
Department of Computer Science University of Helsinki Helsinki, Finland [email protected]
arXiv:2604.07073v1 [cs.SE] 8 Apr 2026
Abstract Assessing the effectiveness of REST API tests in black-box settings can be challenging due to the lack of access to source code coverage metrics and polyglot tech stack. We propose three metrics for capturing average, minimum, and maximum log coverage to handle the diverse test generation results and runtime behaviors over multiple runs. Using log coverage, we empirically evaluate three REST API test generation strategies, Evolutionary computing (EvoMaster v5.0.2), LLMs (Claude Opus 4.6 and GPT-5.2-Codex), and human-written Locust load tests, on Light-OAuth2 authorization microservice system. On average, Claude Opus 4.6 tests uncover 28.4% more unique log templates than human-written tests, whereas EvoMaster and GPT-5.2-Codex find 26.1% and 38.6% fewer, respectively. Next, we analyze combined log coverage to assess complementarity between strategies. Combining human-written tests with Claude Opus 4.6 tests increases total observed log coverage by 78.4% and 38.9% in human-written and Claude tests respectively. When combining Locust tests with EvoMaster the same increases are 30.7% and 76.9% and when using GPT-5.2-Codex 26.1% and 105.6%. This means that the generation strategies exercise largely distinct runtime behaviors. Our future work includes extending our study to multiple systems.
CCS Concepts • Software and its engineering → Software testing and debugging.
Keywords REST API Testing, Log Coverage, Test Generation ACM Reference Format: Nana Reinikainen, Mika Mäntylä, and Yuqing Wang. 2026. Assessing REST API Test Generation Strategies with Log Coverage. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 7 pages. https://doi. org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
The growing adoption of microservice architecture and polyglot tech stacks in modern web software development has increased the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
need for efficient and effective testing of REST APIs. For REST API testing, multiple tools exists [3, 4, 13, 16, 18, 21] and LLM based test generation has been proposed [15, 19]. In black-box REST API testing, test evaluation is typically based on metrics such as request counts, response status codes, and endpoint coverage. While these metrics reflect external API-level behavior, they provide limited insight into the diversity of runtime behaviors exercised by a test suite. Code coverage is a widely used metric in software testing to determine how extensively a system has been tested. However, computing code coverage often requires source code or bytecode instrumentation, which is often impractical in microservice environments. Source code may not be accessible, instrumentation can introduce execution overhead, and deploying code coverage tools can expose engineering challenges [7]. To address this limitation, we propose log coverage as a coverage metric in black-box REST API testing. Logs are widely available in modern systems and provide observable runtime information. We define log coverage as the number of distinct log templates, observed during test execution, where a log template represents a unique structured log message pattern produced by, for example, a log parser such as Drain [14]. We consider each distinct log template as an indicator of differentiated runtime behavior, as log statements are typically associated with specific execution scenarios, system states, or processing branches, making them a reasonable proxy for behavioral diversity. Under this definition, log coverage quantifies the diversity of runtime behaviors exercised by a test suite. Importantly, log coverage does not require code instrumentation or system modification, making it particularly suitable for black-box environments. We have found that the use of execution logs in software testing is quite rare, even though the concept was first proposed at least in the 1990s. The early work by Andrews [2] introduces log file machines, i.e. state machines constructed from log events, which are used as test oracles and to support coverage-driven testing. Prior work of Chen et al. [7] and Xu et al. [22] explored the relation of log coverage and code coverage, demonstrating that execution logs can be used as a proxy for code coverage. This suggests that log coverage can serve as a viable substitute for code coverage in scenarios where source code is unavailable or cannot be instrumented. Tian et al. [20] proposes a method for automatically constructing Markov usage models and test cases based on logged user activity to test web service reliability. Chen et al. [8] uses logs to automatically recover workloads for load testing. Yu et al. [23] introduce a log-based test case prioritization framework and implement seven techniques that combine log representations, including count-based, orderingbased, and semantics-based, and three established prioritization strategies. This demonstrates that logs can serve as behavioral signals for guiding prioritization. We found only two papers focusing
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
an using logs in microservice system context both focusing on regression testing [9, 10]. Work by Chen et al. [9] focused on regression test selection, while Della Corte et al. [10] trained deep learning model that used execution logs to improve regression test suites. Finally, we found no papers where logs would have been used to evaluate test generation done by LLMs. Given that LLM outputs in software testing require constant monitoring, Harman et al. [11, 12] advocate for the development of “assured LLMs,” models whose outputs are accompanied by verifiable claims of utility. We argue that software logs provide an excellent way to ensure the utility of LLM-created tests when metrics such as code coverage are not usable. To assess the applicability of log coverage, we evaluate it with three representative REST API test generation strategies on a small industrial microservice system called Light-OAuth2, which implements the OAuth2 specification and consists of seven services. The strategies include: (i) EvoMaster, a search-based test generation tool using OpenAPI specification, represents the state-of-the-art in automated coverage-oriented API testing; (ii) LLM test generation, an emerging approach that derives tests from semantic understanding of OpenAPI specifications leveraging latest LLMs (Claude Opus 4.6 and GPT-5.2-Codex) to generate test cases; and (iii) manually written Locust load tests, serving as a practitioner-designed baseline. For our evaluation, we define these research questions: • RQ1: How does log coverage differ across black-box API test generation strategies? • RQ2: Do different strategies reveal complementary runtime behaviors as measured by combined log coverage? To answer RQ1, we use log coverage as a relative estimate of behavioral exploration achieved by different test suites. By comparing the number of unique log templates discovered by each strategy, we assess their relative ability to exercise system behaviors. To answer RQ2, we use combined log coverage to assess whether different strategies reveal distinct runtime behaviors. The overlap in discovered unique log templates indicates the extent of shared behavioral exploration, where lower overlap suggests higher complementarity between strategies. Our results indicate that LLM-generated tests by Claude Opus 4.6 discover nearly 30% more unique log templates than the humanwritten Locust baseline, while EvoMaster-generated tests discover approximately 27% fewer. Combining human-written tests with LLM-generated tests substantially increases overall coverage, demonstrating that the strategies exercise partially distinct runtime behaviors. Although our study is limited to a single system, it provides empirical evidence on the usefulness of log coverage for assessing and comparing black-box REST API test generation strategies. This paper makes the following contributions: (1) We propose log coverage, a log-based metric for approximating coverage in black-box API testing. (2) We empirically compare three API test generation strategies using log coverage. (3) We analyze the complementarity of these strategies through combined log coverage.
Reinikainen et al.
2
Methodology
In this Section, we describe the methodology used for our empirical experiments, including a description of the system under test, the API test generation strategies under evaluation, the experimental setup, and approaches used to assess complementary comparison.
2.1
System Under Test
The system under test (SUT) in this study is Light-OAuth21 , an open-source microservice implementation of the OAuth2 authentication and authorization specification developed by networknt. The system consists of seven independent services that communicate via REST APIs and support several database options. The available service API endpoints cover operations such as user login, user registration, service registration, client registration, issuing an access token, and public key certificate distribution. In addition, the system includes the OpenAPI specifications for each of the services. These specifications define endpoint definitions, request schemas, and parameter constraints used for test generation.
2.2
Test Generation Strategies
2.2.1 Locust. Locust 2 is an open-source load testing framework for HTTP and other protocols. It provides an easy and flexible interface for defining basic create, read, update, and delete operations and also more complex use cases of the SUT. In Locust, functions that contain API calls are defined as tasks. At the start of test execution, Locust spawns a virtual user population, where each repeatedly picks a random task to execute, sleeps for a predefined time, and picks a new task. Tasks can have weights, allowing tasks with a higher weight to be executed more frequently. Locust also supports scaling the number of simultaneous users up to millions, enabling testing with a realistic load. For the Locust-based strategy, we use the manually written Locust test suite from the LO2 study that tested Light-OAuth2 system extensively through multiple positive and negative test cases [5]. The test suite was designed to achieve broad API coverage and to enable the collection of execution logs during test execution under both correct and erroneous API usage scenarios. The authors manually constructed the test suite according to the Light-OAuth2 system’s official API documentation and OpenAPI specification, covering standard authorization flows and CRUD operations. Due to their different execution setup and concurrency characteristics, Locust tests serve as a reference baseline rather than a directly controlled experimental condition. 2.2.2 EvoMaster. EvoMaster 3 is an open-source search-based API test generation tool [3]. We used the latest version of EvoMaster in black-box mode to automatically generate API test suites for SUT from its OpenAPI specification. In this black-box mode, EvoMaster applies evolutionary search techniques guided by observable API responses to explore API behaviors and increase endpoint coverage. The test generation process is formulated as a search problem, where the goal is to produce a test set that maximizes coverage while minimizing the number of test cases. The search process is 1 https://doc.networknt.com/service/oauth/ 2 https://locust.io/ 3 https://github.com/WebFuzzing/EvoMaster
Assessing REST API Test Generation Strategies with Log Coverage
guided by fitness functions that guide the exploration toward more comprehensive and efficient tests. Test cases are evolved from an initial population of randomly generated candidates and iteratively refined according to their fitness scores. The inputs of these test cases are derived from the SUT’s API specifications, which must be accurate and up to date to enable successful test generation. The search process terminates when a predefined stopping criterion is met, for example, execution time. The resulting test cases are self-contained and can be executed independently or composed into a full test suite. 2.2.3 LLMs (Claude Opus 4.6, GPT-5.2-Codex). We adopt an LLMbased strategy using two state-of-the-art models, Claude Opus 4.6 and GPT-5.2-Codex. We prompt these LLMs to produce executable API test cases according to the OpenAPI specifications of the SUT. This strategy leverages the generative capabilities of pretrained LLMs to synthesize test cases directly from specifications for a black-box setting. We access both models as out-of-the-box deployments via OpenRouter, without parameter tuning, model adaptation, or task-specific fine-tuning. We use the default configuration, including a fixed temperature of 1.0, which introduces controlled stochasticity into the generation process. Although LLMs have recently attracted attention in software engineering research, relatively few studies [15, 19]. have evaluated their use for API test generation directly from OpenAPI specifications in a black-box setting. Our approach therefore investigates LLM-based test generation as an emerging strategy that relies solely on publicly available API specifications, without access to source code or internal instrumentation.
2.3
Experiment Setup
To enable a fair comparison, we evaluate log coverage across three test generation strategies under consistent experimental conditions, controlling factors unrelated to the generation strategy itself. 2.3.1 Test suite generation. We generated test suites using the OpenAPI specification of the SUT as input for all automated approaches. We slightly modified the OpenAPI specification for the Client, Service, and User services by adding missing success status codes and adjusting required parameters for object creation to match the actual service behavior. For EvoMaster, we executed it in black-box mode with test generation budget of 70 seconds per service. We empirically determined that this time budget was sufficient to produce stable and representative test suites for Light-OAuth2. Increasing the time budget did not consistently yield better results. For the LLM-based strategy, we did not impose an explicit time limit. Instead, we recorded generation times and verified that they remained within a comparable range to those of EvoMaster (see Table 1). We applied the same test generation workflow to both LLMs to ensure a controlled comparison as follows. We first let LLMs generate an initial test suite based on the OpenAPI specification and execute it against the SUT. We allowed LLMs to refine failing tests once by feeding the test set along with the specification and the test execution report. The output formed the final test set. Finally, we manually performed a lightweight quality control step to ensure, e.g., that tests use assertions in a meaningful way and
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
not just increase coverage. Tests that did not pass the check were discarded. We designed the prompts for LLM-based generation using a structured six-component format comprising (1) role definition, (2) context, (3) instructions, (4) input data, (5) constraints, and (6) required output format [1]. The prompts were identical across LLMs and across runs, and no model-specific adjustments were applied. A zero-shot strategy was adopted, meaning that no example test cases were provided in the prompt. To mitigate randomness effects and ensure robust results, we repeated test generation ten times for both EvoMaster and LLM-based approaches, following recommended practices in [6]. For evaluation, all automatically generated test suites were executed against the SUT for 120 seconds under identical deployment, hardware, and logging configurations. For Locust, we did not re-execute the test suite from the prior study [5], but randomly selected 10 test runs and corresponding log files from LO2 dataset as representatives of human-written test execution logs. 2.3.2 Log collection and processing. For each automated test generation strategy, we collect execution logs from running the final test suite against the SUT under a fixed logging configuration. For human-written tests, the logs were collected from the LO2 dataset. Unique log templates were parsed from the logs, and log coverage was computed based on their number, overlap, and distribution among services. Before logs can be efficiently used for coverage estimation, they need to be loaded, masked, and transformed into a structured representation. This transformation process, log parsing [14], aims to group similar log entries that follow the same underlying pattern and constitute a log template by extracting constant parts of the log entry from the dynamic runtime variables. For the purposes of this study, LogLead [17] was used for loading and masking and Drain [14] was selected as the log parser as it has been shown to produce accurate log templates with low parsing time latency [24]. Custom LO2 specific masking was added to LogLead masks by empirically analyzing parsed Drain templates.
2.4
Complementary Comparison Approaches
From the 10 independent runs, we computed the following metrics. 2.4.1 Average Log Coverage (AvgLC). To estimate the typical behavioral profile of a test generation strategy, we computed what we refer to as Average Log Coverage (AvgLC) in two steps. First, we compute the average number of unique log templates discovered across 10 independent runs. But, as we need a set not just a number, we continue in the second step. There we sort log templates from all runs (by each strategy) with decreasing frequency and select the average count of top templates. This frequency-based aggregation represents the average system behaviour exercised by the test suites. Because AvgLC favors the most frequently appearing log templates across all runs, it reduces noise caused by infrequently appearing log templates. However, discarding rare outlier log templates may underestimate the exploration power of a test generation strategy. 2.4.2 Minimum Log Coverage (MinLC). To estimate the lower bound of consistently reproducible log coverage of each test generation
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
strategy, we computed Minimum Log Coverage (MinLC) as the intersection of unique log templates across all 10 independent runs. Only those log templates that appear in every individual run are included. MinLC captures deterministic and repeating runtime behavior while filtering out all rare events and execution paths that occur only sporadically. As a result, MinLC provides a conservative estimate of behavioral coverage and reflects the minimum behavior a strategy reliably exercises. However, because MinLC excludes templates that do not appear in all runs, it may underestimate the true behavioral exploration capability of a test generation strategy even more than AvgLC.
2.4.3 Maximum Log Coverage (MaxLC). To estimate the upper bound of behavioral exploration capability of each test generation strategy, we computed Maximum Log Coverage (MaxLC) as the full union of unique log templates across all 10 independent runs. For each strategy, all unique log templates discovered in individual runs are combined into a single aggregated set, representing the total observable behavior exercised across repetitions. Unlike AvgLC and MinLC, MaxLC captures every distinct runtime behavior identified during experimentation, including rare events and edge cases that may appear only in a subset of runs. However, MaxLC may be inflated by infrequently occurring templates and is therefore more sensitive to randomness. As a result, it does not represent typical performance.
2.4.4 Costs. Here we outline test development effort and cost to the best of our knowledge. These allow us to understand the costs of different test generation strategies and, therefore, provide information for making decisions between test generation strategies. All human effort numbers are estimates by the test authors, given after the task was completed. Locust test [5] were developed during 30 working days with roughly 5 hours per day resulting in human effort of 150 hours. Computing costs for Locust are negligibly as it runs on normal hardware. No LLMs were used to develop Locust tests. Using EvoMaster involves relatively low human effort after the correct configurations were found. The setup mainly consists of downloading the EvoMaster execution file, configuring the tool, deploying the SUT, and executing the generation process, after which a complete test suite is automatically produced. EvoMaster runs locally, meaning that computing costs are limited to local hardware resources and execution time. We estimate that the total human effort required throughout the setup, configuration, and execution process is approximately 4 hours. LLM-based approaches required human effort in designing an effective prompt and performing quick manual quality checks for generated tests. Test generation was performed via external API calls through OpenRouter, which introduce usage-based pricing. The estimated cost for 10 runs is approximately $13 for Claude Opus 4.6 and $3 for GPT-5.2-Codex. We estimate the total human effort to be approximately 10 hours for prompt development (shared with models), plus an additional 2 hours per model, including quality checks.
Reinikainen et al.
3 Results 3.1 RQ1: Log coverage across testing strategies Table 1 summarizes test generation characteristics. Overall Log Coverage. Claude Opus 4.6 achieved the highest log coverage with 113 unique log templates on average, followed by Locust (88), EvoMaster (65), and GPT-5.2-Codex (53). Compared to Locust tests, Claude Opus 4.6 improves log coverage by nearly 30%, and achieves more than twice the coverage of GPT-5.2-Codex. Log Coverage per Service. As reported in Table 2, log coverage exhibits clear service-dependent differences across strategies. Claude Opus 4.6 tests achieve the highest log coverage in six out of seven evaluated services, being particularly strong in Key and Token services. The only exception is the Code service, where Locust outperforms all other strategies. EvoMaster generally remains below both Claude Opus 4.6 and the Locust baseline across most services, while GPT-5.2-Codex achieves the lowest coverage overall. These service-level patterns help explain the coverage differences observed earlier. In particular, the Code service represents a notable exception to the overall trend. It implements complex authorization workflows that cannot be accurately defined in API specifications. This projects directly to a lower number of discovered unique log templates in the LLM and EvoMaster tests. In contrast, Locust executes predefined workflow sequences and therefore achieves higher coverage in the Code service. However, in other services, Locust primarily exercises typical execution paths. Claude Opus 4.6, by comparison, explores more diverse input combinations, edge cases, and less frequent behaviors, which is reflected in its higher log coverage across most services. Factors affecting log coverage. Test suite size. The higher log coverage achieved by Claude Opus 4.6 is partially attributable to the larger size of the test suite. Although Claude Opus 4.6 generates substantially more tests on average, the relative increase in discovered unique log templates is not proportional. When approximating the number of unique log templates discovered per test, Claude Opus 4.6 achieves 113/177 ≈ 0.64, compared to EvoMaster (65/54 ≈ 1.20), GPT-5.2-Codex (53/32 ≈ 1.66), and Locust (88/79 ≈ 1.11). This trend is also reflected in the total lines of generated test code: Claude Opus 4.6 produces 1851 lines on average, substantially more than GPT-5.2 Codex (385 lines) and EvoMaster (1149 lines), and comparable to Locust (1896 lines). Test pass ratio (test flakiness). Although log coverage is computed from all executed tests regardless of their verdict, pass ratio may affect coverage, as tests that terminate prematurely may exercise fewer execution paths. Low pass ratio is also indication of high test flakiness. In our results, Claude Opus 4.6 achieves both high average pass ratio (96.49%) and high log coverage (113 templates). However, GPT-5.2-Codex exhibits substantial variability in pass ratio across runs (69.7%–100%) while maintaining the lowest average log coverage (53 templates). Test generation variability among the strategies also affects the log coverage achieved by each test generation strategy. LLM-based tests show greater variability in both test suite size and pass ratio across runs compared to EvoMaster. This variability directly impacts the number of unique log templates discovered: Claude Opus 4.6 consistently finds more templates than Locust, though the
Assessing REST API Test Generation Strategies with Log Coverage
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Table 1: Test Statistics. Data is presented as AvgLC (MinLC–MaxLC). Method Locust EvoMaster Claude Opus 4.6 GPT-5.2-Codex
Generation Time (s) NA 707 (660–767) 447 (417–492) 743 (547–988)
# Tests 79 (79–79) 54 (52–57) 177 (170–185) 32 (27–36)
# Unique Log Templates 88 (84–92) 65 (62–69) 113 (97–120) 54 (49–60)
Table 2: Average Unique Log Templates Per Service Metric Client Code Key Ref.-token Service Token User
Locust
EvoMaster
Claude 4.6
GPT-5.2
31.8 44.0 4.8 19.7 25.8 28.2 21.5
23.0 28.0 17.3 16.3 23.9 22.2 22.8
36.9 34.4 23.0 24.6 32.0 48.9 24.1
16.5 24.1 17.2 15.7 15.3 18.4 13.0
results vary greatly between runs, while GPT-5.2-Codex occasionally reaches coverage comparable to EvoMaster’s minimum, but generally produces fewer templates. Test generation time does not exhibit a positive relationship with log coverage in our results. Claude Opus 4.6 achieves the highest coverage (113 templates) despite requiring less generation time (447s) than EvoMaster (707s) and GPT-5.2 Codex (743s), both of which produce lower coverage.
3.2
RQ2: Complementary gains in log coverage
Table 3 presents a comparison of human-written Locust tests with other test generation strategies. 3.2.1 Jaccard Similarity. Table 3 shows the unique log templates identified in both the Locust and the respective (Other) test generation approach. After that, the table outlines the intersection and union of these log template sets. Then Jaccard similarity is shown, calculated as the the intersection divided by the union. Jaccard similarity in average case is 33%, 28% 28%, when comparing Locust against Evomaster, Claude and GPT respectively. The observed range spans from a minimum of 13% to a maximum of 38% when considering MinLC and MaxLC. Overall, Jaccard similarity indicates that the generated test sets exhibit substantial dissimilarity when compared to the human-written Locust set. 3.2.2 Gain when combining the approaches. The "Gain over Locust" row indicates the percentage increase in unique templates obtained by supplementing the human-written Locust tests with each respective test generation method. This value is computed by dividing the union of templates by the unique templates identified by Locust alone. Claude Opus demonstrates the highest gain, contributing an additional 78% of unique log templates. This is consistent with its superior log coverage, as shown in Section 3. In contrast, EvoMaster and GPT-5.2 yield more modest gains of 31% and 26%, respectively.
Pass Ratio (%) NA 88.14 (87.04–90.57) 96.49 (93.02–99.43) 94.64 (69.7–100)
Lines of Test Code 1896 (1896–1896) 1149 (1116–1206) 1851 (1717–1947) 385 (314–479)
When examining the minLC and maxLC measures, the "Gain over Locust" metric exhibits substantial variability across setups. When adding Claude Opus test to Locust, the gain ranges from a minimum of 24% to a maximum of 187%. Even EvoMaster, considered the most stable approach, shows a spread of 15% to 85%, while GPT-5.2 ranges from 7% to 52.5%. Reversing the perspective, i.e., assessing the benefit of supplementing each test generation strategy with the Locust template set, reveals that even Claude Opus, which exhibits the highest baseline coverage, still gains an average of 39% additional unique templates. For EvoMaster, integrating Locust templates yields an average gain of 77%, while GPT-5.2 experiences the most significant augmentation, with a 106% increase in unique log templates. An analysis of minLC and maxLC measures further reveals substantial variation. Claude Opus exhibits a gain ranging from 28% (maxLC) to over 75% (minLC) when supplemented with Locust templates. The spread is even more pronounced for GPT-5.2x, with gains spanning from 80% to 330%, while EvoMaster ranges between 57% and 129%.
3.2.3 Summary. The Jaccard similarity and the "Gain over" analyses collectively demonstrate that the test generation strategies are largely complementary. While Claude Opus outperforms others in baseline coverage, as established in Section 3, it still has limitations. Other approaches are behind Locust but can still provide meaningful log template coverage increases to it. This complementarity suggests that integrating multiple approaches will yield the most robust test suites.
4
Threats to Validity
This study has some limitations, mainly related to the use of a single SUT, a single log parser, and the logging quality of the SUT. The experiments were performed only on Light-OAuth2, whose architecture and logging practices may differ from other RESTbased microservice systems. Experimenting with a single system limits the external validity of the proposed method. Drain was the only log parser used for log template extraction. Different parsers may generate different template sets, which could alter the coverage values. Logging distribution, density, and consistency significantly affect the observed log coverage. Systems with sparse, overly dense, inconsistent, or uneven logging may yield misleading coverage values. This affects the validity and reliability of log coverage.
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Reinikainen et al.
Table 3: Combined Log Coverage Comparison showing AvgLC (MinLC–MaxLC). Note that percentages for MaxLC and MinLC are not the same as min and max in statistics.
5
Metric
Locust vs. EvoMaster
Locust vs. Claude Opus 4.6
Locust vs. GPT-5.2-Codex
Unique Log Templates (Locust) Unique Log Templates (Other) Intersection of Templates Union of Templates Jaccard Similarity (%) Gain over Locust (%) Gain over Other (%)
88 (68–160) 65 (34–188) 38 (24–52) 115 (78–296) 33.04 (30.77–17.57) 30.68 (14.71–85.00) 76.92 (129.41–57.45)
88 (68–160) 113 (48–358) 44 (32–59) 157 (84–459) 28.03 (38.10–12.85) 78.41 (23.53–186.88) 38.94 (75.00–28.21)
88 (68–160) 54 (17–133) 31 (12–49) 111 (73–244) 27.93 (16.44–20.08) 26.14 (7.35–52.50) 105.56 (329.41–83.46)
Conclusion
In this study, we conducted experiments on different REST API test generation strategies in black-box settings against the LightOAuth2 microservice system. As our baseline, we utilized humanwritten Locust tests and compared this baseline to EvoMaster, a traditional search-based API test generation tool, as well as two leading LLMs available at the time: Claude Opus 4.6 and GPT-5.2Codex. Our findings indicate that Claude Opus 4.6 outperforms all other strategies, including human-written Locust tests, in terms of log coverage. Human-written tests ranked the second, followed by EvoMaster and GPT-5.2-Codex. Notably, relying on a single test generation strategy resulted in limited log coverage. When combining the two best-performing test generation strategies, Claude Opus 4.6 and human-written tests, we observed that the human-written test contributed an additional 39% coverage beyond what Claude Opus 4.6 achieved alone. This highlights the complementary nature of these strategies in maximizing log coverage. Finally, we believe the most notable finding is that Claude Opus 4.6 was able to outperform the human-written baseline, which had required roughly 150 hours of effort, at a cost of only $13 for compute and 12 hours of human effort for setup on a small industrial system. This suggests that AI-based approaches may produce superhuman results, but we need to validate this with other systems in the future.
6
Acknowledgment
The authors acknowledge CSC-IT Center for Science, Finland, for providing computing resources. This work has been supported by the Research Council of Finland (grant id: 359861, the MuFAno project) and also by FAST, the Finnish Software Engineering Doctoral Research Network, funded by the Ministry of Education and Culture.
References [1] Abbas Ahmad, Gualtiero Bazzana, Alessandro Collino, Olivier Denoo, and Bruno Legeard. 2025. Certified Tester Specialist Level - Testing with Generative AI (CTGenAI) Syllabus. International Software Testing Qualifications Board (ISTQB). https://istqb.org/sdm_downloads/ct-genai-syllabus-v1-0/ [2] J.H. Andrews. 1998. Testing using log file analysis: tools, methods, and issues. In Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239). 157–166. doi:10.1109/ASE.1998.732614 [3] Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 28, 1 (2019), 1–37.
[4] Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. REST-ler: Stateful REST API Fuzzing. In Proceedings of the 41st International Conference on Software Engineering (ICSE). 748–758. doi:10.1109/ICSE.2019.00083 [5] Alexander Bakhtin, Jesse Nyyssölä, Yuqing Wang, Noman Ahmad, Ke Ping, Matteo Esposito, Mika Mäntylä, and Davide Taibi. 2025. LO2: Microservice API Anomaly Dataset of Logs and Metrics. In Proceedings of the 21st International Conference on Predictive Models and Data Analytics in Software Engineering (Trondheim, Norway) (PROMISE ’25). Association for Computing Machinery, New York, NY, USA, 1–10. doi:10.1145/3727582.3728682 [6] Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus. 2026. On Randomness in Agentic Evals. arXiv:2602.07150 [cs.LG] https://arxiv.org/abs/2602.07150 [7] Boyuan Chen, Jian Song, Peng Xu, Xing Hu, and Zhen Ming (Jack) Jiang. 2018. An automated approach to estimating code coverage measures via execution logs. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France) (ASE ’18). Association for Computing Machinery, New York, NY, USA, 305–316. doi:10.1145/3238147.3238214 [8] Jinfu Chen, Weiyi Shang, Ahmed E. Hassan, Yong Wang, and Jiangbin Lin. 2019. An Experience Report of Generating Load Tests Using Log-Recovered Workloads at Varying Granularities of User Behaviour. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 669–681. doi:10.1109/ASE. 2019.00068 [9] Lizhe Chen, Ji Wu, Haiyan Yang, and Kui Zhang. 2021. Microservice Test Suite Minimization Technology Based on Logs Mining. Ruan Jian Xue Bao/Journal of Software 32, 9 (2021), 2729 – 2743. doi:10.13328/j.cnki.jos.006075 Cited by: 4. [10] Raffaele Della Corte, Roberto Pietrantuono, and Stefano Russo. 2025. Log-Driven Testing of Microservice Systems with Transformers. Proceedings of the IEEE International Conference on Web Services, ICWS 2025 (2025), 835 – 837. doi:10. 1109/ICWS67624.2025.00107 Cited by: 0. [11] Mark Harman, Peter O’Hearn, and Shubho Sengupta. 2025. Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1–17. [12] Mark Harman, Jillian Ritchey, Inna Harper, Shubho Sengupta, Ke Mao, Abhishek Gulati, Christopher Foster, and Hervé Robert. 2025. Mutation-Guided LLM-based Test Generation at Meta. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 180–191. [13] Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving Semantics-Aware Fuzzers from Web API Schemas. In Proceedings of the 44th International Conference on Software Engineering (ICSE). 1–12. [14] Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu. 2017. Drain: An Online Log Parsing Approach with Fixed Depth Tree. In 2017 IEEE International Conference on Web Services (ICWS). 33–40. doi:10.1109/ICWS.2017.13 [15] Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2025. Llamaresttest: Effective rest api testing with small language models. Proceedings of the ACM on Software Engineering 2, FSE (2025), 465–488. [16] Huayao Liu, Chang-ai Sun, and Shuo Ma. 2022. RESTCT: Black-box Constraintbased Combinatorial Testing for REST APIs. In 2022 IEEE 15th International Conference on Software Testing, Verification and Validation (ICST). 1–12. doi:10. 1109/ICST53961.2022.00045 [17] Mika V Mäntylä, Yuqing Wang, and Jesse Nyyssölä. 2024. Loglead-fast and integrated log loader, enhancer, and anomaly detector. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 395–399. [18] Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortes. 2021. RESTest: Automated Testing of RESTful Web APIs with Inter-parameter Dependency Support. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 57–60. doi:10.1109/ICSECompanion52605.2021.00034
Assessing REST API Test Generation Strategies with Log Coverage
[19] Robbe Nooyens, Tolgahan Bardakci, Mutlu Beyazıt, and Serge Demeyer. 2025. Test amplification for rest apis via single and multi-agent llm systems. In IFIP International Conference on Testing Software and Systems. Springer, 161–177. [20] Xuetao Tian, Honghui Li, and Feng Liu. 2017. Web Service Reliability Test Method Based on Log Analysis. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). 195–199. doi:10.1109/QRS-C.2017.38 [21] Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTESTGEN: Automated Black-Box Testing of RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Verification and Validation (ICST). 142–152. doi:10.1109/ICST46399.2020.00024 [22] Xiaoyan Xu, Filipe R. Cogo, and Shane McIntosh. 2024. Mitigating the Uncertainty and Imprecision of Log-Based Code Coverage Without Requiring Additional
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Logging Statements. IEEE Transactions on Software Engineering 50, 9 (2024), 2350–2362. doi:10.1109/TSE.2024.3435067 [23] Xiaolei Yu, Kai Jia, Wenhua Hu, Jing Tian, and Jianwen Xiang. 2023. Black-Box Test Case Prioritization Using Log Analysis and Test Case Diversity. In 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW). 186–191. doi:10.1109/ISSREW60843.2023.00072 [24] Tianzhu Zhang, Han Qiu, Gabriele Castellano, Myriana Rifai, Chung Shue Chen, and Fabio Pianese. 2023. System Log Parsing: A Survey. IEEE Transactions on Knowledge and Data Engineering 35, 8 (2023), 8596–8614. doi:10.1109/TKDE.2022. 3222417