arXiv:2604.25862v1 [cs.SE] 28 Apr 2026
RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements Leon Kogler
Stefan Hangler
Maximilian Ehrhart
[email protected] CASABLANCA hotelsoftware GmbH Schönwies, Austria
[email protected] CASABLANCA hotelsoftware GmbH Schönwies, Austria
[email protected] CASABLANCA hotelsoftware GmbH Schönwies, Austria
Benedikt Dornauer
Roland Wuersching
Peter Schrammel
[email protected] University of Innsbruck Innsbruck, Austria
[email protected] Technical University of Munich Munich, Germany
[email protected] Diffblue Ltd Oxford, United Kingdom
Abstract Existing REST API testing tools are typically evaluated using code coverage and crash-based fault metrics. However, recent LLM-based approaches increasingly generate tests from Natural Language (NL) requirements to validate functional behaviour, making traditional metrics weak proxies for whether generated tests validate intended behaviour. To address this gap, we present RESTestBench, a benchmark comprising three REST services paired with manually verified NL requirements in both precise and vague variants, enabling controlled and reproducible evaluation of requirement-based test generation. RESTestBench further introduces a requirements-based mutation testing metric that measures the fault-detection effectiveness of a generated test case with respect to a specific requirement, extending the property-based approach of Bartocci et al. [12]. Using RESTestBench, we evaluate two approaches across multiple state-of-the-art LLMs: (i) non-refinement-based generation, and (ii) refinement-based generation guided by interaction with the running Service Under Test (SUT). In the refinement experiments, RESTestBench assesses how exposure to the actual implementation, valid or mutated, affects test effectiveness. Our results show that test effectiveness drops considerably when the generator interacts with faulty or mutated code, especially for vague requirements, sometimes negating the benefit of refinement and indicating that incorporating actual SUT behaviour is unnecessary when requirement detail is high.
CCS Concepts • Software and its engineering → API languages; Acceptance testing; Software testing and debugging.
Keywords REST API Testing, OpenAPI, Automated Test Generation, Large Language Model, Benchmarking
1
Introduction
Representational State Transfer (REST) APIs have long been essential in enterprise systems, enabling scalable integration between services and supporting digital transformation. The adoption of API-first approaches has accelerated, with a growing number of organizations treating APIs as products that drive business value and foster innovation [30, 35]. If faults in deployed API services occur,
they may lead not only to service outages but also to data integrity violations and costly production incidents, thereby reinforcing the need for effective and scalable automated API testing. This need is reflected in the growth of research on REST API testing tools between 2017 and 2022 [18]. During this period, fuzzingbased approaches became predominant, especially black-box techniques that generate requests from OpenAPI Specification (OAS) and evaluate outcomes using oracles such as server errors (5xx) and schema violations [6, 8, 18, 29]. Recent advances in Large Language Models (LLMs) have shifted the landscape of automated API testing. LLMs can interpret NL artifacts and generate domain-aware assertions, enabling test generators to move beyond purely robustnessoriented checks and towards validating functional correctness and business logic requirements [9, 22]. Emerging tools generate REST API tests from NL requirements or requirement-like scenarios, either directly from human-defined descriptions (e.g., user stories) or via intermediate NL scenarios derived from the OAS [26, 31, 34, 42], as given in Table 1. Thereby, most tools rely on the OAS as the primary source for LLM-derived scenarios treated as functional oracles. Unfortunately, since the OAS only specifies endpoints, parameters, and response schemas [5] while omitting behavioural semantics and business constraints, it might not reliably serve as a basis for functional requirements. Apart from that, the shift in using LLMs to generate requirementsbased tests brings renewed attention to a long-standing challenge in automated testing: the oracle problem [10]. While current LLMbased test generation approaches address the challenge of creating stronger oracles to validate functional requirements, ensuring that tests actually reflect the intended requirements remains an open problem. When tests aim to validate requirements rather than merely detect crashes, traditional metrics such as code coverage or 5xx server error counts are insufficient [12, 21]. Consequently, we see that most approaches still rely on manual evaluation (Section 2.1). Mutation testing offers a stronger adequacy signal than code coverage and crash-based metrics, and has recently been applied to LLM-generated REST API tests [11]. Yet, if mutations are not tied to specific requirements, mutation testing cannot determine whether a test kills mutants for the “right reason”, i.e., because it enforces the intended requirement rather than incidental implementation behaviour. Bartocci et al. [12] introduce Property-Based Mutation Testing (PBMT), emphasizing that mutants are only meaningful when tied to a specific property: a mutant is considered killed in
Kogler L., Hangler S., Ehrhart M., Dornauer B., Wuersching R., and Schrammel P.
Table 1: Overview of LLM-based REST API test generation tools. Ref. = Uses refinement loop, CC = Code Coverage, MC = Mutation Coverage. Tool
Oracle Source
Ref. Evaluation
APITestGenie [34] LogiAgent [42] RestTSLLM [11] SAINT [31] RESTifAI [26]
Human-defined NL req. From OAS (LLM) From OAS (LLM) From code (LLM) From OAS (LLM)
✗ ✓ ✗ ✓ ✓
human CC + #5xx + human CC + MC CC + #5xx + human CC + #5xx + human
a meaningful way only if its execution triggers a violation of that property. Existing collections of services used in REST API testing research (including the Public REST API Benchmark by Decrop et al. [15]) document which APIs have been used for evaluation, but do not provide verified NL requirements or requirement related mutations that would allow controlled measurement of test generation from requirements. Consequently, human judgment remains necessary to assess whether generated tests meaningfully validate intended requirements, limiting reproducibility, and comparability across tools. To address these limitations, we introduce RESTestBench, a benchmark that explicitly separates requirement engineering from test generation and assesses test adequacy using specific mutations relevant to the requirement, drawing inspiration from Bartocci et al. and their PBMT definition. By grounding evaluation in validated requirements, the effectiveness of test generation approaches can be assessed quantitatively without conflating ambiguity in requirement generation with errors in test generation. Assessing the adequacy of generated tests by using PBMT is not only a more challenging metric than normal mutation testing [12], it also allows us to evaluate if requirement based testing approaches that interact with the SUT, are able to distinguish between a valid implementation and an already mutated implementation. While regression-based tools assume the correctness of the current SUT implementation, requirements-based approaches should distinguish invalid from valid behaviour, as their oracle is grounded in the requirements rather than in the implementation. Therefore, our benchmark enables the evaluation of test generation approaches that encounter the actual behaviour of the SUT in two settings: one in which tests are generated based on the valid implementation, and one in which tests are generated based on a mutated implementation. This design enables systematic and reproducible comparison of requirementsbased generation strategies without relying on human judgement as the primary oracle. Overall, this work makes the following contributions: • RESTestBench: A requirements-based REST API test generation benchmark comprising three REST services, 106 human-validated NL-requirements at two different levels of detail, 228 manually designed requirements-based mutations, and a framework that enables straightforward integration and evaluation of new generation approaches. • Two experiments conducted using RESTestBench: (1) An evaluation of a simple single-step approach serving as a baseline for non-refinement-based tools. (2) An evaluation of a refinement-based approach that incorporates knowledge of the actual behaviour of
the SUT. Specifically, we measure whether actual behaviour influences test effectiveness, as observed in prior work on LLM-generated test oracles [27].
2 Background and Related Work 2.1 LLM-Based REST API Test Generation Approaches Over the past decade, the number of automated approaches for testing REST APIs has increased substantially [18]. Until roughly 2022, most practical automated REST API testing approaches were variants of property-based and model-based testing. Representative tools include EvoMaster [6], RESTest [29], and RESTler [8]. Despite the prevalence of property- and model-based testing, automatically deriving test oracles from textual documentation (e.g., API descriptions or requirements in natural language) remained a significant challenge. In general software testing, the oracle problem has long been recognized as difficult when relying on unstructured text alone, since human interpretation is required to map documentation to expected behaviour [10] and existing REST API tools predominantly generate tests from machine-readable specifications rather than textual requirements [3]. Meanwhile, in broader software engineering, Requirements-Based Test Generation (RBTG) has been shown to align testing closely with user intent and quality assurance goals by constructing test cases directly from NL requirements [41]. Recent advances in LLMs have significantly lowered the barrier to leveraging NL artifacts for REST API testing. In the following, we examine current LLM-based REST API testing tools that derive, interpret, and/or translate NL requirements into executable test cases; these approaches are summarized in Table 1. APITestGenie [34] combines an OAS with human-written NL requirements and uses single-shot LLM prompting to generate executable JUnit tests, treating the provided requirements as the oracle. The experimental evaluation focuses on executability and perceived usefulness, with manual validation recommended. LogiAgent [42] adopts a multi-agent workflow that derives NL scenarios from the OAS, executes them stepwise, and validates responses using an LLM as a judge. Effectiveness is measured via code coverage and detected failures, while semantic correctness relies on manual annotation. RestTSLLM [11] employs a two-stage LLM pipeline: first generating a Test Specification Language (TSL)-based test specification from the OAS, then translating it into executable tests. Evaluation includes execution success, coverage, and mutation score, but does not explicitly assess alignment with intended requirements. SAINT [31] is a white-box approach that infers endpoints and dependencies from source code and refines them into tests using LLM agents. Oracles are embedded implicitly in generated scenarios, and evaluation combines automated metrics with developer feedback. RESTifAI [26] generates happy-path and negative test scenarios from an OAS using LLMs, focusing on robustness. Effectiveness is assessed through coverage and detected server errors, with additional expert-based validation. Across the reviewed approaches, two fundamental generation strategies can be distinguished. Non-refinement approaches (e.g.,
RESTestBench
APITestGenie, RestTSLLM) generate test cases in a single-step manner from the OAS and/or NL requirements without interacting with the SUT during generation. In contrast, refinement-based approaches (e.g., LogiAgent, SAINT, RESTifAI) incorporate feedback loops in which intermediate requests are executed against the SUT, and the observed responses are used to iteratively refine scenarios, parameters, or assertions. These strategies therefore differ primarily in whether test generation is performed statically from the specification or dynamically informed by the runtime behaviour of the service. The intention of the experiments in this paper is not to report comprehensive comparison results for existing tools, but to show how the benchmark can be used to compare fundamentally different generation approaches in a controlled way under uniform experimental conditions.
2.2
Metrics and their Problem for Requirements-Based Testing
In 2023, Golmohammadi et al. [19] conducted a systematic literature review (RQ4) to identify the evaluation metrics commonly used by predominantly non-LLM-based REST API testing techniques. Based on that, they group these metrics into three main categories: coverage metrics (e.g., schema-based and code coverage), faultdetection metrics (e.g., HTTP 5xx errors), and performance metrics (e.g., response time and latency). Despite the recent advances in LLM-based REST API test generation (Section 2.1) have not influenced the choice of evaluation strategies, the first two categories are still predominant, as identifiable in Table 1. Coverage metrics are reported by all approaches discussed in Section 2.1, except APITestGenie. Such metrics quantify how extensively the system is exercised, not whether the generated tests actually check the intended functionality [39]. Furthermore, the correlation between test suite effectiveness and coverage is not strongly given, as quantified by Inozemtseva and Holmes [23]. Fault-detection metrics. Li and Offutt [28] define an adequate test case as one that not only triggers and propagates faulty behaviour but also includes a test oracle capable of revealing the fault. Most of the tools discussed in Section 2.1 aim to generate increasingly sophisticated oracles to detect a broader range of faults. In practice, however, many of these approaches still rely on server-side error counts (e.g., HTTP 5xx responses) to evaluate test effectiveness. Such metrics are insufficient for assessing the true effectiveness of test oracles when the goal is to validate the functional behaviour of the SUT. Only RestTSLLM [11] integrates a mutation score, where small, faults known as mutants are systematically introduced directly into the system to evaluate whether the test suite, particularly API tests, can detect the resulting behavioural deviations. A higher mutation score therefore serves as an indicator of test effectiveness and provides a stronger link to the ability of the tests to detect meaningful defects in REST API implementations [32, 33, 37]. Thereby, classical mutation evaluation primarily asks: “If a developer introduced a small coding mistake, would our tests detect it?”. It does not determine whether the tests genuinely enforce the intended system behaviour as specified by requirements, but rather whether they can distinguish the current implementation from syntactically perturbed variants, as is typically done in regression testing.
Therefore, PBMT proposed by Bartocci et al. [12] aims to bridge this gap by evaluating test suites with respect to property-based mutations. In our context, these properties correspond directly to formalized requirements. Compared to the established metrics in REST API testing, PBMT offers the advantage of assessing whether tests validate the intended system requirements rather than merely detecting arbitrary output differences. By treating requirements as explicit constraints during mutant evaluation, PBMT further prevents mutation scores from being inflated by mutants that are irrelevant to the requirement under consideration. Overall, PBMT represents a promising and more requirement-aligned metric that deserves further attention in this and future work.
3
RESTestBench Conception
RESTestBench is designed to evaluate the effectiveness of LLMbased black-box approaches in translating a functional NL requirement, supported by an OAS, into a single executable test case. To this end, the benchmark comprises a curated set of REST API services (Section 3.1) and a manually defined collection of functional NL requirements (Section 3.2). The fault-detection effectiveness of the generated requirements-based test cases is assessed using PBMT [12] with manually defined and validated mutations (Section 3.3). Figure 1 illustrates the overall benchmark workflow, including the generation of test cases from the requirements and the OAS, their execution against the SUT, and the subsequent evaluation of test outcomes using mutation scores. The benchmark is designed to grow over time: additional services, requirements and mutations can easily be integrated into the current benchmark framework. The repository and guidelines can be accessed here: https:// github.com/ casablancahotelsoftware/ RESTestBench. RESTestBench Test Generation
Dataset
Execution & Evaluation
Non-Refinement Approaches LLM
Test Generation FastAPI
TodoApp
RealWorld
Refinement Approaches NL-Requirements Precise
LLM
Vague PBMT
Mutations OAS
Mutation Score
Figure 1: RESTestBench overview
3.1
Selection of suitable REST API Services
The benchmark services were selected to support controlled, reproducible experimentation while remaining representative of realworld REST backends. Practical relevance refers to selecting services that resemble production-style REST backends in terms of API surface, security mechanisms, and that include complex operation dependencies requiring stateful request sequences. At a minimum, candidate services must be open-source and provide an OAS. Moreover, we require sufficient project maturity for
Kogler L., Hangler S., Ehrhart M., Dornauer B., Wuersching R., and Schrammel P.
Table 2: Services used in RESTestBench. Stars/Forks are taken from the exact GitHub repositories (accessed on 2026–01– 29). Endpoint counts and CLOC (counted lines of code) are computed from the corresponding benchmark snapshot.
Name FastAPI [2] TodoApp [17] RealWorld [25]
GET/POST/PUT PATCH/DELETE
Stars
Forks
CLOC
6/10/1/3/3 2/3/1/0/1 7/6/2/0/4
41.2k 3.1k 3.3k
8k 462 701
8,072 1,609 965
realistic engineering practices and stable behaviour. As a transparent proxy we report GitHub stars and forks, given in Table 2. Beyond these minimum constraints, we prioritised services whose domain logic is non-trivial and whose endpoints exhibit meaningful dependencies (e.g., authentication prerequisites, ownership constraints, and state-dependent workflows), because these properties enable requirements that go beyond endpoint reachability and require semantically rich assertions. A further prerequisite is that the OAS aligns with the actual service implementation so that tools can derive valid requests and create correct test cases. For services where the provided OAS deviated from the observed runtime behaviour (e.g., mismatched status codes, missing/incorrect fields, or parameter constraints), we performed targeted manual corrections to obtain a specification that matches the evaluated service implementation, ensuring that all tools can in principle generate correct tests. Future work could investigate how the quality of the OAS influences test quality. Applying these constraints, we selected the following three services as SUTs: FastAPI full-stack template [2] is a production-oriented, Pythonbased FastAPI template service with a PostgreSQL-backed data model, JWT-based security components, and multi-entity workflows (e.g., hierarchical resource ownership and administrative/superuser operations). We selected it because it is widely adopted in the FastAPI ecosystem and provides a realistic, dependency-rich baseline for business-logic requirements. TodoApp [17] is a compact .NET Minimal API service with authentication and domain-specific dependencies that yields a controlled lower-complexity baseline. We selected it because it enables direct comparison between simpler CRUD-centric behaviour and more complex multi-entity workflows and was previously used as an evaluation subject in Barradas et al. [11]. NestJS RealWorld app [25] is a NestJS backend (TypeScript) that implements the RealWorld API specification [1], including articles, comments, profiles, favourites, and follower relationships, making it well-suited for complex requirements-based testing.
3.2
Natural Language Requirements
In our benchmark, an NL requirement is an unstructured textual statement in human language that specifies the expected externally observable functional behaviour of a REST API. The use of natural
language reflects common industrial practice, where requirements are frequently documented informally and exhibit varying degrees of precision and completeness, often leading to ambiguity and underspecification [38, 43]. As the benchmarked services do not provide complete requirements documentation, requirements are reverse-engineered from existing integration tests, where available, or derived through manual inspection of the service source code and its business logic, ensuring that each requirement reflects the actual behaviour of the evaluated system. To validate this alignment, we manually created a golden test for each requirement, verifying that it passes on the original implementation. Each requirement is structured so that, in combination with the OAS, it can be transformed into exactly one REST API test case. To study how requirement detail affects test generation effectiveness, we provide two variants of each requirement: one precise and one vague. This distinction is grounded in established requirements engineering literature differentiating goal-level specifications from operational or scenario-based descriptions [24, 38]. The vague variant is a high-level informal statement that captures only the essential intent of the requirement, providing the minimal information a human tester would need to produce a test case with a still clear and verifiable objective. It omits any preconditions or setup needed for validation, specifying only the testing objective and the expected outcome at a conceptual level. This formulation corresponds to goal-oriented or early-phase requirements that intentionally abstract from operational detail and may remain partially incomplete [38, 43]. Such underspecification reflects realistic industrial requirements and poses known challenges for automated test generation from NL descriptions [7]. The precise variant, in contrast, specifies all preconditions required to validate the intended test objective. It details the exact sequence of operations, their dependencies, and the parameters relevant for subsequent requests. Additionally, it defines the expected results concretely, including response fields and values that must be verified. Compared to the vague variant, this representation functions more like a scenario-based description of a test case. Its structure is more aligned with use cases and scenario-based specifications [13, 24]. By specifying all required preconditions, interaction steps, and expected outputs, the precise variant fulfills established quality characteristics of testable requirements, including completeness, unambiguity, and verifiability [40]. Listing 1 shows a concrete example of both variants. RESTestBench covers a wide range of requirement complexity to support differentiated evaluation. Scenarios range from singleoperation requirements to multi-step sequences with intricate dependencies among operations and parameters. Verification tasks vary from simple status code checks to detailed assertions of nested response fields that depend on earlier operations. The benchmark also includes challenging functional contexts, such as complex CRUD workflows, authentication and authorization procedures, and cascading state changes. By addressing both operational and verification difficulty, RESTestBench ensures that test generation methods are assessed across realistic and representative REST API behaviours.
RESTestBench
listing 1 JSON Requirement Example from FastAPI 1 2 3 4
{
5
6 7 8 9
3.3
}
"id": 28, "service": "fastapi", "requirement_vague": "A superuser ([email protected] / password123) creates 2 random users and retrieves all. Verify the response.", "requirement_precise": "Authenticate as the superuser (use admin@example. com / password123) to obtain an access token. Using that superuser access token, create two distinct users via the public create-user API (each with a unique email and password). Store the returned user IDs and emails. Then, using the same superuser access token, retrieve the list of all users. Verify the response status is 200. Verify the response JSON contains a top-level 'data' collection and a top-level 'count' key. Verify that both created users appear in the 'data' collection by checking that their emails are present. Verify that for each created user, the email in the list matches the email used during creation.", "mutants": [ ... ]
Manual Mutation Construction
To assess the effectiveness of requirements-based test generation, we employ manually defined and validated mutations, following the principles of Property-Based Mutation Testing [12]. Each requirement is formalized as a property, and for each property we define a set of mutations that lead to violations of that property. Compared to automated mutation seeding, manual mutation design ensures the following: (1) Propagation: According to the RIP model [4], a fault must propagate to an observable output to be revealed [28] by an oracle. By explicitly defining and executing mutations, we ensure that violations are observable through the REST API. (2) Relevance: Mutations are constructed to directly impact the satisfaction of the tested property [12]. Existing automated mutation tools are, to our knowledge, unable to generate mutants that meaningfully affect NL requirements. (3) Non-equivalency [16]: Since identifying equivalent mutants is undecidable, mutations are manually defined to ensure behavioural divergence from the original system for at least one input. (4) Non-subsumption [32]: Mutations are designed to avoid subsumption, where non-equivalent mutants produce identical propagated outputs and thus provide no additional evaluation power. (5) Realism: Common mutation operators (e.g., arithmetic, logical, or syntactic changes [14, 20]) are largely semanticsagnostic. In contrast, our manually defined mutants (Listings 3) capture realistic faults that static mutation tools are unlikely to produce. Listing 2 presents an example endpoint definition from the fullstack-fastapi-template service used in our benchmark. Listing 1 specifies the corresponding requirement the endpoint in Listing 2 is expected to fulfill and that the generated test case must validate. Listing 3 illustrates a manually designed mutation that conforms to the above defined criteria. In addition, and in accordance with these criteria, we deliberately restrict the mutation set to a minimal yet representative subset that targets the most critical test assertions
listing 2 Valid implementation of the get_users operation of the FastAPI service @router.get(dependencies=[Depends(get_current_active_superuser)], response_model=UsersPublic) def read_users(session: SessionDep, skip: int = 0, limit: int = 100) -> Any : 3 count_statement = select(func.count()).select_from(User) 4 count = session.exec(count_statement).one() 5 statement = select(User).offset(skip).limit(limit) 6 users = session.exec(statement).all() 7 return UsersPublic(data=users, count=count) 1 2
listing 3 Example mutation of returned user data ensuring the test not only checks the presence of emails in the response, but also if each email is correctly assigned to the corresponding user 1 2 3 4 5 6 7
- return UsersPublic(data=users, count=count) + emails = [u.email for u in users] + swapped = [ + {**u.__dict__, 'email': emails[(i + 1) % len(emails)]} + for i, u in enumerate(users) + ] + return UsersPublic(data=swapped, count=count)
associated with the requirement. This design choice ensures a manageable benchmark runtime, given the substantial computational cost of test generation and execution.
3.4
Evaluation Process
We adopt the PBMT [12] definition of mutation score (Definition III.4) and apply it under a restriction to singleton test sets. In our setting each requirement 𝜙𝑖 is evaluated using a test suite 𝑇𝑖 = {𝑡𝑖 } containing a single test case. A mutant is considered 𝜙𝑖 killed according to Definition III.1 if it is killed by the corresponding test 𝑡𝑖 . We define 𝑀𝜙𝑖 as the set of mutations defined for 𝜙𝑖 , and let 𝑘𝜙𝑖 denote the number of mutations 𝜙𝑖 -killed by the corresponding test 𝑡𝑖 . Let 𝑀 denote the set of all mutations in the benchmark and 𝑘𝜙 denote the number of all 𝜙𝑖 -killed mutations in total. The overall requirements-based mutation score 𝑀𝑆𝜙 is defined as: 𝑀𝑆𝜙 =
𝑘𝜙 |𝑀 |
(1)
To compute the mutation score for refinement-based approaches where the generated test case may depend on the concrete implementation of the SUT, we define 𝑡𝑖valid as the test case generated from requirement 𝜙𝑖 using the valid implementation of the SUT. 𝑚 Furthermore, for each mutant 𝑚 𝑗 ∈ 𝑀𝜙𝑖 , we define 𝑡𝑖 𝑗 as the test case generated based on the 𝑗-th mutated SUT of requirement 𝜙𝑖 . For non-refinement-based approaches, test generation is inde𝑚 pendent of the actual implementation. Thus, 𝑡𝑖valid = 𝑡𝑖 𝑗 , and we evaluate the 𝑘𝜙𝑖 based on 𝑡𝑖valid and calculate total mutation score as defined in Equation 1. For refinement-based approaches, test generation may depend 𝑚 on the implementation, so we assume that 𝑡𝑖valid ≠ 𝑡𝑖 𝑗 . To compare fault-detection effectiveness fairly between the single 𝑡𝑖valid 𝑚 and possible multiple 𝑡𝑖 𝑗 , we compute the average number of mu𝑚𝑗 𝑚 tants killed by all 𝑡𝑖 . Specifically, let 𝑘𝜙 𝑗 denote the number of 𝑖
𝑚
mutations 𝜙𝑖 -killed by 𝑡𝑖 𝑗 , and define
Kogler L., Hangler S., Ehrhart M., Dornauer B., Wuersching R., and Schrammel P.
𝑗
𝜙𝑖
i.e., the average number of 𝜙𝑖 -killed mutations across all mutantspecific tests. Similarly, let 𝑘𝜙valid denote the number of mutations 𝑖
killed by 𝑡𝑖valid . Figure 2 demonstrates the differences between 𝑘𝜙valid 𝑖 and 𝑘𝜙𝑚𝑖 for a single requirement. Calculation of killed mutants
Refinement approach Output
Execution against valid and all mutated SUTs
Input SUT LLM
SUT
SUT
Figure 2: Visualization of the evaluation of 𝑘𝜙valid and 𝑘𝜙𝑚𝑖 for 𝑖 a single requirement 𝜙𝑖 in refinement-based test generation approaches. Summing over all requirements, we define 𝑘𝜙𝑚 =
∑︁
𝑘𝜙𝑚𝑖 ,
𝑘𝜙valid =
∑︁
𝑘𝜙valid . 𝑖
𝑖
𝑖
Based on this distinction, we compute two mutation score values for refinement-based approaches: 𝑀𝑆𝜙valid =
𝑘𝜙valid |𝑀 |
and 𝑀𝑆𝜙m =
𝑘𝜙𝑚 |𝑀 |
By computing both scores for refinement-based approaches, RESTestBench extends standard mutation testing by explicitly analyzing the effect of the actual implementation on test effectiveness. In particular, this distinction allows us to investigate whether a tool that incorporates the concrete behaviour of the SUT can correctly differentiate between valid and mutated implementations for a given requirement, and how differences between the valid and mutated implementations influence the ability of the generated test cases to detect faults. Table 3 summarizes, for each service, Table 3: Requirements and mutations per service used in the benchmark. Service
#req.
#mut.
avg. mut/req
FastAPI [2] TodoApp [17] RealWorld [25]
63 14 29
141 22 65
2.24 1.57 2.24
Total
106
228
2.15
the number of requirements and the total number of associated mutations defined in the benchmark. To enable standardized and comparable evaluation of different approaches, we define a minimal Python interface consisting of the
functions 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑒_𝑡𝑒𝑠𝑡 and 𝑒𝑥𝑒𝑐𝑢𝑡𝑒_𝑡𝑒𝑠𝑡, which must be implemented for each new approach. This abstraction ensures independence from specific test languages and frameworks. To guarantee independent test generations (only relevant for refinement-based generation approaches) and test executions, the SUT is reset to a well-defined initial state prior to each generation process and each test execution. To track the cost of test generation the 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑒_𝑡𝑒𝑠𝑡 function is required to return the generation cost in USD to the caller. To validate the correctness of mutation insertion and benchmark execution, we developed a set of golden tests that achieve a full mutation score on the benchmark. All results are made publicly available here https:// github.com/ casablancahotelsoftware/ RESTestBench/ tree/ master/ results.
4
Experiments
This section presents two controlled experiments evaluating LLMbased REST API test generation on RESTestBench. As discussed in Section 2.1, existing approaches can broadly be classified as either non-refinement or refinement-based. We therefore evaluate one representative strategy for each class under controlled conditions to establish a baseline for subsequent comparisons. To ensure a fair comparison, both experiments use the same core input artifacts (the NL requirement, the OAS, and the service base URL), the same target output format (pytest test cases), and the same evaluation procedure defined in Section 3.4. Across both experiments, we measure the mutation score achieved on RESTestBench as well as the cost of test generation. In addition, both precise and vague requirement variants are evaluated to assess how requirement granularity influences test effectiveness. Table 4: LLM models used in the benchmark with licensing type, parameter count, and cost per million tokens (Azure AI Foundry pricing). Model
Company
Open Source
∑︁ 𝑚 1 · 𝑘 𝑗 |𝑀𝜙𝑖 | 𝑚 ∈𝑀 𝜙𝑖
Llama 3.1 8B Llama 3.3 70B Llama 4 Scout GPT-OSS 120B DeepSeek V3.2 Kimi K2 Thinking
Meta 8B Meta 70B Meta 109B (17B) OpenAI 117B (5.1B) DeepSeek 671B (37B) Moonshot AI 1T (32B)
0.15 / 0.15 0.71 / 0.71 0.25 / 1.00 0.15 / 0.60 0.58 / 1.68 0.60 / 2.50
Closed Source
𝑘𝜙𝑚𝑖 =
Size $/1M (In/Out)
GPT-5 Nano GPT-5 Mini Mistral Medium Claude Sonnet 4.5
OpenAI OpenAI Mistral AI Anthropic
0.05 / 0.40 0.25 / 2.00 0.40 / 2.00 3.00 / 15.00
– – – –
Table 4 lists the ten state-of-the-art LLMs evaluated in both experiments. Models were selected along the Pareto frontier of cost versus quality on the Azure AI Foundry model benchmark, ranging from cost-efficient models (e.g., Llama 3.1 8B) to frontier models (e.g., Sonnet 4.5), and spanning six vendors (Anthropic, DeepSeek, Meta, Mistral AI, Moonshot AI, and OpenAI) with six open-source (OS) and four closed-source (CS) models.1 Due to the 1We used Llama 3.1 8B instead of the newer Llama 3.3 8B because the latter was not
deployable in Azure AI Foundry at the time of our experiments.
RESTestBench
0 X W D W L R Q 6 F R U H I R U Y D J X H 5 H T X L U H P H Q W V
6 L Q J O H 6 W H S 5 H I L Q H P H Q W 9 D O L G 5 H I L Q H P H Q W 0 X W D W H G
0 X W D W L R Q 6 F R U H I R U S U H F L V H 5 H T X L U H P H Q W V
6 R Q Q H W
* 3 7 1 D Q R ' H H S 6 H H N 9
* 3 7 0 L Q L
* 3 7 2 6 6 %
. L P L .
) D V W $ 3 , 5 H D O : R U O G 7 R G R $ S S
0 L V W U D O 0 H G L X P / O D P D 6 F R X W / O D P D % / O D P D %
Figure 3: Mutation scores for vague and precise requirements comparing single-step 𝑀𝑆𝜙 (solid bars) with refinement-based scores 𝑀𝑆𝜙valid (forward-hatched) and 𝑀𝑆𝜙m (cross-hatched). Bars are stacked by service (FastAPI, RealWorld, TodoApp); segment height reflects weighted service contribution and segment percentages are per-service mutation scores. substantial time and computational cost of the full benchmark, each experiment was executed three times; we report averaged results with a maximum observed deviation of 3.4 percentage points (pp) in overall mutation score across runs.
4.1
Experiment 1: Single-Step Generation
The first experiment evaluates a non-refinement, single-step strategy. In this setting, the model receives a zero-shot prompt and generates the final test in one inference call, without interacting with the running SUT during generation. The prompt (Listing 4) takes three inputs: the NL requirement description, the complete OAS, and the service base URL. The generator neither executes the test nor contacts the SUT. Consequently, the produced test is independent of the actual runtime behaviour of the service and is not influenced by potential faults in the current implementation.
Results. The single-step mutation scores for all models are shown in Figure 3, using the solid bars (𝑀𝑆𝜙 ). For precise requirements, scores span a wide range, from 13% (Llama 3.1 8B) to 92% (Sonnet 4.5), with three models exceeding 80% (Sonnet 4.5, GPT-5 Nano, and DeepSeek V3.2). For vague requirements, scores drop substantially for every model, typically by 26 to 40 percentage points, and the strongest models converge to a narrower range of 49%–54%. Sonnet 4.5 performs very strongly on precisely defined requirements, while Llama 3.1 8B nearly fails on vague requirements, reaching only 2%. Overall, these results show that requirement granularity strongly influences single-step generation. More detailed requirements substantially improve the effectiveness of all evaluated models, whereas underspecified requirements remain challenging even for frontier models.
listing 4 Shortened zero-shot generation prompt template. Placeholders are filled per requirement. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
4.2
You are an expert API tester who writes pytest tests to validate that API implementations follow their intended human-written requirements. [...] ## Requirement: <REQUIREMENT_TEXT> ## OpenAPI Specification: <OAS_JSON> ## Instructions (must follow exactly): - Write a complete pytest test function [...] - Use the requests library for HTTP calls - The test MUST be completely self-contained - Hardcode all required values (API base URL should be <SERVICE_BASE_URL>, ...) Generate ONLY the Python test code, no explanations.
Experiment 2: Refinement-Loop Generation
The second experiment evaluates a refinement-based strategy that extends the single-step approach with a feedback loop. Generation begins with the same initial prompt used in Experiment 1. The generated test is then executed against the SUT while a RequestsTracer captures all HTTP request/response exchanges. A judge—a second inference call to the same model—receives the requirement, the OAS, the generated test code, the test execution output, and the HTTP trace, and returns one of two verdicts (Listing 5): FINISH if the test is adequate, or REFINEMENT if another iteration is needed. In the latter case, the generator receives the previous test code, the HTTP trace, and the test execution output and produces a revised test, which is then re-evaluated. This process repeats until the judge issues FINISH or the maximum number of rounds is reached (three rounds in our experiments). Note that the judge solely issues a binary continue/stop decision and is part of the
Kogler L., Hangler S., Ehrhart M., Dornauer B., Wuersching R., and Schrammel P.
listing 5 Shortened judge prompt template. The judge receives test execution results and decides whether refinement is needed. You are a test quality reviewer. [...] ## Requirement Being Tested: <REQUIREMENT_TEXT> ## Test Code: <GENERATED_TEST_CODE> ## Test Execution Output: <STDOUT_STDERR> ## HTTP Trace (actual requests/responses): <TRACE_JSON> ## Your Task: 1. Compare the test case against the requirement 2. Check if the Test Execution Output and HTTP Trace can enhance the test's effectiveness 3. Identify if failures are due to test defects OR correct detection of faulty service behaviour [...] ## Response Format (EXACTLY one of): REFINEMENT: <short description of what needs changing> FINISH: <short description why test is correct>
generation approach, not the external evaluation; using the same model is thus a deliberate choice to evaluate the model performance. Results. Figure 3 shows that refinement generally increases mutation scores over single-step generation. The only notable exception is Sonnet 4.5 on precise requirements, where the single-step score already reaches 92% and refinement yields no further improvement. In general, the gains from refinement are substantially larger for vague requirements than for precise ones. For example, GPT-5 Nano improves by 29 percentage points on vague requirements, reaching 80%, and GPT-5 Mini improves by 24 percentage points, reaching 75%. A consistent pattern across all models is that 𝑀𝑆𝜙𝑚 remains below 𝑀𝑆𝜙valid . This indicates that generating tests against a mutated SUT systematically reduces test effectiveness. The gap is larger for vague requirements than for precise ones, suggesting that underspecified requirements make refinement-based generation more susceptible to adapting to faulty behaviour. Notably, for four models on precise requirements—Sonnet 4.5, GPT-5 Nano, Mistral Medium, and Llama 3.3 70B—the single-step score exceeds 𝑀𝑆𝜙𝑚 . In these cases, singlestep generation without SUT interaction is more effective than refinement performed against a faulty implementation. A plausible explanation is that the LLM adapts its assertions to match the faulty behaviour and thereby treats the defect as correct.
4.3
0 X W D W L R Q 6 F R U H I R U Y D J X H 5 H T
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Costs
Figure 4 shows that all refinement configurations achieve higher mutation scores than their single-step counterparts, but at increased cost. Across models, refinement increases total cost by roughly 2– 4×. However, the gains differ substantially by model. The strongest cost-effectiveness results are achieved by GPT5 Nano, GPT-OSS 120B, and GPT-5 Mini. In the refinement setting, GPT-5 Nano reaches a 70% mutation score at $0.41 per run, surpassing Sonnet 4.5, which reaches 65% at $10.13 per run. This corresponds to a 25× cost reduction while also achieving higher effectiveness. Similarly, GPT-5 Mini reaches 68% at $1.44 per run,
* 3 7 1 D Q R 5 H I
* 3 7 0 L Q L 5 H I
* 3 7 2 6 6 % 5 H I * 3 7 1 D Q R 6 6 * 3 7 2 6 6 % 6 6
* 3 7 0 L Q L 6 6
0 L V W U D O 0 H G L X P 6 6
. L P L . 5 H I
. L P L . 6 6
6 R Q Q H W 5 H I
6 R Q Q H W 6 6
0 L V W U D O 0 H G L X P 5 H I
/ O D P D 6 F R X W 5 H I / O D P D 6 F R X W 6 6 / O D P D % 6 6 / O D P D % 6 6 / O D P D % 5 H I
' H H S 6 H H N 9 5 H I
' H H S 6 H H N 9 6 6
/ O D P D % 5 H I
0 H D Q 7 R W D O & R V W 8 6 '
6 L Q J O H 6 W H S 5 H I L Q H P H Q W 3 D U H W R 6 L Q J O H 6 W H S 3 D U H W R 5 H I L Q H P H Q W
Figure 4: Mean total generation cost (USD) per benchmark run versus mutation score for all models on vague requirements under single-step (circles) and refinement (squares, scores and cost averaged over valid and mutated SUT conditions).
making it about 7× cheaper than Sonnet 4.5 while also outperforming it. More broadly, GPT-5 Nano (+19 pp) and GPT-5 Mini (+17 pp) benefit most from refinement on vague requirements, whereas models such as DeepSeek V3.2 (+12 pp) and Llama 4 Scout (+10 pp) improve more moderately. Sonnet 4.5 improves by 13 percentage points, but at a 2.3× cost increase (from $4.42 to $10.13 per run).
5
Discussion
RESTestBench was designed to evaluate the ability of REST API test generation tools to perform requirements-based test generation. Current approaches often start from the OAS and SUT implementation to generate tests, attempting to implicitly or explicitly reverse-engineer requirements from these artifacts as part of the test generation process (Section 2.1). Requirements-based tests should correctly differentiate between valid and faulty systems, which is only possible if the requirements are not assumed to correspond to the SUT’s behaviour — in contrast to regression testing which assumes the current implementation to be correct. By having explicit requirements our framework allows us to measure the effect of using information about the SUT’s behaviour on requirementsbased test generation. Accordingly, we expect 𝑀𝑆𝜙𝑚 to be zero for pure regression test generation approaches, as they would classify a faulty system as correct. In general, a test suite is considered good if it achieves a mutation score of 90% or more [4]. In our results (Figure 3), only Sonnet 4.5, GPT-5 Nano, and GPT-5 Mini exceed this threshold on precise requirements. Since such detailed requirements are rare in realworld settings or legacy code bases, the mutation score on vague requirements provides a better indicator of real-world usefulness. Nevertheless, no model achieved more than 90% on vague requirements, suggesting that revealing real bugs remains challenging for state-of-the-art models. Despite its strong single-step performance, Sonnet 4.5’s high cost often outweighs its benefits. Smaller models such as GPT-5 Nano
RESTestBench
or GPT-OSS 120B, when used in a refinement loop, achieve comparable or superior performance at minimal cost. This highlights that precise, human-defined requirements can substantially improve test quality, and that refinement loops enable smaller models to compensate for weaker initial reasoning. We observe that, for precise requirements, interacting with the SUT only slightly increases—or in some cases even decreases—the mutation score, as observed for Sonnet 4.5. One possible explanation is that exposing the actual behaviour of the SUT encourages models to validate aspects beyond the specified requirement, producing overly strong oracles that exceed the intended scope and consequently increase false positives, thereby reducing correctness. A closer examination of the false positive outputs reveals a qualitative gap in how models fail. Weak models such as the three Llama variants exhibit construction-level failures: asserting wrong HTTP status codes, hallucinating non-existent functions or API endpoints, and leaving hardcoded placeholder values. In contrast, Sonnet 4.5 produces virtually no code errors but is more prone to specificationinterpretation mismatches—assuming standard API response fields that this implementation omits, or expecting conventional status codes (e.g. 422) where the service returns 400. This underscores that the quality and detail of the requirement specification plays a critical role: weak models fail to construct valid tests regardless of specification quality, while strong models fail when the requirement leaves room for interpretation, causing them to fall back on general REST conventions rather than implementation-specific behaviour. Comparing 𝑀𝑆𝜙valid with 𝑀𝑆𝜙𝑚 confirms the observation of [27] that LLM test oracle accuracy considerably drops in the presence of buggy code, suggesting that predictions are biased towards the actual implementation rather than the desired one. Taking requirement detail into account, we observe that this drop is also larger on vague requirements. Moreover, tests generated on mutated code can negate the benefit of a refinement approach to such an extent that a non-refinement approach becomes more effective, as seen with Sonnet 4.5, GPT-5 Nano, Mistral Medium, and Llama 3.3 70B. This underscores that, for these models, incorporating actual SUT behaviour is unnecessary when requirement detail is high. While RESTestBench provides a manually validated set of requirements, most tools surveyed in Section 2.1 derive requirements automatically using LLMs without validation, suggesting that test correctness may further decrease when relying on unverified requirements.
defect, due to the undecidability of non-trivial program properties [36]. To mitigate this, we created a golden test set covering obvious correct behaviours. External Validity. Our benchmark includes only three services, which may not represent the full diversity and complexity of industrial software projects. Additionally, all generated tests were implemented in Python; results may differ under other programming languages, frameworks, or execution environments. Conclusion Validity. Each experiment was executed only three times, limiting the statistical robustness of our findings.
7
Conclusion
Since the emergence of LLMs, several studies have investigated their effectiveness in REST API testing. Unfortunately, their effectiveness has so far been evaluated primarily using code-based coverage and fault-detection metrics, which provide only limited insight into whether generated tests actually validate the intended functional behaviour of a service. With RESTestBench, we provide a means to close this gap by enabling the systematic and reproducible evaluation of generated tests against human-validated behavioural expectations and manually defined, requirement-aligned mutations, thereby building on and extending the PBMT framework introduced by Bartocci et al. [12]. To illustrate this potential and to revisit assumptions implicit in earlier tool-oriented approaches, we conducted two controlled experiments showing that (1) more precise requirement descriptions lead to substantially more effective tests, (2) refinement loops can improve effectiveness, especially in underspecified settings, (3) interaction with a faulty implementation may negatively influence the resulting tests by causing generation to adapt to incorrect behaviour, and (4) while small models may fail to reach mutated code sections, larger models can overengineer their test oracles, verifying beyond the requirement and reducing overall scores. These findings motivate future work to use the benchmark for comparing new test-generation strategies, investigating the influence of OpenAPI structure and quality, analyzing effectiveness–cost trade-offs across LLM-based approach and a lot more. In addition, we aim to extend the benchmark with other services and add microservice-oriented systems, such as Train Ticket in order to evaluate requirement-based test generation in more realistic multi-service settings.
Acknowledgments 6
Threats to Validity
Internal Validity. Several factors in our experimental setup could confound the observed results. First, prompt design can substantially influence model behaviour. Second, OpenAPI specification mismatches may exist, which can affect reported mutation scores, especially for more capable models. Construct Validity. Our measures rely on proxies for true software correctness and test effectiveness. The mutation operators are human-defined decisions about what constitute meaningful requirement violations, and the mutation-based metrics approximate the effectiveness of requirements-based test generation and oracle quality. Furthermore, we cannot algorithmically determine whether a failing requirements-based test reflects an invalid test or a latent
This work was supported by and done within the scope of the ITEA4 GENIUS project, which was nationally funded by FFG with grant 921454.
References [1] [n. d.]. Realworld-Apps/Realworld: "The Mother of All Demo Apps" — Exemplary Fullstack Medium.Com Clone Powered by React, Angular, Node, Django, and Many More. https://github.com/realworld-apps/realworld?tab=readme-ov-file. [2] 2026. Fastapi/Full-Stack-Fastapi-Template. FastAPI. [3] Juan C. Alonso, Sergio Segura, and Antonio Ruiz-Cortés. 2023. AGORA: Automated Generation of Test Oracles for REST APIs. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Seattle WA USA, 1018–1030. doi:10.1145/3597926.3598114 [4] Paul Ammann and Jeff Offutt. 2017. Introduction to Software Testing (2nd ed ed.). Cambridge university press, Cambridge. [5] Henry Andrews. 2026. OpenAPI Specification.
Kogler L., Hangler S., Ehrhart M., Dornauer B., Wuersching R., and Schrammel P.
[6] Andrea Arcuri. 2021. Automated Black- and White-Box Testing of RESTful APIs With EvoMaster. IEEE Software 38, 3 (May 2021), 72–78. doi:10.1109/MS.2020. 3013820 [7] Chetan Arora, Tomas Herda, and Verena Homm. 2024. Generating Test Scenarios from NL Requirements Using Retrieval-Augmented LLMs: An Industrial Study. In 2024 IEEE 32nd International Requirements Engineering Conference (RE). 240–251. arXiv:2404.12772 [cs] doi:10.1109/RE59067.2024.00031 [8] Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, Montreal, QC, Canada, 748–758. doi:10.1109/ ICSE.2019.00083 [9] Cristian Augusto, Antonia Bertolino, Guglielmo De Angelis, Francesca Lonetti, and Jesús Morán. 2025. Large Language Models for Software Testing: A Research Roadmap. doi:10.48550/ARXIV.2509.25043 [10] Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5 (May 2015), 507–525. doi:10.1109/TSE.2014.2372785 [11] Thiago Barradas, Aline Paes, and Vânia de Oliveira Neves. 2025. Combining TSL and LLM to Automate REST API Testing: A Comparative Study. doi:10.48550/ ARXIV.2509.05540 [12] Ezio Bartocci, Leonardo Mariani, Dejan Ničković, and Drishti Yadav. 2023. Property-Based Mutation Testing. In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, Dublin, Ireland, 222–233. doi:10. 1109/ICST57152.2023.00029 [13] Alistair Cockburn. 2012. Writing Effective Use Cases (24. print ed.). AddisonWesley, Boston. [14] Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java (Demo). In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, Saarbrücken Germany, 449–452. doi:10.1145/2931037.2948707 [15] Alix Decrop, Sara Eraso, Xavier Devroey, and Gilles Perrouin. 2025. A Public Benchmark of REST APIs. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, Ottawa, ON, Canada, 421–433. doi:10. 1109/MSR66628.2025.00072 [16] Serge Demeyer, Ali Parsai, Sten Vercammen, Brent Van Bladel, and Mehrdad Abdi. 2020. Formal Verification of Developer Tests: A Research Agenda Inspired by Mutation Testing. In Leveraging Applications of Formal Methods, Verification and Validation: Engineering Principles, Tiziana Margaria and Bernhard Steffen (Eds.). Vol. 12477. Springer International Publishing, Cham, 9–24. doi:10.1007/978-3030-61470-6_2 [17] David Fowler. 2026. Davidfowl/TodoApp. [18] Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2024. Testing RESTful APIs: A Survey. ACM Transactions on Software Engineering and Methodology 33, 1 (Jan. 2024), 1–41. doi:10.1145/3617175 [19] Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2024. Testing RESTful APIs: A Survey. ACM Transactions on Software Engineering and Methodology 33, 1 (Jan. 2024), 1–41. doi:10.1145/3617175 [20] Alex Groce, Josie Holmes, Darko Marinov, August Shi, and Lingming Zhang. 2018. An Extensible, Regular-Expression-Based Tool for Multi-Language Mutant Generation. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings. ACM, Gothenburg Sweden, 25–28. doi:10.1145/3183440.3183485 [21] Soneya Binta Hossain and Matthew B. Dwyer. 2022. A Brief Survey on Oraclebased Test Adequacy Metrics. doi:10.48550/ARXIV.2212.06118 [22] Linghan Huang, Peizhou Zhao, Huaming Chen, and Lei Ma. 2024. On the Challenges of Fuzzing Techniques via Large Language Models. doi:10.48550/ ARXIV.2402.00350 [23] Laura Inozemtseva and Reid Holmes. 2014. Coverage Is Not Strongly Correlated with Test Suite Effectiveness. In Proceedings of the 36th International Conference on Software Engineering. ACM, Hyderabad India, 435–445. doi:10.1145/2568225. 2568271 [24] Ivar Jacobson. 2005. Object-Oriented Software Engineering: A Use Case Driven Approach. Addison-Wesley Pub, Reading, Mass. [25] Lukas Jakob. 2026. Lujakob/Nestjs-Realworld-Example-App. [26] Leon Kogler, Maximilian Ehrhart, Benedikt Dornauer, and Eduard Paul Enoiu. 2025. RESTifAI: LLM-Based Workflow for Reusable REST API Testing. doi:10. 48550/ARXIV.2512.08706 [27] Michael Konstantinou, Renzo Degiovanni, and Mike Papadakis. 2024. Do LLMs Generate Test Oracles That Capture the Actual or the Expected Program Behaviour? doi:10.48550/ARXIV.2410.21136 [28] Nan Li and Jeff Offutt. 2017. Test Oracle Strategies for Model-Based Testing. IEEE Transactions on Software Engineering 43, 4 (April 2017), 372–395. doi:10. 1109/TSE.2016.2597136 [29] Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: Automated Black-Box Testing of RESTful Web APIs. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Virtual Denmark, 682–685. doi:10.1145/3460319.3469082
[30] Mohammed Mudassir and Mohammed Mushtaq. 2024. The Role of APIs in Modern Software Development. World Journal of Advanced Engineering Technology and Sciences 13, 1 (Oct. 2024), 1045–1047. doi:10.30574/wjaets.2024.13.1.0515 [31] Rangeet Pan, Raju Pavuluri, Ruikai Huang, Rahul Krishna, Tyler Stennett, Alessandro Orso, and Saurabh SInha. 2025. SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents. doi:10.48550/ ARXIV.2511.13305 [32] Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation Testing Advances: An Analysis and Survey. In Advances in Computers. Vol. 112. Elsevier, 275–378. doi:10.1016/bs.adcom.2018.03.015 [33] Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are Mutation Scores Correlated with Real Fault Detection?: A Large Scale Empirical Study on the Relationship between Mutants and Real Faults. In Proceedings of the 40th International Conference on Software Engineering. ACM, Gothenburg Sweden, 537–548. doi:10.1145/3180155.3180183 [34] André Pereira, Bruno Lima, and João Pascoal Faria. 2024. APITestGenie: Automated API Test Generation through Generative AI. doi:10.48550/ARXIV.2409. 03838 [35] Postman. [n. d.]. State of the API Report. Technical Report. [36] H. G. Rice. 1953. Classes of Recursively Enumerable Sets and Their Decision Problems. Trans. Amer. Math. Soc. 74, 2 (1953), 358–366. doi:10.1090/S0002-99471953-0053041-6 [37] Ana B. Sánchez, José A. Parejo, Sergio Segura, Amador Durán, and Mike Papadakis. 2024. Mutation Testing in Practice: Insights From Open-Source Software Developers. IEEE Transactions on Software Engineering 50, 5 (May 2024), 1130– 1143. doi:10.1109/TSE.2024.3377378 [38] A. Van Lamsweerde. 2000. Goal-Oriented Requirements Engineering: A Guided Tour. In Proceedings Fifth IEEE International Symposium on Requirements Engineering. IEEE Comput. Soc, Toronto, Ont., Canada, 249–262. doi:10.1109/ISRE. 2001.948567 [39] Dries Vanoverberghe, Jonathan De Halleux, Nikolai Tillmann, and Frank Piessens. 2012. State Coverage: Software Validation Metrics beyond Code Coverage. In SOFSEM 2012: Theory and Practice of Computer Science, Mária Bieliková, Gerhard Friedrich, Georg Gottlob, Stefan Katzenbeisser, and György Turán (Eds.). Vol. 7147. Springer Berlin Heidelberg, Berlin, Heidelberg, 542–553. doi:10.1007/ 978-3-642-27660-6_44 [40] Karl E. Wiegers, Joy Beatty, and Karl E. Wiegers. 2013. Software Requirements (3. ed. [fully updated and expanded] ed.). Microsoft Press, Redmond, Wash. [41] Zhenzhen Yang, Rubing Huang, Chenhui Cui, Nan Niu, and Dave Towey. 2025. Requirements-Based Test Generation: A Comprehensive Survey. doi:10.48550/ ARXIV.2505.02015 [42] Ke Zhang, Chenxi Zhang, Chong Wang, Chi Zhang, YaChen Wu, Zhenchang Xing, Yang Liu, Qingshan Li, and Xin Peng. 2025. LogiAgent: Automated Logical Testing for REST Systems with LLM-Based Multi-Agents. doi:10.48550/ARXIV.2503.15079 [43] Didar Zowghi and Vincenzo Gervasi. 2004. Erratum to “On the Interplay between Consistency, Completeness, and Correctness in Requirements Evolution”. Information and Software Technology 46, 11 (Sept. 2004), 763–779. doi:10.1016/j.infsof.2004.03.003
Received 02 March 2026; accepted 21 April 2026