REST API Testing with Verified LLM-Inferred Dependencies and Response-Driven Refinement

arXiv:2608.17546v1 [cs.SE] 18 Aug 2026

REST API Testing with Verified LLM-Inferred Dependencies and Response-Driven Refinement Tu Nguyen

Thanh Nguyen

Huy Nguyen

University of Science, VNU-HCM Ho Chi Minh City, Vietnam 23C15017@student.hcmus.edu.vn

University of Science, VNU-HCM Ho Chi Minh City, Vietnam 22127389@student.hcmus.edu.vn

University of Science, VNU-HCM Ho Chi Minh City, Vietnam 22127154@student.hcmus.edu.vn

Viet Nguyen

Tien N. Nguyen

Vu Nguyen*

University of Science, VNU-HCM Ho Chi Minh City, Vietnam 21C11045@student.hcmus.edu.vn

University of Texas at Dallas Richardson, TX, USA tien.n.nguyen@utdallas.edu

University of Science, VNU-HCM; Katalon LLC. Ho Chi Minh City, Vietnam nvu@fit.hcmus.edu.vn

or static analysis, to guide test generation [7]. Despite this progress, REST API testing remains difficult because many APIs expose stateful usage protocols: operations must be invoked in appropriate orders, and values produced by one operation are often required by later operations [8], [9]. These dependencies arise at several levels. At the operation level, one endpoint may need to precede another, such as creating a resource before retrieving, updating, or deleting it. At the parameter level, inputs may need to satisfy constraints involving identifiers, tokens, foreign keys, enum values, or domain-specific formats [10]. Across operations, response fields produced by one operation may serve as required inputs to another. Accurately exercising such producer–consumer relationships is vital for constructing executable API workflows and reaching meaningful server-side behavior [11], [9]. However, existing techniques often rely on heuristics, partial dynamic feedback, or search-based exploration, which can miss implicit dependencies or retain spurious ones. Recent work has explored large language models (LLMs) for REST API testing [12], [13], [14]. KAT [15] uses LLMs to infer operation and inter-parameter dependencies and to generate test sequences from OpenAPI specifications. AutoRestTest [13] constructs a dependency graph from the specification and uses LLM-guided agents to produce test sequences. These techniques show that LLMs can help interpret natural-language I. I NTRODUCTION RESTful APIs become the dominant architectural style for endpoint descriptions, parameter names, and schema informaweb services [1], [2]. Because serving as interfaces between tion that are difficult to capture using purely syntactic rules. However, directly trusting LLM-inferred dependencies can independently developed components, their correctness, robustmake REST API testing fragile. An LLM may infer a depenness, and security are critical to modern software reliability. dency between two operations because their names or descAutomated test generation for RESTful APIs has thus received riptions appear related, even when values produced by the substantial attention [3], [4]. Existing techniques include first operation cannot actually satisfy the second operation at black-box and white-box approaches [4]. Black-box tools runtime. Conversely, it may miss feasible operation chains generate requests from API specifications such as OpenAPI, when the dependency is implicit in the API behavior rather often using specification analysis, dynamic feedback, fuzzing, than explicitly documented in the specification. Such errors or search-based exploration [5], [6]. White-box techniques can lead to hallucinated dependency edges, infeasible request instead analyze server-side code, e.g., via symbolic execution chains, and repeated 4xx responses that provide little coverage * Corresponding author. of meaningful server-side behavior [16]. Generally, an LLM Abstract—Testing RESTful APIs requires generating sequences of API calls that satisfy dependencies among operations, parameters, and runtime-created resources. Recent LLM-based approaches infer such dependencies and generate test sequences from OpenAPI specifications, but they often treat LLM-inferred relationships as correct without execution-based validation. This can introduce spurious dependencies, miss feasible operation chains, and produce infeasible tests. In this paper, we propose APIP ILOT, an execution-validated framework for REST API testing. APIP ILOT first derives candidate producer-consumer dependencies from OpenAPI specifications using structural heuristics and LLM-based semantic reasoning. It then treats these dependencies as hypotheses and validates them through concrete API executions before using them for test generation. The validated dependencies are organized into a dependency graph from which APIP ILOT constructs coverage-aware workflows via bounded topk graph traversal, separating semantic dependency inference from sequence construction. To improve subsequent tests, APIP ILOT further performs response-driven refinement: runtime responses are analyzed to update resource pools, adjust input-generation constraints, and prune or revise invalid dependency mappings. Empirical evaluation on 16 real-world REST API services shows that APIP ILOT achieves 92.3% operation coverage, up to 58.6% code coverage, and an 88.1% workflow execution success rate, outperforming both LLM-based and traditional REST API testing baselines. APIP ILOT also detects 197 unique 5xx failures and specification–execution mismatches, demonstrating the benefit of grounding dependency inference in execution feedback.

1

prediction should be treated as a candidate hypothesis rather 1 openapi: 3.0.1 than as evidence that a dependency is useful for testing. 2 info : title : GitLab Commit API This paper proposes APIP ILOT, an execution-validated 3 framework for REST API testing. APIP ILOT first infers 4 paths : 5 "/ projects " : candidate producer–consumer dependencies from OpenAPI 6 get : summary: Get a list of all visible projects ... specifications using structural heuristics and LLM-based se- 7 post : mantic reasoning. Rather than directly relying on these inferred 8 9 summary: Create a project owned by the authenticated user . dependencies, APIP ILOT validates them through concrete API 10 "/ projects /{ id }/ repository /commits": get : executions and retains only dependencies that can support 11 summary: Get a list of repository commits in a project . executable request chains under the current testing context. The 12 13 post : validated dependencies are organized into a dependency graph, 14 summary: Create a commit by posting a JSON payload. which is then used to construct coverage-aware workflows via 15 "/ projects /{ id }/ repository /commits/{sha}": get : bounded top-k graph traversal. This design separates semantic 16 17 summary: Get a specific commit identified ... dependency inference from sequence construction: LLMs help 18 ... propose candidate dependencies, while workflow generation is 19 components: ... performed systematically over execution-validated evidence. Fig. 1: Endpoints of GitLab’s Repository Commits API APIP ILOT further improves testing through response-driven refinement. During execution, APIP ILOT records status codes, II. M OTIVATION response bodies, and extracted resource values in a contextual resource pool. Runtime responses are then used to refine subse- A. Motivating Example quent testing by updating (in)valid value pools, adjusting inputWe use GitLab’s Repository Commits API [18] to illustrate generation constraints, and pruning or revising dependency the challenges of dependency-aware REST API testing. In mappings that lead to repeated failures. Deterministic checks Fig. 1, we have API operations for listing projects, commits in derived from the schema are used for syntactic constraints, e.g., a project, creating commits, and retrieving a commit. To test the required fields, types, formats, enums, and documented status endpoint GET /projects/{id}/repository/commits/{sha}, codes, while LLM reasoning is reserved for under-specified a tool must construct an executable workflow that supplies both semantic constraints and treated as refinement guidance rather the project id and commit identifier sha. A feasible workflow: than as an oracle. A successful 2xx response is thus interpreted GET /projects → as evidence that a dependency is executable in current context. GET /projects/{id}/repository/commits → We evaluate APIP ILOT on 16 REST API services against GET /projects/{id}/repository/commits/{sha} . four baseline tools: DeepREST [17], a state-of-the-art non-LLM In this workflow, GET /projects produces a project id, approach, and three recent LLM-based tools: AutoRestTest, RESTifAI, and KAT. APIP ILOT achieves the highest aver- which can be used as the path parameter of GET /projects/ age operation coverage (92.3%), outperforming DeepREST {id}/repository/commits. The latter operation returns commit (60.4%), AutoRestTest (70.4%), RESTifAI (79.0%), and KAT objects containing a sha, which can then be used to retrieve a (68.1%). It also obtains the best average code coverage among specific commit. This shows that useful REST API tests require the instrumented services (47.0% branch coverage, 58.6% line producer-consumer reasoning across multiple operations. However, dependency inference from an OpenAPI specificacoverage, and 58.6% method coverage), the highest workflow execution success rate (88.1%), and a competitive token tion is error-prone. First, an approach may infer unnecessary cost of 155 tokens per generated test case. APIP ILOT also ordering constraints. For example, RESTifAI [14] and KAT [15] detects 197 unique 5xx failures and specification–execution may require POST /projects before testing GET /projects, mismatches, demonstrating the benefit of grounding LLM- although GET /projects can be executed independently and can already provide valid project identifiers. Such a dependency guided dependency inference in execution feedback. In summary, this paper makes the following contributions: unnecessarily lengthens the workflow and may make later • We propose an execution-validated dependency infer- operations unreachable if project creation fails. Second, similarity-based dependency inference may reence framework for REST API testing that treats LLMinferred producer–consumer relationships as hypotheses tain spurious edges. For example, AutoRestTest [13] constructs a Semantic Property Dependency Graph using emand validates them through concrete API executions. bedding similarity. This may incorrectly map short_id from • We design a coverage-aware workflow construction algorithm that derives bounded top-k executable work- GET /projects/{id}/repository/commits to the unrelated flows from validated dependencies, separating semantic id_after parameter of GET /projects, while missing the more useful dependency from commit sha to GET /projects/ dependency inference from test sequence generation. • We introduce response-driven refinement that uses run- {id}/repository/commits/{sha}. Such edges can produce time feedback to update resource pools, input-generation syntactically valid but semantically ineffective request chains. constraints, and revise invalid dependency mappings. Third, failed requests require more precise feedback than • We conduct extensive empirical evaluation on APIP ILOT . a status code alone. For example, a 404 response may be

2

caused by an invalid project id, a missing commit sha, or a faulty dependency mapping that injects a value from the wrong source. Treating all 404 responses as generic invalid-input failures can lead a tool to repeatedly mutate values without correcting the underlying dependency. Similarly, required fields and schema formats can often be checked deterministically from the OpenAPI specification, whereas under-specified semantic constraints require additional runtime evidence. These observations motivate a design principle: LLMs should help propose candidate dependencies, but their outputs should not be trusted as final test logic. Instead, candidate dependencies should be treated as hypotheses, validated through concrete API execution, and refined using runtime feedback. Fig. 2: APIPilot Architecture B. Key Ideas We design APIP ILOT around the following key ideas. Key Idea 1: Validate LLM-inferred dependencies by con- extracted resources, and repeated successes or failures is crete execution. Instead of directly accepting dependencies in- used to update whitelists, blacklists, parameter constraints, ferred from names, descriptions, or embeddings, APIP ILOT rep- and dependency-edge statuses. Third, LLM is invoked only resents each dependency as an executable candidate producer- for under-specified semantic constraints or ambiguous rootconsumer mapping. In our example, the candidate edge cause hypotheses that cannot be directly resolved from the from GET /projects to GET /projects/{id}/repository specification. Its output is treated as refinement guidance, not as /commits records that the response field id may supply the path an oracle, and is applied only when validated by later execution. parameter id. Each edge thus carries not only an operation pair, III. APIP ILOT W ORKFLOW but also the specific source field, target parameter, dependency type, and evidence used to infer it. A candidate dependency is Fig. 2 displays APIP ILOT’s workflow. The design prinple retained only when it can support an executable request chain. is to validate inferred dependencies: it first proposes producerKey Idea 2: Generate coverage-aware workflows by graph consumer dependencies from the specification, validates them traversal, not by LLM sequence synthesis. After dependency through concrete executions, constructs coverage-aware workvalidation, APIP ILOT organizes verified dependencies into an flows from the validated dependencies, and refines input Operation Dependency Graph (ODG) and derives executable generation and dependency mappings via runtime responses. workflows through bounded top-k graph traversal. We use the GitLab Commit API in Fig. 1 as a running It can also avoid unnecessary workflows that start with example. To test GET /projects/{id}/repository/commits/ POST /projects when GET /projects already provides valid {sha}, a tool must obtain a valid project id and a commit sha. project identifiers. The workflow construction rather than API One feasible workflow is as the one in the motivating example. execution is deterministic. Given the same validated graph, Here, GET /projects produces a project id; the commitranking function, depth bound, top-k value, random seed, and listing operation consumes this id and produces commit objects initial service state, APIP ILOT produces the same workflow containing sha; the final operation consumes both values to set. However, the actual API responses may still vary if the retrieve a specific commit. APIP ILOT is designed to discover, black-box service state changes outside APIP ILOT’s control. validate, and exploit such workflows without blindly trusting Key Idea 3: Refine values and dependencies from runtime either heuristic matches or LLM-inferred dependencies. responses. During execution, APIP ILOT records response values in a contextual resource pool and updates this pool A. Overview Algorithm 1 summarizes the end-to-end procedure of APIPusing runtime feedback. Successful responses add validated resources, such as project ids and commit shas, to a whitelist ILOT. Details on each component are given later. 1) Candidate dependency inference: APIP ILOT parses the for later reuse. Repeated failures add invalid values or mappings to a blacklist. For example, if a workflow repeatedly produces OpenAPI specification to extract operations, path/query/body 404 responses when a value extracted from one field is injected parameters, request schemas, response schemas, status codes, into an unrelated parameter, APIP ILOT can suppress that value and textual descriptions. It then constructs candidate producer– consumer dependencies using two complementary sources of source or revise the corresponding dependency edge. APIP ILOT refines test generation through a layered feedback evidence. Structural heuristics capture common REST patterns, process, instead of delegating all response analysis to LLM. such as identifier propagation, shared path templates, resource First, deterministic OpenAPI validation checks schema-level hierarchies, lifecycle patterns, and hypermedia links. LLMproperties, e.g., required fields, primitive types, formats, enums, based semantic reasoning is used to analyze cases that are ranges, documented status codes, and response schemas. difficult to resolve from syntax alone, such as when parameter Second, execution evidence from status codes, response bodies, names and response fields are semantically related but not

3

Algorithm 1 APIP ILOT Overview

If GET /projects already provides valid project IDs, it does not need to force an unnecessary prefix such as POST /projects. 4) Response-driven refinement: During execution, APIPILOT records responses and uses them to refine both value generation and dependency mappings. Successful responses contribute reusable values, such as project ids and commit shas, to a contextual resource pool. Repeated failures are used to blacklist invalid values, deprioritize unreliable sources, or prune faulty dependency edges. For example, a 404 response from GET /projects/{id}/repository/commits/{sha} may indicate that the project id is invalid, the commit sha does not belong to that project, or the dependency mapping injected a value from the wrong response field. APIP ILOT distinguishes these cases using the request, response, specification constraints, and dependency provenance, and updates subsequent tests.

1: Input: Specification S, testing budget B, depth limit dmax , top-k 2: Output: Executed tests, validated dependencies, detected failures 3: Parse S to obtain operations, parameters, schemas, and responses 4: Infer candidate dependencies with structural heuristics and LLM reasoning 5: Initialize the Contextual Resource Pool and per-field test configurations 6: for each testing iteration within budget B do 7: Validate candidate dependencies through concrete executions 8: Update the ODG with validated, rejected, and refined dependencies 9: Construct top-k workflows from the validated ODG using bounded traversal 10: for each workflow do 11: Generate request inputs using schema rules, mutation, and resourcepool values 12: Execute the request sequence, record status codes and responses 13: Extract reusable resource values into the Contextual Resource Pool 14: Refine value pools, parameter constraints, and dependency mappings from responses 15: end for 16: end for 17: return executed tests, refined ODG, and detected failures

B. Dependency Representation The ODG is not intended to introduce a new graph abstraction; rather, it is the internal representation used by APIP ILOT to store dependency evidence. Let O be the set of operations extracted from the OpenAPI specification. Each operation o ∈ O has input parameters Po and response fields Ro . A dependency edge is represented as:

lexically identical. For example, in the GitLab API, structural analysis can identify that the response field id from GET /projects is a plausible source for the path parameter id in GET /projects/{id}/repository/commits. Similarly, semantic reasoning can help identify that the sha field returned by the e = ⟨os , ot , r, p, type, evidence, status⟩, commit-listing operation is the appropriate value for the sha parameter in GET /projects/{id}/repository/commits/{sha}. where o is the source operation, o is the target operation, s t 2) Execution validation: Candidate dependencies are r ∈ Ros is the producer response field, p ∈ Pot is the consumer not immediately used as final test logic. Instead, APIP I - parameter, type records the dependency category, evidence LOT validates them by executing concrete API calls and records how the dependency was inferred, and status records checking whether the produced values can support the con- whether the edge is candidate, validated, rejected, or refined. sumer operation. In our example, APIP ILOT executes GET This representation is more precise than a plain operation/projects, extracts a concrete project id, injects it into GET level edge. For example, the edge /projects/{id}/repository/commits, and observes whether ⟨ GET /projects, GET /projects/{id}/repository/commits, the operation can execute under the generated context. If id, id, identifier, heuristic+execution, validated ⟩ successful, the dependency from project id to the path states that the response field id produced by GET /projects parameter id is promoted from a candidate dependency to supplies the path parameter id of the commit-listing operation. a validated dependency. In contrast, a spurious mapping such It does not claim that every value produced by GET /projects as using commit short_id as the id_after parameter of GET can be used by the target operation. /projects is not promoted unless execution supports it. A candidate dependency is created only when the producer field and consumer parameter are compatible with evidence: 3) Coverage-aware workflow construction: After validation, APIP ILOT organizes the validated dependencies cand(e) ⇐⇒ compatible(r, p) ∧ evidence(os , ot , r, p). into an ODG. Each node is an API operation, and each edge records a validated producer–consumer relationship Here, compatible(r, p) checks schema compatibility (type, forbetween a response field and a target parameter. APIP I - mat, enum, path context, and identifier structure). The predicate LOT then constructs executable workflows by traversing this evidence(os , ot , r, p) captures structural, semantic, lifecycle, graph backward from each target operation and retaining hypermedia, or LLM-based evidence showing that r supplies p. A candidate dependency becomes validated only after the top-k shortest and most feasible workflows under a execution provides evidence that a value produced by os can depth bound. This separates dependency inference from sesupport the execution of ot , that is, valid(e) ⇐⇒ quence generation: LLMs help propose candidate dependencies, but workflows are generated systematically from validated ∃v ∈ V al(os , r) : Exec(ot [p ← v]) ∈ Success ∪ P rogress. graph evidence. In our example, the workflow targeting GET /projects/{id}/repository/commits/{sha} is derived by V al(os , r) denotes values extracted from field r in responses of resolving its required inputs. The sha parameter can be sup- os . Success usually corresponds to documented 2xx responses. plied by GET /projects/{id}/repository/commits, whose P rogress captures cases where the injected value moves the id parameter can in turn be supplied by GET /projects. Thus, request beyond a missing-resource or missing-parameter failure, APIP ILOT constructs the workflow as in the motivation section. even if another independent constraint still causes the request

4

To validate a candidate edge ⟨os , ot , r, p, type, evidence, candidate⟩, APIP ILOT executes the source operation os , extracts concrete values from field r, and uses them to instantiate parameter p of the target operation ot . If the operation succeeds under schema-valid inputs, or if the injected value enables progress beyond a previous missing-resource/parameter failure, the edge is marked as validated. If the same mapping repeatedly causes failures, the edge is rejected or deprioritized. A successful 2xx response is treated as evidence that the dependency is executable in the current testing context, not as proof of full business-level correctness. Conversely, a 4xx response does not automatically invalidate an edge: it may result from an unrelated missing parameter, an invalid optional field, authorization, or an inconsistent resource state. Therefore, APIP ILOT records the response together with the request and dependency provenance for later refinement.

to fail. Thus, validation provides evidence that a dependency is executable in the current testing context; it is not a proof of full business-level correctness. We further distinguish validated dependencies as "required" or "optional". A validated dependency is required if the target parameter p is required by the OpenAPI specification or if repeated executions show that omitting or replacing p prevents progress. It is optional if the produced value can improve realism or coverage but the target operation can still execute through another value source or without that parameter. Lifecycle evidence is never treated as required by itself; it only prioritizes candidate orderings until a concrete producerconsumer mapping is validated by execution. We consider four dependency types. An identifier dependency connects a created resource ID to a required input ID. A schema dependency connects a response attribute to a compatible input field. A lifecycle dependency captures REST ordering evidence, e.g., POST before GET, PUT, or DELETE on the same resource; it is treated as candidate evidence rather than sufficient proof of a valid dependency. A hypermedia dependency is derived from explicit links in responses when available.

E. Executable Workflow Graph (EWG) Construction

Once candidate dependencies are validated, APIP ILOT constructs workflows from the validated ODG. For each target operation, it performs a bounded backward traversal to resolve required inputs. Candidate predecessor operations are ranked by dependency status, dependency type, source reliability, path C. Candidate Dependency Inference specificity, and workflow length. The traversal avoids cycles APIP ILOT infers candidate dependencies using structural and and retains only the top-k workflows per target operation. We semantic evidence. The structural phase applies deterministic define the ranking of a workflow W lexicographically as: rules over paths, methods, parameters, and schemas. It matches rank(W)=⟨ unresolved(W), |W|, -validated(W), failed(W) ⟩. response fields to target parameters using exact and normalized Here, unresolved(W ) is the number of required inputs that names, such as id, project_id, and projectId; checks type remain unresolved, |W | is the workflow length, validated(W ) compatibility between producer fields and consumer parameters; is the number of validated dependency edges used in the identifies hierarchical path relationships, such as /projects workflow, and f ailed(W ) is the number of edges or values preand /projects/{id}/repository/commits; and records REST viously associated with failures. APIP ILOT prefers workflows lifecycle patterns and hypermedia links when present. To avoid with fewer unresolved inputs, shorter length, more validated excessive dependencies, generic fields such as name, text, dependencies, and fewer failure-prone mappings. This ranking description, etc. are not retained unless supported by stronger avoids introducing additional weighting parameters and makes evidence, e.g., type compatibility, schema, or execution. the top-k selection deterministic. The semantic phase targets dependencies that structural rules Here, “deterministic” refers to workflow construction rather may miss. APIP ILOT first uses an embedding model to retrieve than API execution. Given the same validated graph, ranking semantically similar parameter–field pairs from the specififunction, depth bound, top-k value, random seed, and initial cation. The LLM then examines the operation descriptions, service state, APIP ILOT produces the same workflow set. parameter descriptions, schema context, and candidate pair However, the actual API responses may still vary if the blackto decide whether the pair plausibly represents a producer– box service state changes outside its control. If several validated consumer relationship. The LLM output is constrained to a sources are available, APIP ILOT ranks them and retains the structured format that identifies the source operation, target top-k shortest workflows that cover the target operation. operation, source field, target parameter, dependency type, and explanation (the prompt templates provided in the replication F. Test Data Generation package [19]). Importantly, this output only creates a candidate For each operation in a workflow, APIP ILOT generates edge; it does not by itself validate the dependency. request inputs from a per-field test configuration. This configuration is derived from the OpenAPI specification and records the D. Execution Validation and Resource Pool generation strategy for each parameter or request body field. The Contextual Resource Pool stores values extracted from Schema-level constraints, such as required fields, primitive successful API responses. Each stored value is associated with types, formats, enums, array bounds, and numeric ranges, its source operation, response field, schema context, observed are handled deterministically. For fields with well-defined status code, and dependency provenance. During execution, constraints, APIP ILOT uses rule-based, boundary, or mutationdependency-based generators retrieve values from this pool based generators. For fields that depend on prior responses, and inject them into parameters of downstream operations. it uses the Dependency Generator to retrieve values from the

5

Contextual Resource Pool. For under-specified fields that cannot while new producer–consumer relationships discovered from be instantiated from schema rules or resource values, APIP ILOT response bodies can be added as candidates for validation. may use an LLM value generator, but these generated values are In the GitLab example, suppose GET /projects/{id}/ treated as candidates and refined through execution feedback. repository/commits/{sha} returns 404. APIP ILOT does not The generator does not enumerate all possible values, which immediately treat the request as a generic invalid input. If would be infinite. Instead, it samples a finite set of field- the project id was previously validated but sha came from a presence combinations and value strategies under a testing different project, the refinement module attributes the failure budget. For example, it can generate required-only requests, to an inconsistent resource pairing and records that sha values required-plus-one-optional requests, all-field requests, boundary must be reused with the project id from which they were cases, and intentionally invalid variants. A request is expected obtained. If the sha came from an unrelated response field, the to succeed when required fields are present, schema-level corresponding dependency edge is deprioritized or rejected. If constraints are satisfied, and dependency-sourced values are the response reveals a new reusable identifier or constraint, the drawn from validated resource pools. A request is expected resource pool and test configuration are updated for later. to fail when APIP ILOT intentionally removes required fields, violates schema constraints, mutates values outside allowed IV. E MPIRICAL E VALUATION domains, or uses blacklisted resource values. In the GitLab workflow, APIP ILOT uses the resource pool to We conduct experiments to answer the following questions: fill the path parameter id with a validated project identifier and 1) RQ1 (Effectiveness and Efficiency). How effective and sha with a validated commit hash. Optional query parameters efficient is APIP ILOT in comparison with the baselines? are then varied according to their schema constraints and 2) RQ2 (Workflow Exploration). How effectively does APIPmutation budget. This allows APIP ILOT to exercise both ILOT generate and execute API request sequences to explore nominal scenarios and error-handling behavior without relying API workflows compared to the baselines? on an LLM to synthesize entire requests from scratch. 3) RQ3 (Error Detection). How effective is APIP ILOT in generating tests to detect API faults? G. Response-driven Refinement 4) RQ4 (Token and Cost Efficiency). How does APIP ILOT After each request, we analyze the response and updates achieve a balance between token efficiency and effectiveness? its testing state. From Key Idea 3, response-driven refinement Experiment Setup: APIP ILOT runs in fully automated mode follows a three-layer design. First, deterministic OpenAPI without any user-supplied configuration overrides, ensuring validation checks schema-level properties, including required a fair comparison with the fully automated baselines. All fields, primitive types, formats, enums, ranges, documented experiments are conducted on two machines, each equipped status codes, and response schemas. Second, execution evi- with an AMD Ryzen 7 3700X processor (3.6 GHz, 8 cores, dence from status codes, response bodies, extracted resources, 16 logical processors), 16 GB of RAM, and a GPU. To and repeated successes or failures is used to update resource ensure consistent testing conditions, services are restarted whitelists, blacklists, parameter constraints, and dependency- and their databases restored at the beginning of each session, edge statuses. Third, LLM reasoning is invoked only for preventing state carryover across runs. All services run under under-specified semantic constraints or ambiguous root-cause default configurations and database settings. To avoid resource hypotheses that are not directly recoverable from the schema. interference, each service and testing tool is allocated dedicated LLM output is treated only as a candidate explanation. resources and executed sequentially. Schema violation: the request violates a documented type, format, enum, range, or required-field constraint. • Invalid resource value: a dependency-sourced value, e.g., a project id or commit sha, is rejected by the operation. • Faulty dependency mapping: the source field is incompatible with the target parameter, even if their names or descriptions appear similar. • Missing or inconsistent context: the target operation requires a resource state not established by the workflow. • Specification–execution mismatch: the observed status code or response is inconsistent with the specification. •

Baselines: We compare APIP ILOT against 4 state-of-the-art REST API testing tools: AutoRestTest [13], RESTifAI [14], KAT [15], and DeepREST [17]. While DeepREST is a stateof-the-art non-LLM approach, the others are LLM-based approaches to REST API testing that collectively cover the primary paradigms: KAT uses GPT-based prompting to construct dependency graphs and generate test scripts; AutoRestTest combines LLMs with a semantic property dependency graph and multi-agent reinforcement learning; and RESTifAI employs LLMs to generate reusable, CI/CD-ready test workflows with error-driven retry. Moreover, their datasets are publicly available to enable a rigorous and reproducible comparison. To ensure a fair comparison, all tools are configured to use the same underlying LLM, OpenAI GPT-4.1-mini, with temperature set to 0 to minimize hallucinations and ensure deterministic output. Each experiment is repeated five times, and we report the average results to mitigate potential randomness.

The resulting feedback is used to update both data generation and the ODG. Successful values are added to whitelists and prioritized in later requests. Values repeatedly associated with failures are blacklisted or deprioritized. Parameter generators are tightened or relaxed based on observed constraints. Candidate edges that repeatedly produce invalid requests are pruned,

6

Datasets: We evaluate APIP ILOT on a diverse benchmark of 16 real-world RESTful API services. The core dataset is drawn from established benchmarks used in prior work, including AutoRestTest, KAT and RESTifAI. We use these services because they already subsume the baselines’ benchmarks, allowing direct and fair comparison under identical experimental conditions used in these studies. To assess structural code coverage, we augment the benchmark with two open-source applications, jhipster-app [20] and petclinic-rest [21], for white-box coverage via JaCoCo [22], which was reported only for the locally instrumentable open-source services. Evaluation Metrics: We use metrics in three dimensions: Operation Coverage (OC) measures the percentage of API operations successfully invoked, i.e., those yielding at least one 2xx response, relative to the total number of operations defined in the specification. It serves as the primary black-box indicator of a tool’s ability to construct valid request sequences. To evaluate response diversity beyond successful invocations, we measure 2xx Coverage and 4xx Coverage. Following the evaluation criteria in KAT [15], 2xx Coverage captures the proportion of unique documented 2xx status codes triggered by the test suite, reflecting the breadth of happy-path scenario coverage. 4xx Coverage measures the proportion of documented 4xx codes triggered, assessing the ability to exercise client-side error handling, e.g., invalid inputs and constraint violations. For white-box evaluation on our open-source services, we collect Line Coverage (LC), Branch Coverage (BC), and Method Coverage (MC) using JaCoCo, with all coverage metrics expressed as percentages. To measure test generation efficiency, we define the Test Efficiency (Eff ) metric as:

(68.1%), and DeepREST (60.4%). At the service level, APIPILOT reaches near-full coverage on Genome Nexus, Languagetool, Restcountries, FDIC, and multiple GitLab services such as Branch and Issues. In more complex, interdependent APIs such as GitLab Groups and GitLab Issues, APIP ILOT substantially outperforms all competing tools, demonstrating its ability to handle diverse and deeply nested operation dependencies. A closer look reveals that APIP ILOT achieves high operation coverage through its ability to identify flexible execution paths via the ODG and the Executable Workflow Graph. Unlike approaches that rely on rigid, linearly ordered dependency chains, APIP ILOT systematically discovers alternative paths to reach target operations. For example, RESTifAI relies on strict dependency chains such as POST /projects → POST /projects/{id}/repository/commits → GET /projects/{id}/repository/commits/{sha}, where a failure at any step renders the target endpoint unreachable. AutoRestTest similarly struggles with such stateful scenarios, often incurring high test case counts or failing to construct valid parameter combinations for deeply nested endpoints. In contrast, APIP ILOT discovers alternative paths, such as GETbased sequences that bypass resource creation prerequisites, enabling robust and accessible coverage for several operations. Among the baselines, RESTifAI achieves a competitive average operation coverage of 79.0% but has unstable performance, with notably low coverage on several GitLab services (e.g., 29.3% for GitLab Commit and 45.2% for GitLab Issues). This instability stems from its heavy reliance on LLM-generated sequences without any verification, making it sensitive to hallucinated dependencies. AutoRestTest, while more consistent, generally achieves lower coverage than APIP ILOT, particularly #T C on large-scale or complex APIs. KAT, despite occasionally Eff = (OC/100 × #Ops) performing well on individual services, records the lowest average operation coverage overall (68.1%), struggling most where #T C is the total number of generated test cases, OC on APIs with complex multi-operation dependencies. Finally, is the achieved operation coverage percentage, and #Ops is the non-LLM baseline DeepREST lag behind other approaches. the total number of operations in a service. A lower Eff value Status Code Coverage. For successful responses, APIP ILOT indicates a more compact and less redundant test suite. leads with an average 2xx coverage of 92.3%, reflecting To assess fault detection capability, we monitor 5xx error effective exploration of valid API behaviors and happy-path responses, specification–execution mismatches, and undocscenarios. It also achieves competitive 4xx coverage at 67.8%, umented status codes. Unlike 2xx and 4xx codes, which demonstrating its ability to trigger client-side error conditions reflect expected API behaviors, 5xx errors indicate unhandled essential for robustness testing. The combination of high 2xx server-side crashes or critical logical flaws. normalized failure and 4xx coverage indicates that APIP ILOT exercises both the key. This key combines the API operation, HTTP method, nominal and error-handling behavior of each API, rather than status code, normalized error response, and response structure. focusing exclusively on one code. Normalization removes volatile values such as identifiers, Source Code Coverage. A key observation from Table II timestamps, random strings, etc. Such mismatches capture is that higher operation coverage does not always translate discrepancies between the status codes listed in the specification to higher code coverage. AutoRestTest slightly surpasses and those observed at runtime, reflecting either execution APIP ILOT in operation coverage on certain services, yet yields failures or outdated specifications. Undocumented status codes consistently lower branch, line, and method coverage. This are those from execution but not in the specification. discrepancy stems from its fuzzing strategy, which generates large numbers of semi-random inputs that predominantly trigger A. Effectiveness and Efficiency (RQ1) 4xx responses without penetrating deeper execution paths. Operation Coverage. As seen in Table I, APIP ILOT achieves APIP ILOT achieves the highest average method coverage the highest average operation coverage at 92.3%, significantly at 58.6%, outperforming DeepREST (42.7%), AutoRestTest outperforming AutoRestTest (70.4%), RESTifAI (79.0%), KAT (56.3%), KAT (53.2%), and RESTifAI (52.2%). It also leads in

7

TABLE I: Operation coverage (2xx) and efficiency comparison. 2xx/4xx: status code coverage(%); Eff: test generation efficiency. GN: Genome Nexus, LT: Languagetool, RC: Restcountries, JA: jhipster-app, PR: petclinic-rest.(RQ1) Service

#Ops LOC

DeepREST #TC 2xx 4xx Eff.

AutoRestTest #TC 2xx 4xx Eff.

RESTifAI KAT #TC 2xx 4xx Eff. #TC 2xx 4xx Eff.

APIP ILOT #TC 2xx 4xx Eff.

211 64.3 12.6 14.3 33 100 100 16.5 211 91.0 92.7 10.5 379 80.0 100 12.5 401 93.7 61.5 12.2

287 87.0 62.6 14.3 9 100 100 4.5 134 80.8 86.9 7.5 530 61.6 86.8 22.7 387 66.3 48.5 16.7

2,374 100 100 103.2 218 100 100 109.0 3,745 100 100 170.2 5,550 89.5 100 163.3 6,864 82.9 54.3 236.7

Canada.H. 4 Bills API 21 GLCom 15 GLGrps 17 GLIss 27 GLBr 9 GLPro 31 GLRepo 10 FDIC 6 PetStore 19 OhSome 134

– 800 100 75.0 200.0 13,000 100 100 3250.0 46 100 100 11.5 51 100 100 12.8 – 264 38.1 41.2 12.6 8,000 77.2 80.9 493.8 200 76.0 57.4 12.5 199 76.2 78.7 12.5 – 11,250 26.7 100 750.0 25,000 73.3 100 2273.6 81 29.3 30.7 18.4 444 16.5 100 179.4 – 14,450 76.5 94.1 850.0 30,000 70.6 100 2500.0 86 45.9 51.8 11.0 429 50.1 100 50.5 – 36,450 77.8 96.3 1350.0 20,000 94.1 100 787.9 204 45.2 51.1 16.7 816 37.8 100 80.0 – 4,050 66.7 66.7 450.0 10,000 100 100 1111.1 101 71.1 100 15.8 196 53.3 100 40.8 – 14,450 70.6 100 466.1 30,000 91.0 100 1063.8 215 50.3 62.6 13.8 815 64.7 100 40.7 – 3,200 50.0 87.5 320.0 15,000 68.0 100 2205.9 92 66.0 93.3 13.9 230 37.2 100 61.8 5,000 100 100 833.3 111 90.0 93.3 20.6 152 100 100 25.3 – 1,800 100 100 300.0 – 2,450 85.7 13.3 128.9 100,000 87.4 73.8 6024.1 208 84.2 64.9 13.0 185 66.3 68.4 14.7 2,518 37.2 22.4 50.6 2,245 95.4 49.5 17.5 2,355 79.6 27.0 22.1 – 5,539 67.9 18.7 41.3

350 100 100 87.5 2,620 85.7 64.3 145.6 1,878 73.3 100 170.9 1,593 80.0 100 117.1 3,617 96.3 100 139.1 803 100 100 89.2 4,030 90.3 100 143.8 1,056 80.0 100 132.0 321 100 100 53.5 2,391 85.2 29.8 147.8 4,141 98.1 29.3 31.5

Average

– 8,250.6 60.4 59.4 380.8 912,518 70.4 68.2 5323.4 4,824 79.0 69.2 14.6 7,219 68.1 65.5 36.5 41,551 92.3 67.8 129.8

23 20k 2,554 82.6 4.3 111.0 4,000 100 100 173.9 2 241k 200 0.0 0.0 100.0 100,000 100 100 50000.0 22 2.4k 3,912 86.4 90.9 177.8 150,000 100 100 6818.2 38 6.3k 18,070 2.6 21.1 475.5 200,000 90.0 100 5848.0 35 5.9k 12,570 34.3 41.9 359.1 200,000 69.7 66.8 8205.5

GN LT RC JA PR

413

TABLE II: Branch(BC)/line(LC)/method(MC) Coverage (RQ1).

covered operation. As shown in Table I, APIP ILOT requires

Ser.

DeepREST AutoRestTest RESTifAI KAT APIPilot 129.8 test cases per covered operation on average, higher BC LC MC BC LC MC BC LC MC BC LC MC BC LC MC than RESTifAI (14.6) and KAT (36.5), but significantly lower

GN LT RC JA PR

21.1 32.4 27.6 44.5 56.9 51.2 34.5 43.8 34.1 34.9 42.6 34.7 49.4 62.5 59.7than DeepREST(380.8), AutoRestTest (5323.4). RESTifAI and 0.1 2.9 4.3 4.5 21.4 15.1 1.7 22.6 13.9 4.2 20.0 16.2 5.4 30.3 16.9KAT achieve lower Eff values primarily due to conservative 84.5 76.4 84.7 91.7 78.0 84.7 81.4 74.2 83.5 81.4 74.2 83.5 86.8 77.3 84.6 29.3 42.5 45.0 54.5 74.8 74.6 47.5 72.2 71.1 46.4 70.5 73.4 55.2 77.3 75.7exploration strategies that rely on LLM-guided generation 34.4 40.3 51.9 38.0 44.2 55.7 40.5 48.4 58.2 39.8 48.1 58.0 38.0 45.5 56.0or localized mutations, producing inputs close to previously

Avg. 33.9 38.9 42.7 46.6 55.1 56.3 41.1 52.2 52.2 41.3 51.1 53.2 47.0 58.6 58.6observed patterns. While this reduces test suite size, it also

limits coverage of complex or less obvious behaviors, as reflected in KAT’s near-zero 5xx error count and its operation coverage 23.3 percentage points below APIP ILOT, despite average line coverage at 58.6%, compared to 55.1%, 51.1%, and comparable token cost per test case (160.8 vs. 154.7). RESTifAI 52.2% for the other tools respectively. For branch coverage, it similarly generates fewer test cases but consumes approximately reaches 47.0% on average, exceeding DeepREST (33.9%), RES19,602.9 tokens per test case, roughly 128 times more than TifAI (41.1%) and KAT (41.3%), and AutoRestTest (46.6%). APIP ILOT making it the least token-efficient approach by a These gains are largely attributable to two complementary wide margin. In both cases, the apparent efficiency of these mechanisms. First, APIP ILOT employs a systematic, algorithmtools comes at the cost of substantially reduced coverage. driven data generation strategy (Algorithm 1, line 11) that explores both required and optional parameter combinations, B. Workflow Exploration (RQ2) including edge cases and incomplete inputs. For instance, Table III reports the number of successfully executed and GET /projects in the GitLab Commit API supports optional total generated workflow sequences for each tool, along with parameters such as id_after for pagination, i.e. cases that their execution success rates. APIP ILOT generates substantially LLM-based approaches like RESTifAI and KAT frequently more execution sequences (875 in total) than RESTifAI (413), overlook, limiting their input diversity and code coverage. reflecting its ability to construct multi-step workflows that Second, APIP ILOT’s response-driven refinement continuously capture dependencies across operations rather than mapping refines input constraints based on observed runtime failures. each test to a single endpoint. Despite this broader exploFor example, although min_access_level in GET /projects ration, APIP ILOT maintains a high execution success rate of accepts values in {0, 10, 20, 30, 40, 50}, values such as 0 88.1%, outperforming RESTifAI (83.1%), and KAT (86.6%). may still produce invalid responses; APIP ILOT detects and DeepREST is excluded from this comparison because it tests suppresses such patterns over successive iterations. Similarly, operations independently rather than constructing multi-step erroneous LLM-inferred dependencies such as misusing id dependency workflows. This indicates that the additional from GET /projects as namespace_id in POST /projects, are sequences generated by APIP ILOT are not speculative, but they identified and corrected, preventing repeated 404 failures. are semantically validated and consistent with API constraints. Test Generation Efficiency. Test generation efficiency is measured as the number of test cases required per successfully

The advantage is most pronounced on complex, statedependent services. In GitLab Projects, APIP ILOT successfully

8

TABLE III: Workflow Execution (RQ2). Total/Succ/Rate: the total and successful numbers, and success rate. Service

RESTifAI KAT APIP ILOT Succ. Total Rate Succ. Total Rate Succ. Total Rate

GN LT RC JA PR

22 2 20 32 31

23 2 22 38 35

95.7 100 90.9 84.2 88.6

Canada.H. 4 Bills API 18 GLCom 4 GLGrps 8 GLIss. 13 GLBr. 9 GLPro. 17 GLRepo 7 6 FDIC PetStore 16 134 OhSome

4 21 15 17 27 9 31 10 6 19 134

100 4 85.7 24 26.7 26 47.1 30 48.2 48 100 16 54.8 48 70.0 10 100 6 84.2 20 100 121

Total

413 83.1 484

343

28 2 22 45 34

31 90.3 52 2 100 2 22 100 22 56 80.4 103 49 69.4 43

71 2 22 119 54

73.2 100 100 86.6 79.6

4 30 28 32 48 16 59 18 6 24 134

12 48 41 47 94 23 124 24 6 54 134

100 89.6 73.2 89.4 95.7 100 90.3 62.5 100 79.6 99.3

100 12 80.0 43 92.9 30 93.8 42 100 90 100 23 81.4 112 55.6 15 100 6 83.3 43 90.3 133

559 86.6 771

TABLE IV: Error detection (5xx, Undocumented) (RQ3). Service

DeepREST AutoRestTest RESTifAI KAT APIPilot 5xx Undoc 5xx Undoc 5xx Undoc 5xx Undoc 5xx Undoc

GN LT RC JA PR

0 0 1 3 32

15 0 1 45 21

0 1 1 13 35

34 2 61 178 23

0 0 0 10 29

32 2 14 52 19

0 0 0 3 5

19 1 24 61 14

0 0 0 25 29

16 2 19 58 4

Canada. Bills API GLCom. GLGrps. GLIss. GLBr. GLPro. GLRepo FDIC PetStore OhSome

0 0 1 1 5 0 0 0 0 2 0

1 0 1 1 10 16 5 10 6 2 0

0 1 15 17 27 9 31 10 6 11 1

8 32 3 2 5 28 5 24 12 39 16

0 1 0 2 2 0 2 0 3 6 96

2 0 6 10 20 12 19 10 0 0 0

0 0 0 0 0 0 0 0 0 0 0

2 2 17 24 44 12 48 11 0 8 0

0 0 1 6 4 1 7 1 1 12 110

2 3 38 18 27 24 35 23 1 1 0

Total

45

134

178

472

151 198

8

287

197 271

875 88.1

DeepREST, RESTifAI, and KAT, but fewer than AutoRestTest. This is expected as APIP ILOT prioritizes dependency-valid workflows and avoids repeatedly exploring invalid or blacklisted values, whereas AutoRestTest’s aggressive mutation strategy generates a large volume of inputs that can reveal more raw 5xx errors. However, on OhSome, AutoRestTest requires substantially more time to fuzz effective inputs and detects fewer 5xx errors before timeout. Overall, APIP ILOT’s strength lies in achieving higher operation and workflow coverage through deeper dependency-valid sequences, rather than maximizing raw 5xx exploration. We plan to improve its 5xx detection by adding more targeted exploration heuristics. Undocumented Status Codes. APIP ILOT detects 271 undocumented responses, fewer than AutoRestTest (472) and slightly below KAT (287), while remaining higher than RESTifAI (198). The reduction relative to AutoRestTest is consistent with APIP ILOT’s ability to suppress previously failing inputs via the Contextual Resource Pool and refine parameter constraints through its feedback mechanism, reducing the generation of requests that trigger specification-undefined behavior. RESTifAI’s lower count reflects its more conservative operation-level generation, which explores a narrower range of input combinations. The pattern is consistent across complex APIs. On GitLab services and petclinic-rest, APIP ILOT maintains moderate 5xx detection while keeping undocumented responses low, suggesting that its workflow-driven exploration reaches meaningful fault-triggering paths without excessive specification deviation. Overall, APIP ILOT achieves a better balance between fault detection and specification adherence than all three baselines.

executes 112 workflow sequences compared to only 17 by RESTifAI, and in GitLab Issues and jhipster-app, APIP ILOT’s test suites are 3X-4X larger while maintaining comparable or higher success rates. RESTifAI, by contrast, constructs its test suite in direct correspondence with the number of API operations, limiting it to validating individual endpoints or simple transitions rather than exploring multi-step dependency chains. As an example, in the GitLab Commit service, where RESTifAI achieves only 29.3% operation coverage, consider testing GET /projects/{id}/repository/commits/{sha} /discussions. RESTifAI generates a single path: POST /projects → POST /projects/id/repository/commits → GET /projects/id/repository/commits/sha/discussions. This sequence is infeasible in practice: no endpoint supports creating discussions for a commit in this manner. Thus, the path fails and the target operation cannot be tested. APIP ILOT infers a valid alternative: GET /projects → GET /projects/{id}/repository/commits → GET /projects/{id}/repo-sitory/commits/{sha}/discussions. This GET-based sequence successfully retrieves existing discussions, allowing the target endpoint to be exercised. More broadly, since RESTifAI generates only a single execution path per operation, any path failure renders that operation untestable. APIP ILOT mitigates this by retaining the top-k valid workflows per operation, ensuring that intermediate failures do not block coverage of downstream operations. C. Error Detection (RQ3) Table IV reports 5xx errors and undocumented status codes detected by each tool across all 16 services. 5xx Errors. APIP ILOT detects 197 unique 5xx errors, outperforming DeepREST (45), AutoRestTest (178), RESTifAI (151), and KAT (8). This advantage is mainly due to APIP ILOT’s ability to expose unhandled faults in the OhSome service. On the other services, APIP ILOT still detects more 5xx errors than

D. Token and Cost Efficiency (RQ4) Table V reports input tokens, output tokens, and tokens per test case (Tt ) for each tool across all 16 services. DeepREST is a non-LLM approach, thus, does not have LLM costs. Token Consumption. AutoRestTest achieves the lowest token consumption per test case (3), owing to its minimal reliance

9

TABLE V: Cost–Efficiency Based on Token Usage (RQ4) Service

AutoRestTest Tin Tout Tt

RestifAI Tin Tout

KAT Tt Tin Tout Tt

Genome. 59 9 17 6055 36 28,864 57 Lang. 6 2 0 175 4 5409 4 Rest. 44 5 0 1858 27 8,934 4 jhipster. 81 20 1 1,729 59 4,718 85 petclinic. 150 13 1 4,048 52 10,226 65 Canada. 16 1 1 1,773 7 38,693 9 Bills 51 8 7 3,851 39 19,453 28 63 19 3 1,980 94 25,607 32 GLCom. GLGrps 64 16 3 1,393 36 16,616 49 GLIss. 101 32 7 5,912 118 29,560 93 36 11 5 1,545 47 15,760 22 GLBr. GLPro. 106 34 5 5,747 183 27,579 94 GLRepo 39 13 3 1,633 64 18,445 29 24 9 6 5,007 16 45,248 23 FDIC PetStore 42 10 1 607 31 3,066 13 OhSome 1,133 314 575 50,148 291 22,468 178

54 387 4 877 27 226 70 292 30 245 3 223 10 192 11 97 17 153 30 151 8 152 31 154 10 172 6 194 5 95 61 102

APIP ILOT Tin Tout Tt 414 17 92 361 412 17 122 117 126 330 85 391 226 60 199 70

230 271 13 139 88 48 341 127 372 114 12 84 76 75 70 100 101 143 237 157 64 185 269 164 52 263 26 268 195 165 62 317

Total 2,013 514 3 93,461 1,104 19,603 785 376 161 3,664 2,764 155 Tin : Input tokens (thousands), Tout : Output tokens (thousands), Tt : Tokens per test case.

on LLMs as it generates test cases at scale through mutation and evolutionary search rather than LLM prompting. At the opposite end, RESTifAI consumes 19,603 tokens per test case, reflecting its heavy dependence on LLM-based generation for every request sequence. APIP ILOT requires 155 tokens per test case, significantly below RESTifAI and comparable to KAT (161), while generating substantially more tests than both, 41,551 versus 7,219 for KAT and 4,824 for RESTifAI. Monetary Cost. KAT incurs the lowest total cost ($0.54), followed by AutoRestTest ($1.11) and APIP ILOT ($3.12), while RESTifAI is much more expensive ($38.05) ($0.4 per 1M input tokens, $0.1 if cached; $1.6 per 1M output tokens). APIP ILOT’s higher absolute cost relative to KAT and AutoRestTest is directly attributable to its broader test suite. It generates roughly 6X more test cases than KAT as it utilizes LLM more. Cost-Effectiveness. Despite its higher absolute cost, APIP I LOT achieves the best coverage outcomes across all dimensions. It attains the highest average operation coverage (92.3%), the best code coverage (47.0% BC, 58.6% LC, and 58.6% MC), and the highest execution success rate (88.1%). KAT and RESTifAI, while cheaper per test case, achieve 23.3 and 13.3 percentage points lower operation coverage respectively, meaning their apparent token efficiency comes at the cost of much reduced effectiveness. AutoRestTest, despite low per-test-case token cost, requires over 40X more test cases per percentage point of operation coverage than APIP ILOT (Eff =5323.4 vs. 129.8). V. T HREATS TO VALIDITY Internal Validity. Results may be influenced by implementation differences across APIP ILOT and the baselines. All LLM-based tools use the same model with temperature 0, but differences in prompts may affect outcomes. Dependency validation and refinement also introduce additional heuristics. Multiple runs and averaging reduce the risk, but residual nondeterminism in LLM outputs and API responses may remain.

10

External Validity. Our benchmark includes 16 real-world APIs from diverse domains, but may not represent all API styles, especially highly stateful, event-driven, or non-RESTful services. Since APIP ILOT and the baselines rely on OpenAPI specifications, the results may not generalize to poorly documented or specification-inconsistent APIs. Conclusion Validity. Baseline selection may affect the observed gains. We compare our tool with recent LLM-based and non-LLM REST API testing approaches closest to ours. VI. R ELATED W ORK The exist several surveys on RESTful API testing [9], [4], [3], [23], [24], [25], [26]. Existing approaches can generally be categorized into two directions: black-box testing [27], [28], [11], [5], [29], [30], [31] and white-box testing [32], [33], [34], [35], [36], [37], [38], [39]. Black-box approaches typically rely on API specifications such as OpenAPI to derive request structures and dependencies, whereas white-box techniques analyze server-side code to guide test generation. Operation dependencies. Modeling dependencies among API operations is essential for generating meaningful request sequences. Viglianisi et al. introduced the Operation Dependency Graph to model data dependencies among operations from OpenAPI specifications [27]. Liu et al. proposed the RESTful-service Property Graph (RPG) to represent producer– consumer relationships among operations and dynamically refine dependencies using execution feedback [11]. RESTler [5] infers dependencies using a grammar-based approach. Other studies explored structural models such as linkage model trees [40] and API forests [41]. We did not compare with RESTler and EvoMaster as DeepREST is a more recent tool. Several works investigate dependencies among parameters within API operations. Martin-Lopez et al. proposed the Interparameter Dependency Language (IDL) to specify parameter constraints explicitly [10], which is used by RESTest to support constraint-based testing [28]. Wu et al. extracted interparameter constraints from API documentation using patternbased NLP [30]. While formalizing parameter dependencies, these approaches often require manual annotations or predefined patterns. Recent work explored learning-based approaches, including reinforcement learning [42] and deep learning models [43]. In contrast, we leverage LLMs to automatically infer such dependencies directly from OpenAPI specifications. NLP in API testing. Classic NLP methods extract testing knowledge from documentation. NLP2REST [44] augments OpenAPI specifications with rules inferred from natural language descriptions. Wanwarang et al. [45] and ARTE [46] generate realistic inputs by querying knowledge bases using semantic concepts extracted from parameter labels and API structure. RESTInfer [47] infers parameter constraints from API descriptions through a two-phase analysis. LLM-based RESTful API Testing. Recent work leverages LLMs to exploit semantic information in OpenAPI specifications. A first line enriches or infers specifications: RESTS PEC IT [48] infers OpenAPI specifications with minimal user input, while RESTGPT [12] augments existing ones with

machine-interpretable rules and example values. A second line uses LLMs for test generation: KAT [15] constructs an ODG and generates test scripts via GPT prompting; AUTO R EST T EST [13] combines LLMs with a semantic dependency graph and multi-agent reinforcement learning; L LAMA R EST T EST [49] fine-tunes small language models using server responses as feedback; and REST IFAI [14] generates reusable CI/CD-ready tests emphasising happy-path workflows. Our work differs in that it treats LLM reasoning as a candidate inference step, verifies inferred dependencies through concrete executions, and generates test sequences deterministically over the verified dependency graph. VII. C ONCLUSION We presented APIP ILOT, an execution-grounded framework for REST API testing. APIP ILOT treats inferred dependencies as hypotheses, validates them through concrete API executions, and constructs coverage-aware workflows from the validated dependency graph rather than relying on LLM-generated sequences. It further uses runtime responses to refine resource pools, input constraints, and dependency mappings across testing iterations. Our results show that grounding LLM-guided dependency inference in execution feedback improves workflow success, operation coverage, code coverage, and fault-revealing capability over LLM-based REST API testing approaches. VIII. DATA AVAILABILITY S TATEMENT Our data and code are publicly available at our website [19]. R EFERENCES [1] R. T. Fielding, “Architectural styles and the design of network-based software architectures,” Ph.D. dissertation, University of California, Irvine, 2000. [2] A. Neumann, N. Laranjeiro, and J. Bernardino, “An analysis of public rest web service apis,” IEEE Transactions on Services Computing, vol. 14, no. 4, pp. 957–970, 2018. [3] A. Ehsan, M. A. M. Abuhaliqa, C. Catal, and D. Mishra, “RESTful API testing methodologies: Rationale, challenges, and solution directions,” Applied Sciences, vol. 12, no. 9, p. 4369, 2022. [4] A. Golmohammadi, M. Zhang, and A. Arcuri, “Testing RESTful APIs: A Survey,” ACM Transactions on Software Engineering and Methodology, 2022. [5] V. Atlidakis, P. Godefroid, and M. Polishchuk, “Restler: stateful rest api fuzzing,” in Proceedings of the 41st International Conference on Software Engineering, ser. ICSE ’19. IEEE Press, 2019, p. 748–758. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00083 [6] A. Arcuri, “Evomaster: Evolutionary multi-context automated system test generation,” in 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), 2018, pp. 394–397. [7] P. Godefroid, M. Y. Levin, and D. Molnar, “Sage: whitebox fuzzing for security testing,” Commun. ACM, vol. 55, no. 3, p. 40–44, Mar. 2012. [Online]. Available: https://doi.org/10.1145/2093548.2093564 [8] M. Zhang and A. Arcuri, “Open problems in fuzzing RESTful APIs: A comparison of tools,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 32, no. 6, pp. 1–40, 2023. [9] M. Kim, Q. Xin, S. Sinha, and A. Orso, “Automated test generation for rest apis: No time to rest yet,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 289–301. [10] A. Martin-Lopez, “Automated analysis of inter-parameter dependencies in web APIs,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings, 2020, pp. 140–142. [11] Y. Liu, Y. Li, G. Deng, Y. Liu, R. Wan, R. Wu, D. Ji, S. Xu, and M. Bao, “Morest: model-based restful api testing with execution feedback,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 1406–1417.

11

[12] M. Kim, T. Stennett, D. Shah, S. Sinha, and A. Orso, “Leveraging large language models to improve rest api testing,” in Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, ser. ICSE-NIER’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 37–41. [Online]. Available: https://doi.org/10.1145/3639476.3639769 [13] M. Kim, T. Stennett, S. Sinha, and A. Orso, “A multi-agent approach for rest api testing with semantic graphs and llm-driven inputs,” in Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, ser. ICSE ’25. IEEE Press, 2025, p. 1409–1421. [Online]. Available: https://doi.org/10.1109/ICSE55347.2025.00179 [14] L. Kogler, M. Ehrhart, B. Dornauer, and E. P. Enoiu, “Restifai: Llm-based workflow for reusable rest api testing,” 2025. [Online]. Available: https://arxiv.org/abs/2512.08706 [15] T. Le, T. Tran, D. Cao, V. Le, T. N. Nguyen, and V. Nguyen, “Kat: Dependency-aware automated api testing with large language models,” in 2024 IEEE Conference on Software Testing, Verification and Validation (ICST), 2024, pp. 82–92. [16] Zhang, “LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation,” in Proceedings of the ACM on Software Engineering (ISSTA). ACM, 2025, identifies API Knowledge Conflicts as a specific hallucination type. [17] D. Corradini, Z. Montolli, M. Pasqua, and M. Ceccato, “Deeprest: Automated test case generation for rest apis exploiting deep reinforcement learning,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 1383–1394. [Online]. Available: https://doi.org/10.1145/3691620.3695511 [18] GitLab Inc., “Gitlab community edition,” https://gitlab.com/gitlaborg/gitlab. [19] “APIPilot Replication package,” https://anonymous.4open.science/r/APIPilot, 2026, jun 2026. [20] JHipster Contributors, “Jhipster: A development platform to quickly generate, develop, and deploy modern web applications and microservice architectures,” https://github.com/jhipster/generator-jhipster, 2026, accessed: 2026-03-22. [21] Spring Community, “Spring petclinic sample application,” https://github.com/spring-projects/spring-petclinic, 2026, accessed: 2026-03-22. [22] “Jacoco,” 2026. [Online]. Available: https://github.com/jacoco/jacoco [23] A. Martin-Lopez, S. Segura, and A. Ruiz-Cortés, “Online testing of RESTful APIs: Promises and challenges,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 408–420. [24] A. Sharma, M. Revathi et al., “Automated API testing,” in 2018 3rd International Conference on Inventive Computation Technologies (ICICT). IEEE, 2018, pp. 788–791. [25] B. Marculescu, M. Zhang, and A. Arcuri, “On the faults found in rest APIs by automated test generation,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 3, pp. 1–43, 2022. [26] A. Martin-Lopez, A. Arcuri, S. Segura, and A. Ruiz-Cortés, “Black-box and white-box test case generation for RESTful APIs: Enemies or allies?” in 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2021, pp. 231–241. [27] E. Viglianisi, M. Dallago, and M. Ceccato, “RestTestGen: automated black-box testing of restful APIs,” in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE, 2020, pp. 142–152. [28] A. Martin-Lopez, S. Segura, and A. Ruiz-Cortés, “RESTest: Automated Black-Box Testing of RESTful Web APIs,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA ’21. Association for Computing Machinery, 2021. [29] N. Laranjeiro, J. Agnelo, and J. Bernardino, “A black box tool for robustness testing of rest services,” IEEE Access, vol. 9, pp. 24 738– 24 754, 2021. [30] H. Wu, L. Xu, X. Niu, and C. Nie, “Combinatorial testing of restful APIs,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 426–437. [31] D. Corradini, A. Zampieri, M. Pasqua, E. Viglianisi, M. Dallago, and M. Ceccato, “Automated black-box testing of nominal and error scenarios in RESTful APIs,” Software Testing, Verification and Reliability, vol. 32, no. 5, p. e1808, 2022.

[32] O. Sahin and B. Akay, “A discrete dynamic artificial bee colony with hyper-scout for RESTful web service API test suite generation,” Applied Soft Computing, vol. 104, p. 107246, 2021. [33] A. Arcuri, “Automated black-and white-box testing of restful APIs with evomaster,” IEEE Software, vol. 38, no. 3, pp. 72–78, 2020. [34] A. Arcuri and J. P. Galeotti, “Enhancing search-based testing with testability transformations for existing APIs,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 1, pp. 1–34, 2021. [35] ——, “Handling sql databases in automated system test generation,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 29, no. 4, pp. 1–31, 2020. [36] M. Zhang and A. Arcuri, “Adaptive hypermutation for search-based system test generation: A study on rest APIs with EvoMaster,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 1, pp. 1–52, 2021. [37] ——, “Enhancing resource-based test case generation for RESTful APIs with SQL handling,” in International Symposium on Search Based Software Engineering. Springer, 2021, pp. 103–117. [38] M. Zhang, B. Marculescu, and A. Arcuri, “Resource-based test case generation for RESTful web services,” in Proceedings of the genetic and evolutionary computation conference, 2019, pp. 1426–1434. [39] ——, “Resource and dependency based test case generation for restful web services,” Empirical Software Engineering, vol. 26, no. 4, p. 76, 2021. [40] D. Stallenberg, M. Olsthoorn, and A. Panichella, “Improving test case generation for rest APIs through hierarchical clustering,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 117–128. [41] J. Lin, T. Li, Y. Chen, G. Wei, J. Lin, S. Zhang, and H. Xu, “foREST: A Tree-based Approach for Fuzzing RESTful APIs,” arXiv preprint

12

arXiv:2203.02906, 2022. [42] M. Kim, S. Sinha, and A. Orso, “Adaptive REST API Testing with Reinforcement Learning,” arXiv preprint arXiv:2309.04583, 2023. [43] A. G. Mirabella, A. Martin-Lopez, S. Segura, L. Valencia-Cabrera, and A. Ruiz-Cortés, “Deep learning-based prediction of test input validity for restful APIs,” in 2021 IEEE/ACM Third International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest). IEEE, 2021, pp. 9–16. [44] M. Kim, D. Corradini, S. Sinha, A. Orso, M. Pasqua, R. Tzoref-Brill, and M. Ceccato, “Enhancing REST API Testing with NLP Techniques,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023, pp. 1232–1243. [45] T. Wanwarang, N. P. Borges Jr, L. Bettscheider, and A. Zeller, “Testing apps with real-world inputs,” in Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test, 2020, pp. 1–10. [46] J. C. Alonso, A. Martin-Lopez, S. Segura, J. M. Garcia, and A. RuizCortes, “ARTE: Automated Generation of Realistic Test Inputs for Web APIs,” IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 348–363, 2022. [47] Y. Liu, “RESTInfer: automated inferring parameter constraints from natural language RESTful API descriptions,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 1816–1818. [48] A. Decrop, X. Devroey, M. Papadakis, P.-Y. Schobbens, and G. Perrouin, “You can rest now: Automated rest api documentation and testing via llm-assisted request mutations,” 2025. [Online]. Available: https://arxiv.org/abs/2402.05102 [49] M. Kim, S. Sinha, and A. Orso, “Llamaresttest: Effective rest api testing with small language models,” Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715737

Related documents

• Multi-Agent LLM-based Metamorphic Testing for REST APIs#241578
• From Exploration to Specification: LLM-Based Property Generation for Mobile App Testing#14098
• DualMine: Static-Dynamic REST API Constraint Discovery with Dual Validation#1064350
• RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements#141516
• Why Do LLMs Fail at OCL Generation? A Graph Reasoning Perspective#1062025