RESTCov: A Tool for Structural Coverage Analysis of REST APIs

arXiv:2608.28114v1 [cs.SE] 28 Aug 2026

RESTCov: A Tool for Structural Coverage Analysis of REST APIs Tolgahan Bardakci

Serge Demeyer

Computer Science University of Antwerp and Flanders Make Antwerp, Belgium tolgahan.bardakci@uantwerpen.be

Computer Science University of Antwerp and Flanders Make Antwerp, Belgium serge.demeyer@uantwerpen.be

Abstract—REST APIs are widely used in modern software systems, but developers and testers often lack visibility into which parts of an API specification are exercised by a test suite. Traditional coverage analysis usually relies on sourcecode instrumentation, which is impractical for REST APIs that are distributed, externally maintained, and hence accessible only through black-box execution. This paper presents RESTCov, a lightweight tool that computes structural REST API coverage from an OpenAPI specification and observed HTTP request/response logs, reporting coverage across paths, operations, parameters, media types, status codes, and status classes. RESTCov produces both machine-readable results and a human-readable HTML report, helping users inspect coverage gaps, diagnose specification-log mismatches, and evaluate REST API test suites without requiring access to the implementation. Screencast: https://youtu.be/mNz2P43OyUc Repository: https://github.com/2tolgahan2/RESTCov Index Terms—Software Testing, API Testing, Test Coverage

I. I NTRODUCTION REST APIs are central interfaces for modern software systems, enabling communication between services, applications, and organizations [1], [2]. As these APIs evolve, their OpenAPI specification [3] and their actual implementation can drift out of sync: endpoints can be added, removed, or change behavior without a corresponding update to the specification. Testing plays an important role in surfacing this drift by verifying that the documented behavior is exercised by existing test suites or observed in client interactions. For developers and testers, this requires visibility into which parts of an API have actually been exercised. Coverage analysis is a widely used method for assessing test sufficiency and understanding which parts of a system have been exercised [4]. Coverage is often computed through source-code instrumentation. However, for REST APIs, this is not always possible or desirable. The implementation may be unavailable, distributed across multiple services, or maintained by another team. In these situations, testers may still have access to two artifacts: an API specification and logs of observed HTTP traffic. Together, these artifacts enable blackbox coverage analysis by comparing executed requests with the documented API behavior. This work is supported by the Research Foundation Flanders (FWO) via the BaseCamp Zero Project under Grant number S000323N.

OpenAPI specifications [3] describe structural elements of REST APIs, such as paths, operations, parameters, media types, and expected response codes. These elements provide a practical basis for measuring how much of a documented API has been exercised. However, in practice, mapping observed traffic back to a specification is not always straightforward. Logs may contain requests that cannot be matched to any documented operation in the OpenAPI specification. We refer to these as unmatched requests. This may happen because the request targets an undocumented endpoint, the path is malformed or encoded differently, or the implementation and specification have evolved out of sync [5]. If such cases are hidden, coverage results become harder to interpret. The missing coverage may reflect either untested API behavior or traffic that falls outside the documented API. To address this need, we present RESTCov, a lightweight tool for black-box coverage analysis of REST APIs. RESTCov takes an OpenAPI specification and observed HTTP logs, maps executed requests to documented API elements, and reports coverage across paths, operations, parameters, media types, status codes, and status classes. It produces machinereadable results and a human-readable HTML report to inspect coverage gaps and diagnose unmatched requests. This supports API maintenance and evolution by helping developers assess regression suites, detect specification-log drift, and prioritize test or documentation updates. This paper makes the following contributions: We present RESTCov, a lightweight tool for computing structural REST API coverage from an OpenAPI specification and observed HTTP traffic, without requiring source-code instrumentation. • We describe the coverage dimensions supported by RESTCov, including paths, operations, parameters, media types, status codes, and status classes. • We demonstrate RESTCov’s machine- and humanreadable outputs, including unmatched request samples to diagnose specification-log mismatches. • We provide a reusable artifact with source code, example inputs, documentation, and expected outputs to support practical use and future experimentation. •

II. R ELATED W ORK AND P OSITIONING A. REST API Testing REST API testing has been widely studied [6]. RESTTESTGEN [7] and RESTest [8] support automated black-box testing by generating test cases from API specifications. Kim et al. further examine automated test generation for REST APIs and highlight the challenges of generating effective tests [5]. In addition to test generation, Nooyens et al. [9] introduce agentic test amplification systems, based on single- and multi-agent LLMs, that extend existing REST API test suites. Other tools extend REST API testing in different directions. RESTler [10] uses OpenAPI specifications for stateful REST API fuzzing, while EvoMaster [11] targets automated systemlevel test generation for RESTful services. QuickREST [12] explores property-based test generation from OpenAPI specifications. Schema-aware and model-based directions have also been explored through tools such as Schemathesis [13], Morest [14], and RestCT [15]. These tools primarily focus on generating or executing REST API tests. B. REST API Coverage Analysis Coverage criteria for REST APIs have been proposed to measure how thoroughly tests exercise API specifications [16]. Restats provides tool support for measuring REST API test coverage [17]. RESTCov shares this motivation: making APIlevel coverage visible without requiring source-code access. The two tools differ most in how they handle traffic that falls outside the documented specification. Restats records some of this, for example through a notDocumentedAndTested section in its operation-coverage report. However, that information is split across separate per-metric JSON files, with no single place to see it all. RESTCov instead collects all such traffic into one explicit list of unmatched requests, shown directly in both its JSON output and its HTML report. This makes it easy to spot specification-log mismatches, whether they stem from specification-implementation drift, such as undocumented endpoints, or from client-side artifacts, such as malformed or encoded paths, without digging through multiple reports. III. RESTC OV OVERVIEW A. Purpose RESTCov is designed for situations where users have an OpenAPI specification and observed HTTP traffic, but need to understand which parts of the documented API were exercised. It also shows traffic that no longer corresponds to the specification, a sign that the implementation and specification may have drifted out of sync. Rather than relying on source code, RESTCov measures coverage directly from execution logs and maps those observations back to the specification. RESTCov reports whether a documented specification element (e.g., a path, operation, or status code) was observed in the traffic. It does not verify that the corresponding functionality behaves correctly, nor does it assess the thoroughness or adequacy of the testing that produced this traffic. As such, RESTCov measures coverage of the specification’s structural elements, not functional correctness of the API’s behavior.

B. Inputs RESTCov operates on two inputs: an OpenAPI specification [3], and a directory of paired request/response log files. It supports different OpenAPI versions, making it suitable for specifications generated by different tools or processes. In the current public version, each request is stored in a *-request.txt file and each response in a matching *-response.txt file. RESTCov scans these and parses them for comparison against the documented API. C. Workflow RESTCov first parses the OpenAPI specification and extracts the documented paths, operations, parameters, response codes, and media types. It then reads the request/response log pairs, and reconstructs the observed traffic, including the HTTP method, request path, visible parameters, request content type, response status code, and response content type. After parsing, RESTCov matches each observed request to a documented API operation based on the request path and method. Each documented path template (e.g., /pet/{petId}) is compiled into a regular expression that replaces path parameters with a wildcard segment, then matched against the request path after stripping any server or base-path prefix declared in the specification or supplied via the --base-path option. Different encodings of the same path are common in REST APIs. For example, a slash may appear literally or encoded as %2F. RESTCov requires an exact match, including casing, trailing slashes, and encoding. A request that differs in any of these is reported as an unmatched request rather than matched to the corresponding operation. In contrast, HTTP methods are compared case-insensitively. Once a request is matched, RESTCov records which documented elements were observed in that interaction. These observations are then aggregated into coverage metrics and written to the output files. D. Coverage Dimensions RESTCov reports coverage across several dimensions, comparing documented API elements against observed traffic, inspired by specification-based REST API coverage criteria [16]. • Path coverage measures whether a documented API path appears. • Operation coverage refines path coverage by considering the combination of an HTTP method and a documented path, such as GET /pet/{petId} or POST /pet. • Parameter coverage measures whether documented parameters are observed. • Request content-type coverage measures whether the documented media types for request bodies are observed. • Response content-type coverage measures whether the documented media types for response bodies are observed. • Status code coverage measures whether documented response codes, such as 200, 400, or 404, are observed. • Status class coverage measures whether documented response codes are observed at the class level, such as 2xx, 4xx, or 5xx, rather than as individual codes.

These dimensions give a diverse view of the system under test. A test suite may exercise many operations while still missing parameters, media types, or response outcomes. RESTCov makes such gaps visible by reporting each dimension.

Table I summarizes coverage results. For Path and Operation, the total counts are distinct documented elements; for other dimensions, it is an aggregate summed across all operations.

E. Outputs RESTCov produces two outputs: a machine-readable coverage.json file and a human-readable report.html file. The JSON file stores the computed metrics and observations for further analysis, test-suite comparison, or CI/CD integration. The HTML report presents the same information in an inspectable form and includes unmatched request samples. The unmatched samples make specification-implementation drift directly visible. Traffic that no longer corresponds to the specification may indicate that an endpoint was added, removed, or changed after the specification was last updated. Not every mismatch reflects drift from the specification. Some of the mismatches may come from client-side artifacts, such as malformed or encoded paths, rather than changes on the API side. Once users know which of the two applies, they can update the specification if the implementation has moved on, or fix the client if the request itself is malformed. IV. D EMONSTRATION This section illustrates how RESTCov analyzes black-box REST API coverage from an OpenAPI specification and observed HTTP traffic. The demonstration uses the Petstore API [18], a small open-source example, together with recorded request/response logs. The goal is not to evaluate a specific test suite, but to show how RESTCov processes its inputs, produces coverage results, and helps users inspect gaps between documented and observed behavior. While this demonstration uses a single API, RESTCov has also been used across five production APIs, including Google Drive and Spotify, in a comparative study [19] and an industrial setting [20]. When applied to the OpenAPI specifications used across these studies, together with Petstore, RESTCov processed all of them successfully, while some of the other coverage tools we tested did not. This underscores the importance of robustness across diverse, real-world OpenAPI documents.

TABLE I C OVERAGE SUMMARY FOR THE P ETSTORE DEMONSTRATION . Dimension Path Operation Parameter Request content type Response content type Status code Status class

RESTCov runs from the command line by specifying the OpenAPI specification, log directory, and output directory:

Listing 1. Example RESTCov command-line invocation

After execution, RESTCov parses the specification, reconstructs HTTP interactions from the logs, and, when possible, matches each request to a documented operation. These observations are aggregated into coverage metrics and written to a machine-readable JSON file and a human-readable HTML report in the output directory.

Coverage 100.0% 100.0% 100.0% 81.8% 86.8% 38.9% 65.2%

B. Output Interpretation Listing 2 shows selected metrics and unmatched request samples from the generated coverage.json file. The total requests field counts only the observed requests that were successfully matched to a documented operation; unmatched requests, such as the one shown in Listing 2, are excluded from this count and reported under unmatched samples separately. 1

{ "spec": "openapi.json", "logs": "logs", "metrics": { "total_requests": 139, "total_paths": 14, "covered_paths": 14, "path_cov_pct": 100.0, "total_ops": 20, "covered_ops": 20, "op_cov_pct": 100.0, "status_total": 36, "status_seen": 14, "status_cov_pct": 38.9 }, "unmatched_samples": [ { "method": "POST", "path": "/pet%2F%2FuploadImage" } ]

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

18 19 20 21 22

python restcov.py \ --openapi examples/petstore/openapi.json \ --logs examples/petstore/logs \ --out output/petstore

Covered 14 20 25 9 33 14 15

The observed traffic covers all documented paths, operations, and parameters. However, coverage is lower for request content types, response content types, status codes, and status classes. This indicates that reaching all documented operations does not guarantee that documented media types and response outcomes have been exercised.

17

A. Command-Line Execution

Total 14 20 25 11 38 36 23

} Listing 2. JSON report from the generated coverage.json file.

Figure 1 shows an excerpt from the report.html file. The report summarizes coverage by dimension and lists unmatched samples. Such samples may correspond to undocumented endpoints, malformed or encoded paths, or other specification-log mismatches. For example, the unmatched path /pet%2F%2FuploadImage contains a double-encoded slash (%2F%2F). This is a client-side artifact, likely caused by concatenating an already encoded path segment, rather than a sign of an undocumented or uncovered operation.

Fig. 1. Excerpt from the generated report.html file for the Petstore demonstration.

V. AVAILABILITY AND R EUSE

C. Insights from the Report

The report helps users move beyond a single coverage number. For example, low parameter coverage may mean tests exercise only default or minimal requests. In contrast, low status code coverage may mean tests mainly cover successful responses and miss error-handling behavior. The unmatched samples also help users decide whether to add tests, improve the specification, or inspect the logging format. Such mismatches often emerge as an API evolves: new endpoints go undocumented, client behavior changes, or specifications fall out of sync with implementation. By surfacing these mismatches directly from traffic, RESTCov supports ongoing API maintenance and evolution, not just a one-time coverage check at release.  The Petstore demonstration shows that RESTCov can reveal differences between exercised endpoints and exercised response behavior, and expose specification-log mismatches through unmatched request samples.

RESTCov is available as an artifact on GitHub1 . The repository includes the source code, documentation, example inputs, expected outputs, and screenshots of the HTML report. It is organized as follows: The root directory contains the analysis script, restcov.py, the dependency file requirements.txt, the documentation README.md, and the license LICENSE. • The e x a m p l e s / p e t s t o r e/ directory contains the Petstore OpenAPI specification, recorded HTTP request/response logs, and expected outputs. • The screenshots/ directory contains views of the HTML report. •

To reuse RESTCov with another REST API, users provide an OpenAPI specification, a request/response log directory, and an output directory. Dependencies and execution instructions are provided in the README.md file. 1 https://github.com/2tolgahan2/RESTCov

VI. L IMITATIONS Several limitations apply to RESTCov’s results. First, the demonstration uses a single API (Petstore) [18] for illustrative clarity. While RESTCov has also been used across five production APIs in a comparative study [19] and an industrial study [20], this paper does not present a multi-API evaluation. Second, RESTCov reports whether documented specification elements were observed in traffic; it does not verify that endpoint behavior was correctly exercised, and its ”coverage” measure should be interpreted accordingly (Section III-A). Third, the current version stores each HTTP interaction as paired request/response text files, which is simple to inspect but may become cumbersome for large-scale log analysis. Supporting consolidated log formats, such as HAR or JSONL, is left as future work rather than a current requirement. VII. C ONCLUSION RESTCov is a lightweight tool for black-box coverage analysis of REST APIs. Given an OpenAPI specification and observed HTTP traffic, it maps executed requests back to documented API elements and reports coverage across several structural dimensions. The JSON output supports automated analysis, while the HTML report helps inspect coverage gaps and unmatched requests. Together, these outputs support test-suite assessment without requiring access to the service’s implementation. R EFERENCES [1] R. T. Fielding, “Architectural Styles and the Design of Network-based Software Architectures,” Ph.D. dissertation, University of California, Irvine, 2000. [Online]. Available: https://www.ics.uci.edu/∼fielding/pu bs/dissertation/fielding dissertation.pdf [2] R. Verborgh, S. Van Hooland, A. S. Cope, S. Chan, E. Mannens, and R. Van De Walle, “The fallacy of the multi-API culture: Conceptual and practical benefits of Representational State Transfer (REST),” Journal of Documentation, vol. 71, no. 2, pp. 233–252, Mar. 2015. [Online]. Available: http://www.emerald.com/jd/article/71/2/233-252/201645 [3] OpenAPI Initiative, “OpenAPI specification,” https://spec.openapi s.org/oas/v3.2.0.html, 2025, version 3.2.0. [Online]. Available: https://spec.openapis.org/oas/v3.2.0.html [4] H. Zhu, P. A. V. Hall, and J. H. R. May, “Software unit test coverage and adequacy,” ACM Computing Surveys, vol. 29, no. 4, pp. 366–427, Dec. 1997. [Online]. Available: https://doi.org/10.1145/267580.267590 [5] M. Kim, Q. Xin, S. Sinha, and A. Orso, “Automated test generation for REST APIs: No time to rest yet,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. Virtual South Korea: ACM, Jul. 2022, pp. 289–301. [Online]. Available: https://dl.acm.org/doi/10.1145/3533767.3534401 [6] A. Golmohammadi, M. Zhang, and A. Arcuri, “Testing RESTful APIs: A Survey,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 1, pp. 27:1–27:41, Nov. 2023. [Online]. Available: https://dl.acm.org/doi/10.1145/3617175

[7] E. Viglianisi, M. Dallago, and M. Ceccato, “RESTTESTGEN: Automated Black-Box Testing of RESTful APIs,” in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). Porto, Portugal: IEEE, Oct. 2020, pp. 142–152. [Online]. Available: https://ieeexplore.ieee.org/document/9159077/ [8] A. Martin-Lopez, S. Segura, and A. Ruiz-Cortés, “RESTest: Automated black-box testing of RESTful web APIs,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. Virtual Denmark: ACM, Jul. 2021, pp. 682–685. [Online]. Available: https://dl.acm.org/doi/10.1145/3460319.3469082 [9] R. Nooyens, T. Bardakci, M. Beyazıt, and S. Demeyer, “Test Amplification for REST APIs via Single and Multi-agent LLM Systems,” in Testing Software and Systems, S. Bonfanti and G. A. Papadopoulos, Eds. Cham: Springer Nature Switzerland, 2026, pp. 161–177. [10] V. Atlidakis, P. Godefroid, and M. Polishchuk, “RESTler: Stateful REST API Fuzzing,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). Montreal, QC, Canada: IEEE, May 2019, pp. 748–758. [Online]. Available: https://ieeexplore.ieee.org/docu ment/8811961/ [11] A. Arcuri, “EvoMaster: Evolutionary multi-context automated system test generation,” in 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), 2018, pp. 394–397. [12] S. Karlsson, A. Causevic, and D. Sundmark, “QuickREST: Propertybased Test Generation of OpenAPI-Described RESTful APIs,” in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). Porto, Portugal: IEEE, Oct. 2020, pp. 131–141. [Online]. Available: https://ieeexplore.ieee.org/document/9159071/ [13] Z. Hatfield-Dodds and D. Dygalo, “Deriving semantics-aware fuzzers from web API schemas,” in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ser. Icse ’22. New York, NY, USA: Association for Computing Machinery, 2022, pp. 345–346. [Online]. Available: https://doi.org/10.1145/3510454.3528637 [14] Y. Liu, Y. Li, G. Deng, Y. Liu, R. Wan, R. Wu, D. Ji, S. Xu, and M. Bao, “Morest: Model-based RESTful API testing with execution feedback,” in Proceedings of the 44th International Conference on Software Engineering. Pittsburgh Pennsylvania: ACM, May 2022, pp. 1406–1417. [Online]. Available: https: //dl.acm.org/doi/10.1145/3510003.3510133 [15] H. Wu, L. Xu, X. Niu, and C. Nie, “Combinatorial testing of RESTful APIs,” in Proceedings of the 44th International Conference on Software Engineering. Pittsburgh Pennsylvania: ACM, May 2022, pp. 426–437. [Online]. Available: https://dl.acm.org/doi/10.1145/3510003.3510151 [16] A. Martin-Lopez, S. Segura, and A. Ruiz-Cortés, “Test coverage criteria for RESTful web APIs,” in Proceedings of the 10th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. Tallinn Estonia: ACM, Aug. 2019, pp. 15–21. [Online]. Available: https://dl.acm.org/doi/10.1145/3340433.3342822 [17] D. Corradini, A. Zampieri, M. Pasqua, and M. Ceccato, “Restats: A Test Coverage Tool for RESTful APIs,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). Luxembourg: IEEE, Sep. 2021, pp. 594–598. [Online]. Available: https://ieeexplore.ieee.org/document/9609156/ [18] Swagger API, “Swagger petstore API,” 2026. [Online]. Available: https://petstore.swagger.io [19] J. Besjes, R. Nooyens, T. Bardakci, M. Beyazit, and S. Demeyer, “Agentic LLMs for REST API test amplification: A comparative study across cloud applications,” 2025. [Online]. Available: https: //arxiv.org/abs/2510.27417 [20] T. Bardakci, A. Faes, M. Beyazit, and S. Demeyer, “Prompt-based REST API test amplification in industry: An experience report,” 2026. [Online]. Available: https://arxiv.org/abs/2601.17903

Related documents

• RESTCov: A Tool for Structural Coverage Analysis of REST APIs#784871
• RESTCov: A Tool for Structural Coverage Analysis of REST APIs#783281
• Hallucination Inspector: A Fact-Checking Judge for API Migration#124119
• Towards Knowledge Alignment in Code LLMs: Contrastive Unlearning for Evolving APIs#324973
• ResTest: Targeted Coverage of Residual Not-Covered Code Using Large Language Models for Web GUI Testing#1063757