Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding DANIEL MANINGER, Technische Universität Darmstadt, Germany and Hessian Center for Artificial Intelligence (hessian.AI), Germany

LEON CHEMNITZ, Pariton AI, Germany and Technische Universität Darmstadt, Germany JANNIS BRUGGER, Technische Universität Darmstadt, Germany and Hessian Center for Artificial Intelligence (hessian.AI), Germany

TUSHAR LAMBA, Technische Universität Darmstadt, Germany AMIR MOLZAM SHARIFLOO, Technische Universität Darmstadt, Germany MIRA MEZINI, Technische Universität Darmstadt, Germany, Hessian Center for Artificial Intelligence (hessian.AI), Germany, and National Research Center for Applied Cybersecurity ATHENE, Germany

arXiv:2607.05936v1 [cs.SE] 7 Jul 2026

Integration of web APIs is a cornerstone of modern software systems, yet writing correct web API invocation code remains challenging due to complex and evolving API specifications. Although LLMs are increasingly used for code generation, previous work has empirically shown that their ability to generate correct web API integrations is limited. At the same time, mitigation techniques and their effectiveness for this setting remain insufficiently understood. In this paper, we propose and systematically evaluate retrieval-augmented generation (RAG) and constrained decoding (CD) as two complementary approaches to improving LLM-generated web API invocation code. For RAG, we design a retriever that processes OpenAPI specifications and retrieves compact endpoint representations to inject into model prompts. For CD, we introduce an automatic translation from OpenAPI specifications to regex-based constraints enforced during generation. We evaluate both approaches on WAPIIBench’s existing synthetic dataset and on a new real-world dataset derived from GitHub repositories. Our results show that RAG reduces hallucinations and improves correctness when generating full API invocations but reduces it when the endpoint is already provided as it encourages the generation of unnecessary parameters. In contrast, CD reliably prevents illegal URLs, HTTP methods, and arguments and substantially improves overall correctness for both starter codes. CCS Concepts: • Software and its engineering → Automatic programming; • Computing methodologies → Neural networks; • Information systems → RESTful web services. Additional Key Words and Phrases: artificial intelligence, software engineering, large language models, code generation, web APIs, benchmarks, retrieval-augmented generation, constrained decoding

Authors’ Contact Information: Daniel Maninger, Technische Universität Darmstadt, Darmstadt, Germany and Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany, [email protected]; Leon Chemnitz, Pariton AI, Berlin, Germany and Technische Universität Darmstadt, Darmstadt, Germany, [email protected]; Jannis Brugger, Technische Universität Darmstadt, Darmstadt, Germany and Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany, [email protected]; Tushar Lamba, Technische Universität Darmstadt, Darmstadt, Germany, [email protected]; Amir Molzam Sharifloo, Technische Universität Darmstadt, Darmstadt, Germany, [email protected]; Mira Mezini, Technische Universität Darmstadt, Darmstadt, Germany and Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany and National Research Center for Applied Cybersecurity ATHENE, Darmstadt, Germany, [email protected].

Maninger et al.

ACM Reference Format: Daniel Maninger, Leon Chemnitz, Jannis Brugger, Tushar Lamba, Amir Molzam Sharifloo, and Mira Mezini. 2026. Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding. ACM Trans. Softw. Eng. Methodol. 37, 4, Article 111 (August 2026), 54 pages. https://doi.org/XXXXXXX.XXXXXXX

Introduction

Web APIs provide services and functionality through a standardized, HTTP-based interface. Developers leverage them to rapidly create applications that seamlessly integrate with a wide range of online services (powered by API integration code). The market surrounding API integrations is experiencing explosive growth [26, 34]. However, writing correct API integration code is a tedious and challenging task. Developers need to understand and correctly utilize a significant amount of information to be able to write an API invocation1 in compliance with its specification [36, 37, 52]. Moreover, the task of ensuring correct API usage is further complicated by the fact that APIs evolve and change over time. Large language models (LLMs) have great potential to boost productivity in software development, e.g., by automatically generating program code from natural language descriptions [4]. However, LLMs may also hallucinate and make mistakes, raising concerns about correctness [8, 47], security [31, 32], and general quality [13] of LLM-generated code. While the capabilities of LLMs have been investigated for many different software engineering tasks, including non-web API invocations [59], their aptitude for web API integrations remained unexplored, until recently, when we introduced WAPIIBench [25], a new benchmark and pipeline for evaluating LLMs on web API invocation code generation, and used it to study state-of-the-art LLMs. Our study revealed significant limitations: LLMs generate incorrect results most of the time with manyfold error types, ranging from selecting the wrong endpoint to leaving out required arguments or hallucinating entirely illegal arguments. Research Gap. Yet how the issues revealed by our previous study [25] can be mitigated remains an open question. While several techniques, including retrieval-augmented generation (RAG) [20], constrained decoding (CD) [5, 16], and fine-tuning, have been shown to effectively mitigate LLM generation errors in other domains, their efficacy has not been investigated for web API invocations. Such invocations present distinct challenges: the interface is defined externally in an OpenAPI specification rather than in the source code, the invocation must be constructed explicitly by assembling a URL, HTTP method, and parameters across multiple different locations (path, query, header, body), and the specification itself evolves independently of the client code (see also Section 2.1). None of these challenges apply to the simpler function-call or tool-use settings for which RAG and CD have been studied. To close this gap, in this paper, we present the results of a systematic study of the effectiveness of RAG, CD, and their combination for improving the performance of LLMs on web API invocation tasks. More specifically, we formulate and answer the following research questions:

1We differentiate between “API integration (code)”, which refers to the general concept of connecting an arbitrary number of different web APIs with each

other through code, and “API invocation (code)”, which refers to the code snippet that calls a single web API. An API integration consists of one or more API invocations plus the code surrounding them. Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

RQ1: To what extent does RAG improve the correctness of LLM-generated web API invocation code? RQ2: To what extent does CD improve the correctness of LLM-generated web API invocation code? RQ3: To what extent does combining RAG and CD improve the correctness of LLM-generated web API invocation code compared to each technique in isolation? We leave fine-tuning out of consideration for two main reasons. (a) Fine-tuning requires a corpus of training data and substantial computational resources; it is unclear how we could obtain sufficient amounts of high-quality training data for the specific kind of web API invocation tasks we are targeting. (b) Improvements achieved by fine-tuning can quickly become obsolete when the underlying data changes; this is especially relevant in the context of our work, as API specifications are constantly evolving. Methodology. To answer Research Questions 1–3, we (a) developed RAG and CD approaches tailored to the specific requirements of web API invocation tasks, along with a combined approach designed to explore their potential synergies, and (b) evaluate their effectiveness against the respective baselines across two complementary settings. In the first setting, we use the synthetic dataset from WAPIIBench [25]. In the second, we evaluate on a new realworld dataset that we constructed in the context of the work presented here by mining public GitHub repositories for JavaScript files containing Axios-based web API invocations and manually converting them into evaluation tasks. To accommodate file-level code context, we extend WAPIIBench accordingly. The resulting dataset covers tasks for 11 real-world APIs, with OpenAPI specifications ranging from a few hundred to over 76,000 lines. These two evaluation settings offer complementary strengths and limitations. The synthetic dataset provides systematic, controlled coverage of all endpoints for a curated set of APIs, but may not fully reflect the diversity of real-world integration scenarios. The real-world dataset, by contrast, captures common usage patterns of widely adopted APIs, yet is necessarily incomplete; it does not exhaustively cover all endpoints of any given API. Results. Our experiments show that RAG reduces hallucinations and can improve the correctness of generated API invocations. On the synthetic dataset, depending on the provided starter code, it increases average correctness by 113% when generating complete API invocations but decreases it by 3% when the endpoint is already provided. On the real-world dataset, the impact of RAG on correctness is almost negligible, but the reduction of hallucinations is consistent across both datasets. The reason for the ambivalent results on correctness is that RAG encourages models to use more parameters than required. The retriever itself is able to identify the sought-after endpoint among all available endpoints with very high precision, implying that remaining errors are largely due to the models’ ability to make proper use of the retrieved information. The study of constrained models confirms that decoding with our constraints enforces compliance with an API’s specification when generating an invocation, reliably reducing the amount of illegal URLs, HTTP methods, and arguments to zero. Moreover, across models, CD significantly improves the overall correctness of the generated code, at relative gains of +209% and +143% on average on the synthetic dataset, depending on the provided starter code. Similar positive trends can be observed on the real-world dataset. The combination of RAG and CD achieves high average gains in correctness—on the synthetic dataset +332% and +111%, depending on the starter code—and reduces hallucinations to zero, similar to CD. However, compared to CD, the gains are less consistent, as several models show a drop in correctness below the baseline. Manuscript submitted to ACM

Maninger et al. Listing 1. General structure of an API invocation using Axios and JavaScript

const axios = require ( ' axios '); axios .< method >( ' https :// server . com / path / to / endpoint / arg1 ', { arg2 : ' request body parameters ' }, { headers : { arg3 : ' header parameters ' }, params : { arg4 : ' query parameters ' } }) ;

Contributions. In summary, we make the following contributions: A specialized retrieval method for OpenAPI specifications that provides models with relevant in-context information about the used API. → Section 3 An automatic constraint generator that translates OpenAPI specifications to regex-based constraints, which ensure that generated code is specification-complaint. → Section 4 A real-world dataset to validate and complement the results on WAPIIBench’s original synthetic dataset. → Section 5 A systematic empirical evaluation of the ability of LLMs to generate correct and hallucination-free web API invocation code with retrieval-augmented generation and constrained decoding. → Section 6 Artifacts are available on GitHub2 , including the RAG and CD implementations presented in this paper as well as the new real-world dataset. In addition, we provide all model-generated codes and evaluation results on Zenodo3 . Additional details and comprehensive result tables are provided in the appendix at the end of this paper. 2

Background

In the following, we will introduce some concepts fundamental to our work. 2.1

Web APIs and OpenAPI

Unlike local function calls, web API invocations require constructing an HTTP request rather than invoking a named function. APIs expose endpoints, identified by the combination of an HTTP method and a URL, and requests may include parameters in multiple locations, such as the path, query string, headers, and request body. As illustrated in Listing 1, this leads to more complex invocation structures with multiple parameter groups, nested data types, and externally defined interfaces. Most modern web APIs are described using the OpenAPI specification4 , the de facto standard for machine-readable API documentation [42]. OpenAPI specifications, written in JSON or YAML, describe available endpoints, their parameters, request and response schemas, authentication requirements, and other metadata. In contrast to local function signatures, this information is maintained separately from the client code and must be interpreted to construct valid requests. This paper focuses on JavaScript, one of the most widely used programming languages [44], and the Axios library5 , a popular HTTP client for JavaScript [14]. The evaluation framework, WAPIIBench, is built around this setup and represents web API invocations as Axios requests to endpoints defined through OpenAPI specifications. 2 https://github.com/stg-tud/WAPIIBench 3 https://doi.org/10.5281/zenodo.13758414 4 https://www.openapis.org/ 5 https://axios-http.com/

Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

API specification

a, s

Dataset creation

a, t

Code generation

5 i

Code execution

c‘

Correctness analysis

Evaluation results

s, c

Fig. 1. Benchmark design for evaluating the capabilities of LLMs in generating web API invocation code. (0) Based on an API 𝑎 and its specifications 𝑠, an API invocation tasks 𝑡 and corresponding correct request configurations 𝑐 are created. (1) For each 𝑡 , the LLM under evaluation generates an API invocation 𝑖. (2) 𝑖 is executed in a controlled environment, yielding a request configuration 𝑐 ′ . (3) 𝑐 ′ is compared to 𝑐 and validated against 𝑠 to obtain various metrics.

2.2

WAPIIBench

For our studies, we rely on WAPIIBench, a benchmark for web API integration code generation that we previously introduced [25]. It includes a dataset of web API invocation tasks and an execution-based evaluation pipeline. A schematic overview is provided in Figure 1. WAPIIBench’s dataset comprises 395 tasks across four real-world APIs: Asana, Google Calendar, Google Sheets, and Slack (one task for each endpoint). Each task includes a natural language task description and a request configuration, which is used as the ground-truth solution of the task and contains the expected URL, HTTP method, and arguments. An example task description can be found in Figure 4a. The dataset was synthetically generated based on the OpenAPI specifications of the respective APIs and then subjected to rigorous automated and manual vetting and correction process. This way, we ensured that each task is well-defined and unambiguously solvable. The evaluation pipeline has three stages: (1) Code generation, (2) Code execution, and (3) Correctness analysis. (1) In the code generation stage, we insert the API name and the task description (as a line comment) into a prompt that instructs the model under evaluation to solve the given task by completing the starter code that is part of the prompt (cf. Listing 3 in the appendix). The prompt also includes some clarifications to avoid instances of misalignment and recurring error patterns observed in preliminary experiments. Experiments can be conducted with two different starter code variants, called setups. The full completion setup includes the beginning of the API invocation (axios.). It is used to assess the ability of the model to identify the correct endpoint (i.e., the combination of URL and HTTP method) and to solve the task as a whole. The argument completion setup already includes the correct endpoint (axios.<method>(’<url>’, ) and aims to evaluate whether the model can use this endpoint in compliance with the API’s specification by selecting only permitted parameters. (2) In the code execution stage, the model-generated API invocation is cut out of the surrounding code and inserted into a controlled environment, which enables the safe execution the generated API invocation without sending requests to external servers. Additionally, this environment serializes and saves the request’s configuration, so it can be used in the following correctness analysis. (3) In correctness analysis stage, the captured request configuration is compared to the ground-truth configuration and validated against the corresponding API specification. The analysis happens in element-wise and aggregated forms to obtain various fine-grained metrics explained in Table 4 in the appendix. 2.3

Retrieval-Augmented Generation

Retrieval augmented generation (RAG) [20] is a commonly used technique to improve model performance on knowledgeintensive tasks. The general idea is to search a knowledge base or collection of documents for relevant pieces of Manuscript submitted to ACM

Maninger et al. // Create new event … and add it to calendar … Task via code comment

Retriever

API specification

Vector store

Augmented prompt

LLM

</> Code

Fig. 2. Retrieval-augmented generation framework for generating web API invocations. Instead of passing the prompt directly to the language model, it is first augmented with information about potentially relevant API endpoints. Chunks of information are extracted from the API specification and persisted in a vector store. The retriever performs a similarity search to find chunks that match the given task description.

information, so-called chunks, and augment the prompt with them before generating a response. This way, the model does not need to rely solely on memorized knowledge and can base its response on the retrieved information, which enhances reliability. However, the effectiveness of RAG depends on the retriever’s ability to find relevant chunks, as well as on the model’s ability to leverage the information from the retrieved chunks. RAG can be implemented in many ways. The main degrees of freedom are: how documents are split into chunks, which embedding model is used to store chunks in a vector database, which algorithm is used for retrieving relevant chunks from the vector database, and how many chunks to retrieve and include in the model’s prompt. Commonly, a larger amount of chunks is retrieved and a so-called reranker model is used to determine the top chunks to pass on to the generator model. 2.4

Constrained Decoding

Constrained decoding (CD) [5, 16] (also known as structured or guided generation) is a general technique that modifies an LLM’s next-token predictions based on predefined constraints to prevent the generation of tokens that would lead to undesirable output sequences. This technique operates without requiring model modifications or prompt adjustments. How constraints are obtained is highly application-specific. For representing constraints, regular expressions or context-free grammars are commonly used [2, 7, 53]. The general constrained decoding algorithm (cf. [27, 33, 48] extends an LLM’s regular autoregressive generation loop with a so-called completion engine [27, 33, 51], which dynamically filters possible next token to enforce compliance with given constraints. More concretely, in each generation step, the current token sequence 𝑠 1, . . . , 𝑠𝑖 and the token vocabulary 𝑡 1, . . . , 𝑡𝑛 are passed to the completion engine, which checks for each token 𝑡 𝑗 whether 𝑠 1, . . . , 𝑠𝑖 , 𝑡 𝑗 satisfies the constraints and would therefore be a valid continuation. The completion engine returns a token mask 𝑚 ∈ {0, 1}𝑛 which is used to set the probability 𝑝 𝑗 of all tokens 𝑡 𝑗 to zero that failed the constraint check. Based on the masked probability distribution, the next token 𝑠𝑖+1 is sampled. Note that this process can be optimized in many ways [7, 33, 48, 53]. 3

Retrieval-Augmented Generation for Web API Invocations

In order to apply RAG to web API invocations, we retrieve chunks from OpenAPI specifications using the respective task description as query, as shown in Figure 2. While RAG systems typically split documents into chunks based on a Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

Listing 2. Example of a retrieved endpoint chunk in a TypeScript-inspired format // GET / tags /{ tag - name }/ media / recent // Get a list of recently tagged media type GetTagsMediaRecentRequest = { path : { " tag - name ": string // The tag name } query : { count ?: number // Count of tagged media to return min_tag_id ?: string // Return media before this ` min_tag_id ` max_tag_id ?: string // Return media after this ` max_tag_id ` access_token ?: string // API key security scheme } header : { Authorization ?: string // OAuth2 security scheme } }

predetermined length or sentence boundaries, our chunks encompass the complete documentation of a single endpoint. We do this because API specifications are highly context-dependent—retrieving smaller chunks out of context would be detrimental. For example, the documentation of a parameter is useless if it is unclear which endpoint it belongs to. To ensure each chunk is self-contained and contains all relevant information, we preprocess OpenAPI specifications prior to chunking, inlining all references and copying API-wide security schemes and path-level parameters into each endpoint’s parameter list. To embed, store, and retrieve chunks, our retriever uses the all-MiniLM-L6-v26 embedding model, a Chroma vector database, and approximate nearest neighbor search, respectively. To increase the hit rate, each chunk is stored with up to five different embeddings, created by including and excluding different parts of the endpoint documentation in the string that is fed into the embedding model. For example, one variant contains only the endpoint’s path and description, while other variants additionally contain the parameter list or the response schema. Across all tasks in the dataset, our retriever achieves 75.7% top-1 accuracy and 95.2% top-5 accuracy (after deduplicating chunks). We also implemented reranking but did not use it, because it could not improve the retriever performance any further. Before injecting the retrieved endpoint documentation into the prompt, it is formatted and truncated in a postprocessing step. The goal of the truncation is to avoid exceeding a model’s context length. We cut the output before the line in which the character threshold is exceeded and append a [TRUNCATED] marker. To reduce the need for truncation, we format the documentation in a way that is compact but also easy to understand for LLMs. We explored different formatting options such as JSON and YAML, but they turned out to be token-inefficient due to a lot of indentation and verbose constructs such as name: <name>, where simply stating the parameter name would be sufficient. Instead, we designed a custom format inspired by TypeScript type declarations, as shown in Listing 2. 4

Constrained Decoding for Web API Invocations

The main challenge we have to address when applying constrained decoding in our setting is constraint generation. Since every web API requires different constraints, we implement an automatic translation from OpenAPI specifications to constraints that are represented as regular expressions. When generating the constraints, the special characteristics and complexities of web API invocations need to be taken into account. Besides respecting the general JavaScript and the specific Axios syntax (cf. Listing 1), our constraints need to handle interdependencies between allowed URLs, HTTP 6 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

Manuscript submitted to ACM

Maninger et al. // Create new event … and add it to calendar … Task via code comment

API specification

LLM

</> Code

Decoder

Constraints

Completion engine

p1 t 1 p2 p3 t 2

t3 …

…

s1 s2 s3 … si Constraint translator

Fig. 3. Constrained decoding framework for generating web API invocations. In regular autoregressive generation, a decoder takes the token sequence 𝑠 1 , . . . , 𝑠𝑖 and predicts the probabilities 𝑝 𝑗 for each possible next token 𝑡 𝑗 , based on which 𝑠𝑖+1 is selected. Constrained decoding augments this loop with a constraint check that sets the probability of tokens that fail the check to zero. We derive our constraints from OpenAPI specifications, and they ensure that generated API invocations comply with the specification.

methods, and different kinds of parameters. Moreover, the number, ordering, and nesting of parameters can be arbitrary, while some parameters are required and some are optional, making advanced constraint formulations necessary. We implemented a simple, model-agnostic constrained decoding framework for our investigations, which is depicted in Figure 3. The constraints are automatically derived from OpenAPI specifications and represented as regular expressions (regex). They are designed to ensure that generated API invocations are always compliant with the API’s specification. Note that our contribution here lies in this transformation of specifications into constraints, and not the completion engine, which is a straightforward implementation of the standard CD algorithm sketched in Section 2.4. We developed a custom completion engine to maintain full control over the system, explore features not supported by existing completion engines (e.g., advanced regex features and constraint orchestration as described below), and add some debugging functionality. For professional use cases, migrating to an off-the-shelf engine (cf. Section 8) would be expedient to enhance performance and robustness. A main challenge when translating OpenAPI specifications to regular expressions in a generalizable and scalable way is the sheer size and complexity of real-world APIs, which is also reflected in the respective specification files. Each API can have dozens if not hundreds of endpoints (in the specifications we worked with, up to 523 in the GitHub API), and each endpoint can have dozens of parameters, distributed on different locations (body, header, path, query). Parameters can be optional or required, and they can be passed in arbitrary order. Additionally, many parameters have complex data types (objects and arrays) with a strictly defined nesting structure. There are also interdependencies between different parts of an API invocation, e.g., only for certain HTTP methods, parameters can be placed in the request body. In summary, some elements of an API invocation must be followed strictly, while other elements are flexible. Additional challenges arise at the syntax-level of the language the API invocation is implemented in. Our regexes should accept valid JavaScript while not limiting variations permitted by the language, such as formatting or the use of different quotation marks. This is important as different LLMs show different stylistic preferences—forcing them away from their preferred style may degrade performance. There are also a lot of edge cases that need to be handled carefully by the constraints, e.g., an escaped quotation mark must not terminate a string literal, and many other seemingly minor details—our experiments showed that models will find and exploit such loopholes. Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

We address the challenges by structuring constraints hierarchically from the HTTP method and the URL to the different parameters. Once the method is matched, the search space is reduced to URLs for which an endpoint with the respective method is defined. Next, once the URL is matched, the search space is reduced to the parameters of the identified endpoint. Finally, once the parameter name is matched, the search space is reduced to values with the right data type. Within the parameter lists of a given endpoint, we use capture groups7 to keep track of parameters that have been defined. Conditional patterns7 are then used to check the existence of a specific group and thus guard against the double definition of a parameter. Moreover, conditionals are used in conjunction with negative lookahead assertions7 to make it impossible to close a parameter list before all required parameters have been defined. While we could represent all our constraints as one monolithic regex, we opted for a more modular and flexible approach8 . We organize constraints as a set of so-called generation rules. Each generation rule has a regex-encoded start condition, stop condition, and body. Once the start condition matches the code generated so far, we constrain the generation according to the body until the stop condition matches. There is one specialized generation rule for each HTTP method, which is triggered when the corresponding Axios invocation is generated (e.g., axios.post(, with the opening parenthesis being part of the trigger). As multiple generation rules may be active at the same time (a token is only admissible if no rule rejects it), we can leverage overlapping constraints. We do so via a special generation rule that becomes active before all other rules and guides the generation towards the start condition of the other rules, ensuring one of them is going to be activated (thus avoiding using methods not supported by the given API). Moreover, we let generation rule bodies overlap with their stop conditions to ensure they properly terminate. This approach can also be generalized to many other applications of constrained decoding to orchestrate multiple constraints in a flexible and efficient manner. Given the breadth of API designs that can be described through OpenAPI specifications plus the complexity of the resulting constraints, we implemented a test suite to ensure that (a) any specification documents can be processed and translated to constraints and (b) resulting constraints allow and disallow exactly the structures they are supposed to. For (a), we successfully parse 34 real-world OpenAPI specifications, including those used in the synthetic and real-world dataset as well as additional ones. For (b), we created 125 test cases, comprising positive and negative code examples, that our regex constraints must, respectively, match or not match—all these test cases pass. Further implementation details about our constraints are discussed in Appendix B. 5

Real-World Dataset Creation

WAPIIBench, introduced in our previous work [25], is based on a synthetic dataset. This dataset enables controlled, large-scale experiments with precise ground truth. However, it may not fully capture the range of real-world API integration scenarios. In this paper, we therefore complement the original synthetic benchmark with a newly constructed dataset derived from real-world repositories. To construct the new dataset we addressed several challenges that were left open in our earlier work. Real-world code rarely contains sufficient information to serve directly as an unambiguous task description, which makes task reconstruction labor-intensive and limits the number of tasks that can realistically be included. Moreover, even when a plausible task can be identified, manual inspection is still necessary to determine whether the code is correct, or at least complete and plausible, which parts are relevant to the task, and which API version it assumes. 7 https://docs.python.org/3/library/re.html 8 They could still be joined if needed for compatibility with other frameworks.

Manuscript submitted to ACM

Maninger et al. Table 1. Number of tasks per API in the real-world dataset

API

# tasks

Etherscan FrankerFaceZ GitHub Google Maps Platform Instagram JSONPlaceholder npm Registry Slack Web Telegram Bot YouTube Data Zephyr Cloud

3 2 8 2 2 3 2 1 1 2 2

Total

The extension of WAPIIBench with the real-world dataset allows us to evaluate in two complementary settings. The synthetic dataset provides comprehensive and systematic coverage of all endpoints for a limited number of APIs under tightly controlled conditions. However, it may not reflect the real-world web API integration scenarios. In contrast, being derived from repository code, the tasks in the real-world dataset reflect common use cases of widely used APIs. The real-world dataset provides less comprehensive—unlike the synthetic dataset, it is necessarily incomplete and does not cover all endpoints of a given API—but more diverse and realistic coverage, including file-level code context of varying complexity. To derive the real-world dataset, we queried BigQuery9 for GitHub JavaScript files that import the Axios library, yielding an initial sample of 277 files. We manually screened each file to identify web API invocations suitable for being converted into a task. We excluded files if they lacked actual Axios invocations, obscured the server address (making the target API unidentifiable), or interacted with APIs lacking an OpenAPI specification. Finally, to prevent skewing the dataset’s API distribution, we partially excluded two files—effectively web API wrapper libraries—that contained disproportionately high numbers of invocations to a single API. After filtering, 18 files remained, from which we constructed a dataset consisting of 28 tasks (some files yielded more than one). The dataset spans 11 APIs, with the task distribution shown in Table 1. All retrieved code files plus the reason for exclusion and other notes can be found in our artifact. To convert a file into one or more tasks we performed the following manual steps: localizing the API invocation in the source code; identifying the corresponding endpoint in the API specification; authoring a natural-language task description; creating a ground-truth request configuration; preparing starter code grounded in the original code context; and capturing relevant definitions within that starter code. An example task is shown in Listing 4b; Starter Code Preparation: By default, the starter code is a slice of the original file from its beginning up to the API invocation site, giving the model realistic file-level context10 . However, a few targeted adjustments are necessary. (a) Removing giveaways: If the file defines constants such as API keys or the request URL, those definitions are stripped out; leaving them in would hand the model part of the answer. (b) Removing similar calls: Code that closely resembles 9 https://cloud.google.com/bigquery/ 10 The length of starter codes in the dataset varies between 4 and 163 lines.

Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

11 const axios = require ( ' axios '); const User = require ( ' ../ users / userModel '). User ;

// Save the project with gid '778899 ' as a template with the name ' New Project Template ' and make it public to its team . const axios = require ( ' axios '); axios .

module . exports = { getUniqueTagPics ( req , res ) { const hashtag = req . body . hashtag ; User . where ({ id : req . body . userId }) . fetch () . then (( user ) => { if (! user ) { res . status (404) . send ( ' User not found '); } else { const accessToken = user . get ( ' instagram_token '); if ( accessToken ) { // Get a list of media recently tagged with the value in ` hashtag `. Authenticate with the access token in ` accessToken `. axios .

(a) Synthetic dataset

(b) Real-world dataset

Fig. 4. Comparison of a task and starter code from the synthetic dataset [25] to a task and starter code from the real-world dataset (Sec. 5). The former targets the Asana API, the latter the Instagram API. The task is included in the starter code as a line comment.

the target invocation (e.g., other calls to the same API) is also removed, since it would encourage the model to copy rather than reason. Handling Variables in Real-World Tasks: Unlike the synthetic dataset where all API parameters are plain literals, the real-world tasks use file-level starter code, meaning some parameters are passed as variables or constants. This introduces two challenges. (a) Injecting definitions at execution time. Because our pipeline only executes the isolated API invocation (not the surrounding starter code), any definitions that must be in scope are captured in advance and injected into the execution environment. For variables with statically unknown values, such as function parameters, we generate random instances of the appropriate data type and schema. This prevents the model from guessing a literal value and forces it to reference the correct variable. These random values are used only during execution and correctness analysis (cf. Section 2); the model never sees them during code generation. (b) Handling unexpected variable references. A model may attempt to use variables from the starter code that were not part of the intended solution. Without those definitions present, execution would throw a ReferenceError. Since we would rather count such cases as an incorrect argument value than a non-executable implementation, we iteratively expand the set of injected definitions until all remaining ReferenceError in the execution logs stem exclusively from genuinely hallucinated identifiers. Beyond handling task-specific starter codes and variable definitions, we extended the evaluation pipeline as well as the retriever and constraint generator to support a larger set of OpenAPI features present in the specifications of the APIs in the new dataset. For instance, to support server variables (similar to path parameters but in the URL’s server part), we needed to (a) normalize model-generated URLs before matching them against the specification to find the endpoint they selected, (b) merge the list of server variables with the list of the endpoint’s path parameters, (c) extract the values of all server variables and path parameters from the URL, and (d) add the resulting name–value pairs to the request configuration, so they can be properly analyzed later in the evaluation pipeline. Manuscript submitted to ACM

Maninger et al. Moreover, the extraction of Axios invocations from the generated codes in preparation of executing them was

reworked to remove any assumptions about the structure of the starter code, as those assumptions would not hold for the real-world dataset. Finally, since many tasks in the new dataset require passing variables from the starter code as path parameters, we implemented constraints that allow inserting variables into URL strings via string interpolation as well as concatenation. The outlined extensions have made WAPIIBench much more flexible w.r.t. task structure and robust w.r.t. API-specific peculiarities, facilitating reuse and further development. Adding a new task to a dataset is as simple as adding a new entry to a JSON file—it will automatically be evaluated together with the previously existing tasks. Adding an entirely new datasets requires less than 70 lines of code (mostly boilerplate), as evidenced by our script to run the evaluation with the new real-world dataset. If a new task or dataset involves a new API, its OpenAPI specification has to be placed in a dedicated directory and a mapping from the API’s file name to the API’s proper name has to be added to an existing dictionary. 6

Evaluation

This section covers our experimental setup and results for different generation settings as well as a discussion about their efficiency and answers the research questions formulated in Section 1. 6.1

Experimental Setup

To assess their efficacy for web API invocation code generation and answer the research questions posed in Section 1, we put our retriever (Section 3) and constraints (Section 4) into action using WAPIIBench’s dataset and evaluation pipeline (Section 2.2) as well as the new real-world dataset (Section 5). Experiment Variations. We evaluate four different generation settings: Vanilla Generation based only on the prompt and starter code RAG Retrieval-augmented generation CD Constrained decoding RAG + CD Retrieval-augmented generation combined with constrained decoding Furthermore, conduct all experiments with two different starter code variants, a.k.a. setups (cf. also Section 2.2): Full completion The model has to select the correct endpoint (URL plus HTTP method) as well as request arguments in order to solve the given task Argument completion The correct endpoint is already provided, the model has to select the correct arguments in order to solve the given task Models. We study the performance of 24 state-of-the-art open-source LLMs, listed in Table 2. These models were selected due to their frequent appearance in coding leaderboards11 . To obtain an approximate upper bound on performance, we also evaluated OpenAI’s commercial GPT-4o and GPT-4o mini models. As they are closed-source, we cannot apply constrained decoding to the GPT models. Due to a model-internal bug, Code Llama (34B) cannot use constrained decoding either. For all experiments, greedy decoding is used to ensure deterministic and reproducible results. In the RAG experiments, five endpoint chunks are retrieved per task. If the total length of the retrieved text exceeds 8000 characters, it is truncated. 11 E.g., https://evalplus.github.io/leaderboard.html

Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

Table 2. Evaluated LLMs and their sizes

Model

Size

CodeT5+ [49] StarCoder [22] StarCoder2 [23] DeepSeek-Coder [15] Qwen2.5-Coder [18] Llama 3.1 [9] Code Llama [38] GPT-4o mini [28] GPT-4o [29]

6B, 16B 1B, 3B, 7B, 15.5B 3B, 7B, 15B 1.3B, 6.7B, 33B 0.5B, 1.5B, 3B, 7B, 14B, 32B 8B, 70B 7B, 13B, 34B, 70B N/A N/A

This threshold is set to strike a balance between supporting models with shorter context lengths and truncating as few retriever outputs as possible. Only for CodeT5+, we have to reduce the threshold to 4000 to avoid inference-time crashes and skip some tasks from the real-world dataset as their starter code alone already exceeds the model’s maximum context length (we count this as a generation-time error). The prompts used for the synthetic and the real-world tasks are given in Listing 3 and 4 in the appendix. Metrics. We focus our discussion on the following four key metrics; a multitude of additional metrics is provided in Appendix F: Correct implementations Ratio of generated codes that are executable and correctly solve the given task, i.e., their request configuration matches the ground-truth Hallucinated endpoints Ratio of generated codes that use a combination of URL and HTTP method that is not defined in the API’s specification12 (only applicable to full completion) Hallucinated implementations Ratio of generated codes that use at least one request parameter that is not defined in the API’s specification (only applicable to argument completion) Executable implementations Ratio of generated codes that are executable, i.e., complete, syntactically correct, and free of runtime errors Here, all metrics will be reported relative to the total number of tasks in the respective dataset. Metrics relative to only the tasks that resulted in executable code are provided in the appendix. For a fair comparison between generation settings compatible and incompatible with closed-source models, we always exclude the GPT-4o models when aggregating metrics. Otherwise, vanilla and RAG would get an advantage over CD and RAG + CD due to the contribution of the powerful GPT-4o models to the aggregate values. Additionally, we exclude models from average calculations if they did not generate any executable implementation. In the following subsections, we elaborate on the experimental results for the four generation settings. The results for the synthetic dataset are presented in Figure 5—showing the ratio of correct implementations for full and argument completion across all four generation settings—and Figure 6—showing the ratio of hallucinated endpoints and implementations, respectively, for the two setups. Similarly, Figures 7 and 8 present the results for the real-world dataset. Comprehensive tables with numeric results for all metrics are provided in Appendix F. 12While we use the intuitive term “hallucinated” for this error category, it subsumes any mismatch with the API specification, including cases where the

model’s intention may be correct, but its spelling is incorrect. Manuscript submitted to ACM

Co d Co eT5+ de (6 T Sta 5+ ( B) rCo 16B Sta der ) rC (1 Sta oder B) Sta rCo (3B rCo der ) d ( Sta er ( 7B) rCo 15.5 Sta der B) rC 2 De Star oder (3B) ep Co 2 ( De Seek der2 7B) ep -C (1 S o De eek der 5B) ep -Co (1. 3 S Qw eek der ( B) en -Co 6.7 Qw 2.5- der B) en Cod (33 2 e B Qw .5-Co r (0. ) en de 5B Qw 2.5- r (1. ) e C 5 Qw n2.5 oder B) en -Co (3B Qw 2.5- der ) en Cod (7B 2.5 er ) -Co (14 Lla der ( B) m 3 Lla a 3. 2B) ma 1 ( Co 3.1 8B) d Co e Lla (70B de ma ) Co Llam (7B de a ) Co Llam (13B de a ) Lla (34 ma B) GP (70 T-4 B) om i GP ni T-4 o

Correct implementations

14 Maninger et al.

Full completion

Argument completion

6.2

Manuscript submitted to ACM

Vanilla

13 Small quantitative deviations are due to minor adjustments of the prompt and implementation.

RAG

RAG + CD

0.6

0.5

0.4

0.3

0.2

0.1

0.0

RAG + CD

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

Fig. 5. Comparison of correct implementations between generation settings on the synthetic dataset

Vanilla Generation

The results for vanilla generation on the synthetic dataset are consistent with our previous findings [25]13 . We will

focus our analysis here on a few key aspects and refer the reader to our prior work for a broader discussion.

The overall benchmark performance for the full completion setup is low, at an average correctness rate among

open-source models of 13% and Code Llama (70B) as the best performing model at 30%. GPT-4o achieves 58%. All

Qwen2.5-Coder and Llama 3.1 models except Qwen2.5-Coder (32B) do not produce any executable implementation as

Mitigating Errors in LLM-Generated Web API Invocations

Full completion

Vanilla

RAG

RAG + CD

0.7

Hallucinated endpoints

0.6 0.5 0.4 0.3 0.2 0.1

Co d Co eT5+ de (6 T Sta 5+ ( B) rCo 16B Sta der ) rC (1 Sta oder B) Sta rCo (3B rCo der ) d ( Sta er ( 7B) rCo 15.5 Sta der B) rC 2 S De tar oder (3B) ep Co 2 ( De Seek der2 7B) ep -C (1 S o De eek der 5B) ep -Co (1. 3 S Qw eek der ( B) en -Co 6.7 Qw 2.5- der B) en Cod (33 2 e B Qw .5-Co r (0. ) en de 5B) Qw 2.5- r (1. e C 5 Qw n2.5 oder B) en -Co (3B Qw 2.5- der ) en Cod (7B 2.5 er ) -Co (14 Lla der ( B) m 3 Lla a 3. 2B) ma 1 ( Co 3.1 8B) d Co e Lla (70B de ma ) Co Llam (7B de a ) Co Llam (13B de a ) Lla (34 ma B) GP (70 T-4 B) om i GP ni T-4 o

0.0

Hallucinated implementations

Argument completion

Vanilla

RAG

RAG + CD

0.5 0.4 0.3 0.2 0.1

0.0

Fig. 6. Comparison of hallucinated endpoints between generation settings on the synthetic dataset

they are unable to continue the given starter code in a syntactically valid way. The remaining models hallucinate on average 31% of all endpoints. On the real-world dataset, we observe a higher baseline performance—47% correctness on average among open-source models producing executable implementations. We attribute this to the tasks in this dataset being drawn from real-world repositories, and thus having a high likelihood of representing common use cases of wide-spread APIs that are also represented in the models’ training corpora. Nonetheless, most Qwen2.5-Coder and Llama 3.1 models that were unable to generate syntactically valid code completions for synthetic tasks fail here as well. The average rate of hallucinated endpoints among the remaining models remains high at 29%. Manuscript submitted to ACM

Maninger et al.

Full completion

Vanilla

RAG

RAG + CD

0.7 Correct implementations

0.6 0.5 0.4 0.3 0.2 0.1

0.0

Argument completion

Vanilla

RAG

RAG + CD

0.8 Correct implementations

0.7 0.6 0.5 0.4 0.3 0.2 0.1

0.0

Fig. 7. Comparison of correct implementations between generation settings on the real-world dataset

On both datasets, models achieve generally higher correctness rates for argument completion than for full completion— 26% and 55% on average, respectively. On the synthetic dataset, Qwen2.5-Coder (14B) scores highest at 47%. Interestingly, GPT-4o-mini performs better than GPT-4o (63% vs. 59%). An important caveat for the real-world dataset is that several tasks included do not require parameters in locations other than the path, i.e., there are no further arguments to complete and all the model has to do is recognize this fact and terminate the statement by generating a closing parenthesis. On average, 39% and 20% of implementations contain hallucinations, depending on the dataset. We hypothesize that this increased hallucination rate compared to the full completion setup stems from the fact that for argument completion, the model is forced to generate arguments for a predetermined endpoint that it might not be familiar with. Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

0.45

Full completion

Vanilla

RAG

RAG + CD

0.40 Hallucinated endpoints

0.35 0.30 0.25 0.20 0.15 0.10 0.05

Co d Co eT5+ de (6 T Sta 5+ ( B) rCo 16 Sta der B) rC (1 Sta oder B) Sta rCo (3B rCo der ) d ( Sta er ( 7B) rCo 15. Sta der 5B) rC 2 De Star ode (3B) ep Co r2 De Seek der2 (7B) ep -C (1 S o De eek der 5B) ep -Co (1. 3 S Qw eek der ( B) en -Co 6.7 Qw 2.5- der B) en Cod (33 2 e B Qw .5-Co r (0. ) en de 5B Qw 2.5- r (1. ) e C 5 Qw n2.5 oder B) en -Co (3B Qw 2.5- der ) en Cod (7B 2.5 er ) -Co (14 Lla der ( B) m 3 Lla a 3. 2B) ma 1 ( Co 3.1 8B) d Co e Lla (70B de ma ) Co Llam (7B de a ) Co Llam (13B de a ) Lla (34 ma B) GP (70 T-4 B) om i GP ni T-4 o

0.00

Argument completion

Vanilla

RAG

RAG + CD

Hallucinated implementations

0.30 0.25 0.20 0.15 0.10 0.05

0.00

Fig. 8. Comparison of hallucinated endpoints between generation settings on the real-world dataset

In contrast, for full completion, it is free to select any endpoint it is familiar with—even if it is the wrong choice, it is less likely to be a hallucination. Summary: Vanilla generation yields low overall correctness, particularly for full completion (13% average across open-source models on the synthetic dataset), with hallucinated endpoints affecting nearly a 1/3 of all generations. Model size is only a weak predictor of performance, and several recent models fail to produce executable code entirely. Real-world tasks yield higher baseline correctness (47%), presumably reflecting the prevalence of common API patterns in training data, yet hallucination rates remain substantial. These results confirm that unassisted Manuscript submitted to ACM

Maninger et al.

LLMs are unreliable for web API invocation code generation and motivate the mitigation approaches studied in the following subsections. 6.3

Retrieval-Augmented Generation

In the case of the full completion setup on the synthetic dataset, RAG yields a clear benefit: it more than doubles the average correct implementation rate compared to vanilla generation, raising it to 22%. The picture changes substantially, however, for the argument completion setup on the synthetic dataset and for both setups on the real-world dataset: in all these cases, RAG’s correctness scores fall within 1% of the vanilla baseline, indicating negligible net impact. Examining per-model results reveals that the effect of RAG is highly model-dependent rather than uniformly beneficial or harmful. Individual models swing between a +562% gain in correctness (StarCoder 1B, full completion, synthetic dataset) and complete collapse to 0% correctness (CodeT5+ 6B, full completion, synthetic dataset), underscoring that RAG’s utility is strongly conditioned on the underlying model’s capacity to leverage retrieved context. The ambivalent outcomes stem from several competing effects. On the positive side, RAG helps models select the correct API endpoint—an advantage that, by definition, vanishes in the argument completion setup, where the endpoint is already given. On the negative side, RAG’s potential benefits in selecting correct parameters are overshadowed by a substantial increase in the number of unnecessary parameters for most models (see Appendix F). These parameters are valid (but optional) for the chosen endpoint according to the API specification yet irrelevant or undesirable for the task at hand. We hypothesize that most LLMs—even those not explicitly instruction-tuned—tend to treat retrieved context as implicit guidance, attempting to incorporate as many listed parameters as possible into the generated API call. This over-inclusion is further amplified when the same parameter is defined across multiple endpoints and thus appears repeatedly in the retrieved context. A prime example is Asana’s optional opt_pretty parameter, which is used to format responses in a human-readable way. Intended for debugging, it can be passed to every API endpoint. Accordingly, it appears in every retrieved chunk, and we observe a high frequency of use among models. How susceptible a given model is to this behavior ultimately determines how much RAG hurts its correctness. For Qwen2.5-Coder and Llama 3.1, RAG has no effect on their tendency to produce only non-executable code. Among the remaining models, however, RAG delivers substantial reductions in hallucinations on average: 82% fewer on the synthetic dataset and 10% fewer on the real-world dataset for full completion, and 60% and 51% fewer, respectively, for argument completion. RQ1: To what extent does RAG improve the correctness of LLM-generated web API invocation code? Answer: RAG effectively reduces hallucinations but does not uniformly improve correctness and can even degrade it. Its benefits are most pronounced for full completion, where it also aids endpoint selection, while its impact on argument completion is negligible. Whether RAG is net-positive depends strongly on the individual model: models prone to incorporating retrieved parameters indiscriminately suffer the greatest correctness losses, making careful model-specific evaluation advisable before deploying RAG in practice. 6.4

Constrained Decoding

On the synthetic dataset, constrained decoding consistently increases correctness for all models and both setups. For full completion—where RAG achieved its highest average gain at +100%—CD outperforms it with +209%, leading to an Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

average correctness rate of 28%. For argument completion, the gain is +143%, corresponding to an average correctness rate of 57%. Code Llama (70B) remains the strongest open-source model at 50% for full completion, closing the gap to GPT-40 at 58%. For argument completion, Code Llama (70B) with CD achieves 72% correctness, outperforming GPT-4o (59%) and GPT-4o mini (63%). The Qwen2.5-Coder and Llama 3.1 models that were previously unable to generate any executable code in the full completion setup, now constrained into the required syntactic structure, achieve executability rates similar to the other models. However, disregarding models with 0% executability, the overall rate of non-executable implementations increases slightly compared to the previous generation settings: from 92% (vanilla) and 86% (RAG) to 81% (CD) for full completion and from 98% (vanilla) and 93% (RAG) to 85% (CD) for argument completion. A potential explanation for this is the distortion an LLM’s next token probability distribution that CD may cause [2]. On the real-world dataset, we see similar trends, but with a smaller amplitude due to the higher baseline performance. CD increases correctness by +69% and +18%, depending on the setup. Unlike on the synthetic dataset, on the real-world dataset, a few models drop in correctness with CD: 6 out of 23 models for full completion and 3 out of 23 for argument completion. The only extreme case is Qwen-2.5-Coder (0.5B), which drops correctness by -82% on full completion—in all other cases, correctness drops by less than 30%. This drop is likely a consequence of the reduced executability rates. The most striking property of CD—one that holds universally across all models, setups, and datasets—is its guarantee of zero hallucinated endpoints and implementations. This is a direct consequence of how the constraints are constructed (cf. Section 4), and our experiments consistently confirm it.

RQ2: To what extent does CD improve the correctness of LLM-generated web API invocation code? Answer: CD eliminates hallucinations of URLs, HTTP methods, and arguments by construction, reducing them to zero across all evaluated settings. Beyond this guarantee, CD consistently and substantially boosts the correctness of generated API invocations—in some cases making smaller constrained models competitive with larger unconstrained ones. Given these clear benefits and the negligible risk of correctness degradations, CD can be recommended unconditionally for web API invocation tasks.

6.5

Combining RAG and CD

The combination of RAG and CD retains CD’s zero-hallucination guarantee while additionally leveraging retrieved context to guide generation. This yields strong average correctness gains over the vanilla baselines: +332% and +111% on the synthetic dataset, and +74% and +14% on the real-world dataset, for full and argument completion, respectively. RAG + CD also achieves the highest correctness scores across all setups and datasets: 67% (DeepSeek-Coder 33B) and 73% (Qwen2.5-Coder 14B) for full and argument completion on the synthetic dataset, and 71% (Llama 3.1 70B) and 71% (Qwen2.5-Coder 32B / Llama 3.1 70B) on the real-world dataset. Compared to CD alone, RAG + CD increases the share of executable implementations by +10% and +7% for full and argument completion on the synthetic dataset, with only a +1% gain on the real-world dataset. Correctness gains over CD, however, are limited to full completion (+43% synthetic, +11% real-world); for argument completion, average correctness declines (-14% synthetic, -3% real-world), consistent with the parameter overuse effect described in Section 6.3. RAG + CD is also less consistent than CD alone regarding the achieved correctness gains. CD improved correctness for all models on the synthetic dataset and reduced it for only 6 (full completion) and 3 (argument completion) models Manuscript submitted to ACM

Maninger et al. Table 3. Generation speed in tokens per second for each API and generation setting

API Asana Google Calendar Google Sheets Slack Web

Vanilla

RAG

RAG + CD

29.85 30.94 31.72 29.23

29.54 26.54 26.26 29.59

7.15 10.36 10.37 7.30

6.27 7.25 2.31 7.49

on the real-world dataset. RAG + CD, by contrast, reduces correctness for 1 and 2 models on the synthetic dataset, and for 9 and 5 models on the real-world dataset—a wider spread of regressions. RQ3: To what extent does combining RAG and CD improve the correctness of LLM-generated web API invocation code compared to each technique in isolation? Answer: RAG + CD achieves the highest correctness scores across all setups and datasets, outperforming both techniques in isolation. However, this peak performance comes at the cost of consistency: compared to CD alone, RAG + CD produces more correctness regressions across individual models, and its average correctness gain over CD is limited to full completion—argument completion suffers due to parameter overuse. RAG + CD is therefore advisable when maximum correctness is the priority and model behavior with retrieved context has been validated in advance. When neither condition holds, CD alone is the safer and more robust choice. 6.6

Efficiency

Depending on the use case, the efficiency of the generation setting may be an important factor as it directly affects inference speed. To estimate the overhead of RAG and CD, we run a series of runtime measurements and calculate the average token per second (TPS) generation speed. The results are shown in Table 3. We selected one representative task for each API of the synthetic dataset and repeated the generation ten times per task on one A100 40GB GPU, using StarCoder2 [23] and the full completion setup. The time to set up and run the retriever for RAG or to create the constraints for CD is excluded from the measurements. Since the variation in runtime between individual runs was low14 , we focus our discussion on the average TPS. The results show that vanilla generation is largely invariant to the API or task (ca. 30 TPS), which is expected since the model’s prompt is almost static except for the API name and the task description, which have a negligible impact on the input length. RAG is overall slightly slower than vanilla due to the increased computational cost of processing the retrieved context. For Asana and Slack, the difference is indistinguishable from measurement noise, for Google Calendar and Sheets, it amounts to a -4 TPS or -5 TPS slowdown, respectively (about -13% reduction). CD has a clear performance impact compared to vanilla due to constraint checking. For Asana and Slack, it is about -76%, for Google Calendar and Sheets about -67%. Naturally, the combination of RAG and CD combines the performance overhead of the two approaches. The strongest reduction can be observed for Google Sheets at roughly -93%, while Slack shows no measurable reduction. The impact of RAG on inference speed might be acceptable in most scenarios given the potential improvement of generation results it offers. The only factor affecting RAG’s overhead is the truncation threshold of the retrieved context as transformers’ inference speed is known to scale with input length (the exact scaling behavior depends on the 14 The highest recorded standard deviation is 0.6s on a 30s run; most are much lower. The complete measurements are available in our artifact.

Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

used attention mechanism and is typically between linear and quadratic). The impact of CD on inference speed, on the other hand, highly depends on the CD implementation. The prototype implementation used for our study (for reasons outlined in Section 4) is clearly not optimized for performance and unsuitable for use cases with real-time requirements. However, existing works show that CD can be implemented in highly efficient ways [53]. If attention masks can be fully or partially precomputed (which is the case for our constraints), CD can run at near-zero overhead [7, 48]. 7

Discussion and Limitations

This section discusses practical implications of our results—including implementation challenges of RAG and CD—and limitations that should be considered when interpreting the findings. Practical Implications. The conducted experiments demonstrate that retrieval-augmented generation, constrained decoding, or both, are effective in reducing or eliminating errors and hallucinations of LLMs in writing web API integration code. In direct comparison, constrained decoding offers more consistent gains than RAG and provides hard guarantees on specification-compliance, while also avoiding prompt bloat [12]. Regardless of which approach is chosen, it is advisable to validate that the model in use actually benefits from it. We were unable to identify meaningful patterns relating model family or size to the degree of benefit from RAG or CD. The correlation between benchmark performance and model size is, for instance, only weak. Manual inspection of generated code suggests that the main reason is the highly idiosyncratic behavior of individual models—even within the same family. Code Llama (34B) is a prime example: it frequently inserts placeholders such as <body>, producing syntactically broken code, while its smaller and larger siblings do not exhibit this behavior. On the implementation side, we experienced different challenges for RAG and CD. RAG benefits from a production-grade ecosystem of components, such as vector databases and embedding models, that can be readily built upon. Its main challenge lies in the degrees of freedom at each stage of a RAG pipeline (e.g., preprocessing, chunking, embedding, retrieval, reranking, formatting, etc.). Even when established best practices are followed, many parameters remain to be tuned: chunk size, overlap, number of retrieved chunks, truncation threshold, and others. This creates a risk of overfitting the pipeline to certain models, which is especially problematic in a benchmarking context where all models must be treated equally. CD’s main challenge lies in constraint design. Care must be taken not to over- or under-constrain a model. In addition, certain rules are difficult or impossible to express with regexes (or grammars), requiring workarounds or approximations. Regarding the completion engine, we implemented a custom one to be able to explore the orchestration of multiple constraint rulesets and to experiment with features of extended regular expressions not supported by off-the-shelf completion engines (cf. Section 4). However, the complexity of the constraint checker is negligible compared to that of the constraint generator. Limitations. Our implementations of RAG and CD require the user to provide the name and specification of the API, for which invocations are to be generated. This limitation could be addressed by automatically inferring the API referred to in a task description and retrieving the corresponding specification. However, our work targets use cases in the API integrations industry, where usually the customer knows which APIs they want to be integrated, and the developer only has to decide which endpoints to use in order to connect the APIs in the desired way. In this scenario, having to provide the correct API specification to our system is not a real limitation. Our work is not intended for highly exploratory development scenarios, but future work could fill this gap. Manuscript submitted to ACM

Maninger et al. Both RAG and, to a larger extent, CD lead to a small increase in non-executable implementation. Since WAPIIBench’s

evaluation pipeline is execution-based, non-executable codes cannot be properly evaluated and do not contribute to correctness and hallucination metrics. To provide a complementary view on the evaluation results, we calculate a second variant of each metric after filtering out any non-executable codes—these metrics are presented in the appendix. However, for practical applications like IDEs, sporadic cases of non-executable codes are no big concern, as code suggestions can be generated in the background and discarded or regenerated if they fail a syntax check. As mentioned earlier, the biggest limitation of our RAG evaluation is the large design space for the retriever implementation. While we tried to make reasonable design decisions and to validate those decisions with preliminary experiments, any change to the RAG pipeline will inevitably influence the evaluation results in a non-uniform way across models. Another minor limitation of our RAG implementation is that during preprocessing of OpenAPI specifications, references in are only resolved up to a certain level to avoid inflating the size of individual chunks. However, this also means that the retriever will be less useful for tasks requiring deeply nested parameters. The limitations of our CD implementation revolve around what structures the constraints do and do not permit. There is a natural tradeoff between allowing everything that is correct and preventing everything that is incorrect. In consequence, effective constraints will often limit the model’s flexibility at the same time. For example, to be able to effectively guide the model through generating a syntactically correct and hallucination-free web API invocation, we constrain the Axios call syntax to the one shown in Listing 1. However, some models may prefer a different syntax (e.g., axios.request(<url>, <method>, <config>)) and perform better when given the liberty to use it. Another limitation is that our constraints act only locally from the beginning of an API invocation to its end. When (instead of literals) variables are used as parameter values, we have no control over such a variable’s value or data type. The main limitation of the new real-world dataset is its small size due to the high manual effort involved in its creation. However, WAPIIBench’s extensibility makes it easy for everyone to contribute tasks, so the dataset may be expanded in the future. The newly implemented support for per-task starter codes is limited to program prefixes, so fill-in-the-middle generation is not possible. Moreover, variable or constant definitions within the starter code and that are used for execution-based evaluation must be primitive data types, objects, or arrays—it is not possible to use imported classes or functions. Despite these limitations, the consistent empirical trends across 24 models, two datasets, two starter code setups, and four generation settings provide a solid foundation for the practical adoption and further refinement of RAG and CD in web API invocation scenarios. 8

Related Work Differentiation of API Invocation Settings. API invocation has been studied in several settings that differ substantially

from the one considered in this work. We distinguish five categories: (1) general web APIs (our setting), (2) domain-specific web APIs, (3) SDK-wrapped web APIs, (4) local function APIs, and (5) tool APIs for AI agents. Our work considers general RESTful web APIs described by OpenAPI, where code must explicitly construct complete HTTP requests. Compared to local function calls, this requires identifying operations by HTTP method and endpoint, configuring multiple request components (path, query, headers, and body), and handling externally documented interfaces. We are not aware of other works besides WAPIIBench [25] that systematically analyze the capabilities of LLMs in this specific setting. Prior work has largely focused on different settings. Domain-specific web APIs target constrained ecosystems such as OData [17, 45, 46], making the task considerably narrower than arbitrary OpenAPI services. SDK-based approaches Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

replace HTTP requests with library-specific abstractions [19, 41], shifting the problem toward SDK usage rather than direct API invocation. Research on local function calls [10, 11, 55] considers a much simpler interface consisting of a function name and a single argument list. Finally, tool-use benchmarks for LLM agents [21, 35, 39, 43, 54] typically expose function-like interfaces optimized for agent interaction rather than production-ready HTTP integration code. Consequently, techniques and evaluation results from these settings are not directly transferable to general web API invocation. Retrieval-Augmented Generation. Retrieval augmented generation (RAG) [20] is commonly used when a model’s memorized knowledge is insufficient to solve a given task. This includes many applications in code generation. To improve correctness in general code generation, either similar code examples are retrieved [24, 56] or excerpts from software documentation [58]. Some RAG approaches specifically target local function API invocations, retrieving examples [3] or documentation [30, 55]. Work on web APIs is limited to SDK wrappers [19]. To our knowledge, we are the first to study RAG in the context of general web API invocations as defined at the beginning of this section. Our work falls into the category of retrieval from documentation, concretely OpenAPI specification documents. In this setting, we are facing some special challenges as chunks from OpenAPI specifications must be carefully crafted (i.e., scoping by endpoint, resolution of references, and inlining of certain global definitions) to preserve crucial context. Moreover, unlike other works, we design a custom format for the retriever output to optimize token usage and LLM comprehension. Constrained Decoding. General-purpose constrained decoding frameworks for code generation [2, 7, 48, 53] primarily enforce syntactic validity using regular expressions or context-free grammars. More recent work has extended constrained decoding to semantic correctness, for example by incorporating database schemas [40], domain-specific semantic rules [33], or programming language type systems [27]. Another line of work addresses hallucinated function or method invocations by constraining generation based on information available in the code base, such as language servers or mined program dependencies [1, 3, 51]. While effective for local APIs, these approaches cannot readily be applied to web APIs, whose interfaces are specified externally and whose invocations are structurally more complex, requiring the generation of complete HTTP requests. Constrained decoding has also been applied to tool use in AI agents [50, 57], where tools are invoked through comparatively simple JSON or Python-based interfaces. In contrast, our work targets general web APIs described by OpenAPI specifications. To our knowledge, we are the first to explore constrained decoding in this setting, introducing constraints that capture the structure of web API invocations and exploit externally provided API specifications to prevent hallucinations. Alternative Approaches. Besides retrieval-augmented generation and constrained decoding, several other approaches have been proposed to mitigate problems LLMs face when generating API invocations. Most of these works focus on local libraries (primarily in Python), but some also consider web APIs, even though they are not as complex as the ones targeted by our work (cf. discussion at the beginning of this section). These approaches are primarily fine-tuning [10, 30, 55] and iterative refinement [6, 11, 56] to reduce hallucinations and improve correctness. All these approaches come with their own advantages and disadvantages: RAG is well established and relatively easy to implement but risks prompt bloat [12]. Constrained decoding is the only approach to offer guarantees but is relatively inflexible and, depending on the implementation, adds inference-time overhead. Fine-tuning is arguably the most natural way to improve a model but has the highest computational cost and its training data is prone to becoming Manuscript submitted to ACM

Maninger et al.

outdated. Iterative refinement might be more reliable than RAG but can be expensive in terms of token consumption. It is worth exploring all these approaches to assess their suitability for web API invocations. We argue that they should be seen as complementary rather than alternative. 9

Conclusion and Future Work

In this paper, we examined approaches to address the challenges faced by large language models when generating web API invocation code: retrieval-augmented generation (RAG), constrained decoding (CD), and a combination thereof. To this end, we implemented a retriever that processes OpenAPI specifications and stores, retrieves, and compactly formats endpoint-level chunks that can be injected into a model’s prompt. Furthermore, we developed a custom translation algorithm that derives constraints, represented as regular expressions, from OpenAPI specifications. Both implementations effectively manage the complexity and scale of real-world APIs. To evaluate the approaches we use the original synthetic dataset of the WAPIIBench benchmark [25] and a new dataset with tasks manually derived from real-world GitHub repositories that we introduce in this paper. Our evaluation results demonstrate that both RAG and CD can reduce hallucinated endpoints and arguments and improve the overall correctness of the generated web API invocations. However, the potential benefit of RAG is much more model- and task-dependent than that of CD. Moreover, only CD can guarantee the absence of hallucinations. The combination of RAG and CD can further boost correctness for some models but degrades correctness for others. Overall, our work underscores the importance of integrating quality assurance techniques into AI-driven code generation workflows as a practical path toward correctness and reliability. Looking ahead, we plan to extend RAG and CD to further code generation tasks and to broaden the quality criteria they target beyond correctness and specification-compliance, for instance, to security properties. At the same time, our benchmark lays the foundation for future work to explore other approaches to improving quality of LLM-based code generation. Although our evaluation pipeline is specialized for JavaScript and Axios, the underlying datasets and methodology are language- and library-agnostic and can be transferred to other ecosystems. WAPIIBench is also designed for extensibility: new datasets can be added, and existing ones expanded to cover additional APIs or completion scenarios with minimal effort. Acknowledgments This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy (EXC-3057/1 Reasonable Artificial Intelligence, Project No. 533677015), the Hessian Ministry of Higher Education, Research, Science and the Arts within the cluster project The Third Wave of Artificial Intelligence (3AI), by the National Research Center for Applied Cybersecurity ATHENE within the project Foundational Models for Secure Software Development, and by the LOEWE initiative (Hesse, Germany) [LOEWE/4a//519/05/00.002(0013)/95]. References [1] Lakshya A. Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram K. Rajamani. 2023. Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/662b1774ba8845fc1fa3d1fc0177ceeb-Abstract-Conference.html [2] Luca Beurer-Kellner, Marc Fischer, and Martin T. Vechev. 2024. Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation. In Fortyfirst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. https://openreview.net/forum?id= pXaEYzrFae Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

[3] Yujia Chen, Mingyu Chen, Cuiyun Gao, Zhihan Jiang, Zhongqi Li, et al. 2025. Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, FSE Companion 2025, Clarion Hotel Trondheim, Trondheim, Norway, June 23-28, 2025, Leonardo Montecchi, Jingyue Li, Denys Poshyvanyk, and Dongmei Zhang (Eds.). ACM, 468–479. doi:10.1145/3696630.3728569 [4] Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, et al. 2025. The effects of generative AI on high-skilled work: Evidence from three field experiments with software developers. Available at SSRN 4945566 (2025). doi:10.2139/ssrn.4945566 [5] Daniel Deutsch, Shyam Upadhyay, and Dan Roth. 2019. A General-Purpose Algorithm for Constrained Sequential Inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019, Mohit Bansal and Aline Villavicencio (Eds.). Association for Computational Linguistics, 482–492. doi:10.18653/V1/K19-1045 [6] Hanxing Ding, Shuchang Tao, Liang Pang, Zihao Wei, Jinyang Gao, et al. 2025. ToolCoder: A Systematic Code-Empowered Tool Learning Framework for Large Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, 17876–17891. https://aclanthology.org/2025.acl-long.874/ [7] Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. 2025. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. In Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025, Matei Zaharia, Gauri Joshi, and Yingyan (Celine) Lin (Eds.). OpenReview.net/mlsys.org. https://openreview.net/forum?id= rjQfX0YgDl [8] Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, et al. 2024. What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study. CoRR abs/2407.06153 (2024). doi:10.48550/ARXIV.2407.06153 arXiv: 2407.06153. [9] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. 2024. The Llama 3 Herd of Models. CoRR abs/2407.21783 (2024). doi:10.48550/ARXIV.2407.21783 arXiv: 2407.21783. [10] Sujan Dutta, Sayantan Mahinder, Raviteja Anantha, and Bortik Bandyopadhyay. 2024. Applying RLAIF for Code Generation with API-usage in Lightweight LLMs. CoRR abs/2406.20060 (2024). doi:10.48550/ARXIV.2406.20060 arXiv: 2406.20060. [11] Aryaz Eghbali and Michael Pradel. 2024. De-Hallucinator: Iterative Grounding for LLM-Based Code Completion. CoRR abs/2401.01701 (2024). doi:10.48550/ARXIV.2401.01701 arXiv: 2401.01701. [12] Tiantian Gan and Qiyao Sun. 2025. RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation. CoRR abs/2505.03275 (2025). doi:10.48550/ARXIV.2505.03275 arXiv: 2505.03275. [13] GitClear 2025. AI Copilot Code Quality: Evaluating 2024’s Increased Defect Rate with Data. GitClear. Retrieved 2025-05-17 from https: //www.gitclear.com/ai_assistant_code_quality_2025_research [14] Sacha Greif and Eric Burel. 2022. State of JavaScript 2022. Devographics. Retrieved 2024-07-30 from https://2022.stateofjs.com/en-US/ [15] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming The Rise of Code Intelligence. CoRR abs/2401.14196 (2024). doi:10.48550/ARXIV.2401.14196 arXiv: 2401.14196. [16] Chris Hokamp and Qun Liu. 2017. Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, 1535–1546. doi:10.18653/v1/P17-1141 [17] Saghar Hosseini, Ahmed Hassan Awadallah, and Yu Su. 2021. Compositional Generalization for Natural Language Interfaces to Web APIs. CoRR abs/2112.05209 (2021). https://arxiv.org/abs/2112.05209 arXiv: 2112.05209. [18] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, et al. 2024. Qwen2.5-Coder Technical Report. CoRR abs/2409.12186 (2024). doi:10.48550/ ARXIV.2409.12186 arXiv: 2409.12186. [19] Nihal Jain, Robert Kwiatkowski, Baishakhi Ray, Murali Krishna Ramanathan, and Varun Kumar. 2025. On Mitigating Code LLM Hallucinations with API Documentation. In 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025. IEEE, 237–248. doi:10.1109/ICSE-SEIP66354.2025.00027 [20] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, et al. 2020. Retrieval-Augmented Generation for KnowledgeIntensive NLP Tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html [21] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, et al. 2023. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 3102–3116. doi:10.18653/V1/2023.EMNLP-MAIN.187 [22] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, et al. 2023. StarCoder: may the source be with you! Trans. Mach. Learn. Res. 2023 (2023). https://openreview.net/forum?id=KoFOg41haE [23] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. CoRR abs/2402.19173 (2024). doi:10.48550/ARXIV.2402.19173 arXiv: 2402.19173. [24] Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022. ReACC: A Retrieval-Augmented Code Completion Framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, 6227–6240. Manuscript submitted to ACM

Maninger et al.

doi:10.18653/v1/2022.acl-long.431 [25] Daniel Maninger, Leon Chemnitz, Amir Molzam Sharifloo, Jannis Brugger, and Mira Mezini. 2025. Benchmarking Web API Integration Code Generation. In 2nd IEEE/ACM International Conference on AI-powered Software, AIware 2025, Seoul, Republic of Korea, November 19-20, 2025. IEEE, 240–248. doi:10.1109/AIWARE69974.2025.00034 [26] MarketsandMarkets Research Private Ltd. 2024. API Management Market by Platform, Service – Global Forecast to 2029. MarketsandMarkets Research Private Ltd. Retrieved 2025-05-18 from https://www.marketsandmarkets.com/Market-Reports/api-management-market-178266736.html [27] Niels Mündler, Jingxuan He, Hao Wang, Koushik Sen, Dawn Song, et al. 2025. Type-Constrained Code Generation with Language Models. Proc. ACM Program. Lang. 9, PLDI (2025), 601–626. doi:10.1145/3729274 [28] OpenAI 2024. GPT-4o mini: advancing cost-efficient intelligence. OpenAI. Retrieved 2026-06-29 from https://openai.com/index/gpt-4o-miniadvancing-cost-efficient-intelligence/ [29] OpenAI 2024. Hello GPT-4o. OpenAI. Retrieved 2026-06-29 from https://openai.com/index/hello-gpt-4o/ [30] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large Language Model Connected with Massive APIs. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). https://proceedings.neurips.cc//paper_files/paper/2024/hash/e4c61f578ff07830f5c37378dd3ecb0d-Abstract-Conference.html [31] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022. IEEE, 754–768. doi:10.1109/SP46214.2022.9833571 [32] Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023. Do Users Write More Insecure Code with AI Assistants?. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Denmark, November 26-30, 2023, Weizhi Meng, Christian Damsgaard Jensen, Cas Cremers, and Engin Kirda (Eds.). ACM, 2785–2799. doi:10.1145/3576915.3623157 [33] Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, et al. 2022. Synchromesh: Reliable Code Generation from Pre-trained Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https://openreview.net/forum?id=KmtVD97J43e [34] Postman, Inc. 2024. 2024 State of the API Report. Postman, Inc. Retrieved 2025-05-17 from https://www.postman.com/state-of-api/2024/ [35] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, et al. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Realworld APIs. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=dHng2O0Jjr [36] Irum Rauf, Elena Troubitsyna, and Ivan Porres. 2019. A systematic mapping study of API usability evaluation methods. Comput. Sci. Rev. 33 (2019), 49–68. doi:10.1016/J.COSREV.2019.05.001 [37] Martin P. Robillard. 2009. What Makes APIs Hard to Learn? Answers from Developers. IEEE Softw. 26, 6 (2009), 27–34. doi:10.1109/MS.2009.193 [38] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, et al. 2023. Code Llama: Open Foundation Models for Code. CoRR abs/2308.12950 (2023). doi:10.48550/ARXIV.2308.12950 arXiv: 2308.12950. [39] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, et al. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html [40] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 9895–9901. doi:10.18653/v1/2021.emnlp-main.779 [41] Raphael Shu, Elman Mansimov, Tamer Alkhouli, Nikolaos Pappas, Salvatore Romeo, et al. 2022. Dialog2API: Task-Oriented Dialogue with API Description and Example Programs. CoRR abs/2212.09946 (2022). doi:10.48550/ARXIV.2212.09946 arXiv: 2212.09946. [42] SmartBear Software 2020. The State of API 2020 Report. SmartBear Software. Retrieved 2024-07-29 from https://smartbear.com/resources/ebooks/thestate-of-api-2020-report/ [43] Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, et al. 2023. RestGPT: Connecting Large Language Models with Real-World Applications via RESTful APIs. CoRR abs/2306.06624 (2023). doi:10.48550/ARXIV.2306.06624 arXiv: 2306.06624. [44] Statista, Inc. 2024. Most used programming languages among developers worldwide as of 2024. Statista, Inc. Retrieved 2024-09-08 from https: //www.statista.com/statistics/793628/worldwide-developer-survey-most-used-languages/ [45] Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, et al. 2017. Building Natural Language Interfaces to Web APIs. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, November 06 - 10, 2017, Ee-Peng Lim, Marianne Winslett, Mark Sanderson, Ada Wai-Chee Fu, Jimeng Sun, J. Shane Culpepper, Eric Lo, Joyce C. Ho, Debora Donato, Rakesh Agrawal, Yu Zheng, Carlos Castillo, Aixin Sun, Vincent S. Tseng, and Chenliang Li (Eds.). ACM, 177–186. doi:10.1145/3132847.3133009 [46] Yu Su, Ahmed Hassan Awadallah, Miaosen Wang, and Ryen W. White. 2018. Natural Language Interfaces with Fine-Grained User Interaction: A Case Study on Web APIs. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 855–864. Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

doi:10.1145/3209978.3210013 [47] Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, and others Desmarais. 2025. Bugs in large language models generated code: an empirical study. Empir. Softw. Eng. 30, 3 (2025), 65. doi:10.1007/S10664-025-10614-4 [48] Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. 2025. SynCode: LLM Generation with Grammar Augmentation. Trans. Mach. Learn. Res. 2025 (2025). https://openreview.net/forum?id=HiUZtgAPoH [49] Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui, Junnan Li, et al. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 1069–1088. doi:10.18653/V1/2023.EMNLP-MAIN.68 [50] Zhuoer Wang, Leonardo F. R. Ribeiro, Alexandros Papangelis, Rohan Mukherjee, Tzu-Yen Wang, et al. 2024. FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, 6179–6191. https://aclanthology.org/2024.findings-emnlp.359 [51] Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023, Satish Chandra, Kelly Blincoe, and Paolo Tonella (Eds.). ACM, 172–184. doi:10.1145/3611643.3616271 [52] Anna-Katharina Wickert, Lars Baumgärtner, Florian Breitfelder, and Mira Mezini. 2021. Python Crypto Misuses in the Wild. In ESEM ’21: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Bari, Italy, October 11-15, 2021, Filippo Lanubile, Marcos Kalinowski, and Maria Teresa Baldassarre (Eds.). ACM, 31:1–31:6. doi:10.1145/3475716.3484195 [53] Brandon T. Willard and Rémi Louf. 2023. Efficient Guided Generation for Large Language Models. CoRR abs/2307.09702 (2023). doi:10.48550/ arXiv.2307.09702 arXiv: 2307.09702. [54] Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, et al. 2025. Survey on Evaluation of LLM-based Agents. CoRR abs/2503.16416 (2025). doi:10.48550/ARXIV.2503.16416 arXiv: 2503.16416. [55] Daoguang Zan, Bei Chen, Yongshun Gong, Junzhi Cao, Fengji Zhang, et al. 2025. Private-library-oriented code generation with large language models. Knowl. Based Syst. 326 (2025), 113934. doi:10.1016/J.KNOSYS.2025.113934 [56] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: RepositoryLevel Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 2471–2484. doi:10.18653/V1/2023.EMNLP-MAIN.151 [57] Kexun Zhang, Hongqiao Chen, Lei Li, and William Yang Wang. 2023. Don’t Fine-Tune, Decode: Syntax Error-Free Tool Use via Constrained Decoding. CoRR abs/2310.07075 (2023). doi:10.48550/ARXIV.2310.07075 arXiv: 2310.07075. [58] Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig. 2023. DocPrompting: Generating Code by Retrieving the Docs. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/ forum?id=ZTCxT2t2Ru [59] Terry Yue Zhuo, Xiaoning Du, Zhenchang Xing, Jiamou Sun, Haowei Quan, et al. 2023. Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names? CoRR abs/2309.07804 (2023). doi:10.48550/ARXIV.2309.07804 arXiv: 2309.07804.

Manuscript submitted to ACM

28 A

Maninger et al. Data Availability

WAPIIBench is available on GitHub15 . In addition, we provide all model-generated codes and evaluation results in our artifact16 . B

Constrained Decoding Details

We implemented our constrained decoding engine as an extension of the Hugging Face Transformers library (cf. Appendix C) by providing a LogitsProcessor subclass that manages the whole constrained decoding process. The implementation is model-agnostic and compatible with all encoder-decoder and decoder-only architectures available on Hugging Face. Our constraints ensure for each argument that its name is defined in the specification for the location it appears in, and that the associated expression is either a valid variable name or a value that matches the specified schema and data type. Complex data structures such as arrays and objects are constrained deeply, i.e., constraints are constructed for their members as well, recursively. Moreover, we ensure that an argument is not passed more than once in the same location and that required arguments are indeed passed. While the general call syntax is predetermined in favor of the most common way to use Axios (even though alternative syntaxes exist), whitespace is handled flexibly by our constraints to permit different code formattings. When using greedy decoding, where always the most likely next token is generated, we optimize the search for allowed next tokens by checking candidate tokens in order of their probability and stop once the first matching token is found. Further performance optimizations are out of scope for this work, as related work has already proposed effective measures to improve it (cf. Section 2.4), which could be incorporated into our implementation. Our approach requires the user to select the API he or she wants to use and provide its specification. For an automated API selection, more research on processing the task description and classifying the implied API would be necessary. The evaluation in this work was limited to one API at a time. Enforcing constraints for multiple different APIs at once requires only implementing OR-semantics in addition to the current AND-semantics for generation rules. The generation rules we implemented are activated when axios. is generated and deactivated when the end of a statement (e.g., by ); or ).then() is generated. This makes it feasible to generate whole programs while only constraining Axios invocations. However, both the start and the stop conditions are heuristics and may fail, e.g., if a matching string is found in a comment or nested expression. Improving the detection of start and stop conditions is only possible to a certain extent due to the fundamental limitations of regular expressions (e.g., it is impossible to decide if parentheses are balanced). When generating whole programs, we also cannot constrain the data types and values of variables that are declared before and used within the Axios invocation, as it cannot be foreseen which variable will be used for what (and if it is relevant to the API invocation in the first place). More research on this problem is needed for constrained decoding approaches in general. Our constraints are designed to match a code snippet if it contains an Axios invocation that respects JavaScript syntax and API usage constraints from an OpenAPI specification. However, due to some simplifications we made to keep the complexity of the generated regexes in bounds, the opposite is not true, i.e., there are syntactically invalid Axios invocations that are matched by our constraints. One example are single, double, and backtick quotation marks, for which we do not always ensure that the opening and the closing one match. However, we did not encounter any cases where a model made such a mistake. Hence, we did not consider it worthwhile to implement such constraints, 15 https://github.com/stg-tud/WAPIIBench 16 https://doi.org/10.5281/zenodo.13758414

Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

which would add even more complexity to the regexes. Another OpenAPI-related example is the schema property of any parameter, which provides additional constraints beyond its data type, e.g., that a string value should be an email address or that an integer value should be in the range from 0 to 100. Although it would be feasible to respect such schema constraints (at least approximately), we did not implement them so far, since by constraining the data type, we already cover the majority of parameter-value-related mistakes. Finally, OpenAPI specifications sometimes contain usage constraints that are only explained in a parameter’s description property, which contains a free form textual description of an operation. For instance, the maxResults parameter in the Google Calendar API contains a passage “The page size can never be larger than 250 entries”, but this information is not reflected in the parameter’s schema (even though schemas support specifying maximum and minimum values for a parameter). More research is needed to enable extracting such constraints from descriptions and using them for constrained decoding and evaluation purposes. For API parameters that are nested objects or arrays, some simplifications were necessary. Since OpenAPI permits recursive schemas, we prevent an infinite regress while generating constraints from self-recursive schemas by limiting the recursion depth to three. This depth is sufficient for all samples in our dataset but could be adapted if necessary. For arrays, we had to set the maximum number of elements to five, due to an incompatibility with the way we prevent duplicate arguments. This is, however, no fundamental limitation. Again, the chosen maximum is sufficient for our dataset, and we argue that arrays of more than five elements should anyway not be defined inline in most cases. C

Technologies

We implemented WAPIIBench using the following technologies: • OpenAPI • OpenAPI 3 parser • Axios • axios-mock-adapter • Hugging Face Transformers • LangChain • Chroma • regex D

Models

These are the exact names of the models we evaluated: • bigcode/starcoderbase-1b • bigcode/starcoderbase-3b • bigcode/starcoderbase-7b • bigcode/starcoderbase • bigcode/starcoder2-3b • bigcode/starcoder2-7b • bigcode/starcoder2-15b • deepseek-ai/deepseek-coder-1.3b-base • deepseek-ai/deepseek-coder-6.7b-base Manuscript submitted to ACM

Maninger et al. • deepseek-ai/deepseek-coder-33b-base • meta-llama/CodeLlama-7b-hf • meta-llama/CodeLlama-13b-hf • meta-llama/CodeLlama-34b-hf • meta-llama/CodeLlama-70b-hf • meta-llama/Llama-3.1-8B • meta-llama/Llama-3.1-70B • openai/gpt-4o • openai/gpt-4o-mini • Qwen/Qwen2.5-Coder-0.5B • Qwen/Qwen2.5-Coder-1.5B • Qwen/Qwen2.5-Coder-3B • Qwen/Qwen2.5-Coder-7B • Qwen/Qwen2.5-Coder-14B • Qwen/Qwen2.5-Coder-32B • Salesforce/codet5p-6b • Salesforce/codet5p-16b We use only base (i.e., non-instruction-tuned) models. While instruction-tuned models tend to perform better on

coding tasks17 , we considered them to be inappropriate for our setting, which is based on code completion. Performing code completion on instruction-tuned models leads to rather unnatural results, as the models often do not directly return the completion and instead start with some response (“Here is your completed code ...”) followed by a code snipped that may contain (1) only the completion, (2) the starter code followed by the completion, or (3) an arbitrarily modified version of the starter code and corresponding completion. These factors make the model output very hard to parse reliably. Therefore, we decided to focus our evaluation on base models. The following hyperparameters were used: • floating point precision = 16 bit • # beams = 1 • temperature = 0.0 E

Prompts Listing 3 shows the prompt used in WAPIIBench’s evaluation pipeline when generating API invocation code based

on tasks from the synthetic dataset. Listing 4 shows the prompt adapted for the new real-world dataset. F

Extended Results

The full set of metrics calculated by WAPIIBench is explained in Table 4. Tables 5 to 20 provide a comprehensive summary of these metrics for all evaluated models (cf. Appendix D) and both datasets. The best values are in bold. Tables 21 to 23 compare the correctness rates of all models between vanilla generation and the other generation settings (RAG, CD, and RAG + CD). Likewise, Tables 24 to 26 compare the hallucination rates. 17 Cf., e.g., the EvalPlus leaderboard (https://evalplus.github.io/leaderboard.html)

Manuscript submitted to ACM

Mitigating Errors in LLM-Generated Web API Invocations

Listing 3. Prompt for generating API invocations used for the synthetic dataset. Words in curly braces are placeholders. { spec } You are an AI programming assistant that helps users write API requests . You are given a comment that describes what the user wants to achieve and are supposed to implement it using the Axios library in JavaScript . For this , write a single call to Axios ( using the syntax ` axios .< method >( url [, config ]) `) that does exactly what was described in the comment . * Make sure to include all parameters in `config ` that are required to solve the given task , but do not include any unnecessary parameters . * Insert all values directly into the place where they belong rather than using intermediate variables . * If the API requires authentication , prefer OAuth2 over other schemes and use `< token >` as a placeholder for the authorization token . * If a request body requires a media type other than ` text / json `, explicitly set the ` Content - Type ` header to the respective type , and Axios will automatically serialize the request body accordingly . * Always use HTTPS . Your next task is about the { api } API . Complete the following code snippet : ``` javascript // { task } const axios = require (' axios ') ; axios .{ method }( '{ url }',

Listing 4. Prompt for generating API invocations used for the real-world dataset. Words in curly braces are placeholders. { spec } You are an AI programming assistant that helps users write API requests . You are given a comment that describes what the user wants to achieve and are supposed to implement it using the Axios library in JavaScript . For this , write a single call to Axios ( using the syntax ` axios .< method >( url [, config ]) `) that does exactly what was described in the comment . * Make sure to include all parameters in `config ` that are required to solve the given task , but do not include any unnecessary parameters . * If a request body requires a media type other than ` text / json `, explicitly set the ` Content - Type ` header to the respective type , and Axios will automatically serialize the request body accordingly . * Always use HTTPS . Your next task is about the { api } API . Complete the following code snippet : ``` javascript { starter_code }

Note that some metrics can only be calculated either for full completion or for argument completion and are therefore not shown in all tables. Values marked with (t) are ratios relative to the total samples (395 for the synthetic dataset, 28 for the real-world dataset), while values marked with (e) are relative to the subset of executable implementations18 , which varies from experiment to experiment. The complete raw data these results are based on can be found in our artifact (cf. Appendix A). Small discrepancies to previously published results [25] are due to minor prompt adjustments and improvements to the evaluation pipeline.

18 To be exact, the calculation for (t) and (e) values is identical except for a filtering step to remove non-executable samples before calculating the latter.

Manuscript submitted to ACM

Maninger et al. Table 4. Complete evaluation metrics for correctness and specification-compliance of API invocations

Metric

Description

Executable implementations Correct implementations Hallucinated implementations Correct endpoints Hallucinated endpoints Correct URLs Hallucinated URLs Correct methods Hallucinated methods

Generated code is complete and contains no syntax or runtime error Generated executable code matches the ground-truth configuration Generated code contains at least one violation of the API specification Generated URL–method combination matches the ground-truth URL Generated URL–method combination is not defined in the API specification Generated URL matches the ground-truth URL Generated URL is not defined in the API specification Generated HTTP method matches the ground-truth HTTP method Generated HTTP method is not defined for the generated URL in the API specification Generated arguments are correct Generated argument values are correct Expected arguments are not generated Generated arguments are not expected Redundant arguments are generated for an API endpoint Generated arguments are not permitted for the generated API endpoint Probability that the generated arguments are correct Probability that the correct arguments are generated Overlap between generated and correct arguments Probability that an argument value is correct if the argument name is correct

Correct argument names Correct argument values Missing arguments Unexpected arguments Unnecessary arguments Hallucinated arguments Mean argument precision Mean argument recall Mean argument Jaccard index Mean argument value conditional accuracy Total errors Incomplete implementations Runtime errors Timeouts Unsatisfiable constraints

Manuscript submitted to ACM

Any type of error prevented execution Generated code did not contain a complete API invocation Generated code produced an error when trying to execute it Constrained decoding was canceled because finding allowed next tokens took too long Constrained decoding was canceled because no allowed next token existed

(1 6B ) ar Co de r( 1B St ) ar Co de r( 3B St ) ar Co de r( 7B St ) ar Co de r( 15 St .5B ar Co ) de r2 (3 St B) ar Co de r2 (7 St B) ar Co de r2 (1 De 5B ep ) Se ek -C o de De r( ep 1. 3 Se B) ek -C od De er ep (6 Se .7B ek ) -C od Q er w en (3 3B 2. 5 ) -C od Q er w (0 en .5B 2. 5 ) -C od Q e r w (1 en .5B 2. 5 ) -C od Q er w (3 en B 2. 5 ) -C od Q er w (7 en B) 2. 5 -C od Q er w en (1 4B 2. 5 ) -C od Ll er am (3 2B a3 ) .1 (8 B) Ll am a3 .1 (7 0B Co ) de Ll am a( Co 7B de ) Ll am a( Co 13 de B) Ll am a( Co 34 de B) Ll am a( GP 70 B) T4o m in i GP T4o

(6 +

0.96 0.04 0.04 0.11 0.11 0.70 0.72 0.13 0.13 0.66 0.69 0.79 0.82 0.14 0.15 0.29 0.30 0.25 0.26 0.71 0.70 0.28 0.28 0.53 0.53 0.32 0.33 0.29 0.30 0.89 0.89 14 5 7

0.96 0.07 0.07 0.25 0.26 0.46 0.48 0.31 0.32 0.38 0.40 0.79 0.82 0.14 0.15 0.40 0.42 0.32 0.33 0.60 0.58 0.29 0.30 0.56 0.56 0.43 0.44 0.36 0.38 0.77 0.77 14 9 5

de T5 Co

Executable implementations (t) Correct implementations (t) Correct implementations (e) Correct endpoints (t) Correct endpoints (e) Hallucinated endpoints (t) Hallucinated endpoints (e) Correct URLs (t) Correct URLs (e) Hallucinated URLs (t) Hallucinated URLs (e) Correct methods (t) Correct methods (e) Hallucinated methods (t) Hallucinated methods (e) Correct argument names (t) Correct argument names (e) Correct argument values (t) Correct argument values (e) Missing arguments (t) Missing arguments (e) Unexpected arguments (t) Unexpected arguments (e) Mean argument precision (t) Mean argument precision (e) Mean argument recall (t) Mean argument recall (e) Mean arg. Jaccard index (t) Mean arg. Jaccard index (e) Mean arg. val. cond. acc. (t) Mean arg. val. cond. acc. (e) Total errors Incomplete implementations Runtime errors

0.96 0.03 0.03 0.21 0.22 0.30 0.31 0.28 0.29 0.24 0.25 0.61 0.63 0.16 0.17 0.26 0.27 0.23 0.24 0.74 0.73 0.31 0.33 0.42 0.42 0.26 0.27 0.23 0.24 0.88 0.88 16 15 1

0.96 0.05 0.05 0.33 0.35 0.33 0.35 0.38 0.40 0.25 0.26 0.77 0.80 0.15 0.15 0.35 0.37 0.31 0.33 0.65 0.63 0.30 0.31 0.53 0.53 0.36 0.37 0.31 0.32 0.87 0.87 15 15 0

0.98 0.12 0.12 0.42 0.42 0.36 0.37 0.47 0.47 0.29 0.30 0.81 0.83 0.15 0.16 0.49 0.50 0.43 0.44 0.51 0.50 0.29 0.30 0.61 0.61 0.49 0.50 0.42 0.42 0.87 0.87 6 6 0

0.99 0.13 0.14 0.49 0.49 0.23 0.23 0.54 0.54 0.18 0.18 0.88 0.89 0.08 0.08 0.55 0.56 0.46 0.47 0.45 0.44 0.27 0.27 0.66 0.66 0.55 0.55 0.48 0.49 0.83 0.83 3 3 0

0.97 0.13 0.14 0.37 0.38 0.28 0.28 0.42 0.43 0.21 0.21 0.78 0.80 0.12 0.12 0.49 0.51 0.43 0.44 0.51 0.49 0.32 0.33 0.58 0.58 0.49 0.50 0.41 0.42 0.84 0.84 11 9 2

0.99 0.10 0.10 0.42 0.43 0.33 0.34 0.46 0.47 0.29 0.30 0.91 0.92 0.06 0.06 0.44 0.44 0.39 0.40 0.56 0.56 0.31 0.31 0.54 0.54 0.43 0.44 0.36 0.37 0.89 0.89 4 4 0

0.98 0.25 0.25 0.56 0.57 0.26 0.26 0.59 0.61 0.21 0.21 0.86 0.87 0.10 0.10 0.59 0.60 0.55 0.56 0.41 0.40 0.25 0.25 0.68 0.68 0.59 0.60 0.52 0.53 0.93 0.93 7 4 2

0.97 0.07 0.08 0.29 0.30 0.38 0.39 0.34 0.35 0.33 0.34 0.83 0.85 0.09 0.10 0.43 0.44 0.36 0.37 0.57 0.56 0.33 0.34 0.53 0.53 0.44 0.45 0.37 0.38 0.83 0.83 10 7 3

0.64 0.14 0.23 0.33 0.52 0.15 0.24 0.36 0.56 0.13 0.20 0.60 0.94 0.03 0.04 0.33 0.53 0.31 0.49 0.67 0.47 0.18 0.26 0.65 0.65 0.34 0.54 0.31 0.48 0.92 0.92 144 4 0

0.97 0.17 0.18 0.60 0.61 0.20 0.21 0.65 0.67 0.14 0.14 0.86 0.89 0.09 0.09 0.58 0.60 0.54 0.55 0.42 0.40 0.24 0.25 0.64 0.64 0.56 0.57 0.47 0.49 0.91 0.91 11 6 0

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 395 5 82

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 395 0 0

0.43 0.17 0.39 0.27 0.64 0.10 0.24 0.29 0.67 0.07 0.17 0.37 0.86 0.05 0.12 0.29 0.74 0.26 0.69 0.71 0.26 0.07 0.17 0.81 0.81 0.33 0.76 0.30 0.71 0.94 0.94 226 0 0

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 395 0 0

0.99 0.03 0.03 0.40 0.40 0.38 0.39 0.46 0.47 0.28 0.29 0.75 0.75 0.19 0.19 0.38 0.38 0.27 0.27 0.62 0.62 0.19 0.19 0.71 0.71 0.40 0.40 0.36 0.36 0.68 0.68 2 2 0

0.98 0.18 0.18 0.50 0.51 0.30 0.31 0.55 0.56 0.23 0.24 0.86 0.88 0.11 0.11 0.62 0.63 0.49 0.51 0.38 0.37 0.23 0.24 0.72 0.72 0.63 0.65 0.57 0.58 0.78 0.78 9 3 6

0.96 0.17 0.17 0.51 0.53 0.27 0.28 0.56 0.59 0.22 0.22 0.84 0.87 0.10 0.10 0.62 0.65 0.48 0.50 0.38 0.35 0.23 0.24 0.73 0.73 0.64 0.67 0.57 0.60 0.77 0.77 15 2 13

1.00 0.30 0.30 0.59 0.59 0.27 0.27 0.68 0.68 0.17 0.17 0.87 0.87 0.11 0.11 0.70 0.70 0.65 0.65 0.30 0.30 0.20 0.20 0.75 0.75 0.70 0.70 0.63 0.63 0.93 0.93 0 0 0

1.00 0.39 0.39 0.49 0.49 0.32 0.32 0.56 0.56 0.24 0.24 0.86 0.87 0.12 0.12 0.77 0.77 0.73 0.74 0.23 0.23 0.15 0.15 0.84 0.84 0.79 0.79 0.74 0.74 0.96 0.96 1 1 0

1.00 0.58 0.58 0.73 0.73 0.15 0.15 0.79 0.79 0.09 0.09 0.91 0.91 0.08 0.08 0.86 0.86 0.83 0.83 0.14 0.14 0.11 0.11 0.88 0.88 0.87 0.87 0.82 0.82 0.97 0.97 0 0 0

Mitigating Errors in LLM-Generated Web API Invocations

Table 5. Complete evaluation results for full completion with vanilla generation on the synthetic dataset

Manuscript submitted to ACM

eT 5+ (6 B) Co de T5 + (1 6B St ) ar Co de r( 1 B) St ar Co de r( 3B St ) ar Co de r( 7B St ) ar Co de r( 15 St .5B ar Co ) de r2 (3 St B ar ) Co de r2 ( 7B St ar ) Co de r2 (1 De 5B ep ) Se ek -C od De er ep (1 Se .3B ek ) -C od De er ep (6 Se .7B ek ) -C od Q w er en (3 2.5 3B -C ) od Q er w en (0 .5B 2.5 ) -C od Q er w en (1 .5B 2.5 ) -C od Q er w en (3 B) 2.5 -C od Q er w en (7 B) 2.5 -C od Q er w en (1 4B 2.5 ) -C od Ll er am (3 a3 2B .1 ) (8 B) Ll am a3 .1 (7 0B Co ) de Ll am a( Co 7B de ) Ll am a( Co 13 de B) Ll am a( Co 34 de B) Ll am a( 70 GP B) T4o m in i GP T4o

Co d

Manuscript submitted to ACM

Table 6. Complete evaluation results for argument completion with vanilla generation on the synthetic dataset

Executable implementations (t) Correct implementations (t) Correct implementations (e) Hallucinated implementations (t) Hallucinated implementations (e) Correct argument names (t) Correct argument names (e) Correct argument values (t) Correct argument values (e) Missing arguments (t) Missing arguments (e) Unnecessary arguments (t) Unnecessary arguments (e) Hallucinated arguments (t) Hallucinated arguments (e) Mean argument precision (t) Mean argument precision (e) Mean argument recall (t) Mean argument recall (e) Mean arg. Jaccard index (t) Mean arg. Jaccard index (e) Mean arg. val. cond. acc. (t) Mean arg. val. cond. acc. (e) Total errors Incomplete implementations Runtime errors

0.98 0.24 0.24 0.57 0.58 0.44 0.45 0.42 0.42 0.56 0.55 0.02 0.02 0.25 0.25 0.59 0.59 0.48 0.49 0.42 0.43 0.94 0.94 7 2 5

0.98 0.21 0.21 0.54 0.55 0.53 0.54 0.48 0.49 0.47 0.46 0.03 0.03 0.23 0.24 0.64 0.64 0.55 0.56 0.48 0.49 0.90 0.90 7 3 4

0.97 0.22 0.23 0.48 0.50 0.46 0.48 0.43 0.45 0.54 0.52 0.04 0.04 0.24 0.24 0.61 0.61 0.49 0.51 0.45 0.46 0.94 0.94 12 11 1

0.97 0.23 0.23 0.49 0.50 0.51 0.53 0.48 0.49 0.49 0.47 0.04 0.04 0.23 0.23 0.64 0.64 0.55 0.56 0.49 0.50 0.93 0.93 10 10 0

0.98 0.29 0.30 0.35 0.36 0.67 0.69 0.60 0.61 0.33 0.31 0.07 0.07 0.16 0.16 0.73 0.73 0.68 0.69 0.61 0.62 0.89 0.89 6 6 0

0.99 0.28 0.29 0.28 0.29 0.73 0.74 0.62 0.63 0.27 0.26 0.06 0.06 0.13 0.13 0.79 0.79 0.74 0.74 0.67 0.68 0.85 0.85 5 5 0

0.99 0.22 0.22 0.45 0.46 0.62 0.63 0.55 0.56 0.38 0.37 0.08 0.08 0.19 0.20 0.65 0.65 0.62 0.62 0.53 0.53 0.88 0.88 3 3 0

0.99 0.19 0.20 0.48 0.48 0.55 0.55 0.50 0.51 0.45 0.45 0.07 0.07 0.20 0.21 0.62 0.62 0.54 0.54 0.46 0.47 0.93 0.93 4 4 0

0.99 0.24 0.24 0.35 0.36 0.64 0.64 0.58 0.59 0.36 0.36 0.06 0.06 0.14 0.14 0.74 0.74 0.64 0.65 0.56 0.57 0.91 0.91 5 2 3

0.98 0.19 0.20 0.50 0.52 0.60 0.62 0.52 0.53 0.40 0.38 0.05 0.05 0.22 0.23 0.66 0.66 0.62 0.64 0.54 0.55 0.86 0.86 9 9 0

0.99 0.29 0.29 0.30 0.30 0.65 0.66 0.59 0.60 0.35 0.34 0.05 0.05 0.14 0.14 0.77 0.77 0.66 0.67 0.60 0.61 0.90 0.90 5 5 0

0.99 0.26 0.26 0.40 0.40 0.70 0.71 0.66 0.66 0.30 0.29 0.06 0.06 0.14 0.14 0.73 0.73 0.68 0.68 0.59 0.59 0.93 0.93 2 2 0

0.90 0.17 0.19 0.54 0.60 0.46 0.52 0.40 0.45 0.54 0.48 0.03 0.03 0.28 0.31 0.57 0.57 0.50 0.55 0.43 0.47 0.87 0.87 39 38 1

0.96 0.24 0.25 0.52 0.54 0.54 0.57 0.51 0.54 0.46 0.43 0.04 0.04 0.26 0.27 0.64 0.64 0.57 0.59 0.49 0.51 0.95 0.95 16 14 2

0.98 0.27 0.27 0.44 0.45 0.55 0.56 0.51 0.52 0.45 0.44 0.03 0.03 0.20 0.21 0.70 0.70 0.59 0.60 0.52 0.54 0.94 0.94 8 8 0

0.99 0.32 0.32 0.41 0.41 0.72 0.72 0.67 0.68 0.28 0.28 0.05 0.05 0.17 0.17 0.74 0.74 0.72 0.73 0.63 0.63 0.95 0.95 2 2 0

0.98 0.47 0.47 0.26 0.27 0.80 0.82 0.74 0.76 0.20 0.18 0.04 0.04 0.11 0.12 0.84 0.84 0.82 0.83 0.76 0.77 0.93 0.93 7 7 0

0.99 0.43 0.43 0.28 0.28 0.82 0.82 0.75 0.76 0.18 0.18 0.06 0.06 0.11 0.11 0.81 0.81 0.82 0.82 0.75 0.75 0.92 0.92 2 2 0

0.98 0.10 0.10 0.34 0.35 0.46 0.47 0.41 0.42 0.54 0.53 0.03 0.03 0.17 0.18 0.73 0.73 0.47 0.48 0.42 0.43 0.90 0.90 7 7 0

0.99 0.30 0.31 0.26 0.27 0.66 0.67 0.61 0.61 0.34 0.33 0.05 0.05 0.12 0.12 0.81 0.81 0.69 0.69 0.63 0.64 0.92 0.92 3 3 0

1.00 0.05 0.05 0.19 0.19 0.52 0.52 0.38 0.38 0.48 0.48 0.05 0.05 0.09 0.09 0.82 0.82 0.56 0.57 0.52 0.52 0.73 0.73 1 1 0

0.99 0.26 0.26 0.31 0.31 0.73 0.74 0.60 0.61 0.27 0.26 0.05 0.05 0.13 0.13 0.80 0.80 0.74 0.75 0.68 0.69 0.83 0.83 5 5 0

0.99 0.40 0.40 0.26 0.26 0.80 0.80 0.68 0.69 0.20 0.20 0.07 0.07 0.11 0.11 0.81 0.81 0.81 0.81 0.73 0.74 0.85 0.85 3 0 3

1.00 0.42 0.42 0.24 0.24 0.79 0.80 0.73 0.74 0.21 0.20 0.05 0.05 0.09 0.09 0.83 0.83 0.80 0.80 0.73 0.74 0.93 0.93 1 1 0

1.00 0.63 0.63 0.19 0.19 0.89 0.89 0.84 0.84 0.11 0.11 0.01 0.01 0.08 0.08 0.91 0.91 0.90 0.90 0.86 0.86 0.96 0.96 0 0 0

1.00 0.59 0.59 0.31 0.31 0.77 0.77 0.74 0.74 0.23 0.23 0.01 0.01 0.15 0.15 0.78 0.78 0.79 0.79 0.75 0.75 0.96 0.96 0 0 0

Maninger et al.

+ (6 B) Co de T5 + (1 6B St ) ar Co de r( 1 B) St ar Co de r( 3B St ) ar Co de r( 7 B) St ar Co de r( 15 St . 5B ar Co ) de r2 ( 3B St ar ) Co de r2 (7 St B) ar Co de r2 (1 De 5B ep ) Se ek -C od De er ep (1 Se . 3B ek ) -C od De er ep (6 Se . 7B ek ) -C od Q er w en (3 3B 2. 5 ) -C od Q er w (0 en . 5B 2.5 ) -C od Q er w (1 en . 5B 2. 5 ) -C od Q er w (3 en B) 2. 5 -C od Q e r( w 7B en 2.5 ) -C o de Q w r( en 14 2. 5 B) -C od Ll er am (3 2B a3 ) .1 (8 B ) Ll am a3 .1 (7 0B Co ) de Ll am a( Co 7B de ) Ll am a( Co 13 de B) Ll am a( Co 34 de B) Ll am a( GP 70 B) T4o m in i GP T4o

Co de T5

0.18 0.00 0.00 0.02 0.08 0.17 0.92 0.02 0.10 0.16 0.90 0.17 0.94 0.01 0.06 0.04 0.30 0.03 0.21 0.96 0.70 0.01 0.09 0.88 0.88 0.05 0.28 0.05 0.27 0.73 0.73 323 89 233

0.43 0.08 0.19 0.29 0.69 0.05 0.11 0.32 0.76 0.02 0.05 0.38 0.90 0.03 0.07 0.21 0.51 0.17 0.40 0.79 0.49 0.08 0.17 0.74 0.74 0.24 0.55 0.22 0.52 0.79 0.79 227 34 192

0.95 0.22 0.23 0.74 0.78 0.10 0.10 0.81 0.85 0.04 0.05 0.85 0.90 0.06 0.06 0.55 0.58 0.47 0.49 0.45 0.42 0.24 0.25 0.68 0.68 0.58 0.61 0.51 0.54 0.86 0.86 19 17 1

0.97 0.28 0.28 0.80 0.82 0.07 0.07 0.83 0.86 0.03 0.03 0.90 0.93 0.04 0.04 0.60 0.62 0.54 0.56 0.40 0.38 0.22 0.23 0.71 0.71 0.63 0.65 0.57 0.59 0.89 0.89 12 10 2

0.99 0.34 0.34 0.81 0.82 0.08 0.08 0.87 0.88 0.02 0.02 0.90 0.91 0.06 0.06 0.80 0.80 0.68 0.68 0.20 0.20 0.21 0.21 0.79 0.79 0.81 0.82 0.73 0.73 0.85 0.85 2 2 0

1.00 0.36 0.36 0.87 0.87 0.05 0.05 0.90 0.90 0.02 0.02 0.96 0.96 0.03 0.03 0.79 0.80 0.69 0.70 0.21 0.20 0.21 0.21 0.79 0.79 0.80 0.80 0.73 0.73 0.85 0.85 1 1 0

0.99 0.33 0.33 0.81 0.82 0.06 0.06 0.86 0.87 0.01 0.01 0.90 0.91 0.05 0.05 0.74 0.75 0.68 0.69 0.26 0.25 0.23 0.23 0.76 0.76 0.76 0.77 0.65 0.66 0.92 0.92 5 3 2

1.00 0.14 0.14 0.86 0.87 0.06 0.06 0.88 0.88 0.05 0.05 0.97 0.97 0.01 0.01 0.76 0.77 0.59 0.59 0.24 0.23 0.24 0.24 0.78 0.78 0.77 0.77 0.68 0.68 0.75 0.75 1 1 0

0.98 0.33 0.34 0.87 0.88 0.05 0.05 0.90 0.92 0.02 0.02 0.93 0.95 0.04 0.04 0.81 0.82 0.75 0.76 0.19 0.18 0.26 0.26 0.74 0.74 0.79 0.81 0.66 0.67 0.93 0.93 7 7 0

0.95 0.06 0.06 0.79 0.83 0.04 0.05 0.82 0.86 0.03 0.03 0.91 0.96 0.02 0.02 0.71 0.75 0.44 0.47 0.29 0.25 0.21 0.22 0.74 0.74 0.74 0.78 0.67 0.70 0.59 0.59 19 12 7

0.90 0.20 0.22 0.83 0.92 0.01 0.01 0.83 0.92 0.01 0.01 0.88 0.98 0.01 0.01 0.72 0.81 0.59 0.66 0.28 0.19 0.16 0.18 0.83 0.83 0.73 0.81 0.66 0.74 0.80 0.80 39 5 0

0.99 0.24 0.24 0.90 0.91 0.03 0.03 0.92 0.93 0.01 0.01 0.96 0.97 0.02 0.02 0.57 0.57 0.54 0.54 0.43 0.43 0.11 0.11 0.85 0.85 0.54 0.55 0.50 0.50 0.96 0.96 3 1 0

0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00 0.00 1.00 0.00 0.00 0.00 0.67 0.00 0.33 1.00 0.33 0.01 0.73 0.20 0.20 0.00 0.67 0.00 0.18 0.50 0.50 394 1 21

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 395 0 0

0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00 0.00 1.00 1.00 0.00 0.00 0.25 0.75 0.75 0.00 1.00 0.00 0.75 1.00 1.00 394 0 0

0.28 0.13 0.48 0.26 0.92 0.01 0.04 0.26 0.94 0.01 0.02 0.27 0.98 0.01 0.02 0.26 0.94 0.25 0.91 0.74 0.06 0.07 0.22 0.82 0.82 0.26 0.94 0.22 0.79 0.98 0.98 285 2 0

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 395 0 0

1.00 0.07 0.07 0.86 0.86 0.04 0.04 0.89 0.89 0.02 0.02 0.93 0.93 0.03 0.03 0.71 0.71 0.51 0.51 0.29 0.29 0.21 0.22 0.81 0.81 0.73 0.73 0.66 0.66 0.70 0.70 1 1 0

0.98 0.04 0.04 0.86 0.87 0.05 0.05 0.89 0.90 0.02 0.02 0.93 0.95 0.03 0.03 0.79 0.80 0.51 0.51 0.21 0.20 0.16 0.16 0.86 0.86 0.81 0.82 0.74 0.75 0.60 0.60 6 4 0

0.97 0.23 0.24 0.83 0.85 0.08 0.08 0.85 0.88 0.06 0.06 0.94 0.96 0.03 0.03 0.62 0.64 0.51 0.53 0.38 0.36 0.30 0.31 0.63 0.63 0.64 0.66 0.54 0.56 0.84 0.84 11 2 7

1.00 0.42 0.42 0.88 0.89 0.04 0.04 0.92 0.93 0.01 0.01 0.95 0.95 0.03 0.03 0.78 0.79 0.74 0.74 0.22 0.21 0.15 0.15 0.82 0.82 0.77 0.78 0.72 0.72 0.95 0.95 1 1 0

0.99 0.37 0.38 0.91 0.92 0.02 0.02 0.92 0.93 0.01 0.01 0.96 0.98 0.01 0.01 0.93 0.95 0.77 0.78 0.07 0.05 0.17 0.17 0.89 0.89 0.94 0.96 0.86 0.87 0.81 0.81 5 0 5

0.99 0.66 0.66 0.94 0.94 0.01 0.01 0.94 0.95 0.00 0.00 0.98 0.99 0.00 0.00 0.95 0.96 0.91 0.92 0.05 0.04 0.12 0.12 0.91 0.91 0.96 0.96 0.90 0.90 0.96 0.96 2 0 1

Mitigating Errors in LLM-Generated Web API Invocations

Table 7. Complete evaluation results for full completion with retrieval-augmented generation on the synthetic dataset

Manuscript submitted to ACM

)

0.99 0.08 0.08 0.10 0.10 0.76 0.76 0.55 0.55 0.24 0.24 0.14 0.14 0.04 0.04 0.86 0.86 0.77 0.78 0.70 0.71 0.72 0.72 2 2 0

0.99 0.05 0.05 0.07 0.07 0.84 0.85 0.55 0.56 0.16 0.15 0.11 0.11 0.03 0.03 0.89 0.89 0.85 0.85 0.78 0.79 0.63 0.63 3 2 1

0.99 0.29 0.29 0.21 0.21 0.70 0.71 0.59 0.60 0.30 0.29 0.16 0.17 0.11 0.11 0.67 0.67 0.71 0.72 0.61 0.62 0.85 0.85 5 1 4

Se e

od rC

B) 70

B) 34

in i o

1.00 0.29 0.29 0.11 0.11 0.61 0.61 0.53 0.53 0.39 0.39 0.07 0.07 0.06 0.06 0.85 0.85 0.64 0.64 0.58 0.58 0.88 0.88 0 0 0

GP T4

0.99 0.09 0.09 0.06 0.06 0.41 0.41 0.38 0.38 0.59 0.59 0.06 0.06 0.05 0.05 0.90 0.90 0.41 0.41 0.37 0.38 0.92 0.92 2 1 1

0.97 0.45 0.47 0.13 0.14 0.88 0.90 0.84 0.86 0.12 0.10 0.17 0.17 0.07 0.07 0.79 0.79 0.87 0.90 0.73 0.76 0.96 0.96 13 12 1

GP T4o

1.00 0.46 0.46 0.04 0.04 0.70 0.70 0.66 0.66 0.30 0.30 0.09 0.09 0.03 0.03 0.89 0.89 0.73 0.73 0.66 0.66 0.95 0.95 1 1 0

1.00 0.46 0.46 0.07 0.07 0.82 0.82 0.77 0.77 0.18 0.18 0.09 0.09 0.03 0.03 0.86 0.86 0.81 0.81 0.76 0.76 0.95 0.95 0 0 0

0.99 0.33 0.34 0.15 0.15 0.83 0.84 0.67 0.68 0.17 0.16 0.09 0.09 0.09 0.09 0.82 0.82 0.86 0.88 0.79 0.80 0.81 0.81 5 5 0

1.00 0.56 0.56 0.28 0.28 0.75 0.75 0.69 0.69 0.25 0.25 0.06 0.06 0.14 0.14 0.75 0.75 0.78 0.78 0.73 0.73 0.90 0.90 0 0 0

de L

1.00 0.44 0.44 0.12 0.12 0.86 0.86 0.81 0.81 0.14 0.14 0.16 0.16 0.04 0.04 0.81 0.81 0.85 0.86 0.75 0.75 0.94 0.94 1 1 0

la m

13 Co de L

la m

0.99 0.37 0.37 0.27 0.27 0.75 0.76 0.72 0.72 0.25 0.24 0.13 0.13 0.12 0.12 0.75 0.75 0.77 0.78 0.66 0.67 0.95 0.95 2 2 0

) 7B a( eL Co d

Co d

0.99 0.07 0.07 0.19 0.20 0.66 0.67 0.53 0.53 0.34 0.33 0.10 0.10 0.10 0.11 0.76 0.76 0.65 0.66 0.60 0.61 0.80 0.80 4 4 0

a3 Ll am

la m

(8 .1

Ll am

0.94 0.19 0.20 0.38 0.40 0.58 0.63 0.49 0.53 0.42 0.37 0.08 0.09 0.23 0.25 0.65 0.65 0.61 0.65 0.51 0.54 0.85 0.85 25 24 1

w en Q

eL la m

)

de r -C o

w en Q

2.5

2B (3

de -C o 2.5

w en Q

0.99 0.28 0.28 0.07 0.07 0.59 0.59 0.56 0.56 0.41 0.41 0.07 0.07 0.03 0.03 0.87 0.87 0.57 0.57 0.53 0.53 0.96 0.96 2 2 0

0.98 0.23 0.23 0.16 0.16 0.83 0.84 0.67 0.69 0.17 0.16 0.08 0.08 0.06 0.06 0.86 0.86 0.83 0.84 0.76 0.77 0.81 0.81 6 5 1

2.5

-C od

(3 er

) B)

.5B

-C o

w en 2.5 Q

2.5

w en Q

-C od

de r

de -C o

0.5

(3 w en Q

0.98 0.09 0.09 0.19 0.20 0.78 0.80 0.51 0.53 0.22 0.20 0.11 0.11 0.09 0.09 0.78 0.78 0.81 0.83 0.73 0.75 0.61 0.61 8 8 0

Se e

ep De

kCo d

(6 er

)

.7B

.3B (1 er ep De

0.99 0.29 0.29 0.11 0.11 0.80 0.80 0.74 0.74 0.20 0.20 0.19 0.19 0.04 0.04 0.77 0.77 0.78 0.78 0.65 0.65 0.92 0.92 4 4 0

Se e ep

kCo d

Co d

5B (1

1.00 0.15 0.15 0.14 0.14 0.82 0.82 0.64 0.64 0.18 0.18 0.16 0.16 0.07 0.07 0.82 0.82 0.82 0.82 0.73 0.73 0.76 0.76 0 0 0

0.99 0.35 0.35 0.20 0.20 0.78 0.79 0.72 0.73 0.22 0.21 0.13 0.13 0.08 0.08 0.79 0.79 0.80 0.80 0.68 0.69 0.92 0.92 3 2 1

er 2

St a

1.00 0.39 0.39 0.21 0.21 0.83 0.84 0.73 0.73 0.17 0.16 0.10 0.10 0.09 0.09 0.82 0.82 0.84 0.84 0.77 0.77 0.85 0.85 1 1 0

0.99 0.38 0.38 0.16 0.16 0.85 0.85 0.73 0.73 0.15 0.15 0.11 0.11 0.07 0.07 0.83 0.83 0.86 0.86 0.78 0.78 0.86 0.86 2 2 0

St a

0.97 0.30 0.31 0.21 0.22 0.66 0.68 0.59 0.61 0.34 0.32 0.09 0.09 0.11 0.11 0.76 0.76 0.69 0.71 0.63 0.64 0.89 0.89 10 9 1

)

B) 5.5

0.95 0.26 0.27 0.25 0.27 0.59 0.63 0.51 0.54 0.41 0.37 0.10 0.10 0.12 0.13 0.71 0.71 0.62 0.65 0.56 0.58 0.87 0.87 19 18 1

0.40 0.09 0.21 0.06 0.14 0.22 0.56 0.18 0.47 0.78 0.44 0.01 0.01 0.03 0.08 0.87 0.87 0.24 0.60 0.24 0.59 0.85 0.85 236 27 209

St a

0.20 0.05 0.26 0.03 0.15 0.10 0.59 0.08 0.49 0.90 0.41 0.00 0.00 0.03 0.14 0.85 0.85 0.11 0.58 0.11 0.57 0.82 0.82 317 84 233

(7 od St a

ar St

Co d

er St a

rC od

B) (3

) er

(1 St a

rC od

B) (6 +

eT 5 Co d

de T5

Manuscript submitted to ACM

Table 8. Complete evaluation results for argument completion with retrieval-augmented generation on the synthetic dataset

Maninger et al.

. 5B ar ) Co de r2 (3 St B) ar Co de r2 (7 St B) ar Co de r2 (1 De 5B ) ep Se ek -C od De er ep (1 Se . 3B ek ) -C o de De r( ep 6.7 Se B) ek -C od Q er w en (3 3B 2.5 ) -C od Q e r( w 0.5 en 2.5 B) -C od Q er w (1 en .5B 2.5 ) -C od Q er w (3 en B) 2.5 -C od Q er w (7 en B) 2.5 -C od Q e r( w 14 en B) 2. 5 -C od Ll er am (3 2B a3 ) .1 (8 B ) Ll am a3 .1 (7 0B Co ) de Ll am a( 7B Co ) de Ll am a( 13 Co B) de Ll am a( 70 B)

)

0.87 0.37 0.43 0.50 0.57 0.00 0.00 0.50 0.57 0.00 0.00 0.79 0.91 0.00 0.00 0.68 0.80 0.64 0.76 0.32 0.20 0.23 0.26 0.77 0.77 0.72 0.83 0.63 0.72 0.95 0.95 51 17 33 1 0

0.86 0.30 0.34 0.50 0.58 0.00 0.00 0.50 0.58 0.00 0.00 0.81 0.94 0.00 0.00 0.68 0.80 0.55 0.66 0.32 0.20 0.19 0.21 0.81 0.81 0.71 0.82 0.64 0.74 0.80 0.80 54 21 25 8 0

0.81 0.35 0.43 0.43 0.53 0.00 0.00 0.43 0.53 0.00 0.00 0.72 0.89 0.00 0.00 0.63 0.80 0.60 0.76 0.37 0.20 0.20 0.25 0.79 0.79 0.67 0.83 0.59 0.73 0.96 0.96 75 29 25 21 0

Co St ar

de r

de r( 15

(7 B

(3 B)

0.85 0.32 0.37 0.42 0.49 0.00 0.00 0.42 0.50 0.00 0.00 0.76 0.89 0.00 0.00 0.61 0.74 0.56 0.69 0.39 0.26 0.19 0.23 0.79 0.79 0.66 0.77 0.59 0.69 0.94 0.94 58 13 39 6 0

Co St ar

0.93 0.19 0.20 0.27 0.29 0.00 0.00 0.30 0.32 0.00 0.00 0.59 0.64 0.00 0.00 0.56 0.60 0.51 0.56 0.44 0.40 0.16 0.17 0.82 0.82 0.60 0.65 0.55 0.59 0.93 0.93 28 9 17 2 0

Co St ar

0.75 0.18 0.23 0.26 0.34 0.00 0.00 0.26 0.34 0.00 0.00 0.63 0.83 0.00 0.00 0.49 0.71 0.43 0.62 0.51 0.29 0.21 0.27 0.73 0.73 0.55 0.73 0.47 0.62 0.90 0.90 97 67 26 4 0

Co St ar

de r

(1 6

1B )

) (6 B +

Co de T5 +

0.58 0.17 0.29 0.22 0.37 0.00 0.00 0.22 0.38 0.00 0.00 0.51 0.87 0.00 0.00 0.35 0.68 0.33 0.63 0.65 0.32 0.16 0.27 0.73 0.73 0.42 0.73 0.36 0.63 0.94 0.94 166 65 68 2 31

de T5 Co

0.81 0.36 0.44 0.49 0.60 0.00 0.00 0.49 0.60 0.00 0.00 0.77 0.95 0.00 0.00 0.65 0.83 0.59 0.76 0.35 0.17 0.15 0.19 0.84 0.84 0.69 0.85 0.63 0.77 0.93 0.93 74 17 44 13 0

0.85 0.46 0.54 0.58 0.69 0.00 0.00 0.58 0.69 0.00 0.00 0.79 0.92 0.00 0.00 0.70 0.84 0.67 0.81 0.30 0.16 0.17 0.20 0.82 0.82 0.74 0.86 0.65 0.77 0.97 0.97 58 25 26 7 0

0.84 0.21 0.25 0.39 0.47 0.00 0.00 0.39 0.47 0.00 0.00 0.75 0.90 0.00 0.00 0.61 0.75 0.47 0.58 0.39 0.25 0.23 0.27 0.73 0.73 0.65 0.78 0.56 0.66 0.71 0.71 64 46 15 3 0

0.79 0.42 0.53 0.52 0.66 0.00 0.00 0.52 0.66 0.00 0.00 0.74 0.94 0.00 0.00 0.64 0.84 0.62 0.80 0.36 0.16 0.17 0.21 0.83 0.83 0.68 0.86 0.61 0.77 0.97 0.97 84 49 32 3 0

0.79 0.46 0.58 0.54 0.68 0.00 0.00 0.55 0.70 0.00 0.00 0.73 0.92 0.00 0.00 0.66 0.84 0.64 0.82 0.34 0.16 0.16 0.19 0.84 0.84 0.69 0.87 0.63 0.80 0.98 0.98 83 55 28 0 0

0.69 0.07 0.11 0.13 0.18 0.00 0.00 0.15 0.22 0.00 0.00 0.34 0.49 0.00 0.00 0.43 0.63 0.31 0.45 0.57 0.37 0.32 0.41 0.63 0.63 0.47 0.67 0.36 0.52 0.73 0.73 122 85 28 9 0

0.76 0.10 0.14 0.15 0.19 0.00 0.00 0.18 0.24 0.00 0.00 0.31 0.40 0.00 0.00 0.52 0.66 0.47 0.60 0.48 0.34 0.36 0.42 0.57 0.57 0.53 0.70 0.38 0.50 0.93 0.93 93 28 33 31 1

0.74 0.17 0.23 0.23 0.31 0.00 0.00 0.28 0.38 0.00 0.00 0.38 0.51 0.00 0.00 0.54 0.74 0.50 0.68 0.46 0.26 0.29 0.35 0.66 0.66 0.58 0.78 0.45 0.60 0.94 0.94 102 67 26 9 0

0.81 0.34 0.42 0.44 0.54 0.00 0.00 0.45 0.56 0.00 0.00 0.63 0.78 0.00 0.00 0.64 0.81 0.60 0.76 0.36 0.19 0.20 0.24 0.79 0.79 0.68 0.84 0.60 0.74 0.95 0.95 74 40 26 8 0

0.80 0.19 0.24 0.29 0.37 0.00 0.00 0.37 0.46 0.00 0.00 0.41 0.50 0.00 0.00 0.65 0.81 0.60 0.75 0.35 0.19 0.27 0.32 0.69 0.69 0.66 0.83 0.52 0.64 0.94 0.94 78 19 31 28 0

0.86 0.42 0.49 0.55 0.64 0.00 0.00 0.56 0.65 0.00 0.00 0.68 0.79 0.00 0.00 0.74 0.89 0.69 0.83 0.26 0.11 0.17 0.20 0.84 0.84 0.77 0.90 0.69 0.80 0.94 0.94 57 12 20 25 0

0.80 0.23 0.28 0.28 0.35 0.00 0.00 0.30 0.37 0.00 0.00 0.45 0.56 0.00 0.00 0.52 0.67 0.49 0.63 0.48 0.33 0.24 0.29 0.69 0.69 0.58 0.72 0.48 0.59 0.95 0.95 78 28 34 12 4

0.80 0.32 0.40 0.41 0.51 0.00 0.00 0.43 0.53 0.00 0.00 0.63 0.79 0.00 0.00 0.61 0.78 0.58 0.75 0.39 0.22 0.18 0.23 0.81 0.81 0.65 0.81 0.58 0.72 0.97 0.97 78 33 31 14 0

0.73 0.09 0.12 0.42 0.57 0.00 0.00 0.42 0.57 0.00 0.00 0.63 0.86 0.00 0.00 0.55 0.80 0.35 0.50 0.45 0.20 0.22 0.29 0.77 0.77 0.60 0.83 0.52 0.72 0.57 0.57 108 87 17 4 0

0.85 0.28 0.33 0.52 0.61 0.00 0.00 0.52 0.61 0.00 0.00 0.78 0.92 0.00 0.00 0.66 0.81 0.55 0.67 0.34 0.19 0.21 0.24 0.79 0.79 0.70 0.83 0.62 0.73 0.83 0.83 60 29 26 5 0

0.81 0.50 0.62 0.57 0.71 0.00 0.00 0.58 0.72 0.00 0.00 0.75 0.93 0.00 0.00 0.69 0.86 0.67 0.84 0.31 0.14 0.12 0.15 0.86 0.86 0.71 0.88 0.66 0.82 0.98 0.98 75 38 34 3 0

Mitigating Errors in LLM-Generated Web API Invocations

Table 9. Complete evaluation results for full completion with constrained decoding on the synthetic dataset

Manuscript submitted to ACM

+ (1 6B St ) ar Co de r( 1B St ) ar Co de r( 3B St ) ar Co de r( 7B St ) ar Co de r( 15 St . 5B ar ) Co de r2 (3 St B) ar Co de r2 (7 St B) ar Co de r2 (1 De 5B ) ep Se ek -C od De er ep (1 Se . 3B ek ) -C od De er ep (6 Se . 7B ek ) -C od Q er w en (3 3B 2.5 ) -C od Q er w (0 en . 5B 2.5 ) -C od Q e r w (1 en . 5B 2.5 ) -C od Q e r( w 3B en 2.5 ) -C od Q e r( w 7B en 2.5 ) -C od Q e r( w 14 en B) 2.5 -C od Ll er am (3 2B a3 ) .1 (8 B ) Ll am a3 .1 (7 0B Co ) de Ll am a( 7B Co ) de Ll am a( 13 Co B) de Ll am a( 70 B)

) (6 B +

Co de T5

Manuscript submitted to ACM

Table 10. Complete evaluation results for argument completion with constrained decoding on the synthetic dataset

0.55 0.36 0.66 0.00 0.00 0.45 0.93 0.42 0.87 0.55 0.07 0.03 0.06 0.00 0.00 0.96 0.96 0.51 0.94 0.50 0.91 0.95 0.95 179 43 84 9 43

0.74 0.52 0.70 0.00 0.00 0.68 0.95 0.64 0.89 0.32 0.05 0.04 0.05 0.00 0.00 0.97 0.97 0.71 0.96 0.69 0.93 0.95 0.95 102 81 17 3 1

0.89 0.56 0.62 0.00 0.00 0.79 0.90 0.73 0.83 0.21 0.10 0.05 0.06 0.00 0.00 0.96 0.96 0.81 0.91 0.79 0.88 0.93 0.93 43 7 33 3 0

0.84 0.57 0.68 0.00 0.00 0.77 0.95 0.72 0.89 0.23 0.05 0.06 0.08 0.00 0.00 0.95 0.95 0.80 0.96 0.77 0.91 0.94 0.94 63 11 41 11 0

0.91 0.66 0.72 0.00 0.00 0.88 0.98 0.85 0.94 0.12 0.02 0.08 0.09 0.00 0.00 0.94 0.94 0.90 0.99 0.85 0.93 0.96 0.96 36 5 30 1 0

0.87 0.46 0.53 0.00 0.00 0.84 0.98 0.71 0.83 0.16 0.02 0.05 0.06 0.00 0.00 0.96 0.96 0.85 0.98 0.82 0.95 0.83 0.83 52 16 28 8 0

0.89 0.63 0.71 0.00 0.00 0.86 0.98 0.82 0.94 0.14 0.02 0.08 0.09 0.00 0.00 0.94 0.94 0.88 0.98 0.83 0.93 0.97 0.97 43 8 22 13 0

0.89 0.67 0.75 0.00 0.00 0.87 0.98 0.83 0.94 0.13 0.02 0.07 0.08 0.00 0.00 0.95 0.95 0.88 0.98 0.84 0.94 0.96 0.96 43 11 27 5 0

0.86 0.69 0.79 0.00 0.00 0.84 0.98 0.81 0.96 0.16 0.02 0.05 0.06 0.00 0.00 0.96 0.96 0.85 0.99 0.82 0.94 0.97 0.97 54 15 33 6 0

0.90 0.42 0.47 0.00 0.00 0.85 0.97 0.69 0.79 0.15 0.03 0.08 0.09 0.00 0.00 0.94 0.94 0.88 0.97 0.83 0.92 0.79 0.79 40 20 12 6 2

0.81 0.59 0.74 0.00 0.00 0.78 0.98 0.76 0.95 0.22 0.02 0.08 0.10 0.00 0.00 0.94 0.94 0.79 0.98 0.75 0.93 0.97 0.97 77 49 25 3 0

0.81 0.66 0.81 0.00 0.00 0.78 0.98 0.76 0.96 0.22 0.02 0.05 0.06 0.00 0.00 0.96 0.96 0.80 0.98 0.77 0.95 0.98 0.98 74 45 27 2 0

0.88 0.39 0.44 0.00 0.00 0.80 0.93 0.64 0.74 0.20 0.07 0.06 0.07 0.00 0.00 0.95 0.95 0.83 0.94 0.79 0.90 0.79 0.79 46 24 7 15 0

0.89 0.65 0.72 0.00 0.00 0.85 0.98 0.81 0.93 0.15 0.02 0.07 0.07 0.00 0.00 0.95 0.95 0.87 0.98 0.83 0.93 0.96 0.96 42 15 18 9 0

0.85 0.62 0.73 0.00 0.00 0.81 0.97 0.78 0.93 0.19 0.03 0.08 0.10 0.00 0.00 0.94 0.94 0.83 0.97 0.79 0.92 0.97 0.97 58 30 21 7 0

0.87 0.63 0.73 0.00 0.00 0.84 0.98 0.82 0.96 0.16 0.02 0.09 0.10 0.00 0.00 0.93 0.93 0.86 0.99 0.80 0.92 0.98 0.98 53 10 37 6 0

0.90 0.67 0.74 0.00 0.00 0.89 0.99 0.87 0.96 0.11 0.01 0.09 0.10 0.00 0.00 0.94 0.94 0.89 0.99 0.84 0.93 0.98 0.98 39 18 15 6 0

0.89 0.66 0.74 0.00 0.00 0.87 0.98 0.85 0.96 0.13 0.02 0.09 0.10 0.00 0.00 0.93 0.93 0.88 0.99 0.82 0.92 0.98 0.98 42 9 24 9 0

0.84 0.60 0.71 0.00 0.00 0.80 0.98 0.77 0.94 0.20 0.02 0.09 0.11 0.00 0.00 0.93 0.93 0.83 0.98 0.77 0.92 0.97 0.97 63 27 19 17 0

0.85 0.65 0.77 0.00 0.00 0.81 0.98 0.79 0.95 0.19 0.02 0.07 0.08 0.00 0.00 0.96 0.96 0.83 0.98 0.80 0.94 0.97 0.97 61 17 28 16 0

0.78 0.14 0.18 0.00 0.00 0.74 0.98 0.52 0.68 0.26 0.02 0.09 0.12 0.00 0.00 0.93 0.93 0.77 0.98 0.72 0.92 0.65 0.65 86 61 23 2 0

0.88 0.49 0.55 0.00 0.00 0.86 0.98 0.73 0.84 0.14 0.02 0.06 0.07 0.00 0.00 0.95 0.95 0.87 0.99 0.84 0.95 0.86 0.86 46 26 17 3 0

0.83 0.72 0.86 0.00 0.00 0.80 0.98 0.79 0.97 0.20 0.02 0.03 0.03 0.00 0.00 0.97 0.97 0.82 0.99 0.80 0.96 0.98 0.98 66 34 29 3 0