arXiv CSopen access

SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

arXiv:2605.22564v1 [cs.CL] 21 May 2026

SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations Shuaiqi Wang

Aadyaa Maddi

Zinan Lin

Carnegie Mellon University

Carnegie Mellon University

Microsoft Research

[email protected]

[email protected]

[email protected]

Giulia Fanti Carnegie Mellon University [email protected]

Abstract Today, tool-calling agents are commonly evaluated or tested on static datasets of execution traces, including input commands, agent responses, and associated tool calls. However, internal production datasets are often insufficient or unusable for testing; for example, they may contain sensitive or proprietary data, or they may be too sparse to support comprehensive testing (especially pre-deployment). In these settings, practitioners are increasingly replacing or augmenting real datasets with synthetic ones for evaluation purposes. A key challenge is quantifying the relation between these synthetic datasets and the real data. We introduce SynAE, an evaluation framework for assessing how well synthetic benchmarks for multi-turn, tool-calling agents replicate and augment the characteristics of real data trajectories. SynAE assesses the validity, fidelity, and diversity of synthetic data across four metric categories: (i) task instructions and intermediate responses, (ii) tool calls, (iii) final outputs, and (iv) downstream evaluation. We evaluate SynAE using recent agent benchmarks and test common synthetic data failure modes via realistic and controlled generation schemes. SynAE detects fine-grained variations in data validity, fidelity and diversity, and shows that no single metric is sufficient to fully characterize synthetic data quality, motivating a multi-axis evaluation of synthetic data for agent testing. A demo of SynAE is available at https://synae-2026-synae-demo.static.hf.space/index.html, with code at https://github.com/wsqwsq/SynAE.

1

Introduction

Agent evaluation and testing is a nascent but critical component of pre-deployment processes for production agentic workflows [26, 2]. Today, tool-calling agent evaluations are often (but not always) run on static baseline datasets 1 consisting of a trace generated from agent interactions; these typically include user inputs, tool calls, intermediate interactions with the agent, and a final output [44, 22]. Such datasets are often collected from real-user interactions with an environment and/or synthesized by measuring scripted interactions with a dynamic environment. While the design of proper agent evaluations is its own active research area [12, 1, 48], common evaluation metrics measure whether the agent selects the correct tools and produces the desired final output [44, 22, 2]. In many practical situations, existing baseline datasets cannot be directly used or are insufficient for agent evaluation, either because they contain sensitive user data (e.g., emails, travel details) subject to privacy restrictions [31, 3] or because they are too small for comprehensive testing [26]. Practitioners therefore increasingly use synthetic datasets as replacements for or augmentations to real execution traces in evaluation pipelines (Figure 1a) [28, 32, 9]. Synthetic data may be generated by directly synthesizing trajectories or by synthesizing inputs to an interactive environment. Although synthetic datasets are increasingly used for agent evaluation, typical workflows lack systematic quality checks against the real baseline data. Indeed, the current literature on testing agents with 1 These evaluation datasets are also commonly referred to as benchmarks; we use both terms interchangeably in this paper.

1

Inputs

SynAE Evaluation Framework

Real Dataset Instructions & Responses

Tool Calls

Final Output

Synthetic Dataset Instructions & Responses

Final Output

Tool Calls

Metric Configs Data-specific Prior Knowledge

LLM Agent(s)

Validity

Fidelity

Diversity

Strong

Moderate

Poor

Outputs

(a) Today, agents are commonly evaluated on synthetic (b) The SynAE evaluation pipeline takes as input real and datasets modeled after a real dataset of agent trajectories, synthetic datasets (task completion trajectories), along with e.g., for privacy or augmentation reasons. However, it is optional metric configurations that may specify data-specific often unclear whether these synthetic datasets are repre- prior knowledge and one or more LLM agents. The LLM sentative of the real dataset. Our evaluation framework, agents are used only to compare downstream performance SynAE, measures the semantic and structural similarity between real and synthetic input datasets. The framework between real and synthetic datasets of execution trajecto- evaluates synthetic data along three dimensions: validity, ries, and the validity and diversity of the synthetic data. fidelity, and diversity.

Figure 1: The SynAE framework evaluates the quality of synthetic data used in agent evaluations. synthetic data provides almost no quantitative methods for evaluating the quality of such synthetic data, leaving operators with little visibility into evaluation gaps. In this work, we develop a comprehensive evaluation framework, SynAE, to assess how well synthetic trajectories replicate and augment the characteristics of real data trajectories, including task instructions and responses, and the associated reference tool calls and outputs. As illustrated in Figure 1b, SynAE takes as input a real baseline dataset, a synthetic dataset, and optional metric configurations that may specify data-specific prior knowledge and one or more LLM agents; the input agent(s) need not be the same as the one being evaluated—each agent input to SynAE is used purely to compare downstream performance between real and synthetic input datasets. SynAE evaluates synthetic data along three properties: (1) validity, (2) fidelity, and (3) diversity. For validity, we evaluate whether synthetic tool calls and outputs successfully fulfill the given instructions, using LLM-as-a-judge by default or rule-based checkers when available. We measure fidelity by assessing how similar the real and the synthetic data are; specifically, we compute metrics on both real and synthetic data and evaluate their similarity. We quantify the diversity of synthetic data with entropy-based metrics for various representations of the dataset. For each property, SynAE computes a suite of metrics to evaluate (a subset of): (1) task instructions and responses, (2) tool calls, (3) final outputs, and (4) downstream evaluation. We demonstrate the utility of SynAE with three recent agent benchmarks playing the role of the “real dataset” in Fig. 1a: T1 [4], BFCL [27], and ACP [13]. We then construct synthetic datasets using NVIDIA NeMo [23], an industry-standard synthetic data tool, as well as custom synthetic data generation methods designed to simulate common real-world pitfalls, such as degraded data fidelity and limited diversity. Our experiments show that SynAE captures fine-grained variations in the validity, fidelity, and diversity of synthetic task-completion trajectories. They further suggest that no single metric is sufficient to fully characterize synthetic data quality, underscoring the need for SynAE. Overall, we view SynAE as a plug-in component for agentic workflows, allowing operators to automatically evaluate the quality of synthetic benchmark datasets.

1.1

Related Work

We provide a brief overview of related works, and provide a detailed discussion in §A. Several prior works evaluate synthetic benchmarks for LLMs using factors such as task difficulty, realism, and whether model rankings are preserved [7, 18, 38, 19]. However, these works mainly focus on standard NLP settings rather than agent benchmarks, where evaluation is more difficult due to multi-step decision making, interaction with tools, and the need to attribute failures to different parts of the agent pipeline. Recent evaluations on synthetic benchmarks for tool-calling agents has largely focused on single-turn settings or instruction-level quality [30, 9, 48, 1, 26], without systematically evaluating associated reference tool calls, outputs, or multi-step dependencies. 2

Multi-turn tool-calling agents are commonly evaluated using end-to-end task success on state-based checkers [42, 16, 46, 41, 37, 11], response-based criteria [27, 14, 45], or LLM-as-a-judge [28, 25, 39, 17]. These methods evaluate agent performance on a given benchmark, whereas our goal is to evaluate the benchmark itself by providing quantitative metrics for validity, fidelity, and diversity across instructions and responses, tool calls, final outputs, and downstream evaluation.

2

SynAE Framework

The SynAE framework assesses how well synthetic agent trajectories replicate and augment the characteristics of a real dataset. It quantifies the validity, fidelity, and diversity of synthetic data. Sample 𝑫𝒊 Instructions & Responses 𝑹𝒊 𝒓𝒊,𝟏

I want to find some art attractions in USA.

𝒓𝒊,𝟐

Are you looking at specific cities?

𝒓𝒊,𝟑

I’m thinking of visiting NYC or LA.

Tool Calls 𝑭𝒊

Final Output 𝑶𝒊

𝒇𝒊,𝟏 attractions = search_attractions(type

="Art", country=”USA")

𝒇𝒊,𝟐 result1 = filter_attractions(prior_res =attractions, city=”NYC")

Both NYC and LA are great choices for art. Here’s a breakdown to help you decide: …

𝒇𝒊,𝟑 result2 = filter_attractions(prior_res =attractions, city=”LA")

Figure 2: Agent trajectory from T1 [4] benchmark dataset, with notation for each component. m

Notation and setup Consider a dataset D = {Di }i=1 of m samples (or agent trajectories). Fig. 2 illustrates a single sample trajectory Di , which consists of a set of instructions and responses Ri , tool calls Fi , and a textual output Oi . Ri is a sequence containing multiple instructions and responses Ri = (ri,1 , ri,2 , . . . , ri,ℓi ), where ℓi denotes the number of instructions and responses. The tool call sequence Fi consists of tool calls Fi = (fi,1 (φi,1 ) , fi,2 (φi,2 ) , . . . , fi,qi (φi,qi )), where fi,j ∈ F corresponds to an executable function, F denotes the set of all possible tools, φi,j represents the input to fi,j , and qi is the total number of tool calls in the trajectory. Note that instructions and responses, tool calls, and outputs can be interleaved in time; in a slight abuse of notation, we use Di to refer to the time-ordered sequence of events, and use Ri , Fi , Oi to refer to the corresponding filtered subsequences containing only instructions and responses, tool calls, and final outputs, respectively. Summary of inputs SynAE takes as input (1) a real dataset D; (2) a synthetic dataset D′ ; and (3) optional metric configurations that may specify data-specific prior knowledge and agents A1 , . . . , Ah (details below). Both datasets contain instructions and responses, reference tool calls and outputs; SynAE also supports datasets missing responses, tool calls, or outputs.

2.1

Evaluation Metrics

SynAE divides evaluation metrics into three pillars: validity (§2.1.1), fidelity (§2.1.2), and diversity (§2.1.3). Each pillar includes multiple sub-metrics that require varying amounts of prior knowledge about the real data, and evaluate different aspects of the synthetic dataset: (1) task instructions and responses, (2) tool calls, (3) outputs, and (4) downstream tasks. 2.1.1

Validity Metrics

Validity is important to assess since synthetic data can appear faithful yet be unusable if the tool calls or final outputs fail to complete the task. In practice, invalid samples may arise from hallucinated tool names or arguments, or from plausible-looking outputs that do not satisfy the instructions. While validity may be dataset-specific, we use a broad definition: whether the provided tool calls or outputs accomplish the task instruction, serving as a basic self-consistency check. By default, we assess validity for each sample Di ∈ D with an LLM-as-a-judge, whose prompt and agreement with human annotations are 3

reported in §B; users may instead define rule-based checkers when available. We report the overall Validity Rate (VR) as the proportion of valid tool-call sequences and outputs. 2.1.2

Fidelity Metrics

We design fidelity metrics to measure similarity between synthetic and real data, which is especially useful when synthetic data is used as a direct replacement for real data, e.g., due to privacy constraints. Fidelity is the only property in SynAE that uses real data to compute its associated metrics, covering (1) task instructions and responses, (2) tool calls, (3) outputs, and (4) downstream tasks. 1. Fidelity metrics for task instructions and responses Task instructions and responses have an alternating structure, with each instruction followed by a response. Due to this, we draw inspiration from StructBench [34], a framework for evaluating structured synthetic data, and use metrics below. For fidelity metrics that capture the structural relations in the data, we use Key Node Dependency and Attribute Match. Key Node Dependency (KND) measures semantic dependencies between parts of each sample by computing embedding cosine similarities between each instruction and its corresponding response, and between each response and the subsequent instruction. We then compare real and synthetic datasets by measuring the distributional distance between their similarity-score distributions. Attribute Match (AM) measures how closely real and synthetic datasets align on predefined statistical and semantic attributes by computing distributional distances over their attribute distributions, using Wasserstein-2 distance for numerical attributes and Total Variation (TV) distance for categorical ones. We consider attributes including the number of instruction turns, instruction and response token lengths, and dataset-specific semantic properties (specified in Table 1). For non-structural fidelity metrics, we use KNN-Precision and KNN-Recall to measure the semantic quality and coverage. KNN-Precision (respectively, KNN-Recall) is the fraction of synthetic (respectively, real) samples whose embedding distance to a real (respectively, synthetic) sample is smaller than the distance to the k-th nearest neighbor within their own distribution. We also use Fréchet Inception Distance (FID) to measure semantic closeness between real and synthetic data. 2. Fidelity metrics for tool calls We evaluate whether synthetic tool-usage and planning patterns match the real dataset. Each metric computes a distributional distance between real and synthetic data for a specific property. (1) Tool Usage Match (TUM) compares overall tool-usage patterns, TUM = Dis(ωf , ωf′ ), where ωf and ωf′ denote the tool-usage distributions in the real and synthetic data, respectively, and Dis is the TV distance. (2) Tool Call Number Match (TCNM) compares the distributions of tool-call counts qi per sample Di , TCNM = Dis(ωq , ωq′ ), where Dis is the Wasserstein-2 distance and ωq is the distribution over the tool-call number. (3) k-Step Tool Planning Match (k-Step Planning) measures tool-planning similarity by comparing conditional distributions of the next tool given the previous k − 1 tool calls, i.e., ωf |f1 ···fk−1 , ∀f1 · · · fk−1 ∈ F k−1 . It is a weighted sum of TV distances between real and synthetic conditional distributions:   X k-Step Planning = nf1 ···fk−1 · Dis ωf |f1 ···fk−1 , ωf′ |f1 ···fk−1 , f1 ···fk−1 ∈F k−1

where nf1 ···fk−1 is the number of k-step tool-call sequences in the real data with prefix f1 · · · fk−1 . We set k ∈ {1, 2} by default and allow larger values for long-horizon tasks. 3. Fidelity metrics for outputs We evaluate the task fidelity of the final outputs by measuring their KNN-Precision, KNN-Recall, and FID relative to the real dataset. 4. Fidelity metrics for downstream evaluation To assess the downstream utility of a benchmark constructed from synthetic data, we compare agent performance on real and synthetic tasks, measuring whether the two benchmarks induce similar tool-call and output-generation behavior. Recall that the user may optionally provide one or more agents A1 , . . . , Ah for the downstream evaluation; these agents can differ from the one being evaluated in Fig. 1a. For each agent Aj and reference trajectory Di , the agent generates tool calls at each instruction turn, conditioned on the current instruction, prior instruction turns, 4

and the history of executed tool calls and results, using the dataset-specific tool set. The final output is then generated from the full instruction and executed tool-call history. We adopt two downstream metrics. (1) Task Difficulty Difference (TDD) measures the average absolute difference in task-completion performance across agents between real and synthetic tasks. For each reference trace Di from either dataset and each input agent Aj , we generate an agent trace D̃i,Aj and use an LLM-as-a-judge to determine whether Di and D̃i,Aj are functionally equivalent (prompt in §C). We then compute the absolute difference in task success rates for tool-call selection or final-output generation between the real and synthetic benchmarks, averaged across agents. (2) Ranking Divergence (RD) measures whether agent performance rankings are preserved across the real and synthetic datasets by computing the Spearman rank correlation between rankings based on tool-call or output correctness. Lower TDD and higher RD indicate better downstream utility. 2.1.3

Diversity Metrics

Diversity is particularly important when synthetic data is used to improve real-dataset coverage. For example, the real dataset may be collected in a controlled setting with little representation of outliers or real-world user requests; practitioners may thus add synthetic outliers to test product robustness. Measuring diversity is challenging: without prior domain-specific knowledge, it is unclear whether a dataset adequately covers the (high-dimensional) space of data traces. We therefore use two reference-free metrics, Vendi Score and Attribute Diversity, which do not require access to real data, though users of SynAE can compare the diversity of real and synthetic data if desired. Vendi Score (Vendi) [6] is widely used to measure diversity in machine learning, including data curation and generative modeling [10, 36]. It computes diversity from a similarity matrix K: for instructions & responses and final outputs, Ki,j is the cosine similarity between embeddings of Ri , Rj or Oi , Oj , respectively, Levenshtein(Fi ,Fj ) using text-embedding-3-small in our experiments. For tool calls, we define Ki,j = 1 − , max(qi ,qj ) where qi is the number of tool calls in Fi , and we adopt Levenshtein distance to measure sequence dissimilarity. Vendi then computes Pm the exponential entropy of the eigenvalues λ1 , . . . , λm of the normalized matrix K/m: Vendi = exp (− i=1 λi log λi ) . Higher Vendi Score indicates more dissimilar samples and thus greater diversity; it is upper-bounded by m, achieved when all pairwise similarities are zero. Vendi Score is agnostic to dataset format and content but may be harder to interpret. For users with domain knowledge, we also include Attribute Diversity, which is more interpretable but requires specifying attributes of interest. Attribute Diversity (AD) measures instruction-response diversity using user-specified attributes that capture dimensions the operator wants to diversify, such as attraction type and city in T1 or anomalous vs. benign behavior in a computer security context. Each sample is assigned, by a human or LLM, to one attribute-value combination, e.g., (“attraction type=museum", “city=Austin"). We then compute the PC entropy of the resulting attribute distribution: AD = − i=1 pi log pi , where pi is the proportion of samples in attribute combination i, and C is the number of possible combinations. Higher AD indicates that no single attribute value dominates the population, reflecting greater diversity; it is upper-bounded by log C, attained when all combinations are equally represented. SynAE metrics and examples of dataset-specific prior knowledge are summarized in Table 1.

3

Experiments

We demonstrate SynAE on the following baseline datasets and synthetic data generation methods. Evaluation datasets We evaluate SynAE on T1 [4], BFCL [27], and ACP [13] benchmark datasets, and treat them as the real data. T1 uses the T1-attraction dataset with 225 samples, where each sample contains multi-turn attraction-recommendation instructions, reference tool calls, and final outputs. BFCL uses the BFCL-V3-Base-Multi-Turn dataset with 200 samples, covering domains such as file operations, mathematical calculation, and travel booking, with reference tool calls. ACP uses the ACPBench-Applicability&Progression dataset with 260 samples, covering planning domains such as transportation and robot action planning. Metrics across datasets are summarized in Table 1, and the computational costs of SynAE across datasets are reported in §D.

5

Table 1: Evaluation metrics across datasets. Metrics in orange require dataset-specific prior knowledge; “—” indicates no input is used. Evaluation Aspect Validity

Evaluation Target Tool Call Output

Metric Validity Rate Validity Rate Key Node Dependency

Instruction Attribute Match Fidelity

Tool Call Output Downstream Task

KNN-Precision KNN-Recall FID Tool Usage Match Tool Call Number Match k-Step Planning KNN-Precision KNN-Recall FID Task Difficulty Difference Ranking Divergence

Instruction Diversity

Tool Call Output

Vendi Score Attribute Diversity Vendi Score Vendi Score

T1 Instantiation LLM-as-a-judge LLM-as-a-judge 1. (instruction, response) pair 2. (response, instruction) pair 1. number of instructions 2. instruction token length 3. response token length 4. city 5. attraction type — — — — — k ∈ {2, 3} — — — 1. Tool Call 2. Output 1. Tool Call 2. Output — (city, attraction type) pair — —

BFCL Instantiation LLM-as-a-judge N/A

ACP Instantiation N/A LLM-as-a-judge

(instruction, instruction) pair

(context, instruction) pair

1. number of instructions 2. instruction token length 3. task domain 4. task subdomain

instruction token length

— — — — — k ∈ {2, 3} N/A N/A N/A

— — — N/A N/A N/A — — —

Tool Call

Output

Tool Call

Output

— (domain, subdomain) pair — N/A

— planning domain N/A —

Synthetic data generation To evaluate the sensitivity of SynAE to structured, controlled changes in synthetic data quality, we design interpretable synthetic data algorithms that modify original trajectories to model common problems in synthetic data (details below): Blank Filling, Oversampling, and In-Context Generation models combined fidelity and diversity degradation, and Invalidation models degraded validity. To evaluate SynAE on industry-standard synthetic data techniques, we use NVIDIA NeMo [23], a synthetic data tool that is used in part to test agents. (1) Blank Filling : We randomly mask tokens in the original instructions with probability p and prompt a language model to fill them in (prompt in §E). Larger p masks more information, causing greater deviation from the original instructions. We set p ∈ {0.1, 0.3, 0.5, 0.7, 0.9}. (2) Oversampling : We construct a synthetic dataset of fixed size m by first selecting one instructionresponse sequence R ∈ R, where R ≜ ∪j Rj is the set of all such sequences in D. We use R for the first rm samples and fill the remaining (1 − r)m samples by sampling without replacement from R \ {R}. Larger r creates more duplicates and thus lower diversity. We set r ∈ {0.1, 0.3, 0.5, 0.7, 0.9}. (3) In-Context Generation: We generate synthetic instructions and responses by prompting language models with k in-context examples (prompt in §E). When k = 0, generation relies only on the prompt and is weakly grounded in the original data, reducing fidelity. When k > 0, fixed in-context examples limit diversity, while randomly varying them improves the instruction-space coverage. We set k ∈ {0, 1, 3, 5} and consider both fixed and randomly sampled examples. (4) Invalidation: We keep the original instructions but replace a fraction v of samples with modified tool calls or outputs, leaving the rest unchanged. For benchmark datasets with tool calls f (φ), namely T1 and BFCL, we alter each selected tool call by replacing its input φi,j with an input φi,k that does not satisfy the instruction. For datasets with final outputs O, namely T1 and ACP, we replace each selected output Oi with an alternative output Ok that does not satisfy the instruction. (5) Industry-standard synthetic data: We generate synthetic agent trajectories using NVIDIA NeMo Data Designer, based on the specified input schemas, constraints, and backend language model. We use GPT-4o-mini, Nvidia-Nemotron-Nano-9B-v2, Mistral-24B-Instruct, and Llama3.1-8B-Instruct as backends, and vary the temperature over {0.1, 0.3, 0.5, 0.7, 0.9}. For methods that generate synthetic instructions and responses, namely Blank Filling, Oversampling, and In-Context Generation, we also need corresponding tool calls and outputs, which we generate using Llama3.1-8B-Instruct as the backend for each dataset-specific agent.

6

Agents for downstream evaluation We conduct downstream evaluations on each real or synthetic dataset using LLM agents. For each trace, an agent iterates over the instruction sequence, receiving the full prior context, including previous instructions, responses, tool calls, and results, and predicts the next response and/or tool call, which is then compared with the provided trace. We evaluate three agents using the same benchmark runtime context, with backends gemma-3-1b-it, Qwen3-4B-Instruct, and Llama3.1-8B-Instruct. We use Mistral-7B-Instruct as the LLM-as-a-judge for functional equivalence of tool calls and outputs (prompt in §C). Metric visualization We visualize synthetic-data performance across metrics using line and radar plots. For radar plots, all metrics are rescaled to [20, 100], where 20 is the worst performance among compared methods and 100 is the metric upper bound (e.g., KNN-Precision = 1 or 2-Step Planning = 0). To summarize overall diversity, fidelity, and validity, we compute an aggregate score for each by first normalizing the corresponding metrics to [0, 1] (with higher values indicating better performance) and then averaging them. Although averaging normalized fine-grained metrics is not necessarily the most meaningful way as they may scale differently, this practice is sometimes used in benchmarks for easier visualization and interpretation [33].

3.1

Experimental Results

We present the evaluation results of synthetic data generation methods proposed in this work on T1 in Figs. 3 to 7, with detailed results on all datasets in §F and NVIDIA NeMo results on T1 in §G. 3.1.1

SynAE can capture fine-grained variations in fidelity, diversity, and validity of synthetic data. Blank Filling: Precision & Recall ( )

1.0

.1' LQVWUXFWLRQUHVSRQVH .113UHFLVLRQ .1' UHVSRQVHLQVWUXFWLRQ .115HFDOO

0.8 0.6



$0&LW\

0.0

0.1

0.3

0.5

p









7&10 .113UHFLVLRQ 2XWSXW

0.7

0.9

Tool Call Eval

1.0

(a) KNN-Precision and KNN-Recall of Blank Filling with different masking probability p.

780 6WHS3ODQQLQJ S 

Output Eval

6WHS3ODQQLQJ

S 

S 

(b) Blank Filling. As p increases, data fidelity degrades across nearly all metrics.

Oversampling: Precision & Recall ( )

1.0



.115HFDOO 2XWSXW

KNN-Precision (instruction) KNN-Recall (instruction) KNN-Precision (output) KNN-Recall (output)

0.2

.1' LQVWUXFWLRQUHVSRQVH .113UHFLVLRQ .1' UHVSRQVHLQVWUXFWLRQ .115HFDOO

0.8

5'7RRO&DOO

Instruction Eval ),'

0.6

Downstream Eval

7''7RRO&DOO

0.4



$0&LW\

0.0

0.1

0.3

0.5

r











.115HFDOO 2XWSXW

KNN-Precision (instruction) KNN-Recall (instruction) KNN-Precision (output) KNN-Recall (output)

0.2 0.0

Downstream Eval

7''7RRO&DOO

0.4

0.0

5'7RRO&DOO

Instruction Eval ),'

!"#$

0.7

0.9

Tool Call Eval

1.0

(c) KNN-Precision and KNN-Recall of Oversampling with different duplication rate r.

%##&'()*+,+-. /0123124 780

Output Eval

!"#$%& '()**+*, 5&62)3 '78..+.9

-./01

-./02

(:;<=

(d) Oversampling. As r increases, data fidelity degrades across nearly all metrics.

Figure 3: Fidelity of Blank Filling and Oversampling on the T1 dataset. 7

Fidelity: Fig. 3 shows the fidelity of Blank Filling and Oversampling on T1. For Blank Filling (Fig. 3a), as p increases, the KNN-Precision for both instructions and outputs decreases from nearly 1 to close to 0, indicating progressive degradation in semantic quality, consistent with the intuition that higher masking probability lowers fidelity. In contrast, when p ≤ 0.9, the KNN-Recall for both instructions and outputs remains relatively high (above 0.7), suggesting that Blank Filling largely preserves semantic coverage. For Oversampling (Fig. 3c), increasing r significantly reduces output KNN-Recall, while instruction and output KNN-Precision remain near perfect, consistent with the intuition that duplication reduces task coverage while largely preserving task quality. Moreover, Figs. 3b and 3d show that increasing p or r degrades performance across nearly all fidelity metrics, aligning with the intuition that heavier masking or duplication produces lower-fidelity synthetic data. Diversity: Fig. 4 shows the diversity of Blank Filling and Oversampling on T1. As the masking probability p increases, Blank Filling becomes more diverse (see Fig. 4a), consistent with the intuition that masking more words introduces novel content and diversifies task topics and execution trajectories. In contrast, Oversampling diversity decreases as the duplication rate r increases (see Fig. 4b), consistent with the intuition that more duplicated samples reduce overall dataset diversity. Vendi: Instruction

Vendi: Instruction

!""#$%&"' D$versity 0

Vendi: Tool Call

20

40

60

80

!""#$%&"' Diversity

100

0

Vendi: Tool Call

Vendi: Output p=0.1

p=0.3

p=0.5

p=0.7

20

40

60

80

100

!"#$%& '()*() p=0.9

r=0.1

(a) Vendi Scores and Attribute Diversity of Blank Filling with different masking probability p. As p increases, data diversity improves.

r=0.3

+,-./

+,-.0

!"#$%

(b) Vendi Scores and Attribute Diversity of Oversampling with different duplication rate r. As r increases, data diversity decreases.

Figure 4: Diversity metrics for Blank Filling and Oversampling on the T1 dataset.

3.1.2 No single metric can fully characterize synthetic data performance.

1.0 0.8

Validity Rate

Validity: Fig. 5 illustrates the Validity Rate of Invalidation on T1. As the invalidation ratio v increases, Validity Rates for both tool calls and final outputs decrease monotonically with an approximate slope of −1, as expected by construction. This indicates that SynAE effectively captures validity variations.

Tool Call Final Output Ground-truth Validity Rate (Upper Bound)

0.6 0.4 0.2

Fig. 6 illustrates the fidelity-diversity trade-offs for Blank 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Filling and Oversampling. For each dimension, we comv pute an aggregate score to summarize overall performance. Increasing the duplication rate r in Oversampling degrades Figure 5: Validity of Invalidation on T1. As both fidelity and diversity, whereas increasing the masking invalidation ratio v increases, Validity Rates for probability p in Blank Filling lowers fidelity but improves both tool calls and outputs decrease. diversity. These results highlight that no single metric category fully characterizes synthetic data performance, as different metrics capture distinct aspects.

8

Even within fidelity evaluation itself, one parameter setting may outperform another on some metrics while underperforming on others. For example, in InMask more Context Generation (Fig. 7a), increasing the number of in-context examples k does not uniformly improve all fidelity metrics: on T1, k = 3 improves KNNPrecision over k = 1 but lowers KNN-Recall. Fixing k and comparing fixed versus randomized examples (Fig. 7b), randomization improves instruction- and output-level fidelity while maintaining similar toolcall performance. We also show in §H that simple Oversample more baseline metrics, such as corpus-level statistics and embedding closeness, can be misleading when used alone. Another example appears under Blank Filling (Fig. 3b): as p increases, tool-call metrics degrade, indicating larger discrepancies in tool-planning patFigure 6: Fidelity vs. diversity for Blank Filling and terns between real and synthetic tasks. In contrast, Oversampling. For Blank Filling, a higher masking the downstream metric RD: Tool Call remains strong, probability p leads to lower fidelity but greater diversity. since higher-capacity models continue to outperform For Oversampling, a higher duplication rate r results lower-capacity ones even when planning patterns in declines in both fidelity and diversity. shift, resulting in largely consistent model rankings. These observations further motivate the need for a comprehensive evaluation framework such as SynAE. .1' LQVWUXFWLRQUHVSRQVH .113UHFLVLRQ .1' UHVSRQVHLQVWUXFWLRQ

.1' LQVWUXFWLRQUHVSRQVH .113UHFLVLRQ .1' UHVSRQVHLQVWUXFWLRQ

.115HFDOO

.115HFDOO 5'7RRO&DOO

Instruction Eval ),'

Downstream Eval

5'7RRO&DOO

Instruction Eval ),'

7''7RRO&DOO

7''7RRO&DOO 

$0&LW\













$0&LW\









Tool Call Eval

Output .113UHFLVLRQ 2XWSXW Eval 780 6WHS3ODQQLQJ N 

N  IL[HG



.115HFDOO 2XWSXW

.115HFDOO 2XWSXW 7&10

Downstream Eval

7&10 .113UHFLVLRQ 2XWSXW

Tool Call Eval

6WHS3ODQQLQJ

780 6WHS3ODQQLQJ N  IL[HG

N  IL[HG

Output Eval

6WHS3ODQQLQJ

N  UDQGRP

(b) In-Context Generation with k = 5. Randomized in-context examples improve instruction and output fidelity over fixed examples, while maintaining similar tool-call performance.

(a) In-Context Generation with different number of incontext examples k. Increasing k does not consistently improve all fidelity metrics, including KNN-Recall, KND, and AM.

Figure 7: Fidelity of In-Context Generation under T1 with fixed or randomized in-context examples. 3.1.3

Case study: Practitioners can leverage SynAE to iteratively diagnose and improve synthetic data generation.

We present a case study simulating a common workflow: a data holder starts with skewed data, uses SynAE to diagnose issues, then iteratively refines an augmentation strategy with SynAE’s feedback. We start with a modified version of T1 where some properties are under-represented: we select half of the attraction types, including culture, sport, culinarian, and guide, and downsample instances containing these types to 10% of the original count by discarding the remaining samples. The data holder then aims to augment the dataset to better represent these target types. Step 1: Diagnose the bottleneck with SynAE. The data holder first evaluates the skewed data with SynAE, which identifies diversity as the primary issue (first column of Table 2) and pinpoints under-represented attributes via attribute distributions (reported in §I), motivating targeted augmentation.

9

Step 2: Attempt a lightweight fix via relabeling, then re-evaluate. As a fast intervention, the data holder applies Relabeling, which replaces attraction-type keywords in the original instructions with target types. For example, we might change “Please find some art attractions in Canada” to “Please find some sporting attractions in Canada.” SynAE confirms that diversity improves (2nd column of Table 2); however, validity drops, revealing that naive keyword substitution can introduce semantic inconsistencies. For example, the modified sporting-attraction instruction may still be followed by an art-related response such as “Do you prefer museums or concerts?”. This can make the modified instructions incompatible with the tool calls and outputs, causing failures in task completion. Step 3: Escalate to model-based synthesis via NVIDIA NeMo, then re-evaluate. To improve diversity while preserving validity, the data holder next adopts NVIDIA NeMo. They first use smaller backend models, Llama3.1-8B-Instruct and Nvidia-Nemotron-Nano-9B-v2, with temperature 0.6, and provide prompts specifying the target attraction types. For example, given the target type sport, NeMo generates the instruction “Please find some sporting attractions in Canada.” followed by responses such as “Do you prefer basketball or baseball ?”, preserving semantic consistency. SynAE shows that NVIDIA NeMo with Llama or Nemotron preserves validity, but substantially sacrifices fidelity, while diversity remains moderate (columns 3 & 4 of Table 2). The data holder then escalates to the higher-capacity backend GPT-4o-mini. SynAE shows that this improves diversity while preserving validity, without substantially sacrificing fidelity (column 5 of Table 2). Table 2: Validity, fidelity, and diversity of the skewed real data, Relabeling, and NVIDIA NeMo. Green indicates strong performance, red indicates poor performance, and black indicates moderate performance. The diversity of the original (unskewed) data is 0.72. Skewed Real Data Validity Fidelity Diversity

Relabeling

1.0 1.0 0.48

0.82 0.95 0.65

NVIDIA NeMo Llama

Nemotron

GPT

0.98 0.71 0.67

0.99 0.79 0.61

0.99 0.94 0.70

Table 2 summarizes fidelity, diversity, and validity using aggregate scores. We define “strong" performance as metrics within 5% of the original (unskewed) real dataset and “poor" performance as metrics at least 15% lower. Overall, this case study shows how SynAE supports iterative refinement: diagnose the limiting factor, apply an intervention, and use metric feedback to adjust the strategies.

4

Conclusions and Limitations

In this paper, we introduced SynAE, a multi-axis framework for evaluating how well synthetic data replicates and augments real agent trajectories across task instructions and responses, tool calls, final outputs, and downstream performance. Through experiments on recent agent benchmarks with realistic and controlled synthetic generation schemes, SynAE consistently detects fine-grained failures in validity, fidelity, and diversity. Overall, no single metric suffices to capture synthetic benchmark quality, motivating multi-axis evaluation with SynAE before using synthetic data for agent testing. A limitation of our work is that SynAE mainly focuses on multi-turn tool-calling agentic settings. Extending the framework to agent benchmarks involving interactive environments or multi-agent coordination is an important direction for future work.

Acknowledgments This work was supported in part by the NSF RINGS program, grant CNS-2148359.

10

References [1] Omar Alonso and Kenneth Church. Evaluating the evaluations: A perspective on benchmarks. In ACM SIGIR Forum, volume 58, pages 1–27. ACM New York, NY, USA, 2025. [2] Anthropic. Demystifying evals for ai agents. demystifying-evals-for-ai-agents, 2026.

https://www.anthropic.com/engineering/

[3] CapitalOne. Synthetic data matters for machine learning innovation. https://www.capitalone.com/ tech/machine-learning/synthetic-data-research/, 2022. [4] Amartya Chakraborty, Paresh Dashore, Nadia Bathaee, Anmol Jain, Anirban Das, Shi-Xiong Zhang, Sambit Sahu, Milind Naphade, and Genta Indra Winata. T1: A tool-oriented conversational dataset for multi-turn agentic planning. arXiv preprint arXiv:2505.16986, 2025. [5] Enkrypt AI. What are specialized task ai agents? benefits, features & use cases explained. Enkrypt AI Blog (Guest Post), March 2024. [6] Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.02410, 2022. [7] Alexander Gill, Abhilasha Ravichander, and Ana Marasović. What has been lost with synthetic evaluation? arXiv preprint arXiv:2505.22830, 2025. [8] Ben Hutchinson, Negar Rostamzadeh, Christina Greer, Katherine Heller, and Vinodkumar Prabhakaran. Evaluation gaps in machine learning practice. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pages 1859–1876, 2022. [9] Shadi Iskander, Sofia Tolmach, Ori Shapira, Nachshon Cohen, and Zohar Karnin. Quality matters: Evaluating synthetic data for tool-using llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4958–4976, 2024. [10] Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.00703, 2025. [11] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. [12] Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, et al. Holistic agent leaderboard: The missing infrastructure for ai agent evaluation. arXiv preprint arXiv:2510.11977, 2025. [13] Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. Acpbench: Reasoning about action, change, and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26559–26568, 2025. [14] Renhao Li, Jianhong Tu, Yang Su, Yantao Liu, Fei Huang, Hamid Alinejad-Rokny, Derek F Wong, Junyang Lin, and Min Yang. Toolrm: Towards agentic tool-use reward modeling. arXiv preprint arXiv:2510.26167, 2025. [15] Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd workshop on knowledge extraction and integration for deep learning architectures, pages 100–114, 2022. [16] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023. 11

[17] Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J Pal, and Siva Reddy. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories. arXiv preprint arXiv:2504.08942, 2025. [18] Gaurav Maheshwari, Dmitry Ivanov, and Kevin El Haddad. Efficacy of synthetic data as a benchmark. arXiv preprint arXiv:2409.11968, 2024. [19] Michael Majurski and Cynthia Matuszek. Grounding synthetic data evaluations of language models in unsupervised document corpora. arXiv preprint arXiv:2505.08905, 2025. [20] Amanda McGrath and Amanda Downie. What are vertical ai agents? IBM Think, n.d. [21] Sohum Mehta and Saaketh Bhojanam. Prompt genotyping: Quantifying the evaluation gap between synthetic benchmarks and real llm performance. In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling. [22] Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. Evaluation and benchmarking of llm agents: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6129–6139, 2025. [23] NVIDIA. NVIDIA NeMo. https://www.nvidia.com/en-us/ai-data-science/products/nemo/. [24] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [25] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06474, 2024. [26] Melissa Z Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, et al. Measuring agents in production. arXiv preprint arXiv:2512.04123, 2025. [27] Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. [28] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023. [29] Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, and Monica Lam. Assisting in writing wikipedia-like articles from scratch with large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6252–6278, 2024. [30] Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. Advances in Neural Information Processing Systems, 37:4540–4574, 2024. [31] Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, et al. Clio: Privacy-preserving insights into real-world ai use. arXiv preprint arXiv:2412.13678, 2024. [32] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023.

12

[33] B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaeffer, et al. DecodingTrust: A comprehensive assessment of trustworthiness in GPT models. In Conference on Neural Information Processing Systems, 2023. [34] Shuaiqi Wang, Vikas Raunak, Arturs Backurs, Victor Reis, Pei Zhou, Sihao Chen, Longqi Yang, Zinan Lin, Sergey Yekhanin, and Giulia Fanti. Struct-bench: A benchmark for differentially private structured text generation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [35] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. Livebench: A challenging, contamination-limited llm benchmark. arXiv preprint arXiv:2406.19314, 2025. [36] Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou. Self-evolved diverse data sampling for efficient instruction tuning. arXiv preprint arXiv:2311.08182, 2023. [37] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024. [38] Lang Xiong, Nishant Bhargava, Jianhang Hong, Jeremy Chang, Haihao Liu, Vasu Sharma, and Kevin Zhu. Probe-rewrite-evaluate: A workflow for reliable benchmarks and quantifying evaluation awareness. arXiv preprint arXiv:2509.00591, 2025. [39] Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382, 2025. [40] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. [41] John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36:23826–23854, 2023. [42] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. [43] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. [44] Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. Survey on evaluation of llm-based agents. arXiv preprint arXiv:2503.16416, 2025. [45] Zeyu Zhang, Guohao Li, Zhenchang Xing, Alexandros Apostolopoulos, Yu Lin Lee, and Liang Zheng. Gecko: A simulation environment to ground agent tool calls with stateful feedback for refinement. [46] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. [47] Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. arXiv preprint arXiv:2309.17167, 2023. [48] Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, et al. Establishing best practices for building rigorous agentic benchmarks. arXiv preprint arXiv:2507.02825, 2025. 13

[49] Kaijian Zou, Muhammad Khalifa, and Lu Wang. On many-shot in-context learning for long-context evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25605–25639, 2025.

14

A

Related Work

Robust benchmarks for interactive tool-use are necessary both for generalist agents (e.g., code generation, research assistance, and open-domain conversation [44, 2, 40, 29, 43, 24]) and domain-specific agents with narrowly scoped workflows (e.g., in finance, technology, and corporate services [26, 5, 20]). However, constructing and maintaining such benchmarks is costly and requirements evolve [22, 44, 8], motivating continuously updated and synthetic benchmark generation approaches [35, 47]. Evaluating synthetic benchmarks for LLMs. There have been several synthetic benchmark datasets for evaluating sequential, conversational interactions between a user and an LLM; a few papers have quantitatively evaluated the quality of these benchmarks [7, 18, 38, 19]. However, these lines of work do not tackle agent benchmarks, where evaluation is more complicated due to multi-step decision making, interaction with environments/tools, and the need to attribute failures to specific components of an agentic pipeline. Examples include [7, 18, 19], which study synthetic benchmarks for standard NLP tasks (e.g., reading comprehension, intent detection). They assess synthetic data via factors like difficulty (i.e., whether an agent has a similar completion rate on tasks in the real/synthetic data) and whether model rankings are preserved. SynAE generalizes these ideas to the multi-turn tool-calling setting in the fidelity evaluation of downstream tasks. Relatedly, [38] and [21] demonstrate that LLM behavior—both response content and trustworthiness—can differ between synthetic-benchmark settings and real-world deployment, motivating the need for realistic synthetic data. Evaluating synthetic benchmarks for tool-calling agents. To our knowledge, prior works on evaluating synthetic benchmarks for tool-calling agents have only considered single-turn benchmarks, in which each dataset instance consists of a single client query followed by a single agent response [30, 9]. This significantly simplifies evaluation, as it does not require the synthetic data to capture sequential dependencies among task instructions present in the real data; measuring the quality of these transitions is a major component of SynAE. Moreover, these prior works only evaluate the instructions (human inputs) in the synthetic data (e.g., naturalness, coherence), avoiding systematically evaluating the reference tool calls and outputs correspond to those instructions [30, 9]. [9] further evaluates synthetic samples by how much they help in-context learning, which serves as an indirect signal of data quality. Overall, these approaches do not provide quantitative metrics for systematically measuring the statistical and semantic properties of synthetic tasks directly, nor do they evaluate the associated tool calls and final outputs within task-completion trajectories. Recent work has also proposed qualitative checklists for practitioners to evaluate benchmarks. For example, [48] advocate for verifying task validity, i.e., each sample in the benchmark should have the correct ground truth, while [1] emphasize that a benchmark should contain realistic samples that test a diverse range of agent behaviors. However, these checklists typically requires practitioners to manually review samples, making benchmark creation and validation time-consuming and iterative [26], especially when practitioners replace or augment real datasets with synthetic samples [35, 47]. SynAE speeds up these workflows by providing fine-grained quantitative metrics that pinpoint which parts of the benchmark can be improved. We demonstrate how SynAE can help practitioners through a case study in §3.1.3. Evaluating multi-turn tool-calling agents Multi-turn tool-calling agents are typically evaluated using end-to-end task success on interactive benchmarks, where agents execute multi-step tool calls and are scored by state-based checkers that compare the final environment or database state to an annotated goal state [42, 16, 46, 41, 37, 11]. While these works primarily evaluate agents on a given benchmark, in contrast, we evaluate the benchmarks themselves. We assume text-based outputs in this work, but the SynAE framework could easily be extended to incorporate a state check as part of the output evaluation. Many benchmarks additionally enforce turn-level validity by jointly checking tool outputs and dialogue via state-based and response-based criteria across all turns [27, 14, 45]. Increasingly, evaluations also incorporate LLM-as-a-judge, using a strong judge model to determine pass/fail or preference-style win rates over full tool-use trajectories when multiple solutions are plausible [28, 25, 39, 17]. SynAE also employs an LLM-as-a-judge to assess tool calls and final outputs by jointly checking them against the instructions across turns in our downstream evaluation. However, this alone is insufficient for evaluating the quality of a synthetic benchmark, as it focuses only on end-to-end agent performance and does not capture the intrinsic properties of task instructions or 15

their associated tool calls and outputs. Hence, we introduce additional metrics that quantitatively measure the semantic and statistical properties of task instructions, tool-usage and planning patterns, and the quality and diversity of final outputs, providing richer diagnostic signals for developers. We demonstrate the value of these metrics in §3.

B

Prompts for LLM-as-a-Judge on Validity

B.1

Prompts for T1

B.1.1

Tool Call Evaluation

System Prompt: 1

You are an evaluator . You must answer with ONLY ’yes ’ or ’no ’. Never provide explanations or reasoning .

User Prompt: 1

Check the validity of the tool call sequence for an attractions recommendation assistant :

2 3 4

Conversation : { instr }

5 6 7

Tool call : { tool_call }

8 9 10 11

Evaluation rules : - Tool call must accomplish the goal of the conversation - Partial correctness = NO

12 13 14

Does the tool call correctly implement the conversation ? Answer ( yes / no ) :

B.1.2

Output Evaluation

System Prompt: 1

You are an evaluator . You must answer with ONLY ’yes ’ or ’no ’. Never provide explanations or reasoning .

User Prompt: 1

Check the validity of the output for an attractions recommendation assistant :

2 3 4

Conversation : { instr }

5 6 7

Output : { expected_output }

8 9 10 11

Evaluation rules : - Output must accomplish the goal of the conversation - Partial correctness = NO

12 13 14

Does the output correctly implement the conversation ? Answer ( yes / no ) :

16

B.2

Prompts for BFCL

B.2.1

Tool Call Evaluation

System Prompt: 1 2

You are an evaluator . You must answer with ONLY ’yes ’ or ’no ’. Never provide explanations or reasoning .

User Prompt: 1

Check the validity of the tool call sequence for a function calling assistant :

2 3 4

User Requests : { instr }

5 6 7

Tool calls : { exp ec te d_tool_call }

8 9 10 11

Evaluation rules : - Tool call must accomplish the goal of the conversation - Partial correctness = NO

12 13 14

Does the tool call correctly implement the conversation ? Answer ( yes / no ) :

B.3

Prompts for ACP

B.3.1

Output Evaluation

System Prompt: 1

You are an evaluator . You must answer with ONLY ’yes ’ or ’no ’. Never provide explanations or reasoning .

User Prompt: 1

Check the validity of the output :

2 3 4

Conversation : { instr }

5 6 7

Output : { expected_output }

8 9 10 11

Evaluation rules : - Output must accomplish the goal of the conversation - Partial correctness = NO

12 13 14

Does the output correctly implement the conversation ? Answer ( yes / no ) :

B.4

Agreement Rate with Human Annotation

We conduct human annotation on 100 randomly selected synthetic T1 samples to assess whether the generated tool calls and outputs correctly fulfill the given tasks. We then compare the human labels with judgments produced by an LLM-as-a-judge using Mistral-7B-Instruct. The LLM judge achieves an F1 score of 0.86, with Cohen’s κ as 0.61, indicating substantial agreement with human annotations. This suggests that the LLM-as-a-judge provides a reliable default validity checker when task-specific rule-based checkers are unavailable. 17

C

Prompts for LLM-as-a-Judge in Downstream Evaluation

C.1

Prompts for T1

C.1.1

Tool Call Evaluation

System Prompt: 1

You are an evaluator . You must answer with ONLY ’yes ’ or ’no ’. Never provide explanations or reasoning .

User Prompt: 1

Compare these tool call sequences for an attractions recommendation assistant :

2 3 4

Conversation : { instr }

5 6 7

Expected tool calls : { exp ec te d_tool_call }

8 9 10

Actual tool calls : { actual_tool_call }

11 12 13 14

15 16

Evaluation rules : - Actual must accomplish the same goal as expected - Semantic equivalence is OK ( e . g . , " OR " vs " Oregon " , reordered operations with same result ) - Partial correctness = NO - Different variable / cache names = OK if functionality identical

17 18 19

Does actual correctly implement the conversation based on expected ? Answer ( yes / no ) :

C.1.2

Output Evaluation

System Prompt: 1

You are an evaluator . You must answer with ONLY ’yes ’ or ’no ’. Never provide explanations or reasoning .

User Prompt: 1

Compare these outputs for an attractions recommendation assistant :

2 3 4

Conversation : { instr }

5 6 7

Expected output : { expected_output }

8 9 10

Actual output : { actual_output }

11 12 13 14 15 16

Evaluation rules : - Actual must convey same information and meaning as expected - Different wording is OK if content equivalent - Partial correctness = NO - Focus on semantic content , not syntax

17

18

18 19

Does actual correctly respond to the conversation based on expected ? Answer ( yes / no ) :

C.2

Prompts for BFCL

C.2.1

Tool Call Evaluation

System Prompt: 1 2

You are an evaluator . You must answer with ONLY ’yes ’ or ’no ’. Never provide explanations or reasoning .

User Prompt: 1

Compare these outputs for a function calling assistant :

2 3 4

User Requests : { instr }

5 6 7

Expected tool calls : { exp ec te d_tool_call }

8 9 10

Actual tool calls : { actual_tool_call }

11 12 13 14 15 16

Evaluation rules : - Actual must accomplish the same goal as expected - Semantic equivalence is OK ( e . g . , reordered operations with same result ) - Partial correctness = NO - Different variable names = OK if functionality identical

17 18 19

Does actual correctly implement the user requests based on expected ? Answer ( yes / no ) :

D

Computational Cost of SynAE Across Datasets

For the LLM-as-a-judge in both validity checks and downstream evaluations, we use Mistral-7B-Instruct. For the agents in downstream evaluations, we use gemma-3-1b-it, Qwen3-4B-Instruct, and Llama3.1-8B-Instruct across all datasets. • T1 (225 samples): 450 LLM calls for validity checking of tool calls and outputs; 1,350 LLM calls for downstream evaluation per agent. • BFCL (200 samples): 200 LLM calls for validity checking of tool calls; 800 LLM calls for downstream evaluation per agent. • ACP (260 samples): 260 LLM calls for validity checking; 0 LLM calls for downstream evaluation per agent, since agent responses can directly be compared with benchmark ground truth (answers are either ‘yes’ or ‘no’). All experiments use open-source LLMs. For reference, using GPT-5.4-mini instead would cost under $5 per dataset.

19

E

Prompts for LLM-Based Synthetic Data Generation Methods

E.1

Prompts for T1

E.1.1

Blank Filling

System Prompt: 1

2

3 4

5

6

You are a helpful conversation generator . When given a conversation with blanks ( underscores ) , fill them in naturally . IMPORTANT RULES : 1. The conversation MUST start with ’ assistant : ’ ( not ’ Assistant : ’ or any variation ) 2. Lines MUST alternate strictly between ’ user : ’ and ’ assistant : ’ 3. Each line must follow the format : ’ role : content ’ where role is either ’ user ’ or ’ assistant ’ 4. Output ONLY the completed conversation with no preamble , explanation , or extra text 5. Maintain the same number of conversation turns as the input

User Prompt: 1

Example input for fill in the blanks :

2 3

4 5 6 7

8

assistant : H_____ What ____ of attractions are you looking for ? Are you interested in _______ , a__ , or something else ? user : I ’ m interested in ___ and ____ attractions in __ . assistant : G_ has a lot to offer . Are you looking at specific ______ or re_____ ? user : Yeah , I ’ m thinking of visiting Fre___ and M______ . assistant : Both _____o and ____his have great A__ and S_____ attractions . Let me tell you about some of them . user : That sounds _____ .

9 10 11

12 13 14 15

16

Completed conversation for example input : assistant : Hello ! What kind of attractions are you looking for ? Are you interested in history , art , or something else ? user : I ’ m interested in Art and Scenic attractions in GA . assistant : GA has a lot to offer . Are you looking at specific cities or regions ? user : Yeah , I ’ m thinking of visiting Fresno and Memphis . assistant : Both Fresno and Memphis have great Art and Scenic attractions . Let me tell you about some of them . user : That sounds great .

17 18

Now fill in the blanks to complete this conversation :

19 20 21

{ masked_conv } Completed conversation :

E.1.2

In-Context Generation

System Prompt: Same as the system prompt in Section E.1.1. User Prompt: 1 2

Here are example conversations : { conv_list_str }

3 4 5 6

Generate 1 new similar conversation that follows the same structure . Do not include anything other than this conversation . Similar conversation :

20

E.2

Prompts for BFCL

E.2.1

Blank Filling

System Prompt: 1

You are a request completion assistant . Fill in blanks using ONLY the provided APIs and Resources . Output only the completed requests , one per line . Do not add explanations or extra text .

User Prompt: 1 2 3

Fill in the blanks ( underscores ) to complete the user requests . Example : { example_context }

4 5 6

Input with blanks : { example_masked }

7 8 9

Completed requests : { exa mple_completed }

10 11 12 13

Now fill in the blanks for this : APIs : { target_classes } Resources : { target_resources }

14 15 16

Input with blanks : { masked_conv_str }

17 18

Completed requests :

E.2.2

In-Context Generation

System Prompt: 1 2 3

You are a test case generator . Output each request as : Request 1: < request text > Request 2: < request text >

User Prompt: 1

{ examples_str_with_APIs_resources_requests }

2 3 4

5

Generate EXACTLY { n_turns } requests for the situation below . Requests can query information , perform operations , modify resources , or search / filter data . Keep requests realistic . Later requests should build on earlier ones .

6 7 8 9 10

ONLY refer to the APIs and Resources below in the requests . APIs : { target_classes } Resources : { target_resources } Requests :

E.3

Prompts for ACP

E.3.1

Blank Filling

System Prompt:

21

1

2

You are a question completion assistant . Fill in the blanks ( underscores ) to complete the question . Use ONLY entities and terms from the provided context . Output ONLY the completed question text ."

User Prompt: 1 2

Fill in the blanks to complete the question . Use ONLY entities from the context .

3 4 5

6 7 8

9

Example 1: Context : This is a ferry domain , where the task is to transport cars from their start to their goal locations , using a ferry . Each location is accessible by ferry from each other location . The cars can be debarked or boarded , and the ferry can carry only one car at a time . There are 3 locations and 10 cars , numbered consecutively . Currently , the ferry is at l1 , with the car c2 on board . The cars are at locations as follows : c6 , c3 , and c0 are at l2 ; c4 , c9 , and c7 are at l0 ; c1 , c8 , and c5 are at l1 . Group : a p p l ic ab le _ ac ti o ns _b oo l Answer : yes Masked question : Is the fol___ing action appl___able in this state : deb_rk the car c2 fr_m the ferry to loc_tion l1 ? Completed question : Is the following action applicable in this state : debark the car c2 from the ferry to location l1 ?

10 11 12

13 14 15

16

Example 2: Context : This is a ferry domain , where the task is to transport cars from their start to their goal locations , using a ferry . Each location is accessible by ferry from each other location . The cars can be debarked or boarded , and the ferry can carry only one car at a time . There are 3 locations and 10 cars , numbered consecutively . Currently , the ferry is at l1 location and it is empty . The cars are at locations as follows : c9 , c4 , and c6 are at l0 ; c0 , c8 , c1 , c7 , and c2 are at l1 ; c3 and c5 are at l2 . Group : progression_bool Answer : no Masked question : Will the f_ct " The ferry is emp__ " hold aft__ perf___ing the act_on " emb_rk the car c0 at loc_tion l1 on to the ferry " in the cur_ent state ? Completed question : Will the fact " The ferry is empty " hold after performing the action " embark the car c0 at location l1 on to the ferry " in the current state ?

17 18 19 20 21 22 23

Now complete this : Context : { target_context } Group : { target_group } Answer : { target_answer } Masked question : { masked_question } Completed question :"

E.3.2

In-Context Generation

System Prompt: 1 2

3

You are a test case generator for planning domain tasks . Given a context and examples , generate a single question that has the specified answer . Output ONLY the question text with no preamble or label .

User Prompt: 1

Examples :

22

2

{ examples_str_with_context_group_answer }

3 4 5

Generate a question of type ’{ target_group } ’ for the context below " such that the answer is ’{ target_answer } ’.

6 7

Context : { target_context }

8 9

f " Question :"

F

Detailed Evaluation Results

The evaluation results on T1 across synthetic data generation methods are summarized in Table 3, the results on BFCL are shown in Table 4 and Fig. 9, and the results on ACP are summarized in Table 5. Table 3: Synthetic data generation evaluation results on SynAE under T1. Instruction Eval

Tool Call Eval

Output Eval

Method

Parameter KND ↓

AM ↓

Vendi ↑

AD ↑

TUM ↓

3-Step Planning ↓

Vendi ↑

Blank Filling

p=0 p = 0.1 p = 0.3 p = 0.5 p = 0.7 p = 0.9 p=1

0 0.019 0.025 0.037 0.035 0.059 0.075

0 0.113 0.096 0.213 0.295 0.564 0.746

9.768 10.790 12.664 14.885 18.029 25.029 21.788

4.260 4.205 4.246 4.223 3.595 2.546 0.840

0 0.018 0.074 0.049 0.022 0.041 0.031

0 0.027 0.134 0.048 0.167 0.135 0.346

2.920 6.928 9.067 12.430 15.159 15.312 10.256

1 0.640 0.587 0.498 0.142 0.240 0.044

Oversampling

r=0 r = 0.1 r = 0.3 r = 0.5 r = 0.7 r = 0.9 r=1

0.004 0.012 0.038 0.060 0.087 0.110 0.122

0.051 0.073 0.229 0.351 0.511 0.658 0.750

8.966 9.200 6.865 4.755 2.842 1.520 1.000

3.979 4.062 3.423 2.657 1.675 0.560 0.000

0 0.009 0.023 0.031 0.058 0.059 0.094

0 0.085 0.086 0.079 0.043 0.127 0.238

6.719 6.843 7.282 6.488 6.126 5.869 5.621

k=0 k=1 (fixed) k=1 (random) k=3 (fixed) k=3 (random) k=5 (fixed) k=5 (random)

0.075

0.746

21.788

0.840

0.031

0.346

0.065

0.645

12.475

2.425

0.036

0.054

0.444

15.354

3.466

0.117

0.866

7.346

0.021

0.523

0.031 0.019

In-Context Generation

G

KNNKNNPrecision ↑ Recall ↑

Downstream Eval Vendi ↑

TDD ↓

RD ↑

1 0.933 0.813 0.791 0.796 0.693 0.164

20.085 24.667 26.354 26.590 41.878 36.059 30.941

0 0.052 0.090 0.056 0.147 0.160 0.033

1.0 1.0 1.0 1.0 0.5 1.0 0.5

0.994 0.984 0.953 0.984 0.971 0.882 0.992

0.923 0.818 0.693 0.724 0.631 0.489 0.004

24.309 22.385 20.983 15.497 11.506 15.686 5.986

0.095 0.048 0.117 0.053 0.156 0.165 0.183

1.0 1.0 0.5 1.0 1.0 0.5 1.0

10.256

0.044

0.164

30.941

0.033

0.5

0.040

12.035

0.067

0.311

38.490

0.076

0.5

0.017

0.008

12.200

0.027

0.729

13.259

0.126

1.0

2.817

0.011

0.016

6.929

0.311

0.271

19.037

0.062

0.5

10.496

4.168

0.038

0.034

7.096

0.062

0.853

35.427

0.121

0.5

0.875

9.129

3.692

0.005

0.070

7.534

0.369

0.524

24.447

0.124

0.5

0.460

10.378

4.305

0.010

0.085

6.547

0.471

0.707

25.269

0.138

1.0

Evaluation Results of Nvidia NeMo

To evaluate a more realistic synthetic dataset, we use NVIDIA NeMo, an industry-standard synthetic data tool (including for testing agents), to generate synthetic datasets using three backbone models, mistral-small-24b-instruct, nvidia-nemotron-nano-9b-v2 and Llama3.1-8B-Instruct, and evaluate them with SynAE on the T1 dataset. Our results in Table 6 are consistent with prior findings that the three models have similar performance on standard LLM benchmarks: SynAE reports similar fidelity scores for all. However, SynAE also shows that Nemotron and Llama exhibits greater diversity than Mistral. We use this observation to explore the distribution of the attraction type attribute in Table 7, where Nemotron and

23

Table 4: Synthetic data generation evaluation results on SynAE under BFCL. Instruction Eval

Tool Call Eval

Downstream Eval

Method

Parameter KND ↓

AM ↓

KNNKNNVendi ↑ Precision ↑ Recall ↑

Blank Filling

p=0 p = 0.1 p = 0.3 p = 0.5 p = 0.7 p = 0.9 p=1

0 0.087 0.079 0.087 0.105 0.163 0.124

0 0.665 1.030 1.320 1.470 1.445 0.695

22.153 26.683 28.295 30.630 32.895 37.555 24.867

1 0.715 0.675 0.640 0.430 0.225 0.245

1 0.900 0.905 0.920 0.910 0.805 0.290

0 0.452 0.388 0.438 0.656 0.471 0.601

0 6.805 5.445 5.895 16.335 6.335 6.515

114.256 118.149 109.984 88.581 67.985 65.832 69.318

0 0.684 0.684 0.709 0.654 0.760 0.747

0 0.595 0.632 0.623 0.567 0.583 0.585

0.146 0.157 0.163 0.175 0.215 0.250 0.150

1.0 0.5 0.5 0.5 0.5 0.5 0.5

Oversampling

r=0 r = 0.1 r = 0.3 r = 0.5 r = 0.7 r = 0.9 r=1

0.020 0.024 0.033 0.053 0.080 0.106 0.118

0.110 0.130 0.305 0.525 0.725 0.955 1.060

22.058 23.380 14.432 8.099 3.804 1.654 1.000

1.000 0.995 0.995 0.995 0.995 0.995 1.000

0.875 0.980 0.965 0.920 0.910 0.950 0.005

0.112 0.150 0.384 0.583 0.739 0.850 0.912

0.285 0.485 1.395 2.175 2.985 3.955 4.290

52.287 59.895 30.895 13.555 5.361 1.824 1.000

0.156 0.122 0.261 0.371 0.466 0.557 0.627

0.148 0.085 0.180 0.258 0.315 0.368 0.411

0.005 0.018 0.022 0.048 0.073 0.093 0.108

1.0 1.0 0.5 0.5 0.5 0.5 0.5

k=0 k=1 (fixed) k=1 (random) k=3 (fixed) k=3 (random) k=5 (fixed) k=5 (random)

0.124

0.695

24.867

0.245

0.290

0.601

6.515

69.318

0.747

0.585

0.150

0.5

0.103

0.330

15.082

0.450

0.175

0.613

4.820

66.247

0.752

0.586

0.138

0.5

0.129

0.570

29.312

0.370

0.760

0.571

5.045

53.051

0.698

0.596

0.182

0.5

0.079

0.375

20.832

0.385

0.235

0.582

10.770

91.796

0.644

0.571

0.135

0.5

0.115

0.515

29.471

0.500

0.765

0.557

7.430

65.258

0.699

0.574

0.162

0.5

0.081

0.365

24.618

0.405

0.330

0.457

5.360

94.033

0.691

0.621

0.122

0.5

0.064

0.312

27.739

0.505

0.720

0.467

5.685

56.282

0.745

0.577

0.152

0.5

In-Context Generation

$0,QVWUXFWLRQ7RNHQ/HQJWK

.1' LQVWUXFWLRQLQVWUXFWLRQ

TUM ↓

TCNM ↓

Vendi ↑

2-Step Planning ↓

3-Step Planning ↓

TDD ↓

RD ↑

$0,QVWUXFWLRQ7RNHQ/HQJWK

.1' LQVWUXFWLRQLQVWUXFWLRQ

$01XPEHURI,QVWUXFWLRQV

$01XPEHURI,QVWUXFWLRQV

.113UHFLVLRQ

.113UHFLVLRQ

Instruction Eval

Instruction Eval

),' 

.115HFDOO











780 6WHS3ODQQLQJ

S 

.115HFDOO

Tool Call Eval

S 











780 6WHS3ODQQLQJ

7&10

6WHS3ODQQLQJ

),' 

!"#$

6WHS3ODQQLQJ

S 

U 

Tool Call Eval

!"#$%

%&'()

(a) Blank Filling. As p increases, data utility degrades (b) Oversampling. As r increases, data utility degrades across nearly all metrics on fiedelity. across nearly all metrics on fidelity.

Figure 8: Fidelity of Blank Filling and Oversampling on the BFCL dataset.

24

$0,QVWUXFWLRQ7RNHQ/HQJWK

.1' LQVWUXFWLRQLQVWUXFWLRQ

$0,QVWUXFWLRQ7RNHQ/HQJWK

.1' LQVWUXFWLRQLQVWUXFWLRQ

$01XPEHURI,QVWUXFWLRQV

$01XPEHURI,QVWUXFWLRQV

.113UHFLVLRQ

.113UHFLVLRQ

Instruction Eval

Instruction Eval

),' 

.115HFDOO











780 6WHS3ODQQLQJ

N 

.115HFDOO

Tool Call Eval

N  IL[HG











780 6WHS3ODQQLQJ

7&10

6WHS3ODQQLQJ

),' 

7&10

6WHS3ODQQLQJ

N  IL[HG

N  IL[HG

Tool Call Eval

N  UDQGRP

(a) In-Context Generation with different k. Increasing k (b) In-Context Generation with k = 5. Compared to fixed does not consistently improve data utility across several in-context examples, randomized in-context examples metrics, including KNN-Recall, KNN-Precision, and AM. across generations lead to higher data utility under both instruction evaluation metrics.

Figure 9: Fidelity of In-Context Generation under BFCL with fixed and randomly sampled in-context examples across generations.

Table 5: Synthetic data generation evaluation results on SynAE under ACP. Instruction Eval

Output Eval KNNKNNPrecision ↑ Recall ↑

Method

Parameter KND ↓

AM ↓

FID ↓

Vendi ↑

Blank Filling

p=0 p = 0.1 p = 0.3 p = 0.5 p = 0.7 p = 0.9 p=1

0.011 0.049 0.081 0.150 0.157 0.176 0.203

3.319 10.981 8.650 9.142 13.346 18.992 21.746

0.000 0.000 0.000 0.000 0.000 0.000 3.981

10.822 10.822 10.822 10.822 10.822 10.822 10.447

1.000 1.000 1.000 1.000 1.000 0.996 0.996

Oversampling

r=0 r = 0.1 r = 0.3 r = 0.5 r = 0.7 r = 0.9 r=1

0.007 0.018 0.060 0.094 0.135 0.170 0.188

10.769 10.327 31.735 52.988 80.908 99.827 114.935

5.195 5.235 33.454 83.330 150.814 263.393 382.144

10.386 10.506 8.430 5.450 3.177 1.582 1.000

k=0 k=1 (fixed) k=1 (random) k=3 (fixed) k=3 (random) k=5 (fixed) k=5 (random)

0.203

21.746

3.981

0.184

16.085

0.213

In-Context Generation

Downstream Eval VR↑

TDD ↓

RD ↑

1.000 1.000 1.000 1.000 1.000 1.000 0.985

0.627 0.638 0.662 0.642 0.619 0.600 0.538

0.042 0.088 0.106 0.101 0.103 0.126 0.142

0.5 0.5 0.5 0 0.5 0 0.5

0.988 0.996 0.992 0.981 0.977 0.969 0.969

0.988 1.000 0.985 0.992 0.962 0.962 0.019

0.827 0.731 0.638 0.565 0.438 0.323 0.242

0.044 0.036 0.133 0.206 0.301 0.403 0.455

1.0 1.0 1.0 1.0 1.0 1.0 1.0

10.447

0.996

0.985

0.538

0.142

0.5

3.981

10.447

0.981

0.985

0.515

0.191

0.5

30.408

2.736

10.505

0.992

0.977

0.538

0.160

0.5

0.225

17.681

4.096

10.435

0.992

0.985

0.558

0.142

0

0.186

15.085

3.760

10.507

0.996

1.000

0.581

0.127

0.5

0.146

15.181

4.760

10.362

0.988

0.973

0.612

0.109

0.5

0.130

11.792

2.464

10.773

0.992

0.977

0.550

0.150

0.5

25

Llama show greater variability than Mistral. This illustrates that SynAE can uncover non-obvious properties of synthetic agent trajectories. We also observe from Table 6 that, for both backbone models, increasing the generation temperature lowers KNN-Precision and increases KNN-Recall (coverage). This is intuitive, since higher temperatures introduce more randomness into generation and thus increase coverage. To show that SynAE captures real evaluation outcomes, we compare agent performance on Nemotron datasets generated with temperatures 0.1 and 0.5 in Table 8. Relative to tmp = 0.1, agent performance on Nemotron tmp = 0.5 deviates more from the real dataset, which is reflected by the larger TDD and lower RD in Table 6. This indicates larger performance discrepancies and less consistent rankings. Table 6: NeMo evaluation results on SynAE under T1. Validity

Fidelity: Instruction Eval

VR: output ↑

KND: (instruction, response) ↓

Mistral

tmp = 0.1 tmp = 0.3 tmp = 0.5 tmp = 0.7 tmp = 0.9

0.798 0.831 0.898 0.811 0.893

Nemotron

tmp = 0.1 tmp = 0.3 tmp = 0.5 tmp = 0.7 tmp = 0.9

Llama

tmp = 0.1 tmp = 0.3 tmp = 0.5 tmp = 0.7 tmp = 0.9

Model

Temperature

AM: city ↓

Fidelity: Tool Call Eval 3-step planning ↓

Fidelity: Output Eval

Fidelity: Downstream Eval

KNNprecision ↑

KNNrecall ↑

TDD ↓

RD ↑

Vendi score ↑

AD ↑

0.131 0.149 0.135 0.143 0.113

0.828 0.869 0.805 0.932 0.715

0.540 0.522 0.626 0.519 0.492

0.382 0.302 0.222 0.227 0.013

0.089 0.067 0.338 0.369 0.533

0.110 0.044 0.011 0.096 0.274

1.0 1.0 1.0 1.0 1.0

4.692 4.759 5.472 6.305 7.472

2.449 2.592 2.899 2.718 3.077

0.764 0.787 0.820 0.778 0.856

0.112 0.117 0.115 0.106 0.104

0.903 0.843 0.806 0.819 0.780

0.551 0.404 0.482 0.520 0.476

0.209 0.244 0.237 0.031 0.049

0.271 0.298 0.302 0.902 0.667

0.129 0.141 0.323 0.203 0.397

1.0 1.0 0.5 0.5 1.0

5.145 6.013 7.619 8.246 9.524

2.347 2.676 3.125 3.160 3.487

0.809 0.832 0.881 0.804 0.859

0.123 0.142 0.129 0.138 0.135

0.789 0.765 0.795 0.798 0.770

0.729 0.817 0.571 0.497 0.660

0.120 0.037 0.259 0.286 0.100

0.662 0.511 0.160 0.236 0.427

0.145 0.161 0.135 0.157 0.154

0.5 1.0 0.5 0.5 0.5

7.480 8.299 9.236 8.788 9.397

3.184 3.187 3.257 3.408 3.481

Diversity

Table 7: Distribution of attraction types. Model mistral-small-24b-instruct (tmp = 0.7) nvidia-nemotron-nano-9b-v2 (tmp = 0.7) Llama3.1-8B-Instruct (tmp = 0.7)

Art

Historical

Cultural

Scenic

Other

35.73% 25.76% 25.47%

20.62% 23.80% 24.07%

9.08% 13.21% 12.84%

5.76% 12.77% 11.01%

28.80% 24.46% 26.61%

Table 8: Agent performance on real and synthetic datasets.

H

Dataset

Agent 1

Agent 2

Agent 3

real dataset nvidia-nemotron-nano-9b-v2 (tmp = 0.1) nvidia-nemotron-nano-9b-v2 (tmp = 0.5)

1.0000 ± 0 1.0000 ± 0 1.0000 ± 0

0.6356 ± 0.0098 0.5644 ± 0.0102 0.3200 ± 0.0131

0.9600 ± 0.0067 0.6444 ± 0.0049 0.2967 ± 0.0081

No single baseline metric can fully characterize synthetic data performance

We report several simple baselines for evaluating synthetic instructions and responses generated by In-Context Generation with different numbers of in-context examples, k. Specifically, we consider vocabulary-overlap F1, total variation (TV) distance between unigram distributions, TV distance on length distributions, Fréchet Inception Distance (FID). The results are shown in Table 9. 26

Table 9: Simple baseline metrics for synthetic instructions and responses generated by In-Context Generation with different numbers of in-context examples k. Metric

k=0

k=1

k=3

k=5

Vocabulary overlap (F1) ↑ Unigram distribution (TV) ↓ Length distribution ↓ FID ↓

0.3856 0.6132 9.2444 204.0479

0.4788 0.5281 8.7733 166.2299

0.6030 0.3891 8.7422 131.6828

0.6809 0.3111 8.7022 107.5157

Taken in isolation, these baselines may suggest that increasing k monotonically improves synthetic data quality, since lexical overlap increases while several distributional distances decrease. However, this conclusion is misleading. As shown in Fig. 7a, using more in-context examples does not necessarily improve overall generation quality. This is consistent with prior work showing that in-context learning is highly sensitive to the choice of demonstrations, and that adding more demonstrations does not uniformly improve performance across tasks [15, 49]. More broadly, these simple baselines are useful as sanity checks, but they mainly capture surface-form similarity or coarse marginal statistics, and therefore can miss higher-order aspects of synthetic data quality such as validity and diversity.

I

Attribute Distribution of the Skewed and Augmented Datasets

We report the attraction-type distributions of the skewed and augmented datasets in Table 10. The skewed dataset contains four under-represented attraction types, defined as those with proportions below 5%, whereas the augmented datasets produced by Relabeling or NVIDIA NeMo contain no under-represented attraction types. Table 10: Distribution of attraction types. Red indicates under-represented attraction types. Dataset Skewed Real Data Relabeling NVIDIA NeMo (Llama) NVIDIA NeMo (Nemotron) NVIDIA NeMo (GPT)

Art

Historical

Cultural

Scenic

Touristy

Culinary

Guided

Social

Sporting

16.67% 12.60% 27.35% 19.56% 17.74%

15.56% 11.02% 13.76% 14.92% 14.07%

2.22% 9.45% 11.59% 12.83% 12.74%

21.11% 16.54% 12.13% 11.28% 12.74%

17.78% 12.60% 9.42% 11.80% 12.07%

4.44% 11.02% 7.24% 9.74% 9.40%

4.96% 9.45% 6.70% 8.19% 6.07%

12.82% 8.66% 6.70% 6.13% 7.40%

4.44% 8.66% 5.07% 5.55% 7.74%

27

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500