Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents Noam Koren* Noam.Koren1@ibm.com

Roy Bar-Haim* Abigail Goldsteen IBM Research roybar@il.ibm.com abigailt@il.ibm.com

arXiv:2608.06329v1 [cs.CL] 6 Aug 2026

Abstract

has motivated growing interest in automatically generated benchmarks (Li et al., 2023; Qin et al., 2024; Levi and Kadar, 2025), which forgo manual curation and its implicit quality control. This shift raises a fundamental question: how should benchmark quality itself be evaluated? Synthetic benchmarks may contain inconsistent tasks, simplistic scenarios, or limited coverage of the capabilities and constraints they aim to test. Prior work shows that synthetic evaluation data can differ from human-curated data in representativeness, difficulty, and induced model rankings (Maheshwari et al., 2024). Benchmark quality issues are not limited to synthetic data: a review of 445 benchmarks found pervasive construct-validity problems even in widely used, manually curated benchmarks (Bean et al., 2025), and τ -bench, the basis for the benchmarks studied in this work, has also been shown to contain flawed and ambiguous tasks (Cuadron et al., 2025).12 Although recent studies have begun to assess benchmark quality more systematically (Qian et al., 2026; Iskander et al., 2024), policy-grounded evaluation of conversational-agent benchmarks remains underexplored. Undetected benchmark flaws may therefore lead to unreliable conclusions about agent capabilities. We address this gap by proposing reference-free metrics for evaluating conversational benchmarks with LLM judges (Zheng et al., 2023). The metrics capture multiple quality dimensions, including consistency, complexity, and coverage, without requiring comparison to existing datasets. They assess both task coherence and alignment with the domain policy that defines the intended agent behavior. Beyond aggregate scores, they identify concrete issues, such as inconsistent tasks and gaps

Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations. We introduce a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy coverage, while providing actionable diagnostics of weaknesses. We validate the framework by demonstrating agreement with independent human annotations and by evaluating benchmarks generated by LLMs of varying capabilities, as well as benchmarks subjected to controlled quality-degrading perturbations. Across domains and judge models, the proposed metrics consistently distinguish between benchmark quality levels. We further demonstrate the framework’s applicability to manually curated benchmarks. Our framework offers a practical approach for evaluating synthetic and manually curated conversational-agent benchmarks.

1

Introduction

Task-oriented conversational agents are increasingly deployed in customer support, travel, and enterprise workflows (Mohammadi et al., 2025). Unlike static question-answering systems, they must interact over multiple turns, follow domain-specific policies, operate in real-world environments, and use external tools. As their autonomy and complexity grow, reliable evaluation becomes increasingly important and challenging (Yehudai et al., 2026). A common approach is to evaluate agents using curated benchmarks with realistic tasks, user requests, environment states, and expected outcomes (Yao et al., 2024; Barres et al., 2025). However, manual construction is costly, requiring domain expertise, scenario design, tool and database simulation, and specification of expected behavior. This

1 3

τ -bench, which we use for evaluation in §6, fixes issues found in earlier versions of τ -bench. 2 Sierra Blog also documents specific task fixes: https: //taubench.com/blog/tau3-task-fixes.html.

* Equal contribution.

1

in policy coverage, providing actionable diagnostics for benchmark developers. We demonstrate empirically that our metrics provide valuable insights across both synthetic and manually-created benchmarks. A key challenge is validating that the metrics reflect meaningful quality differences. We construct benchmark suites with expected quality variation using two complementary approaches: generating benchmarks with LLMs of different capabilities, used as a proxy for expected quality, through the I N TELL AGENT pipeline (Levi and Kadar, 2025), and applying controlled perturbations that deliberately degrade benchmark quality. Across both settings, the metrics consistently distinguish quality levels across domains and LLM judges. We further validate the LLM-judge scores against independent human annotations, demonstrating medium-to-strong, statistically significant correlation between the two. Our main contributions are: 1. We identify and study the problem of evaluating conversational-agent benchmarks. 2. We propose multidimensional metrics that assess consistency, complexity, and coverage with respect to the domain policy. 3. We validate the metrics using benchmarks generated by different LLMs, controlled qualitydegrading perturbations, and human evaluations. 4. We demonstrate the metrics on a manually curated benchmark (τ 3 -bench).

2

domain also includes a policy that specifies the rules, constraints, and procedures governing agent behavior. The benchmark is designed to assess and evaluate the behaviors required by this policy. Within each domain, a benchmark contains a collection of test cases, referred to as tasks. Each task provides the information required to simulate the interaction and evaluate the agent trajectory. This includes a description of the user scenario, an initial database state, and expected agent behavior, including adherence to the domain policy, and, when available, a sequence of tool calls.

3

Benchmark Metrics

This work focuses on evaluating the conversational benchmarks used to test agents, rather than the agents themselves. While agent evaluation measures an agent’s performance on a given benchmark, benchmark evaluation assesses whether the benchmark is coherent, challenging, and representative of the behaviors defined by its domain policy. We introduce four LLM-judge-based metrics that evaluate benchmark consistency (§3.1), complexity, and coverage (§3.2). Together, these metrics provide a comprehensive view of benchmark quality and difficulty. Figure 1 provides an overview of the benchmark evaluation process. Notation. Let B = (P, t1 , . . . , tn ) denote a benchmark consisting of a policy document P and a set of n tasks. Each task t ∈ B represents a user scenario that the agent must handle in accordance with P and is defined as

Benchmarks for Task-Oriented Conversational Agents

t = (dt , et , it ), Recent task-oriented conversational-agent benchmarks, including τ -B ENCH, τ 2 -B ENCH, and I N TELL AGENT (Yao et al., 2024; Barres et al., 2025; Levi and Kadar, 2025), evaluate agents through dynamic interactions with an LLM-based user simulator. The agent must fulfill user requests by combining natural-language dialogue with tool calls that access or update organizational information, while adhering to domain-specific policies. Our work assumes this benchmark structure and uses I NTELL AGENT for benchmark generation. A benchmark typically focuses on a specific domain, such as airline or retail, which defines the environment in which the agent operates. This environment comprises a set of available tools and an associated database containing relevant information, such as users, products, or orders. Each

(1)

where dt is the task description, including the user request; et specifies the expected agent behavior and should be consistent with the policy P ; and it is the initial database state. For any metric m, let Sm (B) denote the score assigned to benchmark B. Figure 2 illustrates a task and its components: the description dt , expected behavior et , and initial state it , which is omitted for brevity. 3.1

Consistency Metrics

Consistency measures whether the task description, domain policy, and expected behavior are aligned and free of contradictions. This helps identify and filter low-quality, ambiguous, or inconsistent tasks. We define consistency through LLM-judged alignment between task components. Let Jm (·) 2

! , + * ) ( ' & % $ # " , + * ) ( ' & % $ # " ! , + * ) ( ' & % $ # " ! Benchmarks

< ; : 9 8 7 6 5 4 3 2 1 Benchmarks Benchmarks Benchmarks

,Comparison 0 / . & Insights

* Metrics ) ( ' & % $ # " !

" Task Consistency !

: Task Complexity 9 8

? Benchmark Coverage

Description ⟷ Expected Behavior Policy ⟷ Expected Behavior

> Policy Viola*ons Count = < ;

> Policy Viola*ons Coverage = < ;

Figure 1: Benchmark evaluation process

Task #0 Description: User asks to modify reservation RES12345, asks about adding travel insurance, and later decides to cancel the reservation and ask about the refund. Expected Behavior: The chatbot should request the user ID, explain that travel insurance cannot be added during modification, ask for the cancellation reason, and state that the refund will be sent to the original payment method within 5–7 business days. Initial state: <...>

3.2

A key aspect of conversational-agent evaluation is testing whether the agent correctly identifies and handles requests that conflict with domain policies. For example, under the policy “The user may modify passenger details but not the number of passengers,” a task may ask the agent to remove a passenger from an existing reservation. A high-quality benchmark should include broad and systematic coverage of such policy-violating scenarios. A high-quality benchmark should also test compliant (“positive”) scenarios, not only violations. We focus on violations because they are more discriminative: a probe that only checks whether a policy item is relevant to a scenario would yield near-complete coverage for almost any benchmark, since nearly every task engages most policy items, causing such a signal to saturate. Requiring a task to actively violate a specific policy item is a much harder condition, and is therefore more informative. Let LP = (p1 , . . . , pN ) denote a list of policy items, represented as annotated spans in the policy document P . Let Jviolate (·) be an LLM judge that identifies the policy items violated by a task based on its description and initial database state. We denote this set by

Figure 2: Benchmark task t = (dt , et , it ), showing its description, expected behavior, and initial state.

denote the LLM judge for metric m. Given the task components, the judge assigns a score from 1 to 10, with higher scores indicating stronger alignment. Description–Expected Behavior Alignment. This metric measures whether the expected behavior is consistent with the task description and adequately addresses the user’s intended goals. Mdesc (t) = Jdesc (dt , et ).

(2)

Policy–Expected Behavior Alignment. This metric measures whether the expected behavior complies with the domain policy for the described scenario. It penalizes unsupported actions and omissions of policy-required steps. Formally, Mpol (t) = Jpol (dt , et , P ).

Complexity and Coverage Metrics

V (t) = Jviolate (P, LP , t) = {k : (dt , it ) violates pk }.

(3)

(5)

Let T (pk ) be the set of tasks that violate policy item pk :

For each m ∈ pol, desc, the benchmark-level score is 1X Sm (B) = Mm (t). (4) n

T (pk ) = {t : k ∈ V (t)}

t∈B

(6)

We define two benchmark-level metrics. The Policy Violations per Task score measures task complexity as the average number of policy viola-

Higher scores indicate greater consistency. The LLM-judge prompts are provided in App. A. 3

tions identified per task: Sv_count (B) =

1 X |V (t)|. |B|

sampled through a random walk over the graph and used to generate the task description and expected behavior. Finally, the scenario is augmented with symbolic representations and instantiated entities, such as a user or reservation, and a corresponding initial database state is constructed. By default, I NTELL AGENT uses GPT-4o throughout the generation pipeline3 . To create benchmarks of varying quality, we replace GPT4o in selected stages with LLMs spanning a broad range of capabilities. We use generator capability as a proxy for expected benchmark quality, a plausible assumption widely used in prior work: stronger models generally produce more accurate and coherent outputs, and are often used to generate supervision for weaker models (Grattafiori et al., 2024; Cui et al., 2024; Bubeck et al., 2023; Kim et al., 2023). While weaker models can extract policies and generate task descriptions and expected behaviors, the final construction stages require a stronger model to complete reliably. We therefore retain GPT-4o for these stages in all configurations. Each generated event is converted into a task t with components dt , et , and it , as defined in Eq. 1. Figure 3 illustrates the synthetic benchmark-generation process.

(7)

t∈B

The Policy Violations Coverage score measures the fraction of policy items violated by at least K tasks: Sv_cov (B) =

1 |{k : |T (pk )| ≥ K}| N

(8)

We set K = 3 in our experiments. The policy items were extracted semiautomatically. An initial set of policy spans was generated using T OOL G UARD (Zwerdling et al., 2025) and subsequently reviewed and revised by one of the authors. The LLM judge prompt is provided in App. B.

4

Metric Evaluation Benchmarks

To assess how well the proposed metrics distinguish quality differences among benchmarks, we construct a sequence of benchmarks of varying quality within the same domain – a nontrivial challenge in itself. We consider two approaches for generating such test benchmarks. First, we generate multiple benchmarks using a synthetic benchmark-generation pipeline. In each case, we vary selected LLM components while keeping the others fixed, using LLMs that span a range of capabilities (§4.1). Second, we apply controlled perturbations to a given benchmark to degrade its quality (§4.2). The two approaches are complementary: generating benchmarks with LLMs of varying capabilities yields more realistic benchmarks and finergrained quality differences, while the perturbationbased approach provides greater control over the differences in quality between the generated datasets. 4.1

4.2

Perturbed Benchmarks

Building on the synthetic benchmarks introduced in §4.1, we construct perturbed benchmarks by applying controlled perturbations to induce degradation in quality. We then evaluate whether the metric scores decrease in response. We apply the following perturbations to each synthetic benchmark. Expected Behavior Swapping. To break the alignment between the task description and the expected behavior, we swap the expected behavior in 20%, 40%, 60%, and 80% of the tasks, while leaving the remainder of the benchmark unchanged. A useful metric should decrease as the perturbation rate increases.

Synthetic Benchmark Generation

We construct synthetic benchmarks for each target domain using the I NTELL AGENT framework (Levi and Kadar, 2025). Given a domain policy and database schema, I NTELL AGENT first decomposes the policy into flows, such as booking a flight or canceling a reservation, and extracts the policies relevant to each flow. It then builds a policy graph whose nodes represent policy items and whose edges model their likelihood of co-occurring in the same task. For each task, a subset of policies is

Cross-Domain Policy Swapping. To evaluate policy-related metrics, we replace the policy of one domain with that of another domain and evaluate tasks under the mismatched policy. For example, airline tasks are evaluated against the retail policy, and vice versa. A useful policy metric should 3

4

Edge scoring uses GPT-4o-mini.

🌐 Domain

?IntellAgent > =

1 Generated < ; : 9 8 7 6 5 4 3 2 Benchmark

Specifica9on

Generator models

Tasks

3Policy (τ-bench) 6 5 4 🗄 DB Schema

GPT-5.4, Claude-4.5, Llama-70B, Llama-8B, Llama-1B

!Descrip(on ) ( ' & % $ # " 2Expected Behavior 1 0 / . , + * 🗄 Database state

Figure 3: Benchmark generation setup

5.2

assign substantially lower scores under this perturbation.

5

Metric Evaluation Experiments

5.1

Experimental Setup

Benchmark Ordering Across Generators

Figure 4 shows substantial score gaps between benchmarks, with metrics consistently assigning higher scores to benchmarks generated by stronger models (e.g., GPT-5.4 and Claude-4.5-Sonnet), across metrics, domains, and judges. The Benchmark Ordering Scores, summarized in Table 2 (App. D), quantify this observation: all metrics achieve perfect ordering (1.0) in the Airline domain, and a mean of 0.92 in Retail. Policy Violations Count achieves perfect ordering across all judges and domains, and the overall mean ordering score is slightly higher for GPT-5.4 and ClaudeSonnet-4.5 than for Gemini-2-Flash (0.97 vs. 0.94). These results provide strong evidence that our benchmark-quality metrics reliably capture quality differences among synthetic benchmarks.

We use I NTELL AGENT to generate five benchmarks for each of the two τ -B ENCH domains, Airline and Retail. The benchmarks are generated using GPT-5.4, Claude-4.5-Sonnet, Llama-3.3-70B, Llama-3.1-8B, and Llama-3.2-1B. All configurations use the same generation pipeline (§4.1) and differ only in the LLM used during the selected generation stages. Each Airline benchmark contains 100 tasks, and each Retail benchmark contains 95, yielding 975 synthetic tasks in total. We evaluate the synthetic benchmarks using three LLM judges: GPT-5.4, Claude-4.5-Sonnet, and Gemini-2-Flash. To quantify how well a metric induces a qualitybased ordering over benchmarks, we define a metametric, the Benchmark Ordering Score, applied to each of our four benchmark metrics. We partition the synthetic benchmarks in each domain into three tiers according to the capabilities of their generating LLMs: • Top: GPT-5.4 and Claude-4.5-Sonnet • Medium: Llama-3.3-70B and Llama-3.1-8B • Small: Llama-3.2-1B We expect top-tier benchmarks to receive the highest metric scores, with medium-tier benchmarks scoring lower than top-tier ones but higher than small-tier benchmarks. This partial order induces a set of eight pairwise constraints that a metric score S(·) should satisfy, e.g., S(BGPT-5.4 ) > S(BLlama-70B ), S(BGPT-5.4 ) > S(BLlama-1B ), and S(BLlama-8B ) > S(BLlama-1B ). We define the Benchmark Ordering Score of a metric as the fraction of constraints satisfied. A score of 1.0 indicates that all pairwise order constraints are satisfied.

5.3 Detecting Controlled Quality Degradation We next evaluate the sensitivity of the proposed metrics to the controlled perturbations defined in §4.2, using Gemini-2-Flash as the LLM judge. Expected Behavior Swapping. We apply this perturbation to the Description–Expected Behavior Alignment metric, which measures task-level consistency. Figure 5 shows that the metric score decreases consistently as the proportion of tasks with swapped expected behaviors increases. This pattern holds across both domains and all benchmark generators, demonstrating that the metric is sensitive to controlled degradation in task coherence. Cross-Domain Policy Swapping. We apply this perturbation to the three policy-grounded metrics: Policy–Expected Behavior Alignment, Policy Violations Count, and Policy Violations Coverage. As shown in Figure 6, all three metrics decrease substantially when tasks are evaluated against a policy from the other domain. The consistent de5

Description–Expected Behavior Alignment Retail

8 6 4 2 0

Score

Score

Airline

GPT-5.4

Claude-4.5

8 6 4 2 0 GPT-5.4

Gemini-2

Claude-4.5

Gemini-2

Judge Model

Judge Model Policy–Expected Behavior Alignment

Retail

Airline 10

Score

Score

10 5

5

0

0 GPT-5.4

Claude-4.5

Gemini-2

GPT-5.4

Judge Model

Claude-4.5

Gemini-2

Judge Model Policy Violations Count Retail

2

Score

Score

Airline

1

2 1 0

0 GPT-5.4

Claude-4.5

GPT-5.4

Gemini-2

Claude-4.5

Gemini-2

Judge Model

Judge Model Policy Violations Coverage

Retail

Airline Score

Score

0.6 0.4 0.2 0 GPT-5.4

Claude-4.5

Gemini-2

Judge Model GPT-5.4

0.6 0.4 0.2 0 GPT-5.4

Benchmark Generator Claude-4.5

Llama-70B

Claude-4.5

Gemini-2

Judge Model

Llama-8B

Llama-1B

Figure 4: Benchmark quality metrics across domains, generated benchmarks, and judge models. Colors denote the benchmark generator model, while groups correspond to the judge model. We report Description–Expected Behavior Alignment, Policy–Expected Behavior Alignment, and Policy Violations for the Airline and Retail domains, as judged by GPT-5.4, Claude-Sonnet-4.5, and Gemini-2-Flash. Description–Expected Behavior Alignment Retail Airline 10

Score

Score

10 8 6 4

8 6 4

0

20

40

60

80

0

Perturbed tasks (%) GPT-5.4

Claude-4.5

20

40

60

80

Perturbed tasks (%) Llama-70B

Llama-8B

Llama-1B

Figure 5: Mean Description–Expected Behavior Alignment-metric scores assigned by Gemini under increasing perturbation rates, where expected behaviors are swapped in 0%, 20%, 40%, 60%, and 80% of the tasks.

cline across domains and benchmark generators shows that these metrics detect mismatches be-

tween benchmark tasks and their associated domain policy.

6

0.6

8

1.5

0.4

Score

Score

6

Score

2

1

Score

Policy Violations Coverage Airline Retail

Policy Violations Count Airline Retail

10

Score

Score

Policy–Expected Behavior Alignment Airline Retail

0.2

0.5 4 0 Orig. Swapped

Orig. Swapped

GPT-5.4

0 Orig. Swapped

Claude-4.5

Orig. Swapped

Llama-70B

Llama-8B

Orig. Swapped

Orig. Swapped

Llama-1B

Figure 6: Cross-domain perturbation effects on metric scores, comparing original benchmarks to a “corrupted” setting in which airline and retail policies are swapped. Metric scores are computed using Gemini-2 as the judge.

5.4

Human Validation

patterns, and then manually verify and refine the resulting categories. The analysis identifies five recurring issues in the generated expected behaviors: 1. Hallucinated workflow stages or unsupported actions; 2. Missing required information collection; 3. Missing explicit confirmation before an action; 4. Incorrect compensation handling; and 5. Forbidden actions. These categories demonstrate how the metric can distinguish among different benchmark weaknesses rather than providing only an aggregate score. App. C presents representative examples of each issue type.

As an independent validation of the LLM-based metrics, we conducted a human evaluation study. A human annotator rated 50 tasks, sampled across benchmarks generated by all five models, along the three task-level measures in our framework: Description–Expected Behavior Alignment, Policy– Expected Behavior Alignment, and Policy Violations Count. We computed Kendall’s τb correlation (Kendall, 1945) between the human ratings and the scores assigned by each LLM judge, pooling all 50 tasks into a single sample. As shown in Table 3 (App. E), all nine correlations (3 metrics × 3 LLM judges) are positive and statistically significant. The τb values indicate moderate-to-strong agreement, ranging from 0.32 to 0.67, with raw p-values ranging from < 10−6 to 0.011. Agreement is strongest for Policy Violations Count, with correlations ranging from 0.55 to 0.67. Overall, these results provide independent evidence that the LLM judges produce task-level rankings that align with human assessments. Together with the benchmark ordering results in §5.2, this human validation supports our assumption in §4.1 that more capable models tend to generate benchmarks of higher quality and complexity. Additional details are provided in App. E. 5.5

6

Evaluating Manually-Constructed Benchmarks

Beyond synthetic benchmarks, our metrics also apply to manually constructed ones. We illustrate this on τ 3 -B ENCH, evaluating all 50 Airline tasks4 with Claude-4.5-Sonnet and GPT-5.4 as judges, using metric prompts minimally adapted for structural differences from I NTELL AGENT (App. F). Table 1 reports the results. The τ 3 -Bench scores broadly fall within the range observed for the lower-scoring I NTELL AGENT benchmarks generated by Llama-3.1-8B and Llama-3.2-1B. For the consistency metrics, this should be read in light of differences in expectedbehavior granularity: in τ 3 -Bench, the expected behavior typically specifies only the final outcome (e.g., whether a cancellation request is approved),

Qualitative Analysis

Following the quantitative validation of our metrics, we examine their value as diagnostic tools, using Policy–Expected Behavior Alignment as a case study. We analyze low-scoring Airline tasks and the corresponding rationales produced by the Gemini-2-Flash judge. We manually review the examples, use GPT-5 to cluster recurring failure

4

We focused on the Airline domain, as expected behavior annotations were missing from most Retail tasks.

7

Desc.–Expected Policy–Expected Violation Coverage Violations Count

Claude-4.5-Sonnet

GPT-5.4

3.50 5.28 0.31 0.38

4.54 3.80 0.46 0.60

erences. For example, Levi and Kadar validate I NTELL AGENT by measuring correlations between agent performance on its tasks and on τ -B ENCH. Similarly, Maheshwari et al. (2024) compare synthetic and human-curated datasets using modelranking agreement, fidelity measures, and statistical similarity. However, these approaches provide only indirect evidence of benchmark quality and require suitable reference datasets, which may not always be available. Outside the conversationalagent setting, Qian et al. (2026) assess closedform LLM benchmarks (e.g., MMLU, ARC) using model-ranking consistency across benchmarks, discriminative power across models, and detection of anomalous within-family model comparisons; however, their approach requires evaluating many LLMs on each benchmark, which is considerably more costly for open-ended, task-oriented conversational settings than for closed-form benchmarks, and is not required by our metrics.

Table 1: Metric scores on the manually curated τ 3 B ENCH benchmark (Airline domain).

whereas I NTELL AGENT benchmarks generated by stronger models describe the full policy-compliant workflow, including information collection, confirmation steps, and applicable procedures. Since Policy–Expected Behavior Alignment penalizes missing policy-required steps, it scores concise outcome-level annotations lower even when they are correct. τ 3 -Bench also shows substantially fewer policy violations per task (0.38–0.60) than the strongest I NTELL AGENT benchmarks (1.5– 2.3), indicating that top-generator I NTELL AGENT tasks are considerably more adversarial and policycomplex. Finally, policy-violation coverage (0.31– 0.46) leaves room for improvement, indicating that a substantial share of policy items are not challenged by any task in the benchmark.

7

To the best of our knowledge, our work is the first to systematically evaluate benchmarks for conversational agents. Unlike previous benchmarkevaluation approaches, ours does not require a reference benchmark or running a large set of agents or LLMs on the benchmark being evaluated. Building on τ -B ENCH and I NTELL AGENT, our metrics quantify benchmark quality and complexity while also identifying concrete weaknesses, such as inconsistent tasks and insufficient policy coverage.

Related Work

Existing LLM-agent benchmarks evaluate a broad range of capabilities (Mohammadi et al., 2025; Yehudai et al., 2026), from tool use (Yan et al., 2024) and planning (Valmeekam et al., 2023) to specific settings such as web interaction (Yao et al., 2022; Zhou et al., 2023) and software engineering (Jimenez et al., 2023; Miserendino et al., 2025), and more recently, general-purpose agents across multiple tasks and environments (Trivedi et al., 2024; Xu et al., 2024). Within this broader landscape, we focus on benchmarks for task-oriented conversational agents, such as τ -B ENCH and I NTELL AGENT (Yao et al., 2024; Levi and Kadar, 2025), which evaluate agents through dynamic interactions with an LLMbased user simulator (§2). Whereas τ -B ENCH contains manually constructed tasks, I NTELL AGENT generates synthetic tasks from a domain policy and database schema (§4.1). Although these benchmarks support agent evaluation, their own quality has received comparatively little systematic attention. This issue is especially important for synthetic benchmarks, which often lack direct human validation. A common approach is to compare them with manually constructed ref-

8

Conclusion

We introduced a framework for evaluating the quality of conversational-agent benchmarks, defining metrics that capture task-level consistency and complexity, and benchmark-level policy coverage. Beyond aggregate scores, these metrics provide actionable diagnostics, surfacing low-quality tasks and gaps in policy coverage. We validated the metrics using synthetic benchmarks from LLMs of varying capabilities, controlled perturbations, and an independent human evaluation, and illustrated their use on a manually constructed benchmark. The metrics reliably capture quality differences, remain sensitive to controlled degradation, and align with human judgments. Together, they promote more systematic, reliable evaluation of conversational-agent benchmarks. 8

Limitations

Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

This work focuses on conversational-agent benchmarks following τ -B ENCH- and I NTELL AGENTstyle evaluation frameworks. Accordingly, our metrics primarily assess task consistency, complexity, and alignment with domain policies. Our synthetic-benchmark experiments rely on the policy-driven I NTELL AGENT generation pipeline. Other strategies, such as tool-driven generation, may produce different task distributions. Future work should therefore evaluate the proposed metrics across a broader range of benchmarkgeneration methods and domains. Finally, we do not consider non-conversational agent settings, such as code generation or computer interaction, which involve different interaction structures and evaluation challenges.

Shadi Iskander, Sofia Tolmach, Ori Shapira, Nachshon Cohen, and Zohar Karnin. 2024. Quality matters: Evaluating synthetic data for tool-using llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4958– 4976. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? ArXiv, abs/2310.06770. Maurice G. Kendall. 1945. The treatment of ties in ranking problems. Biometrika, 33(3):239–251. Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Yoo, and Minjoon Seo. 2023. Aligning large language models through synthetic feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13677–13700, Singapore. Association for Computational Linguistics.

References Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. τ 2 -bench: Evaluating conversational agents in a dual-control environment. Preprint, arXiv:2506.07982.

Elad Levi and Ilan Kadar. 2025. Intellagent: A multiagent framework for evaluating conversational ai systems. Preprint, arXiv:2501.11067.

Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan Eghlidi, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, María Grandury, Sophia Han, Valentin Hofmann, Lujain Ibrahim, and 23 others. 2025. Measuring what matters: Construct validity in large language model benchmarks. In Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc.

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 3102–3116.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. Preprint, arXiv:2303.12712.

Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. 2025. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115.

Gaurav Maheshwari, Dmitry Ivanov, and Kevin El Haddad. 2024. Efficacy of synthetic data as a benchmark. Preprint, arXiv:2409.11968.

Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. 2025. Evaluation and benchmarking of llm agents: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ’25, page 6129–6139, New York, NY, USA. Association for Computing Machinery.

Alejandro Cuadron, Pengfei Yu, Yang Liu, and Arpit Gupta. 2025. Saber: Small actions, big errors – safeguarding mutating steps in llm agents. Preprint, arXiv:2512.07850. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Ultrafeedback: boosting language models with scaled ai feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.

Qi Qian, Chengsong Huang, Jingwen Xu, Changze Lv, Muling Wu, Wenhao Liu, Xiaohua Wang, and 1 others. 2026. Benchmark2 : Systematic evaluation of LLM benchmarks. arXiv preprint arXiv:2601.03986. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM:

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten,

9

Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations.

for building autonomous agents. arXiv:2307.13854.

arXiv preprint

Naama Zwerdling, David Boaz, Ella Rabinovich, Guy Uziel, David Amid, and Ateret Anaby Tavor. 2025. Towards enforcing company policy adherence in agentic workflows. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 595–606, Suzhou (China). Association for Computational Linguistics.

Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16022–16076, Bangkok, Thailand. Association for Computational Linguistics.

A

Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36:38975–38987.

LLM-as-a-Judge Prompts for Consistency Metrics

To evaluate benchmark consistency, we use LLMbased judges that score the alignment between different task components. Each judge receives the relevant components of a task and produces a score on a 1–10 scale, where higher scores indicate stronger alignment and consistency. The judges are prompted with structured evaluation instructions and asked to return a numeric score reflecting the degree of alignment.

Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, and 2 others. 2024. Theagentcompany: Benchmarking llm agents on consequential real world tasks. Preprint, arXiv:2412.14161.

A.1

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard. https://gorilla.cs.berkeley. edu/blogs/8_berkeley_function_calling_ leaderboard.html.

Description–Expected Behavior Alignment

This judge evaluates whether the expected behavior accurately reflects the task description and intended user goals. Inputs • Task description dt

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable realworld web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.

• Expected behavior et Prompt Template """ You are a benchmark auditor.

Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. Preprint, arXiv:2406.12045.

Given description and expected_behavior, ,→ evaluate whether expected_behavior is ,→ aligned with the description.

Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2026. Survey on evaluation of llmbased agents. Preprint, arXiv:2503.16416.

Evaluate: (1) Consistency: expected_behavior must not ,→ contradict any explicit facts in description. (2) No invention: expected_behavior must not ,→ introduce new specific scenario facts not ,→ supported by description (e.g., names, dates, ,→ cities, flight numbers, membership tiers, ,→ payment methods, prior actions, constraints). (3) Coverage: expected_behavior should include ,→ all key required outcomes implied by ,→ description. (4) Relevance: expected_behavior should not add ,→ unnecessary actions that are not requested or ,→ implied by description.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023. Webarena: A realistic web environment

Only judge description and expected_behavior ,→ alignment. Do NOT judge writing quality or ,→ realism.

10

- If expected behavior omits a mandatory step ,→ that policy requires for the described ,→ workflow, that is also a violation ,→ (missing-required-step). - If expected behavior includes acts not required ,→ by description, they must still be permitted ,→ by policy; otherwise they are invented acts.

Generic statements with no operational meaning ,→ are OK. Return STRICT JSON only. Score 1–10 for alignment between ,→ expected_behavior and description. Rubric: - 10: Fully aligned; no invented facts; covers ,→ all key user goals; no contradictions. - 7–9: Mostly aligned; minor missing detail or ,→ minor overreach without new specific facts. - 4–6: Multiple issues: missing key goals OR ,→ introduces specific invented facts OR at ,→ least one contradiction. - 1–3: Major mismatch: contradicts description or ,→ mostly unrelated behavior.

Only judge policy and expected behavior alignment. ,→ Do NOT judge writing quality or realism. Generic statements with no operational meaning ,→ (e.g., "be helpful") are OK. Return STRICT JSON only. Score 1–10 for alignment between expected ,→ behavior and policy. Rubric: - 10: Fully compliant; no invented acts; no ,→ missing required steps. - 7–9: Mostly compliant; minor ambiguity or one ,→ small missing step that doesn't change ,→ allowed actions. - 4–6: Clear violations (invented acts and/or ,→ missing required steps). - 1–3: Major violations; expected behavior ,→ largely contradicts policy or adds many ,→ disallowed actions.

Return EXACT JSON: {{ "score": <1-10>, "reasoning": ["3-7 short bullets"] }} TASK: description: {description} expected_behavior: {expected_behavior} """

A.2

Return EXACT JSON: {{ "score": <1-10>, "reasoning": ["3-7 short bullets"], }}

Policy–Expected Behavior Alignment

This judge evaluates whether the expected behavior follows the policy constraints for the given scenario.

• Task description dt

TASK: description: {description} expected behavior: {expected_behavior} policy: {policy} """

• Policy specification P

A.3

• Expected behavior et

For each task, the judge produces a single score. Aggregate benchmark scores are computed by averaging task-level scores, while also reporting the median, standard deviation, minimum, and maximum across tasks.

Inputs

Prompt Template """ You are a benchmark auditor. Given description, policy, and expected behavior, ,→ evaluate whether expected behavior is ,→ strictly compliant with policy for the ,→ described user request.

B

Scoring Procedure

LLM-as-a-Judge Prompt for Policy Violations

This judge outputs the IDs of all policy items that are violated by the given task, considering the task description and the initial DB state.

Definitions: - "Act" = any user-facing step or backend step ,→ implied by the expected behavior (e.g., asking for identifiers, quoting fees, calling ,→ ,→ an API / updating a reservation, offering ,→ compensation, escalating to a human). - An act is ALLOWED only if: (a) it is explicitly stated in policy, OR (b) it is a necessary implication of a ,→ policy-required workflow (e.g., policy ,→ says "extra baggage is $50" => telling the ,→ user "$50" is allowed). - Any act not allowed by (a) or (b) is an ,→ "invented act" (policy violation).

Inputs • Policy specification P , annotated with policy item spans IP • Task description dt • Initial DB state it 11

Prompt Template

"""

""" You are an auditor for conversational agent ,→ benchmarks.

C

The agent guidelines are defined in the following ,→ policy document. The document includes ,→ annotated spans of specific policy items. ,→ Each span is marked as <policy ,→ id=...>...</policy>, and has a unique id.

Qualitative Examples of Policy–Expected Behavior Alignment Issues

This appendix presents qualitative examples drawn from the Policy–Expected Behavior Alignment metric. Each example includes a task, its assigned score, and the accompanying judge rationale, illustrating why the task received a low policyalignment score. Representative low-scoring policy-alignment examples are shown in Figures 7, 8, and 9. Figure 7 illustrates a case with invented workflow stages, forbidden or unsupported actions, and omission of required information collection. Figure 8 shows a different failure mode, where the expected behavior includes a forbidden action and skips explicit confirmation before modification. Figure 9 highlights unsupported operational details, invented workflow constraints, and overly specific refund handling. Overall, these examples show that the policyalignment metric is informative not only as a scalar score, but also as a diagnostic signal: low-scoring tasks expose concrete benchmark defects such as unsupported workflow invention, omission of required steps, and direct contradictions of policy rules. Color legend. Hallucinated policy stages or nonexistent actions, missing required information collection, missing explicit confirmation before action, incorrect compensation handling, forbidden actions.

<policy_document> {policy} </policy_document> --You are given a user scenario that tests the ,→ agent's adherence to the above policies. The ,→ scenario is a script that guides the user's ,→ conversation with the agent. The scenario is ,→ visible to the user but not to the agent. You are also given relevant rows from the ,→ system's database, representing its initial ,→ state. Your goal is to assess whether any of the user actions in the scenario are not allowed, ,→ under the given policy items, and considering ,→ the information provided in the initial ,→ database state. The policy items should be ,→ interpreted in the context of the whole ,→ policy document. Output only the policy IDs, ,→ not the policies themselves. ,→ Guidelines: 1. If no user intent is identified in the ,→ scenario, you should not output any policy ,→ IDs. 2. Do not include policies that are related to the agent's actions rather than to the user's ,→ actions, unless the user's actions explicitly ,→ ,→ violate these policies, or if the user asks ,→ the agent to bypass or violate them. 3. Do not include policies that discuss user ,→ authentication, providing information by the ,→ user, asking the user to provide information ,→ or getting the user's confirmation, unless ,→ the scenario explicitly mentions that the ,→ user does not provide the required ,→ information, or provides incorrect or ,→ invalid information. 4. You should output identified violations even ,→ if the scenario indicates that they were ,→ correctly handled by the agent.

Figure 7: Policy–Expected Behavior misalignment example. The low score is driven by an invented modification workflow, a forbidden request for refund method selection, and omission of required cancellation information collection.

Example 1: Description. User john_doe_8923 contacts the chatbot regarding their reservation RES202310011200, which includes flight F12345. The user decides to cancel the reservation and inquires about the refund process and cancellation rules.

Response must be a strictly formatted JSON in the ,→ following format: {{ "policy_ids": [<a list of violated policy IDs, ,→ as a list of comma separated integers. ,→ Leave empty if no relevant policies were ,→ found.>] }} Respond with only JSON, no preambles.

Expected Behavior. The chatbot should first assist the user in modifying their flight reservation by requesting a payment or refund method,

TASK: task_description: {description} initial_database_state: {db_state}

12

inform the user that changing passenger information (wife’s name) is possible and can be processed. (3) When the user requests adding another passenger (the brother), the chatbot should explain that modifying the number of passengers is not possible; this cannot be done through the system and is not something even a human agent can assist with. The user would need to make a separate booking for the additional passenger. (4) Regarding the flight delay mentioned by the user, the chatbot should not proactively offer any compensation since the user only mentioned the inconvenience but did not explicitly ask for compensation. (5) For the question about cancellation and refunds, the chatbot should explain the standard refund policy (refunds go to the original payment method within 5–7 business days) and should take this opportunity to offer travel insurance, explaining that it costs $30 per passenger and would enable a full refund if they need to cancel due to health reasons. (6) Throughout the interaction, the chatbot should handle all aspects of the request that fall within its capabilities and should not transfer to a human agent unless specifically requested to handle something outside its scope.

such as a gift card or credit card. When the user decides to cancel the reservation, the chatbot should inform them that the refund will be processed to the original payment method within 5 to 7 business days. Additionally, the chatbot must ensure that the cancellation rules are strictly followed and communicate these rules to the user before proceeding with the cancellation. Score. 6 Reasoning. The expected_behavior invents an act by assisting the user in modifying their reservation and requesting a payment or refund method before the user has requested a modification; the user only wants to cancel. Policy requires, for cancellation, obtaining user id, reservation id, and reason for cancellation, then communicating and enforcing cancellation rules before proceeding. The expected_behavior correctly states that the refund goes to the original payment method in 5–7 business days and that cancellation rules must be communicated, but the modification/payment method step is not allowed here and is an invented act.

Score. 6

Figure 8: Policy–Expected Behavior misalignment example. The low score reflects a forbidden postbooking insurance offer and missing explicit confirmation before modifying passenger information.

Reasoning. The expected_behavior invents an act by offering travel insurance after booking, which is explicitly prohibited by policy (insurance can only be added at initial booking). It correctly denies adding a passenger, does not offer compensation for the delay, and provides the standard refund policy. However, it misses the mandatory step of obtaining explicit user confirmation before modifying passenger information, which is required before updating the booking database. These are clear policy violations.

Example 2 Description. User john_doe_1980 with ID ABC12345 has a reservation NYLON123 for a round-trip flight from New York to London (NYL123) and back (F2). The reservation includes passengers Alice Smith (the user) and Bob Johnson (the user’s wife). The user requests to update Bob Johnson’s name due to a recent marriage and to add P3 (the user’s brother) to the reservation. Additionally, the user mentions a previous delay of 3 hours on F2 and inquires about refund policies in case of cancellation due to health concerns.

Figure 9: Policy–Expected Behavior misalignment example. The low score reflects invented workflow constraints, unsupported payment assumptions, and overly specific refund handling.

Expected Behavior. The chatbot should handle this complex request by addressing multiple aspects according to the policies: (1) The chatbot should acknowledge the user ID (ABC12345) provided and proceed with the modification request. (2) The chatbot should

Example 3 Description. User john_doe_1234 wants to book flight F1 from New York to Los Angeles for passengers {’first_name’:

13

D

’Alice’, ’last_name’: ’Smith’, ’dob’: ’1985-07-15’} and {’first_name’: ’Bob’, ’last_name’: ’Johnson’, ’dob’: ’1992-03-22’}. The user plans to use payment methods {’source’: ’credit_card’, ’brand’: ’mastercard’, ’last_four’: ’5678’, ’id’: ’credit_card_1234567’}, {’source’: ’certificate’, ’amount’: 150, ’id’: ’certificate_9876543’}, {’source’: ’credit_card’, ’brand’: ’visa’, ’last_four’: ’4321’, ’id’: ’credit_card_7654321’}, and {’source’: ’credit_card’, ’brand’: ’amex’, ’last_four’: ’2223’, ’id’: ’credit_card_1112223’} for the payment. The user is interested in purchasing travel insurance. After booking, the user decides to cancel the reservation due to health reasons and requests a refund.

Benchmark Ordering Scores

Table 2 reports the Benchmark Ordering Score (§5.1) for each metric, judge, and domain.

E

Human Validation

Sampling and annotation. We sampled 50 tasks from benchmarks produced by all five benchmarkgenerator models. A human annotator, one of the paper’s authors, evaluated each task along the same three task-level measures used by our framework: Description–Expected Behavior Alignment, Policy– Expected Behavior Alignment, and Policy Violations Count, following the same criteria as the corresponding LLM-judge prompts. The task-level evaluations were pooled across benchmark generators.

Expected Behavior. The chatbot should first ask for the user’s ID, then collect the first name, last name, and date of birth for each passenger. It should then ask if the user wants to buy travel insurance for $30 per passenger. The chatbot should inform the user that only one travel certificate, one credit card, and up to three gift cards can be used for the reservation, and that the new gift card needs to be added to the user’s profile first. After the user adds the new gift card, the chatbot should process the payment and book the flight. When the user wants to cancel the flight, the chatbot should offer a full refund since the user bought travel insurance. The chatbot should not offer any additional compensation unless the user explicitly asks for it.

Correlation analysis. For each task-level measure and LLM judge, we computed Kendall’s τb correlation between the human evaluations and the corresponding LLM-judge scores. Kendall’s τb was selected because the evaluations contain ordinal scores and ties. We report two-sided p-values. All nine correlations remain statistically significant at α = 0.05. Table 3 reports the results. All correlations are positive and statistically significant, ranging from τb = 0.32 to τb = 0.67. Agreement is strongest for Policy Violations Count, for which correlations range from 0.55 to 0.67. For the two alignment measures, correlations range from 0.32 to 0.55. Overall, these results indicate modest-to-strong rank agreement between the human evaluations and all three LLM judges across the three tasklevel measures.

Score. 5

F

Reasoning. The expected behavior introduces several specific constraints and actions not mentioned in the description, such as the limit on payment methods, the need to add a new gift card, and the travel insurance cost. It also assumes the user will add a new gift card, which is not explicitly stated. The chatbot offering a full refund due to travel insurance is reasonable, but the level of detail is not fully supported by the description.

Evaluation on τ 3 -Bench

This section describes the metric adaptations for evaluating the manually constructed τ 3 -Bench. We mapped the nl_assertions field of τ 3 -Bench to our “expected behavior”. We adapted the metrics as follows: • Unlike I NTELL AGENT, which generates an independent, task-relevant initial database state per task, τ 3 -Bench tasks rely on a single shared database. We therefore removed the initial DB state it from the input to the policy violations judge, and adjusted the prompt accordingly. 14

Judge GPT-5.4

Claude-Sonnet-4.5

Gemini-2-Flash

Mean

Airline Description–Expected Behavior Alignment Policy–Expected Behavior Alignment Policy Violations Count Policy Violations Coverage Average

1.00 1.00 1.00 1.00 1.00

1.00 1.00 1.00 1.00 1.00

1.00 1.00 1.00 1.00 1.00

1.00 1.00 1.00 1.00 1.00

Retail Description–Expected Behavior Alignment Policy–Expected Behavior Alignment Policy Violations Count Policy Violations Coverage Average

0.88 1.00 1.00 0.88 0.94

0.88 0.88 1.00 1.00 0.94

0.75 1.00 1.00 0.75 0.88

0.83 0.96 1.00 0.88 0.92

Overall Mean

0.97

0.97

0.94

0.96

Table 2: Benchmark Ordering Score per metric, evaluated across judge models and domains.

Metric

Judge

τb

p-value

Description–Expected Description–Expected Description–Expected

Claude GPT Gemini

0.46 0.43 0.35

3.5 × 10−5 8.1 × 10−5 1.2 × 10−3

Policy–Expected Policy–Expected Policy–Expected

Claude GPT Gemini

0.48 0.55 0.32

7.3 × 10−5 3.0 × 10−6 1.0 × 10−2

Policy Violations Policy Violations Policy Violations

Claude GPT Gemini

0.56 0.67 0.55

3.0 × 10−6 < 10−6 5.0 × 10−6

Table 3: Task-level Kendall’s τb correlations between human evaluations and LLM-judge scores. All correlations are positive and statistically significant.

• We revised the policy items to match the updated Airline policy document, using Claude Sonnet 4.6, followed by human review. • In I NTELL AGENT, task description and database state are generated together and are expected to contain the same information. In τ 3 -Bench, the database and task description are authored separately, so the expected behavior may draw on database information absent from the description. We therefore removed the “no invention” criterion from the Description–Expected Behavior metric (App. A.1).

G

Usage of AI Assistants

We have used Claude Sonnet 5 as a writing assistant, in accordance with ACL/ARR guidelines.

15

Related documents

• Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations#1116853
• Conversation as data: Extracting features from naturalistic conversation datasets#1108174
• Conversation as data: Extracting features from naturalistic conversation datasets#1108120
• Role-aware Heuristic Episodic Attention for Conversational LLMs#1155893
• What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal#627749