arXiv:2607.29175v1 [cs.SE] 31 Jul 2026
Execution-First Synthetic Tool-Use Trace Generation for LLM Agents Hafsa Ouajdi∗
Francesco Giannuzzo∗
Alaa Boukhary
EURECOM Sophia Antipolis, France Aily Labs Munich, Germany [email protected]
EURECOM Sophia Antipolis, France Aily Labs Munich, Germany [email protected]
Aily Labs Munich, Germany [email protected]
Paolo Papotti
Gerard Conangla
Adam Elwood
EURECOM Sophia Antipolis, France [email protected]
Aily Labs Munich, Germany [email protected]
Aily Labs Munich, Germany [email protected]
Abstract
Keywords
Agentic software-engineering and industrial systems increasingly operate through executable workflows rather than code generation alone: they search artifacts, invoke tools, inspect structured observations, and query databases. Training these agents requires supervision data that captures valid tool interactions and executable workflows. However, traditional query-first data synthesis can fail because plausible user requests may not correspond to valid tool sequences, compatible parameters, or available data. To address this limitation, we propose SyntheticAgentTraceQA, an executionfirst framework for generating scalable supervision data for toolaugmented agents. Our framework first constructs high-level workflow structures, maps them to available tools through dependencyaware assignment, executes and validates the resulting traces in controlled environments, and only then synthesizes natural-language user tasks, teacher-generated reasoning annotations, and reference answers. We evaluate the framework across four tool ecosystems and use the resulting data to fine-tune and evaluate Qwen model variants. The results show that execution-grounded supervision improves tool execution behavior, reference-trace agreement, and answer-generation performance on the evaluated tasks. Further analysis reveals a supervision trade-off: masked supervision, which excludes reasoning annotations from the training objective, improves final-answer metrics, whereas full supervision, computing loss over the complete assistant output including reasoning tokens, underperforms on answer quality and does not consistently improve reference-trace agreement, particularly at the 9B scale. These findings highlight the importance of designing synthetic supervision according to the desired capabilities of tool-augmented agents.
agentic AI, tool-use agents, synthetic traces, execution-grounded data, supervised fine-tuning
CCS Concepts • Software and its engineering → Software testing and debugging; • Computing methodologies → Natural language generation; Planning and scheduling. ∗ Both authors contributed equally to this research.
AgenticDev ’26, Munich, Germany 2026.
ACM Reference Format: Hafsa Ouajdi, Francesco Giannuzzo, Alaa Boukhary, Paolo Papotti, Gerard Conangla, and Adam Elwood. 2026. Execution-First Synthetic Tool-Use Trace Generation for LLM Agents. In Proceedings of International Workshop on Agentic AI for Next-Generation Software Development (AgenticDev ’26). ACM, New York, NY, USA, 12 pages.
1
Introduction
Large language models (LLMs) have enabled a new generation of autonomous agents capable of interpreting complex user requests and interacting with external environments. A particularly effective paradigm is that of tool-augmented agents, which extend the capabilities of LLMs by invoking external tools such as APIs, data-analysis engines, search systems, and other specialized services [28, 49]. By integrating these resources, LLM-based agents move beyond next-token prediction toward dynamic, interactive problem solving [34, 39]. Despite these advances, these agents can still fail in complex, multi-step technical workflows. Common failure modes include selecting inappropriate tools [22, 47], generating invalid tool arguments (e.g., incorrect parameter names or data types), failing to incorporate intermediate execution results into subsequent reasoning, and producing responses that are not fully supported by the evidence gathered during execution [5, 22, 31]. These failures become particularly consequential in software engineering tasks involving artifact retrieval, programmatic tool execution, structured data access, and the orchestration of multiple API calls, where correctness depends on accurate tool use and faithful reasoning over intermediate observations [15, 22, 27, 44]. They also compound broader usability and oversight challenges in deployed agent systems [30]. This paper focuses on a specific aspect of agent performance rather than general coding ability. Specifically, we study whether agents can construct and execute multi-step tool workflows whose correctness can be objectively verified through tool outputs and environment states. Across all tasks, the central capability under
AgenticDev ’26, October 12, 2026, Munich, Germany
evaluation is reliable reasoning over tools, schemas, and intermediate execution results. Existing benchmarks and data-synthesis methods often rely on manually crafted [9, 25] or forward-generated [20, 32, 35] scenarios, which are costly, difficult to scale, and may not guarantee alignment between user requests and executable tool traces. In a conventional query-first pipeline, a plausible user request may require unavailable tools, invalid parameters, or execution paths that cannot actually solve the task [53]. To address this limitation, we introduce SyntheticAgentTraceQA, a backward-chaining data-generation pipeline that retains only traces that execute successfully in the generation environment and uses them as supervision signals. Rather than starting from a user request, our approach first constructs and validates executable tool traces in a controlled environment before synthesizing naturallanguage tasks. This execution-first strategy filters or reduces common synthesis errors, such as tool-capability mismatches and invalid parameterizations, while producing training data grounded in executable workflows. Research Questions. • (RQ1) Does execution-grounded fine-tuning improve tool behavior, reference-trace agreement, and answer-generation performance? • (RQ2) How does model scale influence the effects of executiongrounded supervision across these evaluation metrics? • (RQ3) How does supervising reasoning tokens affect tool behavior, reference-trace agreement, and answer-generation performance? Contributions.
Ouajdi, Giannuzzo, et al.
with external environments, supporting task automation and execution beyond text generation [25, 55]. Tool use can also improve response grounding, transparency, and interpretability by exposing intermediate reasoning and evidence sources [2]. Several approaches have been proposed to facilitate tool use in language models. Toolformer introduced a self-supervised framework for learning API invocation behavior from data [28], while ReAct demonstrated the effectiveness of interleaving reasoning and actions during task execution [49]. These paradigms have subsequently influenced a broad range of tool-augmented and agentbased systems [21, 25]. Although these approaches significantly improve the capabilities of language agents [1, 3, 6], they primarily focus on inference-time tool use. Comparatively less attention has been given to using tool interactions as supervision signals, particularly through executiongrounded traces for training and evaluating tool-augmented agents. Structured reasoning and planning. Recent work has increasingly emphasized process-centric reasoning over outcome-only evaluation. Approaches such as Chain-of-Thought (CoT) and Tree of Thoughts (ToT) demonstrate that exposing intermediate reasoning steps and exploring alternative trajectories can improve problem solving by enabling decomposition, self-verification, and search [38, 48]. Building on this idea, reasoning traces have become an important source of supervision for training and evaluating reasoning-oriented models. Evidence suggests that structured trajectories capture procedural knowledge beyond final answers alone [23, 40]. Recent work has also explored synthetic trace generation and trajectory modeling, where traces explicitly represent goals, plans, tool invocations, observations, and recovery actions. Frameworks such as AgentSynth and ISE generate execution-grounded trajectories for analyzing and supervising complex reasoning behavior [23, 40]. These works support our focus on execution-grounded tool-use traces as a scalable source of supervision and evaluation beyond final-task correctness.
• The SyntheticAgentTraceQA Pipeline: An execution-first framework for generating synthetic supervision data for toolaugmented agents through validated execution traces with varying levels of workflow and task complexity. • Multi-Domain Taxonomy & Extension: An operational taxonomy for tools and parameters that unifies heterogeSynthetic data and instruction tuning. Most instruction-tuning neous tool environments—including internal company tools pipelines follow a forward-generation paradigm: an LLM is prompted (finance, research portfolio, supply chain) and the genius_song_lyrics with a small seed set to generate new instructions, inputs, and tool group from ToolBench for music search and metadata reoutputs, which are subsequently filtered and used for supervised trieval—under a single abstract template and profiling framefine-tuning [33, 35]. Approaches such as Self-Instruct, Alpaca, and work. Tool execution is performed through local wrappers GPT-4-based distillation differ in the teacher model and promptaround the ToolBench APIs using cached responses. ing strategy but largely preserve the same instruction → input → • Controlled Evaluation: A fine-tuning study showing meaoutput generation process [24, 35, 36, 52]. Tool-use data generation sured improvements in tool-use reliability, Reference-Trace extends this paradigm by conditioning on tool or API descriptions Agreement, and answer quality across multiple model variand synthesizing tool-using tasks together with candidate call seants. quences or solution paths [25, 41, 54]. While effective for scaling
2
Related Work
Tool-augmented language models. Large language models (LLMs) have demonstrated strong performance across a wide range of tasks. However, their capabilities remain limited by the static knowledge acquired during pretraining. Tool augmentation overcomes this limitation by enabling access to external resources such as search engines, databases, computational systems, and APIs. This extends LLMs with up-to-date information and specialized functionalities [7, 8, 13, 17, 26]. In addition, tools allow LLMs to interact
supervision, these approaches often rely on static prompts and model-generated trajectories, which can introduce noise, unrealistic behaviors, and invalid multi-tool interactions [12, 16, 25]. Execution-first approaches reverse this pipeline by treating valid executions or trajectories as the primary artifact. Rather than generating a task and inferring a solution path, they begin with verified interactions and derive traces or user-facing tasks from successful executions [2, 37]. Recent trajectory-centric methods share a similar motivation but often rely on proxy signals instead of verified executions. ToolGrad generates and refines candidate workflows
Trace Synthesis for LLM Agents
using textual feedback, which may increase workflow complexity without guaranteeing correctness [20]. ToolMind constructs trajectories through similarity-based tool graphs and random walks, which can produce unrealistic or non-executable workflows that do not reflect real user goals [42]. HardGen leverages API dependency graphs to generate challenging training examples, but its effectiveness depends on accurate dependency metadata and predefined tool relationships [10]. Agent benchmarks. Recent benchmarks for tool-augmented agents have evolved from broad API-centric evaluations toward more reproducible and production-oriented execution environments. Early efforts such as ToolLLM and API-Bank were built around diverse real-world APIs and evaluated planning, retrieval, and tool invocation through executable interactions [19, 25]. Subsequent work identified important reproducibility challenges, showing that benchmark performance can be affected by API drift, unstable tool availability, and complex multi-turn interactions [4, 9, 47]. More recent benchmarks address these issues through virtualized APIs, simulated tool ecosystems, and controlled execution environments that better reflect long-horizon interactions and realistic user workflows [9, 29, 43, 51]. A complementary line of research investigates synthetic data generation for tool-augmented agents. Recent work has shown that producing high-quality supervision often requires generating a large pool of candidate examples before selecting those that satisfy the desired quality criteria. For example, Autodata and Toucan [18, 41] employ an iterative framework in which candidate examples are repeatedly generated, evaluated, and filtered through multiple validation stages. While this process can improve dataset quality, it also increases generation cost by requiring substantial oversampling before selection. Collectively, these efforts highlight the importance of executable supervision for tool-augmented agents. Existing approaches either focus on evaluating agent behavior or generate execution traces from predefined dependencies or proxy supervision. In contrast, SyntheticAgentTraceQA combines an operational tool and parameter taxonomy, active-domain profiling, LLM-generated abstract execution DAGs, dependency-aware tool assignment, executiondriven validation before task synthesis, and application across four heterogeneous tool ecosystems within a single generation framework. This combination enables scalable generation of executiongrounded supervision while improving sample efficiency through early validation of executable traces.
3
AgenticDev ’26, October 12, 2026, Munich, Germany
Scope. Although our task space extends beyond conventional software engineering benchmarks centered on source code editing and repository bug fixing, it remains fundamentally softwareengineering in nature. Tasks require agents to interact with software artifacts such as code repositories, system logs, structured data, domain-specific APIs, database tables, and analytical scripts. Success depends on the same core competencies required for agentic software engineering: composing multi-step tool workflows, respecting interface and schema constraints, executing syntactically valid operations, incorporating execution feedback into subsequent reasoning, and avoiding unsupported or unverifiable responses.
4
The SyntheticAgentTraceQA Pipeline
To overcome the scalability limitations and capability gaps of languagefirst data generation, we introduce SyntheticAgentTraceQA, an execution-first pipeline. Instead of starting from a natural-language query, our approach first constructs and executes valid tool traces, then synthesizes user tasks only after execution-based validation. Figure 1 summarizes the five stages of the pipeline: Profiler extracts tool metadata; Template Generator produces abstract execution templates; Depth-First Search (DFS) constructs executable tool traces; Trace Execution executes the generated traces and Validation & Task Synthesis, which validates the executed traces and synthesizes corresponding user tasks from the validated executions.
4.1
O = {Matcher, Retriever, Analyzer, Aggregator, Metadata, Access}. Each operational class defines a distinct functional role: • Matcher: Resolves ambiguous user-provided entities into unique system identifiers. • Retriever: Retrieves data and attributes associated with resolved entities. • Analyzer: Processes retrieved data to derive insights, detect patterns, or generate predictions. • Aggregator: Combines information from multiple entities or observations through statistical or logical operations. • Metadata: Provides schema information, field definitions, and configuration details. • Access: Loads and validates data sources for downstream processing.
Problem Setup & Formalization
Let 𝑇 = {𝑇1, . . . ,𝑇𝑛 } be a set of available tools. Each tool 𝑇𝑖 maps a set of input arguments 𝑥𝑖 to an observation output 𝑦𝑖 . An execution trace 𝜏 is defined as a sequence of tool invocations: 𝜏 = [(𝑇1, 𝑥 1, 𝑦1 ), . . . , (𝑇𝑚 , 𝑥𝑚 , 𝑦𝑚 )]. Given a natural-language user task 𝑧, a tool-augmented agent must generate a predicted execution trace 𝜏ˆ and a final text answer 𝑟ˆ grounded in the accumulated observations. Our objective is to systematically generate high-quality training tuples (𝑧, 𝜏, 𝑟 ), where 𝜏 is fully executable, 𝑟 is supported by the execution outputs of 𝜏, and 𝑧 reflects a realistic user request.
Operational Taxonomy
To bridge the gap between abstract user intent and executable tool invocations, we organize the available tools into a finite set of Operational Classes. These classes provide the first level of abstraction in our execution-first framework. Instead of reasoning over individual tools, the pipeline constructs execution templates using functional categories, decoupling high-level planning from tool-specific implementations while preserving the logical dependencies between successive operations.
4.2
Tool and Data Profiling
The first stage of our pipeline is the Profiler, which constructs an enhanced metadata catalog for each tool. It is implemented using Claude Sonnet 4.6; no manual annotation is performed, and the
AgenticDev ’26, October 12, 2026, Munich, Germany
Ouajdi, Giannuzzo, et al.
Figure 1: Overview of SyntheticAgentTraceQA. The execution-first pipeline constructs and validates tool-use traces before synthesizing user tasks. (1) Profiler extracts tool metadata and parameter roles; (2) Template Generator creates abstract DAGbased workflows; (3) DFS instantiates executable traces under data-flow constraints; (4) Trace Execution validates candidate workflows; and (5) Validation & Task Synthesis generates user tasks and answers from validated traces. resulting metadata are used during template generation and compatibility checking. The profiler automatically assigns each tool to an operational class (Matcher, Retriever, Analyzer, Aggregator, Metadata, or Access) and classifies every input parameter and output field according to its functional role: • Control (𝑃ctrl ): task-defining inputs that remain consistent throughout an execution trace. • Data-flow (𝑃flow ): inputs supplied by upstream tool outputs. • Temporal (𝑃time ): parameters defining temporal constraints. • Configuration (𝑃set ): tool-specific settings independent of task intent. For a tool 𝑇𝑖 , Params(𝑇𝑖 ) = 𝑃ctrl (𝑇𝑖 ) ∪ 𝑃 flow (𝑇𝑖 ) ∪ 𝑃 time (𝑇𝑖 ) ∪ 𝑃 set (𝑇𝑖 ). Each parameter is represented by its name, data type, semantic role, and optionality. The profiler also constructs an Active Domain, i.e., a set of feasible values obtained by sampling the underlying data or extracting values from tool specifications. Formally,
4.3
Abstract Template Generation
Rather than generating tool calls directly from a user query, our execution-first pipeline first constructs an abstract execution template 𝐴𝑖 as a Directed Acyclic Graph (DAG), 𝐺𝐴𝑖 , whose nodes represent abstract operational blocks rather than concrete tools. Formally, let O denote the set of operational classes defined in Section 4.1, M ∗ the enhanced metadata catalog produced by the Profiler, and A the space of abstract templates. An abstract template 𝐴𝑖 ∈ A is represented as 𝐺𝐴𝑖 = (𝑉𝐴𝑖 , 𝐸𝐴𝑖 ), where • 𝑉𝐴𝑖 = {𝑏 1, . . . , 𝑏 𝐿 } is the set of operational blocks; • 𝐸𝐴𝑖 ⊆ 𝑉𝐴𝑖 ×𝑉𝐴𝑖 is the set of directed execution and data-flow dependencies. Each operational block 𝑏𝑙 ∈ 𝑉𝐴𝑖 is defined as 𝑏𝑙 = (𝑜𝑙 , pool𝑙 , dep𝑙 , desc𝑙 ), where 𝑜𝑙 ∈ O denotes the operational class, pool𝑙 ⊆ T is the set of compatible tools,
Φ : (𝑇𝑖 , Params(𝑇𝑖 ), Schema(𝑇𝑖 )) → M ∗ (𝑇𝑖 ),
dep𝑙 = { 𝑏𝑟 | (𝑏𝑟 , 𝑏𝑙 ) ∈ 𝐸𝐴𝑖 },
where M ∗ (𝑇𝑖 ) augments the tool schema with parameter roles, active domains, and explicit compatibility metadata used during template instantiation. Compatibility between tool calls is determined by semantic roles, data types, and profiler-discovered source mappings rather than parameter names alone.
is the set of predecessor blocks, and desc𝑙 is a domain-specific description used during task synthesis. To encourage structural diversity, templates are generated at three complexity levels according to the number of operational blocks:
Trace Synthesis for LLM Agents
AgenticDev ’26, October 12, 2026, Munich, Germany
Easy, |𝑉𝐴𝑖 | ∈ {1, 2}, D = Medium, |𝑉𝐴𝑖 | ∈ {3, 4}, Hard, |𝑉𝐴𝑖 | ∈ {5, 6}. Template generation is performed by an LLM, ∗
𝐺 : O × M × D → A, where D specifies the target template complexity. The generated templates satisfy three constraints: (i) data-flow consistency, ensuring required inputs are provided by predecessor blocks; (ii) tool interchangeability, associating each operational block with a pool of compatible tools; and (iii) complexity control, constraining the DAG topology to the selected complexity level.
4.4
Template Instantiation
Given an abstract template 𝐴𝑖 , this stage performs tool selection: each abstract operational block is replaced with a concrete tool. Since multiple tools may satisfy the same operational class, instantiation produces a set of candidate execution traces: I𝐴𝑖 = {𝐼𝑐,1, 𝐼𝑐,2, . . . , 𝐼𝑐,𝑛 }. For example, consider an abstract template of three operational blocks, Matcher → Retriever → Analyzer. Instantiation replaces these abstract operations with concrete tools, e.g., match_company → retrieve_financial_statement → calculate_growth. At this stage only the tool identities are fixed; the concrete argument values they operate on (e.g., a specific company and fiscal year) are not chosen here but are bound later, during trace execution (Section 4.5). Because the number of possible assignments grows combinatorially, we employ a Depth-First Search (DFS) strategy to explore valid tool combinations efficiently while pruning incompatible assignments. The procedure consists of three steps: • Sequential processing: The template DAG is traversed in topological order. For each operational block 𝑏𝑙 , the search iterates over the candidate tools in pool𝑙 . An instantiation is complete once every block has been assigned a compatible tool. • Compatibility checking: For each candidate tool 𝑇𝑖 ∈ pool𝑙 , the algorithm verifies that all required input fields are available from predecessor outputs. Formally, 𝑃flow (𝑇𝑖 ) ⊆ 𝑉visible, where 𝑉visible contains the outputs of previously instantiated tools. Compatibility is determined using the profiler metadata: an upstream output is considered compatible with a downstream input when their semantic roles, data types, and explicit source mappings are consistent. Parameter names alone are not used to establish compatibility. • State propagation: The visible state 𝑉visible is the set of data fields currently available from all previously instantiated tools. After selecting a compatible tool 𝑇𝑖 , its output fields are added to the visible state, 𝑉visible ← 𝑉visible ∪ OutputFields(𝑇𝑖 ),
making them available as inputs for subsequent tools. The search then proceeds recursively to the next operational block. Whenever no compatible assignment exists, the DFS backtracks and explores an alternative branch.
4.5
Trace Execution
Each candidate instance is executed to verify that the abstract plan is feasible in the target data environment. Whereas instantiation determines which tools are selected, execution determines which values they are invoked with and runs them. Execution maintains a context 𝐶 = (𝐶 in, 𝐶 out,𝑊 ), where 𝐶 in stores bound input parameters, 𝐶 out stores tool outputs for downstream use, and 𝑊 maintains temporal consistency throughout the workflow. Execution proceeds in topological order. At each step, the required arguments for tool 𝑇𝑘 are resolved from the current context or sampled when unavailable, producing bind(𝑥𝑘 ). The tool is then executed, yielding output 𝑦𝑘 , and the context is updated as (𝑘 ) (𝑘 ) 𝐶 (𝑘+1) = 𝐶 in ∪ bind(𝑥𝑘 ), 𝐶 out ∪ {(𝑇𝑘 , 𝑦𝑘 )}, 𝑊 (𝑘 ) ∪ 𝑊𝑘 . A candidate is retained only if all tool calls pass runtime validation, including tool availability, schema-compliant arguments, successful execution, usable outputs, and satisfied dependencies; otherwise it is discarded. To improve robustness, the execution engine re-samples non-deterministic parameters and adaptively relaxes matcher thresholds when appropriate.
4.6
Validation & Task Synthesis
The final stage transforms a successfully executed trace 𝐼𝑒 into a validated dataset instance 𝐼 𝑣 using two LLM-based components: a Validator, which verifies the executed trace, and an Annotator, which generates the final dataset entry. Validation. Crucially, because our pipeline generates data in reverse, no user question exists at this point. Validation therefore assesses only the consistency of the executed trace, rather than its agreement with a user query. Specifically, it verifies the structural consistency of the execution graph, correct parameter propagation across tool calls, compliance with tool schemas, temporal consistency, and the overall semantic coherence of the workflow. Traces that fail any validation criterion are discarded. The user question, reasoning trace, and reference answer are synthesized only after a trace has successfully passed validation. This validation is performed using the same LLM used to generate the trace. Task synthesis. For each validated execution trace, an LLM generates three user tasks (Basic, Intermediate, and Challenging), together with the corresponding teacher-generated reasoning annotations and reference answers. These difficulty levels characterize the formulation of the user request rather than the complexity of the underlying execution trace. The Basic task explicitly specifies the operational intent and required entities, the Intermediate task omits some execution details that must be inferred from context, and the Challenging task expresses a high-level or under-specified objective
AgenticDev ’26, October 12, 2026, Munich, Germany
requiring greater interpretation, while all three correspond to the same validated execution trace. This design exposes the same executable workflow through multiple natural-language formulations, increasing linguistic and behavioral diversity while preserving the underlying task intent and execution semantics.
5
Multi-Domain Extensions & Tool Sets
SyntheticAgentTraceQA is designed to be domain-agnostic: the same execution-first pipeline operates across environments with different schemas and tool sets without modifying the generation process. We evaluate the framework on four domains spanning enterprise analytics and an external benchmark (Table 1). Three domains (finance, research portfolio, and supply chain) represent structured enterprise workflows over financial assets, internal R&D projects, and supply chain entities. The fourth domain is the genius_song_lyrics tool group from ToolBench, which provides an external API-based environment for music search and metadata retrieval with a distinct entity model and tool ecosystem derived from the ToolBench benchmark [9, 25]. Shared operational abstraction. All domains share the operational taxonomy introduced in Section 4.1. During profiling, each tool is assigned to an operational class and its parameters are annotated according to their functional roles. The resulting metadata are consumed by the same template generation, instantiation, execution, validation, and task-synthesis stages, independent of the underlying application domain. External benchmark integration. To evaluate portability beyond our enterprise domains, we incorporated the genius_song_lyrics tool group from ToolBench. Integration required only lightweight interface wrappers that normalize entity identifiers and parameter formats before profiling. No modifications were made to template generation, trace instantiation, execution, or validation, demonstrating that the framework transfers to external tool ecosystems through interface adaptation alone.
6
Experiments
We investigate whether execution-grounded synthetic traces improve tool-augmented agents under realistic settings involving multi-domain tool use, execution feedback, and varying task difficulty.
6.1
Hypotheses and Ablation Axes
Our experimental design is structured around four hypotheses: • H1 (Data utility): Supervised fine-tuning on executionvalidated traces improves tool-use reliability, Reference-Trace Agreement, and final-answer quality relative to the corresponding base models. • H2 (Reasoning initialization): At a fixed model size, Thinking base models outperform their No-Thinking counterparts on execution-grounded reasoning tasks. • H3 (Process vs. outcome supervision): For Thinking models, Masked supervision, which excludes <think> tokens from the training loss, leads to different performance than Full supervision, which computes the training loss over the complete assistant output.
Ouajdi, Giannuzzo, et al.
• H4 (Scale interaction): Model size interacts with executiongrounded supervision, with larger models benefiting differently across evaluation metrics. Our study considers the 4B and 9B variants of Qwen3.5. We first benchmark the pretrained checkpoints with and without Thinking enabled. We then fine-tune the models using two supervision strategies: • Masked: Excludes tokens within <think> blocks from the training loss while retaining supervision over executable tool calls and final-answer tokens. • Full: Computes the training loss over both the reasoning trace and the final answer.
6.2
Dataset Construction and Splits
Generation protocol. Tool metadata (Profiler) were generated using Claude Sonnet 4.6. Abstract templates were generated with eight LLMs: Claude Sonnet 4.6, Claude Sonnet 4.5, Amazon Nova Pro, Amazon Nova Lite, Qwen3-235B-A22B, Gemini 2.5 Pro, Claude Opus 4.1, and GPT-5.5. For each domain, templates were generated in batches of up to 30 instances (10 per difficulty level) with a temperature of 0.7. Generating templates independently for each domain reduces cross-domain contamination. The same LLM that generates a template is subsequently used for trace validation and task synthesis. Grounded depth-first search (DFS) expanded 676 abstract templates (287 easy, 282 medium, and 107 hard) into 2,587 valid execution traces and 616 invalid traces, corresponding to a valid-trace rate of 80.8%. Task synthesis. Each validated trace is converted into three naturallanguage tasks (Basic, Intermediate, and Challenging), yielding 7,761 samples evenly distributed across difficulty levels. To preserve stylistic diversity, each task is generated by the same LLM that produced its source template, while the use of multiple generators prevents any single model from dominating the linguistic distribution. The annotator also generates a textual rationale describing how the recorded tool calls contribute to solving the task and producing the reference answer. Dataset splits. For computational efficiency, we fine-tune on a subset of the generated corpus. The data are partitioned at the execution-trace level using a fixed random seed and stratified by trace complexity (easy, medium, and hard). All questions derived from the same validated execution trace are assigned to the same split, preventing data leakage. Because each execution trace generates one question at each difficulty level, stratifying by trace complexity while keeping traces intact also preserves a balanced distribution of question difficulties across the training, validation, and test sets. All reported results are evaluated on the held-out test set (𝑛 = 200) and reported as mean ± standard error (SE) across tasks.
6.3
Training Protocol
To assess the quality and usefulness of the generated synthetic traces, we fine-tuned the target models on the training split of our dataset and evaluated their performance on the held-out test set. Our objective is not only to measure whether synthetic data can
Trace Synthesis for LLM Agents
Domain
Target Entities
Finance
Tickers, financial metrics
Research portfolio Supply Chain Music
AgenticDev ’26, October 12, 2026, Munich, Germany
Representative Tools
Operational Focus
Symbol matching, statement retrieval, equity screening Projects, domains, depen- Project matching, portfolio filtering, readiness dencies ranking SKUs, inventory, suppli- Product matching, demand forecasting, bottleers neck diagnosis Songs, artists, albums Content search, detail retrieval, chart ranking
Comparative financial analysis R&D portfolio planning and prioritization Logistics and demand planning Music search
Table 1: Operational domains and representative tool sets used to evaluate SyntheticAgentTraceQA. The evaluation spans four domains: three enterprise analytics environments (finance, research portfolio, and supply chain) and the genius_song_lyrics tool group from ToolBench. For each domain, the table summarizes the target entities, representative tools, and primary operational focus.
improve tool-use performance but also to understand how different supervision strategies over the reasoning process affect learning. All fine-tuning runs share the same optimization setup to ensure that observed differences are attributable solely to the supervision strategy rather than to changes in hyperparameters. Specifically, we use LoRA adapters [11] with rank 𝑟 = 16 and scaling factor 𝛼 = 32, train for three epochs using a learning rate of 2 × 10−4 , and quantize the base model to 4-bit NF4 precision. Due to computational resource constraints, training was performed on a fixed 1,000-example training split. Following standard instruction-tuning practice, the loss is computed only over assistant-generated tokens. To isolate the effect of reasoning supervision (H3), we compare two training objectives. Full supervision computes the loss over the entire assistant output, including the <think> blocks, whereas Masked supervision excludes the reasoning blocks and computes the loss only over tool calls and final answers. All other training settings are kept identical.
6.4
6.5
Following prior work on LLM-agent evaluation [50], we evaluate each model using three complementary groups of metrics: answer quality, tool behavior, and Reference-Trace Agreement. Answer quality. • Answer-completion rate: Percentage of tasks for which the agent produces a final answer. • Token-level F1: Macro-averaged token-overlap F1 between generated and reference answers across all the tasks. Answers are lowercased and tokenized by whitespace before computing set-based precision, recall, and F1. • Numeric Match: Macro-averaged fraction of reference numerical values correctly predicted across all the tasks. Numbers are extracted in order of appearance and matched positionally using an absolute tolerance of 0.01. Tool behavior. • Tool Attempt: Average number of tool-call attempts per task. • Tool Success: Average number of successfully executed tool calls per task. • Attempt-Success Gap (Δ):
Dynamic Evaluation Protocol
Evaluating tool-augmented language models requires more than comparing final answers, as agents may recover from failed tool calls, revise their plans, or reach the correct solution through alternative execution trajectories. We therefore evaluate all models in a multi-turn execution sandbox that executes generated tool calls and feeds the resulting observations back to the model. Each evaluation episode begins with a system prompt and a benchmark task. Models interact with the environment for up to eight turns using greedy decoding (temperature 0). Whenever a valid tool call is generated, the corresponding tool is executed and its output is appended to the conversation before the next generation step, enabling iterative planning and error recovery. To match our evaluation semantics, tool calls generated inside <think> blocks are never executed. Only actions emitted after the closing </think> tag are interpreted as executable API calls. The sandbox is fault tolerant: malformed arguments, invalid tool names, or schema violations return structured error messages instead of terminating the episode, allowing the model to recover from execution errors in subsequent turns.
Metric Groups
Δ=
Tool Attempt − Tool Success × 100. Tool Attempt
• Hallucinated Tool Rate: Fraction of tool-call attempts that invoke an unregistered tool. • Syntax Error Rate: Fraction of tool-call attempts rejected because of malformed tool-call syntax or invalid arguments. Reference-Trace Agreement. • Tool-set F1: F1 score between the predicted and reference sets of tool calls. • Sequence Similarity: Longest-common-subsequence (LCS) similarity normalized by the longer of the predicted and reference tool sequences. • Prefix Match: Harmonic-weighted fraction of the reference prefix matched before the first deviation.
7
Experimental Results
Following the evaluation protocol described in Section 6.4, we report results in three complementary categories: answer quality,
AgenticDev ’26, October 12, 2026, Munich, Germany
tool behavior, and Reference-Trace Agreement. We compare the base Qwen3.5-4B model with and without Thinking enabled, the larger Qwen3.5-9B model under the same inference settings, and two execution-grounded fine-tuned variants: Full, which computes the training loss over the entire assistant output, including the reasoning trace, executable tool calls, and final-answer tokens, and Masked, which excludes tokens within <think> blocks from the training loss while retaining supervision over executable tool calls and final-answer tokens.
7.1
Answer Quality
Table 2 reports final-answer performance in terms of Answer Completion, Token F1, and Numeric Match. Effect of reasoning at inference. Enabling reasoning consistently improves all three answer metrics before fine-tuning. For the 4B model, Answer Completion increases from 21.0% to 41.0%, while Token F1 rises from 0.04 to 0.07 and Numeric Match from 2.09% to 5.51%. Similar improvements are observed for the 9B model, where Answer Completion increases from 25.0% to 52.8%, accompanied by higher Token F1 and Numeric Match. These improvements coincide with higher Reference-Trace Agreement and more successful tool use. Effect of execution-grounded supervision. Execution-grounded fine-tuning further improves all three answer metrics. FT-4B-Masked increases Answer Completion to 59.5% while achieving the highest Token F1 (0.15) and Numeric Match (9.83%) among the 4B models. Likewise, FT-9B-Masked reaches the highest Answer Completion overall (86.0%) while also improving Token F1 and Numeric Match relative to the corresponding reasoning baseline. These results support H1, showing measured improvements in answer production and agreement with the reference answers following executiongrounded supervision. Nevertheless, Token F1 and Numeric Match remain relatively low across all models, indicating substantial room for improvement in matching the reference answers. Process supervision and model scale. Within both model sizes, Masked supervision consistently outperforms Full supervision across the reported metrics. For the 9B model, the Full variant performs below the Thinking baseline in Answer Completion and Token F1. Together, these results suggest that supervising executable tool calls and final answers, without optimizing over <think> tokens, is more effective than supervising the complete reasoning trace, partially supporting H3. The effect of model scale is mixed. FT-9B-Masked achieves the highest Answer Completion (86.0%), whereas FT-4B-Masked attains slightly higher Token F1 (0.15 vs. 0.14) and Numeric Match (9.83% vs. 8.25%). Thus, larger model size primarily improves answer production, while answer agreement with the reference remains comparable between the two masked models, partially supporting H4.
7.2
Tool Behavior
Table 3 summarizes tool-execution behavior, reporting the average number of attempted and successful tool calls together with the Attempt-Success Gap, Hallucinated Tool Rate, and Syntax Error Rate.
Ouajdi, Giannuzzo, et al.
Table 2: Final answer quality on the test set. Ans Compl. denotes the percentage of tasks with generated answers, Token F1 measures token-level overlap with reference answers, and Numeric Match measures the percentage of correctly predicted numerical values. Values are reported as mean ± standard error (SE), averaged over all test tasks. Higher is better (↑). Ans Compl. (%) ↑
Token F1 ↑
Numeric Match (%) ↑
Base-4B Base-4B-Thinking FT-4B-Masked FT-4B-Full
21.00 ± 2.89 41.00 ± 3.49 59.50 ± 3.48 35.00 ± 3.38
0.04 ± 0.01 0.07 ± 0.01 0.15 ± 0.01 0.10 ± 0.01
2.09 ± 0.81 5.51 ± 1.39 9.83 ± 1.78 5.90 ± 1.48
Base-9B Base-9B-Thinking FT-9B-Masked FT-9B-Full
25.00 ± 3.07 52.83 ± 3.97 86.00 ± 2.46 17.50 ± 2.69
0.04 ± 0.01 0.09 ± 0.01 0.14 ± 0.01 0.05 ± 0.01
2.19 ± 0.82 5.85 ± 1.60 8.25 ± 1.66 6.35 ± 1.61
Model
Effect of reasoning at inference. Reasoning substantially improves tool-use behavior before fine-tuning. Although the non-thinking models attempt more tool calls, they also exhibit substantially higher Attempt-Success Gaps and Hallucinated Tool Rates. Enabling reasoning reduces the average number of tool calls while increasing the proportion of successful executions. For example, the Attempt-Success Gap decreases from 18.9% to 6.1% for the 4B model and from 19.6% to 0.07% for the 9B model, with similar reductions in Hallucinated Tool Rate. These improvements coincide with higher Answer Completion and Reference-Trace Agreement. Effect of execution-grounded supervision. Execution-grounded fine-tuning further improves tool behavior. Both Masked models reduce the Attempt-Success Gap, Hallucinated Tool Rate, and Syntax Error Rate while requiring fewer tool calls than their corresponding reasoning baselines. FT-4B-Masked decreases the Attempt-Success Gap from 6.1% to 0.23%, while FT-9B-Masked maintains the low error rates of Base-9B-Thinking using fewer tool calls on average (4.04 vs. 5.52). These results support H1, showing measured improvements in tool execution following execution-grounded supervision. Process supervision and model scale. For both model sizes, the Masked and Full supervision strategies achieve low execution-error rates, with the Masked variants obtaining slightly lower AttemptSuccess Gaps and Hallucinated Tool Rates. This observation is consistent with the answer-quality results in Section 7.1, where Masked supervision also produced stronger performance. The effect of model scale is limited. Among the masked models, the 9B model performs fewer tool calls on average (4.04 vs. 5.33) while maintaining similarly low execution-error rates. Notably, the low Attempt-Success Gap observed for FT-9B-Masked is already present in the Base-9B-Thinking model, indicating that most of the improvement in execution reliability for the 9B model arises from inference-time reasoning, whereas execution-grounded supervision primarily reduces the number of tool calls. These observations partially support H4.
Trace Synthesis for LLM Agents
AgenticDev ’26, October 12, 2026, Munich, Germany
Table 3: Tool execution performance on the test set. Attempt and Success denote the average number of attempted and successfully executed tool calls per query. Δ represents attempt–success gap, while Halluc. and Syntax denote invalid tool calls and malformed tool calls, respectively. Values are mean ± (SE) across test tasks; lower is better (↓) for error metrics. Model
Attempt
Success
Δ (%) ↓
Halluc. (%) ↓
Syntax (%) ↓
Base-4B Base-4B-Thinking FT-4B-Masked FT-4B-Full
7.11 ± 0.14 6.37 ± 0.16 5.33 ± 0.18 4.99 ± 0.23
5.64 ± 0.23 5.99 ± 0.17 5.32 ± 0.18 4.95 ± 0.23
18.87 ± 2.72 6.11 ± 1.12 0.23 ± 0.16 0.64 ± 0.57
18.05 ± 2.69 3.19 ± 0.72 0.23 ± 0.16 0.50 ± 0.50
0.45 ± 0.38 2.58 ± 0.88 0.00 0.00
Base-9B Base-9B-Thinking FT-9B-Masked FT-9B-Full
6.80 ± 0.16 5.52 ± 0.21 4.04 ± 0.15 6.68 ± 0.18
5.31 ± 0.24 5.51 ± 0.21 4.04 ± 0.15 6.68 ± 0.18
19.60 ± 2.76 0.07 ± 0.06 0.07 ± 0.07 0.05 ± 0.05
17.25 ± 2.63 0.08 ± 0.08 0.07 ± 0.07 0.06 ± 0.06
1.56 ± 0.83 0.00 0.00 0.00
7.3
Table 4: Reference-Trace Agreement on the test set. Tool-set F1 measures overlap between the predicted and reference tool sets, Sequence Sim. measures normalized execution-order similarity, and Prefix measures agreement with the reference trace before the first deviation. Values are reported as mean ± standard error (SE) across test tasks; higher is better (↑). Model
Tool-set F1 ↑
Sequence Sim. ↑
Prefix ↑
Base-4B Base-4B-Thinking FT-4B-Masked FT-4B-Full
0.43 ± 0.02 0.53 ± 0.02 0.57 ± 0.02 0.54 ± 0.02
0.20 ± 0.01 0.27 ± 0.01 0.34 ± 0.02 0.34 ± 0.02
0.36 ± 0.02 0.36 ± 0.02 0.43 ± 0.02 0.34 ± 0.02
Base-9B Base-9B-Thinking FT-9B-Masked FT-9B-Full
0.46 ± 0.02 0.53 ± 0.02 0.57 ± 0.02 0.52 ± 0.02
0.22 ± 0.01 0.29 ± 0.02 0.38 ± 0.02 0.23 ± 0.02
0.32 ± 0.02 0.35 ± 0.02 0.42 ± 0.02 0.40 ± 0.02
Reference-Trace Agreement
Reference-Trace Agreement is evaluated independently of the final answer because multiple execution traces may correctly solve the same task. Table 4 reports structural agreement between the predicted execution trace and the reference trace using Tool-set F1, Sequence Similarity, and Prefix. Effect of reasoning at inference. Reasoning consistently improves Reference-Trace Agreement before any fine-tuning. Across both model sizes, the thinking variants achieve higher Tool-set F1, Sequence Similarity, and Prefix than their corresponding base models. These improvements indicate that explicit reasoning produces execution traces that more closely match the reference traces. In particular, the higher Sequence Similarity indicates closer agreement with the reference ordering of tool invocations. These findings support H2, showing that reasoning at inference provides a stronger foundation for execution-grounded reasoning by producing execution traces that more closely align with the reference traces. Effect of execution-grounded supervision. Execution-grounded fine-tuning further improves Reference-Trace Agreement. The masked models achieve the highest Tool-set F1 (0.57 for both model sizes) while substantially improving Sequence Similarity relative to the corresponding reasoning baselines. These results support H1, showing measured improvements in agreement with the reference execution traces following execution-grounded supervision. Process supervision and model scale. For the 4B models, both supervision strategies improve Reference-Trace Agreement relative to the reasoning baseline, although the Masked variant achieves higher Tool-set F1 and Prefix. The Full variant matches the Masked model in Sequence Similarity while performing worse on the remaining metrics, suggesting that supervising reasoning traces does not consistently improve agreement with the reference trace. This observation is consistent with the answer-quality results in Section 7.1, providing further evidence for H3. The effect of model scale is mixed. FT-9B-Masked achieves higher Sequence Similarity (0.38 vs. 0.34), whereas FT-4B-Masked attains a slightly higher Prefix score (0.43 vs. 0.42), and both models obtain the same Tool-set F1 (0.57). Thus, larger model size primarily improves agreement with the reference execution order, while other
Reference-Trace Agreement metrics remain comparable between the masked models, partially supporting H4.
7.4
Comparison with the Toucan Pipeline
We compare SyntheticAgentTraceQA with the direct-generation pipeline of Toucan [41]. The generated datasets are evaluated using an LLM-as-a-Judge (Gemini 3.1 Flash Lite) with a rubric assessing question realism, clarity, naturalness, and the contribution of tool use to the final answer. As shown in Table 5, the judge assigns higher scores to SyntheticAgentTraceQA across all four criteria. In addition, SyntheticAgentTraceQA achieves a higher NovelSum score (0.50 vs. 0.39), indicating greater semantic diversity among the generated user queries. Our reverse-generation pipeline also produces valid samples more efficiently, increasing the valid-trace rate to 80.8% compared with 15% for Toucan. As a result, the average generation cost ($0.05 vs. $0.12) and end-to-end latency (30 s vs. 114 s) per accepted trace are substantially reduced. Finally, we fine-tune a reasoning-enabled Qwen3.5-4B model on each pipeline’s dataset and evaluate it on the corresponding benchmark. As shown in Table 6, both datasets improve performance over their respective base models. SyntheticAgentTraceQA yields larger gains in Tool-set F1, Sequence Similarity, and Tool Hallucination Rate, whereas Toucan produces a larger improvement in Answer Completion Rate (+0.230 vs. +0.195). Because each model is evaluated on its own benchmark, these results should be interpreted as within-pipeline improvements rather than a controlled cross-pipeline comparison.
8
Discussion
Execution-grounded supervision improves tool-use behavior. Our experiments show that execution-grounded supervision improves tool behavior, Reference-Trace Agreement, and final-answer generation across the evaluated environments and model configurations. These gains suggest that validated execution traces provide supervision beyond task-answer pairs by exposing models to consistent tool choices, parameter dependencies, and intermediate outputs. This observation aligns with prior work showing that tool-use
AgenticDev ’26, October 12, 2026, Munich, Germany
Ouajdi, Giannuzzo, et al.
Table 5: Quality and diversity evaluation of synthetic data generated by SyntheticAgentTraceQA and Toucan [41]. LLMas-a-Judge scores are reported on a 1-10 scale, and NovelSum [46] measures semantic diversity of generated user queries. Higher values indicate better quality or diversity (↑). Metric
SyntheticAgentTraceQA
Toucan
Question Realism ↑ Question Clarity ↑ Question Naturalness ↑ Tools Improved Answer ↑
9.85 9.85 9.89 8.56
9.35 9.70 8.02 8.38
NovelSum ↑
0.50
0.39
Table 6: Comparison of fine-tuning results using datasets generated by SyntheticAgentTraceQA and Toucan [41]. Values report Qwen3.5-4B performance after fine-tuning, with improvements over the corresponding base model shown in parentheses. Each model is evaluated on its respective benchmark; therefore, the reported scores should be interpreted as within-pipeline results rather than as a controlled cross-pipeline comparison. Higher is better (↑) except for Tool Hallucination Rate (↓). Metric Tool-set F1 ↑ Sequence Similarity ↑ Tool Hallucination Rate ↓ Answer Completion Rate ↑
SyntheticAgentTraceQA
Toucan
0.612 (+0.086) 0.492 (+0.117) 0.000 (−0.003) 0.795 (+0.195)
0.596 (+0.061) 0.470 (+0.086) 0.002 (+0.001) 0.685 (+0.230)
demonstrations and interaction trajectories can improve agent capabilities [20, 25, 41]. Our contribution is not execution-first generation alone, but the combination of operational tool and parameter profiling, activedomain validation, abstract workflow generation, dependency-aware tool assignment, and execution-grounded task synthesis. The comparison with Toucan should be interpreted cautiously because the two approaches use different evaluation benchmarks; however, the results indicate that verified executions are a useful source of supervision for tool-augmented agents. Reasoning supervision introduces a process-outcome trade-off. Our results reveal a trade-off between supervising full assistant outputs and masking reasoning tokens. Masked supervision achieves stronger answer-generation performance across the evaluated settings, whereas Full supervision yields weaker answer completion and does not consistently improve reference-trace agreement—most notably for the 9B model, where Full supervision falls below the Thinking baseline on several metrics (e.g., Answer Completion drops to 17.5%). A possible explanation is that the reasoning traces used for supervision are post-hoc rationales describing how recorded tool calls support the answer, rather than necessarily the model’s unique reasoning process. Prior work suggests that reasoning supervision depends on trace quality, teacher-student compatibility, and the training objective [14, 45].
Execution-first synthesis provides validated supervision data. SyntheticAgentTraceQA constructs and validates executable workflows before synthesizing user tasks and answers, reducing invalid tool interactions during data generation. Similar trajectory-based approaches have highlighted the value of execution-grounded supervision for tool-augmented agents [2, 37]. Our dataset analysis further shows higher judge-assessed realism, clarity, and naturalness scores from an LLM evaluator, as well as higher NovelSum semantic diversity [46]. These results reflect automated evaluation measures rather than human-validated quality. Model scale effects are metric-dependent. Increasing model size does not uniformly improve all metrics. FT-9B-Masked achieves the highest answer-completion rate and sequence similarity, while FT-4B-Masked obtains slightly higher Token F1, Numeric Match, and Prefix agreement. Tool-set F1 is comparable between scales, suggesting that model capacity benefits specific aspects of agent behavior rather than providing uniform improvements. Limitations and future directions. Our evaluation focuses on controlled tool ecosystems with fixed schemas and execution. Due to computational constraints, fine-tuning used only a subset of the generated dataset. Future work will study scaling across different training set sizes, dynamic environments with changing APIs and longer-horizon workflows, and cross-domain tool composition.
9
Conclusion
We introduced SyntheticAgentTraceQA, an execution-first framework for generating supervision data for tool-augmented agents. Rather than generating tasks and inferring tool trajectories afterward, our framework first constructs and validates executable tooluse traces, then synthesizes user tasks, teacher-generated reasoning annotations, and reference answers from successful executions. This design enables controlled generation of tool-use examples with validated dependencies and executable workflows. Experiments across four tool ecosystems and multiple model configurations show that training on execution-grounded traces improves tool behavior, Reference-Trace Agreement, and final-answer generation on the evaluated benchmarks. Our analysis further reveals a trade-off between supervision strategies: Masked supervision, which excludes <think> tokens while retaining tool-call and answer supervision, achieves stronger answer-quality metrics, whereas Full supervision over the complete assistant output yields weaker answer completion and does not consistently improve reference-trace agreement—most notably for the 9B model, where Full supervision falls below the Thinking baseline on several metrics. These results suggest that the optimal supervision strategy depends on the target capability being optimized.
References [1] Lameck Mbangula Amugongo, Pietro Mascheroni, Steve Brooks, Stefan Doering, and Jan Seidel. 2025. Retrieval Augmented Generation for Large Language Models in Healthcare: A Systematic Review. PLOS Digital Health 4 (2025). doi:10.1371/ journal.pdig.0000877 [2] Jordi Armengol-Estapé, Quentin Carbonneaux, Tianjun Zhang, Aram H. Markosyan, Volker Seeker, Chris Cummins, M. Kambadur, Michael F. P. O’Boyle, Sida Wang, Gabriel Synnaeve, and Hugh Leather. 2025. What I Cannot Execute, I Do Not Understand: Training and Evaluating LLMs on Program Execution Traces. arXiv preprint arXiv:2503.05703 (2025). doi:10.48550/arXiv.2503.05703
Trace Synthesis for LLM Agents
[3] Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. 2024. Augmenting Large Language Models with Chemistry Tools. Nature Machine Intelligence 6 (2024), 525–535. doi:10.1038/s42256-02400832-8 [4] Zhiwei Dong, Ruihao Gong, Yang Yong, Shu Wu, Yongqiang Yao, Song-Lu Chen, and Xu-Cheng Yin. 2025. Tool Playgrounds: A Comprehensive and Analyzable Benchmark for LLM Tool Invocation. In ICASSP 2025 – IEEE International Conference on Acoustics, Speech and Signal Processing. 1–5. doi:10.1109/ICASSP49660. 2025.10890828 [5] Leon Engländer, Sophia Althammer, Ahmet Üstün, Matthias Gallé, and Tom Sherborne. 2026. Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity. arXiv preprint arXiv:2604.17609 (2026). [6] Alex J. Goodell, Simon N. Chu, D. Rouholiman, and Larry F. Chu. 2025. Large Language Model Agents Can Use Tools to Perform Clinical Calculations. npj Digital Medicine 8 (2025). doi:10.1038/s41746-025-01475-8 [7] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, et al. 2024. CRITIC: Large Language Models Can Self-Correct with ToolInteractive Critiquing. In International Conference on Learning Representations. [8] Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, and Yu Su. 2024. Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 7646–7663. [9] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. StableToolBench: Towards Stable LargeScale Benchmarking on Tool Learning of Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024. 11143–11156. arXiv:2403.07714 [10] Bingguang Hao, Zengzhuang Xu, Yuntao Wen, Xinyi Xu, Yang Liu, Tong Zhao, Maolin Wang, Long Chen, Dong Wang, Yicheng Chen, et al. 2026. From Failure to Mastery: Generating Hard Samples for Tool-Use Agents. arXiv preprint arXiv:2601.01498 (2026). [11] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. arXiv:2106.09685 [12] Caishuang Huang, Yang Qiao, Rongyu Zhang, Junjie Ye, Pu Lu, Wen Wu, Mengyu Zhou, Xiku Du, Tao Gui, Qi Zhang, and Xuanjing Huang. 2026. FinToolSyn: A Forward Synthesis Framework for Financial Tool-Use Dialogue Data with Dynamic Tool Retrieval. arXiv preprint (2026). [13] Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Gong, et al. 2024. MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use. In International Conference on Learning Representations. [14] Zixian Huang, Kaichen Yang, Xu Huang, Feiyang Hao, Qiming Ge, Bowen Li, He Du, Kai Chen, and Qipeng Guo. 2026. How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data. arXiv preprint arXiv:2604.14164 (2026). [15] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve RealWorld GitHub Issues?. In International Conference on Learning Representations. [16] Abdullatif Köksal, Marion Thaler, Ayyoob Imani, Ahmet Üstün, Anna Korhonen, and Hinrich Schütze. 2025. MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions. Transactions of the Association for Computational Linguistics 13 (2025), 1032–1055. [17] Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-Augmented Dialogue Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 8460–8478. [18] Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, and Jason Weston. 2026. AutoData: An Agentic Data Scientist to Create High Quality Synthetic Data. arXiv preprint arXiv:2606.25996 (2026). [19] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. arXiv:2304.08244 https: //arxiv.org/abs/2304.08244 [20] Zihan Li, Yifan Zhang, Ming Chen, Tianyi Zhao, et al. 2025. ToolGrad: Efficient Tool-Use Dataset Generation with Textual Gradients. arXiv preprint arXiv:2508.04086 (2025). arXiv:2508.04086 [21] Xukun Liu, Zhiyuan Peng, Xiaoyuan Yi, Xing Xie, Lirong Xiang, Yuchen Liu, and Dongkuan Xu. 2024. ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph. arXiv preprint arXiv:2403.00839 (2024). [22] Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. 2025. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025. 1160–1183.
AgenticDev ’26, October 12, 2026, Munich, Germany
[23] Siyuan Luo, Nairong Zheng, Lin Zhou, Tiankuo Yao, Shengyou Yuan, Haojia Yu, Cong Pang, Jiapeng Luo, and Lewei Lu. 2026. ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories. arXiv preprint arXiv:2606.11520 (2026). [24] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction Tuning with GPT-4. arXiv preprint arXiv:2304.03277 (2023). [25] Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xiang Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. arXiv preprint arXiv:2307.16789 (2023). doi:10.48550/arXiv.2307.16789 [26] Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2025. Tool Learning with Large Language Models: A Survey. Frontiers of Computer Science 19, 8 (2025), 198343. [27] Priyam Sahoo, Gaurav Mittal, Xiaomin Li, Shengjie Ma, Benjamin Steenhoek, Pingping Lin, and Yu Hu. 2026. AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation. arXiv preprint arXiv:2605.12925 (2026). [28] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, Vol. 36. arXiv:2302.04761 [29] Yuanzhe Shen, Zisu Huang, Zhengyuan Wang, Muzhao Tian, Zhengkang Guo, Chenyang Zhang, Shuaiyu Zhou, Zengjie Hu, Dailin Li, Jingwen Xu, Kaimin Wang, Wenhao Liu, Tianlong Li, Fengpeng Yue, Fenglu Hong, Cao Liu, and Ke Zeng. 2026. TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios. arXiv preprint arXiv:2602.01675 (2026). doi:10.48550/arXiv. 2602.01675 [30] Pradyumna Shome, Sashreek Krishnan, and Sauvik Das. 2026. Why Johnny Can’t Use Agents: Industry Aspirations vs. User Realities with AI Agents. In Proceedings of the ACM Conference on AI and Agentic Systems (CAIS ’26). ACM, 597–619. doi:10.1145/3786335.3813140 [31] Harsh Soni. 2026. ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents. arXiv preprint arXiv:2607.04686 (2026). [32] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. arXiv:2306.05301 [33] Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu. 2024. A Survey on Data Selection for LLM Instruction Tuning. arXiv preprint arXiv:2402.05123 (2024). [34] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A Survey on Large Language Model Based Autonomous Agents. Frontiers of Computer Science 18, 6 (2024), 186345. [35] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. arXiv:2212.10560 [36] Yue Wang, Xinrui Wang, Juntao Li, Jinxiong Chang, Qishen Zhang, Zhongyi Liu, Guannan Zhang, and Min Zhang. 2023. Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Models. arXiv preprint arXiv:2308.12711 (2023). doi:10.48550/arXiv.2308.12711 [37] Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Jing Huang, Jiri Gesi, Xianfeng Tang, Chen Luo, Yisi Sang, Hanqing Lu, Manling Li, and Dakuo Wang. 2026. Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents. arXiv preprint arXiv:2601.20144 (2026). doi:10.48550/arXiv. 2601.20144 [38] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou, et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, Vol. 35. 24824–24837. [39] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2025. The Rise and Potential of Large Language Model Based Agents: A Survey. Science China Information Sciences 68, 2 (2025), 121101. [40] Jingxu Xie, Dylan Xu, Xuandong Zhao, and Dawn Song. 2025. AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents. arXiv preprint arXiv:2506.14205 (2025). [41] Zhangchen Xu, Adriana Meza Soria, Shawn Tan, Anurag Roy, Ashish Sunil Agrawal, Radha Poovendran, and Rameswar Panda. 2025. Toucan: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments. arXiv preprint arXiv:2510.01179 (2025). [42] Chen Yang, Ran Le, Yun Xing, Zhenwei An, Zongchao Chen, Wayne Xin Zhao, Yang Song, and Tao Zhang. 2025. ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset. arXiv preprint arXiv:2511.15718 (2025). [43] Jie Yang, Honglin Guo, Lifeng Ji, Jiazheng Zhou, Rui Zheng, Zhikai Lei, Shuo Zhang, Zhiheng Xi, Shichun Liu, Yuxin Wang, Bo Wang, Y. Zheng, Tao Gui, and Xipeng Qiu. 2026. ABC-Bench: Benchmarking Agentic Backend Coding in
AgenticDev ’26, October 12, 2026, Munich, Germany
Real-World Development. arXiv preprint arXiv:2601.11077 (2026). doi:10.48550/ arXiv.2601.11077 [44] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems, Vol. 37. 50528–50652. [45] Yuming Yang, Mingyoung Lai, Wanxu Zhao, Xiaoran Fan, Zhiheng Xi, Mingqi Wu, Chiyue Huang, Jun Zhao, Haijun Lv, Jian Tong, et al. 2026. Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 42123–42150. [46] Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025. Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and a Reliable Metric. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 18530–18549. [47] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. 𝜏 bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045 (2024). arXiv:2406.12045 [48] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv preprint arXiv:2305.10601 (2023). arXiv:2305.10601 [cs.CL] https://arxiv.org/abs/2305.10601 [49] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations. arXiv:2210.03629
Ouajdi, Giannuzzo, et al.
[50] Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2026. A Survey on Evaluation of LLM-based Agents. In Findings of the Association for Computational Linguistics: ACL 2026. 26690–26714. [51] Peijie Yu, Wei Liu, Yifan Yang, Jinjian Li, Zelong Zhang, Xiao Feng, and Feng Zhang. 2026. Benchmarking LLM Tool-Use in the Wild. arXiv preprint (2026). [52] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. 2025. Instruction Tuning for Large Language Models: A Survey. Comput. Surveys 58 (2025), 1–36. doi:10.1145/3777411 [53] Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, and Hayato Yamana. 2024. ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Miami, Florida, USA, 11388–11422. doi:10.18653/ v1/2024.emnlp-main.637 [54] Yifan Zhang, Jingxuan Li, Zhiyun Wang, Zhengting He, Qingfeng Guan, Jianfeng Lin, and Wenhao Yu. 2024. Geospatial Large Language Model Trained with a Simulated Environment for Generating Tool-Use Chains Autonomously. International Journal of Applied Earth Observation and Geoinformation 136 (2024), 104312. doi:10.1016/j.jag.2024.104312 [55] Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. In Advances in Neural Information Processing Systems, Vol. 36. 50117–50143.