arXiv CSopen access

RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents Ruishan Fang1,2,4,∗

arXiv:2606.19047v1 [cs.AI] 17 Jun 2026

1 Inclusion AI, Ant Group

Siyuan Lu1,2,3,4

2 Zhejiang University

Model

Chenyi Zhuang1,†

Tao Lin4,1,†

3 Shanghai Innovation Institute

4 Westlake University

§ AWorld-RL/RODS

Abstract Multi-turn tool-use RL is bottlenecked by the rapid depletion of informative samples in static datasets. We observe that the gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound. Consequently, samples near the agent’s capability boundary—where successes and failures are roughly balanced—contribute disproportionately large policy gradients. As training progresses, this boundary continuously shifts, which gradually depletes the pool of informative samples in a static dataset. We propose RODS (Reward-driven Online Data Synthesis) to resolve this depletion. RODS closes the loop between RL training and data generation by repurposing the progress reward variance as a practical, zero-cost boundary detector that requires no extra inference beyond the rollouts already computed for training. It continuously identifies such boundary samples, synthesizes new multi-turn variants matching their structural complexity (e.g., API topology and dependency depth) via a skill-aligned resampling pipeline, and manages a dynamic replay buffer that co-evolves with the policy. Starting from 400 human seeds and maintaining an active training pool of ∼800 samples, RODS achieves comparable performance to a 17K-sample offline pipeline while requiring roughly 20× fewer trajectories, and improves over fixed-data RL and environment augmentation in our controlled setting.

1

Introduction

Large Language Model (LLM)-based agents (Wang et al., 2024; Weng, 2023) have demonstrated strong potential in solving complex tasks by using external tools and environment interactions (Anthropic, 2026; OpenAI, 2026). Recent work frames tool-use agent training as an RL problem (Agentic RL), optimizing policies through direct environment interaction (Jin et al., 2025; Feng et al., 2025a; Wang et al., 2025; Liu et al., 2025). However, extending Agentic RL to multi-turn tool-use environments introduces unique challenges that have not been fully addressed by existing methods, as it simultaneously demands large training corpora and long-horizon structural coherence (Patil et al., 2025; Yao et al., 2024; Barres et al., 2025). Current RL systems face three main challenges in this regime (see Figure 1). First, high-quality multi-turn datasets are scarce (C1) due to prohibitive annotation and validation costs, as seen in BFCL V3 (Patil et al., 2025) which contains only 800 samples. Second, as agent capabilities evolve, static datasets suffer from shifting boundaries (C2), leading to signal depletion and wasted compute on mastered or unreachable tasks (Li et al., 2025a; Dai et al., 2025). Third, on-the-fly synthesis often causes semantic disjointedness (C3) by lacking a unifying goal, which produces trajectories without coreference or coherence that fail to teach reliable reasoning. Despite these hurdles, existing efforts generally fall into two categories: large-scale offline synthesis and online environment augmentation. Offline pipelines (Prabhakar et al., 2025; Xu et al., 2025b;a) address data scarcity (C1) by generating massive corpora upfront, yet they remain decoupled from the training loop ∗ This work was supported by Ant Group Research Intern Program. † Corresponding Authors.

1

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Figure 1: Current limitations in static data training vs. RODS dynamic synthesis paradigm. (a) Data scarcity (C1): High-quality multi-turn tool-use datasets require massive human annotation effort. (b) Capability vs. static Data (C2): As the model learns, static data becomes mastered, leading to a loss of gradient signal. (c) Semantic disjointedness (C3): Naively stitching single-turn queries creates disjointed interactions. (d) The RODS solution: A closed-loop data engine continuously ingests static data to detect the evolving capability boundary, synthesizing coherent, targeted active corpora where the model needs them.

and thus fail to track the model’s evolving capability boundary (C2). Conversely, RL training methods like EnvTuning (Lu et al., 2025) enable learning from minimal data but are ultimately constrained by the signal depletion of their fixed seed corpora (C2). While online self-play and self-evolution approaches (Acikgoz et al., 2026; Li et al., 2025c; Zhai et al., 2025) theoretically close this loop, unconstrained zero-data generation frequently fails to maintain semantic coherence in complex multi-turn settings (C3), leading to disjointed trajectories that offer little pedagogical value. How can we train multi-turn tool-use agents under extreme data scarcity, by dynamically synthesizing data that strictly tracks the evolving capability boundary while maintaining multi-turn semantic coherence? We propose RODS (Reward-driven Online Data Synthesis) to bridge this gap, a framework that tightly couples data generation with the RL training loop. Our approach rests on a simple insight: the Progress Reward in policy gradient methods like GRPO serves as a boundary detector that reuses existing rollout statistics—since rollouts are already computed for advantage estimation—because rollout variance is highest near the agent’s capability boundary (µ ≈ 0.5, as suggested by the Popoviciu upper bound). By synthesizing data in this high-variance region and managing its lifecycle through a co-evolving replay buffer, RODS maintains a continuous stream of gradient-informative samples. This paper makes three main contributions. To address signal starvation inherent to static datasets (C2), we propose RODS, a reward-driven boundary expansion method that repurposes the RL progress reward to identify and expand boundary tasks in real time. To preserve multi-turn semantic coherence (C3), we introduce a skill-aligned resampling synthesis pipeline that anchors novel trajectories to the complexity profiles of verified seeds. Instead of simple entity substitution, it preserves the functional dependency structure of the seed while generating novel narratives and environment states. Finally, we demonstrate significant data efficiency (C1): starting from 400 human seeds and maintaining an active training pool of ∼800 samples, RODS achieves performance comparable to a 17K-sample offline pipeline (using roughly 20× less data) and improves over fixed-data RL in our controlled setting.

2

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

2

Related Work

Data synthesis for tool use. Although high-quality human-annotated benchmarks (Li et al., 2023; Yao et al., 2024; Barres et al., 2025; Chen et al., 2025; Xie et al., 2024; Maekawa et al., 2025) provide rigorous evaluation protocols for multi-turn tool use, their limited scale of typically hundreds of instances is insufficient for training reliable agentic policies. This data scarcity has prompted a shift toward large-scale offline synthesis. Frameworks such as APIGen-MT (Prabhakar et al., 2025), TOUCAN (Xu et al., 2025b), and Magnet (Yin et al., 2025) prioritize corpus scale and structural complexity, often generating millions of trajectories upfront. While effective for pre-training, these static pipelines remain decoupled from the training process, producing uniform data distributions that cannot track the model’s evolving capability boundary (C2). Directed synthesis methods attempt to narrow this focus by targeting specific failure modes. FunReasonMT (Xu et al., 2025a) utilizes environment-API graphs and advanced tool-query synthesis to tackle hard query generation, while LoopTool (Zhang et al., 2025) employs a feedback loop to correct algorithmic errors. Although these represent a clear advance in targeting known weaknesses, they remain offline snapshots that cannot adapt to the shifting capability boundary during active RL training. RODS addresses this limitation by introducing a reward-driven synthesis loop that operates during training, ensuring that the generated data remains at the model’s immediate capability boundary. Online RL and curriculum learning. Data efficiency in agentic RL is traditionally addressed through environment simulation or corrective feedback. Simulation-based approaches like ScaleEnv (Tu et al., 2026), Agent World Model (Wang et al., 2026), and Simia-Agent (Li et al., 2025b) construct high-fidelity interactive loops to maximize the signal extracted from existing tasks. EnvTuning (Lu et al., 2025) further improves efficiency by orchestrating a four-stage curriculum with actionable environment augmentation. However, these methods are constrained by the fixed diversity of their seed corpora; as the agent improves, the proportion of gradient-informative samples in the static pool shrinks. Self-play paradigms (Acikgoz et al., 2026; Liu et al., 2026) attempt to resolve this via absolute zero-data generation, where a generator proposes tasks from scratch. While theoretically appealing, such unconstrained generation frequently struggles with the long-horizon logical chains and interdependent API calls required for multi-turn tool use (C3). RODS takes a highly complementary approach by using a critically small set of human data as structural anchors. By reusing the inherent rollout variance of GRPO as a boundary detector and grounding synthesis in skill-aligned resampling, RODS enables a targeted, complexity-aligned curriculum that bypasses the high-variance search of generating multi-turn logic from scratch. Our approach aligns with prioritized data selection methods, such as prioritized experience replay (Schaul et al., 2016), hard-example mining (Shrivastava et al., 2016), and competence-based curricula (Platanios et al., 2019), which focus learning on the most informative samples. The key distinction is that these classical methods select or re-weight existing experiences, whereas RODS generates new data at the reward-defined boundary, combining the targeting principle of prioritized replay with the distributional expansion of generative synthesis.

3

RODS

To resolve multi-turn data scarcity and signal depletion, we introduce RODS, a reward-driven data synthesis framework. RODS maintains a saturated learning signal through three co-evolving modules. First, rewardbased seed detection identifies high-variance boundary tasks (§3.2). Second, skill-aligned synthesis generates coherent, structurally-isomorphic variants (§3.3). Third, dynamic replay buffer management tracks the shifting capability boundary (§3.4). 3.1 Problem Formulation and Design Motivation Multi-turn tool use is formalized as a POMDP where an agent resolves interdependent queries through API calls and environment feedback (details in Appendix A). We use GRPO (Shao et al., 2024) for optimization, though our variance-based boundary tracking extends to other trajectory-sampling methods like PPO (see Appendix K). Reward sparsity and progress reward. Because sparse binary rewards fail to assign credit across complex long-horizon trajectories (Feng et al., 2025b), we adopt the progress reward (R P ∈ [0, 1]) from Lu et al. (2025): R P = 1/N ∑tN=1 (rtstate · rtexec ). This assigns continuous credit for partial completion, enriching the advantage signal. Note that ground truth is used exclusively for simulation-based reward computation; the policy never observes it.

3

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Figure 2: The RODS closed-loop RL-data synthesis architecture. (Top) Reward Calculation & Agent Training: The agent trains on a mixed dataset via GRPO; the Progress Reward (R P ) identifies boundary seeds and feeds them back to the data engine. (Bottom-left) Plan & Execute: The Planner Agent selects a function sequence from the API graph; the Execution Orchestrator instantiates it on a simulation environment with environment feedback (Repeat × N), producing an executable raw trajectory. (Bottom-right) Refine & Judge: A Query Agent converts the trajectory into per-turn naturallanguage queries; a Rewrite Agent grounds all queries in the Planner’s narrative for cross-turn coherence; a Critique Agent validates semantic quality with a feedback loop (Repeat × N), yielding validated synthetic datasets that are injected into training.

Design heuristic: variance peaks near the capability boundary. In policy gradient methods like GRPO (Shao et al., 2024), the gradient signal density of a sample is governed by its rollout reward variance. While Popoviciu’s inequality (Popoviciu, 1935) shows that the variance of a bounded variable is upper-bounded by µ(1 − µ) (maximizing at µ ≈ 0.5), this provides a theoretical upper bound rather than an exact description of the continuous reward variance. We nonetheless validate this as an empirically supported heuristic: the continuous progress reward variance peaks near the capability boundary (µ ≈ 0.5), as confirmed by the per-task variance analysis in Figure 3 (right). Thus, this heuristic underpins the design of RODS: identifying these boundary samples serves as an effective, cheap proxy for finding the highest concentration of informative gradient signals (see Appendix B.5 for theoretical details). 3.2 Reward-Based Seed Data Detection To operationalize the boundary-targeting heuristic identified in §3.1, we use the average Progress Reward r̄i across K rollouts as a density-based probe to partition the task space D into three dynamically evolving regions: Dmastered = { xi : r̄i > α+ } , Dboundary = { xi : α− ≤ r̄i ≤ α+ } , Dhard = { xi : r̄i < α− } ,

(1)

where α− and α+ are boundary thresholds (set to 0.20 and 0.85 respectively, see Appendix B). We define the tasks within the intermediate region xi ∈ Dboundary as boundary seeds, as they represent the model’s immediate capability boundary and harbor the highest potential for informative gradient signals. Type-quota seed emission. At each training step, we select up to M seeds from Dboundary . To ensure diverse skill coverage, we enforce a per-type quota Mτ (e.g., base/long-context, missing-function, missing-parameter) such that ∑τ Mτ = M. Within each type, candidates are ranked by the variance-proxy ϕ(r̄i ) = 4r̄i (1 − r̄i ) in descending order, prioritizing seeds closest to the capability midpoint (r̄i = 0.5) where gradient signal 4

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

potential is highest (cf. C2). A temporal exclusion window of c steps prevents redundant sampling, and the selected seeds are asynchronously dispatched to the synthesis pipeline. 3.3 Skill-Aligned Data Synthesis The synthesis pipeline transforms boundary seeds into novel, structurally valid variants that preserve the informative complexity of the original task (see Appendix P for detailed seed-to-variant examples). Rather than simple paraphrasing, we enforce structural similarity: we extract the complexity profile Φ( xseed ) (e.g., the directed acyclic graph of API dependencies and parameter flows) and sample a new task x ′ ∼ p(·|Φ( xseed )) such that Φ( x ′ ) approximates Φ( xseed ) in dependency depth and API topology. This constraint ensures that x ′ has structural difficulty similar to the seed, placing it near the capability boundary (C2) as validated by the injection reward analysis in Figure 3 (middle), while forcing the model to generalize across novel abstract logic and environment states, preventing overfitting to static execution paths (C3). We implement this via a five-stage multi-agent pipeline (see Appendix C for full prompts): In Stage I (schema-guided planning, addressing C2 and C3), the planner agent ingests xseed to design an execution plan S ′ and an underlying narrative N , using failure histories (Favoid ) to bypass known execution bottlenecks. In Stage II (feedbackdriven execution, addressing C1), an execution orchestrator instantiates S ′ in a simulated environment C0 to produce a trajectory with ground-truth dependencies. An error critic applies multi-tier mitigation to find a valid instantiation across Kmax = 3 attempts, while a query agent maps each turn to natural-language queries without exposing function details. In Stage III (holistic semantic grounding, addressing C3), a rewrite agent renders the entire multi-turn trajectory simultaneously based on N . Unlike greedy turn-by-turn generation, this approach anchors all turns to a single goal, resolving semantic disjointedness and ensuring natural conversational flow. In Stage IV (critique and refinement), variants undergo rule-based checks and LLM scoring via a critique agent. A feedback loop with the rewrite agent prunes logic flaws and fixes phrasing. Finally, Stage V (optional adversarial augmentation) injects structural exceptions, such as missing tools or blurred parameters, to force clarification turns and improve out-of-distribution robustness. 3.4 Dynamic Replay Buffer Management To maintain a high-fidelity training distribution, we implement a dual-control lifecycle that balances expansion flow with stock relevance. Expansion flow control: staged injection. To prevent instability from abrupt distributional shifts—where a sudden influx of synthetic data destabilizes RL gradients—we adopt a staged injection protocol. Synthesized variants are initially held in an asynchronous candidate queue and merged into the active replay pool strictly at epoch boundaries. Specifically, tasks seeded during epoch n are staged and injected at the start of epoch n+1. To ensure gradual integration, the per-epoch injection volume is capped at β · |Dactive |; any excess is deferred to a persistent staging buffer for subsequent epochs. Upon each injection, the sampling distribution is re-initialized to ensure uniform coverage of the expanded pool. Active stock management: multi-layer retirement. We maintain pool informativeness through a three-layer retirement mechanism that tracks the shifting capability boundary. The first layer, burn-in filtering, discards new variants with initial rewards below ϵtrial to prune tasks beyond the immediate exploration horizon. + The second layer, boundary-drift eviction, removes tasks that have drifted into mastered (r̄i > αretire ) or − unsolvable (r̄i < αretire ) zones after nmin observations. The third layer applies variance-prioritized pruning: if the pool exceeds Pmax , samples are retired based on reward variance ϕ(r̄i ) = 4r̄i (1 − r̄i ), preserving the highest gradient signals (C2). We also prune generated variants that remain unsampled for an extended period to prevent stale data accumulation. This dual-control architecture ensures the replay buffer remains a high-fidelity mirror of the capability boundary while maintaining the stability of the RL optimization loop. We employ default hyperparameters across all settings to ensure a reliable and fair evaluation; details are provided in Appendix J.

4

Experiments

Our experiments are designed to answer five questions: (Q1) Does boundary-targeted expansion outperform both fixed-data RL and environment augmentation? (Q2) Does boundary-expanded training generalize to OOD tasks? (Q3) How does RODS dynamically expand the data space without catastrophic distribution shifts? (Q4) How data-efficient is RODS compared to large-scale offline synthesis? (Q5) Which components of RODS contribute most to performance?

5

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Model

Size

Overall

Base

Miss Func

Miss Param

Long Context

Reference Models GPT-4o-2024-11-20 DeepSeek-V3.2-Exp FunReason-MT-4B (17K offline)

671B 4B

42.50 44.88 56.50

55.50 55.00 63.00

34.50 49.00 53.00

29.00 27.00 40.00

51.00 48.50 55.00

Qwen2.5-7B-Instruct + Static dataset + EnvTuning + RODS (ours) Llama-3.1-8B-Instruct + Static dataset + EnvTuning + RODS (ours) Qwen3-4B-Instruct + Static dataset + EnvTuning + RODS (ours)

7B 8B 4B -

7.00 36.92 (+29.92) 37.75 (+30.75) 40.25 (+33.25) 5.48 28.25 (+22.77) 28.38 (+22.90) 30.88 (+25.40) 22.13 50.00 (+27.87) 50.50 (+28.37) 56.00 (+33.87)

9.33 50.33 (+41.00) 51.50 (+42.17) 54.00 (+44.67) 6.15 28.20 (+22.05) 28.00 (+21.85) 32.00 (+25.85) 26.50 62.00 (+35.50) 64.00 (+37.50) 68.00 (+41.50)

9.33 40.33 (+31.00) 41.00 (+31.67) 43.50 (+34.17) 6.80 25.85 (+19.05) 25.50 (+18.70) 28.00 (+21.20) 21.00 51.00 (+30.00) 52.00 (+31.00) 59.00 (+38.00)

6.33 29.33 (+23.00) 30.50 (+24.17) 33.50 (+27.17) 3.20 22.15 (+18.95) 23.00 (+19.80) 24.50 (+21.30) 15.50 35.00 (+19.50) 35.00 (+19.50) 44.00 (+28.50)

3.00 27.67 (+24.67) 28.00 (+25.00) 30.00 (+27.00) 5.75 36.80 (+31.05) 37.00 (+31.25) 39.00 (+33.25) 25.50 52.00 (+26.50) 51.00 (+25.50) 53.00 (+27.50)

Table 1: In-distribution performance on BFCL V3 multi-turn (Tier 1: controlled RL comparisons). All RL methods share the same 400 training samples and GRPO setup. FunReason-MT-4B (Tier 2) is included for data-scaling reference. For a detailed comparison with 20+ models, see Appendix Q. Red text indicates improvement over the base model.

Benchmark. We evaluate on the multi-turn subset of BFCL V3 (Patil et al., 2025), which includes 800 samples across four balanced splits: Base, Missing Function, Missing Parameter, and Long-Context. Following Lu et al. (2025), we reserve 400 samples (100 per split) for training and use the remaining 400 for held-in evaluation. For OOD testing, we adopt the BFCL V4 multi-turn tracks, τ 2 -bench (Barres et al., 2025), and the ACEBench Agent split (Chen et al., 2025) as our held-out test sets. Detailed benchmark and evaluation details are provided in Appendix O. Training configuration. We train Qwen3-4B-Instruct using GRPO (Shao et al., 2024) (K = 16 rollouts) on 8×A100 GPUs via a three-stage curriculum (shared across baselines for fair comparison, details in Appendix J). Stage transitions occur when validation performance plateaus (changing <1% over one full epoch) and gradient norms converge, following the protocol of Lu et al. (2025). The protocol isolates syntactic and logical acquisition: (1) Stage 1 (Format): Training on 100 Base samples using a format reward Rformat to isolate syntactic acquisition (XML, function names, arguments) before task reasoning (see Appendix H); (2) Stage 2 (Base Reasoning): Continuing on the Base split with the Progress Reward R P to build a stable reasoning anchor without expansion; and (3) Stage 3 (Full data + expansion): Scaling to 400 samples across all splits. For RODS, the synthesis engine targets high-variance boundary tasks, using the Stage 2 foundation to drive generalization. To evaluate cross-model generalizability, we also report results on Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. Data synthesis. The RODS synthesis pipeline uses Qwen3-32B deployed via vLLM on a separate 8×A100 cluster running asynchronously alongside training. It achieves a seed-to-injection latency of ∼1 training step, introducing no idle time into the RL loop (boundary thresholds α− = 0.20, α+ = 0.85, Pmax = 400). Full configuration and detailed cost breakdown are provided in Appendix J and N. Baselines and references. We structure our comparisons into two tiers. Tier 1: Controlled RL comparisons (consistent 400-sample seeds) includes S TATIC DATASET (baseline trained only on the fixed seed set without generation), E NV T UNING (Lu et al., 2025) (actionable environment enrichment), and RODS (our full dynamic boundary expansion system). Tier 2: Data efficiency references (varying scales) includes F UN R EASON -MT4B (Xu et al., 2025a) (20× more data) and state-of-the-art models like GPT-4o and DeepSeek-V3.2-Exp. 4.1 Results (Q1): Boundary Expansion vs. Fixed-Data RL and Environment Augmentation Table 1 presents the controlled comparison among Tier 1 methods. All three methods share the same 400 training samples, the same GRPO configuration, and the same progress reward; the only variable is the strategy for addressing gradient signal depletion in Stage 3. Boundary expansion achieves the best results in our controlled setting. RODS attains the highest overall scores across the three model families in our experiments. On the Qwen3-4B-Instruct base model, RODS improves overall multi-turn performance by +33.87% (reaching 56.00%), surpassing both the “Static dataset” 6

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Data Space Partition Evolution

800 600 400 200 0

0

100

200

300

400

500

Training Steps

600

700

800

0.035

0.8

0.6

0.4

0.2

0.0

Generated Data (Score) Optimal Boundary Zone Center (p = 0.5) 0

100

200

300

400

500

Training Steps

600

700

800

Mean Reward Variance (Var[RP])

Mean Progress Reward (RP)

1000

Cumulative Number of Tasks

Reward Trajectory of Synthesized Variants 1.0

Static Capacity Limit Mastered (Evicted) Data Active Generated Data Original Static Data

0.030

n=2383

0.025 0.020 0.015

n=229

2.2×

2.0× n=2188

0.010 0.005 0.000

Low (RP < 0.25)

Boundary (0.25 RP 0.75)

High (RP > 0.75)

Figure 3: The RODS dynamic synthesis mechanism during stage 3 training. (Left) Evolution of the data space partition. RODS breaks the static capacity limit (400 tasks) by continuously synthesizing active boundary data and evicting mastered tasks, effectively expanding the training curriculum without exploding memory. (Middle) Reward trajectory of newly synthesized variants upon injection. The continuous data generation strictly anchors the mean progress reward within the boundary zone ([0.25, 0.75]), confirming successful targeting of the capability frontier. (Right) Empirical validation of the variance heuristic. Across 4,800 per-task measurements (K = 16 rollouts each), rollout reward variance in the boundary zone is 2.0–2.2× higher than in the low-reward or high-reward regions, confirming that gradient signal concentrates near the capability boundary.

baseline (50.00%) and the EnvTuning baseline (50.50%). This suggests that dynamically synthesized boundary data provides complementary gradient signals beyond what fixed datasets or enriched environment feedback alone supply. The gains are consistent across all four sub-splits, indicating that boundary targeting benefits diverse task complexities rather than overfitting to a specific category. RODS achieves comparable performance to large-scale offline synthesis (FunReason-MT-4B) while using roughly 20× fewer trajectories. Furthermore, it improves over fixed-data RL and environment augmentation in our reported runs under the same 400-sample controlled setup. While RODS also achieves competitive benchmark performance against larger models (e.g., DeepSeek-V3.2), we emphasize that the controlled comparisons within the same 400-sample setup constitute the primary evidence for the mechanism’s efficacy. Data expansion vs. environment augmentation: two orthogonal strategies. The comparison between RODS and EnvTuning is informative, as both address the same problem (gradient sparsity under data scarcity) but via opposite mechanisms. EnvTuning enriches the feedback signal on existing data by providing corrective hints upon failure, effectively extracting more value from each fixed sample. RODS instead expands the data distribution itself by injecting new variants precisely at the capability boundary. The observed empirical advantage of RODS (e.g., a +5.50% absolute lead over EnvTuning on Qwen3-4B) suggests that expanding the data distribution at the capability boundary is more effective than deepening feedback on fixed samples in extreme data-scarce RL settings. The structural diversity of boundary-targeted variants provides gradient signals that an exhausted static pool cannot. Because RODS achieves this without any structural modification to the training environment’s feedback mechanism, it is environment-agnostic and readily applicable to new API domains. We investigate whether the two strategies are complementary in Appendix I. OOD generalization (Q2). We evaluate on BFCL V4, τ 2 -bench, and the ACEBench Agent split to test whether boundary-expanded training yields generalizable reasoning or merely in-distribution pattern matching. The OOD improvements of RODS over the base model (detailed in Appendix L) are consistent with our design hypothesis in Section 3.3: by preserving API execution plans while heavily randomizing surface variables and environment states, structural isomorphism prevents the policy from overfitting to specific textual cues, forcing it to internalize abstract, generalizable multi-turn reasoning patterns. 4.2 Mechanism validation (Q3) via Data Space Evolution Breaking the static capacity limit via boundary anchoring. Figure 3 illustrates the internal mechanics driving the data efficiency reported in Table 1. Rather than accumulating data indefinitely, RODS treats the active training pool as a sliding window over the capability space. The left panel shows how the system breaches the 400-task static limit by introducing new variants while retiring mastered ones, generating over 800 unique tasks in total while keeping the active pool bounded by Pmax . The middle panel confirms that newly synthesized variants land inside the boundary zone (R P ∈ [0.25, 0.75]). The right panel empirically validates the underlying heuristic: across 4,800 per-task measurements, rollout reward variance in the boundary zone is 2.0–2.2× higher than in the mastered or too-hard regions, confirming that gradient signal 7

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Sub-splits Performance

Overall Average Performance 58 57

BFCL V3 Performance (%)

65

56.00

56

60

55

55

54.50

54

50

53

53.25

53.75

52

45 Base Missing Func. Missing Param. Long-Context

40 35 0

50

100

200

Generated Pool Cap (Pmax)

51 50 49

400

50.00 0

Avg. (Overall)

50

100

200

Generated Pool Cap (Pmax)

400

Figure 4: Data scaling analysis. Performance on BFCL V3 as a function of the generated pool cap Pmax . Pmax = 0 corresponds to the static baseline (static pool of 400 items). (Left) Performance across sub-splits (Base, Missing Functions, Missing Parameters, Long-Context). (Right) The overall average performance (solid orange) steadily improves as Pmax increases, with specific average scores annotated.

concentrates near the capability boundary and thus alleviating gradient starvation (C2). 4.3 Data efficiency analysis (Q4): boundary targeting vs. blind scaling Our central claim is that where to synthesize data matters more than how much. We evaluate this by (a) scaling the boundary-targeted data pool in RODS, and (b) comparing its efficiency against FunReason-MT (Xu et al., 2025a), a state-of-the-art large-scale offline synthesis pipeline. Scaling the maximum generated pool size (Pmax ). We vary Pmax ∈ {0, 50, 100, 200, 400} on Qwen3-4BInstruct while maintaining the standard three-stage curriculum. • Even Pmax = 50 (∼12% expansion) yields a meaningful improvement over the static baseline, demonstrating the high marginal value of a small amount of targeted boundary data. • Performance scales with Pmax but exhibits diminishing returns beyond Pmax = 200, as the boundary region of the 400 original samples becomes fully covered. • This validates Pmax = 400 as a practical operating point and establishes that boundary data has a significantly higher per-sample training value than uniformly sampled data. Comparison with large-scale offline synthesis. To contextualize the data efficiency of boundary targeting, we compare against FunReason-MT-4B (Xu et al., 2025a) in Table 1, which was trained on 17K offline trajectories. RODS achieves a highly competitive 56.00% overall utilizing an active training pool of ∼800 samples (400 original seeds + up to Pmax = 400 generated variants), matching FunReason-MT on LongContext and significantly exceeding it on Missing Functions and Missing Parameters. This represents an ∼20× reduction in data volume to achieve comparable or superior performance, demonstrating that boundary-targeted synthesis is a highly data-efficient alternative to massive offline corpora. We further isolate the value of boundary-aware seed selection from the mere benefit of additional data volume via an ablation on random expansion (Table 2, row “w/ random seed selection”). 4.4 Ablation study (Q5) We ablate the three pillars of RODS—boundary detection, synthesis pipeline, and lifecycle management—to quantify their individual contributions (Table 2). All ablations use Qwen3-4B-Instruct with Pmax = 400 under the same three-stage curriculum. Detailed ablation configuration descriptions are provided in Appendix R. Key findings. (a) Removing coherence rewrite (−5.13%) causes the largest overall drop, collapsing the Quality Judge pass rate from ∼63% to ∼12% and drastically reducing usable variants. Among boundary detection ablations, random seed selection (−4.75%) confirms that boundary targeting—not merely additional data—drives improvement. Replacing progress reward with binary accuracy (−3.25%) further validates the importance of continuous credit for boundary identification. (b) The synthesis pipeline is robust to backbone choice: replacing Qwen3-32B with GLM-4.5-Air yields only −0.75% (see Appendix M). (c) Disabling 8

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents Table 2: Ablation study on BFCL V3. Each row modifies one component of RODS while keeping the rest intact. ∆: change relative to the full system. BFCL V3 Multi Turn

Configuration Avg.

Base

M. Func

M. Param

L. Ctxt

∆Avg.

RODS (full system)

56.00

68.00

59.00

44.00

53.00

—

(a) Boundary detection w/ random seed selection w/ binary acc instead of progress reward

51.25 52.75

63.50 65.00

52.50 55.00

37.00 36.50

52.00 54.50

↓ 4.75 ↓ 3.25

(b) Synthesis pipeline w/o coherence rewrite w/o narrative planning w/o feedback loop (blind retry)

50.87 52.37 53.87

63.00 64.50 66.00

52.00 54.00 57.00

36.00 42.00 40.50

52.50 49.00 52.00

↓ 5.13 ↓ 3.63 ↓ 2.13

(c) Lifecycle management w/o retirement mechanism w/ static pool (no dynamic refresh)

52.62 53.12

64.50 65.50

55.00 56.00

39.00 39.50

52.00 51.50

↓ 3.38 ↓ 2.88

retirement (−3.38%) confirms that mastered data accumulation dilutes gradient signal; continuous pool refresh is necessary.

5

Conclusion

Training multi-turn tool-use agents via RL faces a tension between data scarcity and signal relevance: static datasets lose informativeness as the agent improves. This paper introduces RODS, a framework that recasts this bottleneck as a dynamic curriculum design problem. Because progress reward variance peaks near the agent’s shifting capability boundary, RODS reuses this signal as a practical heuristic for boundary detection to sustain informative policy gradients throughout training. Combined with a narrative-driven, structurally isomorphic synthesis pipeline, our approach achieves competitive in-distribution and OOD improvements using roughly 20× less data than massive offline pipelines. These results suggest that boundary-targeted synthesis achieves higher per-sample training value than uniformly scaled static corpora, though at additional synthesis compute cost. Limitations & Future Work. While RODS provides a highly efficient curriculum, its current synthesis pipeline relies on deterministic simulation environments (implemented via executable Python objects) to verify execution correctness and provide feedback. Adapting this framework to inherently opaque environments or remote Model Context Protocol (MCP) servers remains an area for refinement. Future work will explore extending our simulation abstraction to robustly wrap and interact with stateful MCP endpoints, allowing the synthesis engine to safely capture input-observation dynamics without direct access to the underlying internal state. In addition, investigating multi-backbone synthesis ensembles to inject diverse structural priors at the boundary presents a promising direction for scaling agentic capabilities.

9

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

References Emre Can Acikgoz et al. Tool-r0: Self-evolving llm agents for tool-learning from zero data. arXiv preprint arXiv:2602.21320, 2026. Anthropic. System Card: Claude Opus 14e4fb01875d2a69f646fa5e574dea2b1c0ff7b5.pdf, 2026.

4.6.

https://www-cdn.anthropic.com/

Victor Barres et al. τ 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. Chen Chen et al. Acebench: Who wins the match point in tool usage? arXiv preprint arXiv:2501.12851, 2025. Yanqi Dai, Yuxiang Ji, Xiao Zhang, Yong Wang, Xiangxiang Chu, and Zhiwu Lu. Harder is better: Boosting mathematical reasoning via difficulty-aware GRPO and multi-aspect question reformulation. arXiv preprint arXiv:2601.20614, 2025. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536, 2025a. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978, 2025b. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025. Minghao Li et al. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023. Renda Li, Hailang Huang, Fei Wei, Feng Xiong, Yong Wang, and Xiangxiang Chu. Adacurl: Adaptive curriculum reinforcement learning with invalid sample mitigation and historical revisiting. arXiv preprint arXiv:2511.09478, 2025a. Yuetai Li et al. Simulating environments with reasoning models for agent training. arXiv:2511.01824, 2025b.

arXiv preprint

Yuwen Li, Wei Zhang, Zelong Huang, Mason Yang, Jiajun Wu, Shawn Guo, Huahao Hu, Lingyi Sun, Jian Yang, Mingjie Tang, and Byran Dai. Close the loop: Synthesizing infinite tool-use data via multi-agent role-playing. arXiv preprint arXiv:2512.23611, 2025c. Wei Liu et al. Self-play only evolves when self-synthetic pipeline ensures learnable information gain. arXiv preprint arXiv:2603.02218, 2026. Zeyuan Liu, Jeonghye Kim, Xufang Luo, Dongsheng Li, and Yuqing Yang. Exploratory memory-augmented LLM agent via hybrid on- and off-policy optimization. arXiv preprint arXiv:2602.23008, 2025. Siyuan Lu, Zechuan Wang, Hongxuan Zhang, Qintong Wu, Leilei Gan, Chenyi Zhuang, Jinjie Gu, and Tao Lin. Don’t just fine-tune the agent, tune the environment. arXiv preprint arXiv:2510.10197, 2025. Seiji Maekawa et al. Towards reliable benchmarking: A contamination free, controllable evaluation framework for multi-step llm function calling. arXiv preprint arXiv:2509.26553, 2025. OpenAI. GPT-5.4 Thinking System Card. https://deploymentsafety.openai.com/gpt-5-4-thinking/ gpt-5-4-thinking.pdf, 2026. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom Mitchell. Competence-based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1162–1172, 2019. Tiberiu Popoviciu. Sur les équations algébriques dont toutes les racines sont réelles. Mathematica, 9:129–145, 1935.

10

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay. arXiv preprint arXiv:2504.03601, 2025. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 761–769, 2016. Dunwei Tu et al. Scaleenv: Scaling environment synthesis from scratch for generalist interactive tool-use agent training. arXiv preprint arXiv:2602.06820, 2026. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024. Zhaoyang Wang et al. Agent world model: Infinity synthetic environments for agentic reinforcement learning. arXiv preprint arXiv:2602.10090, 2026. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025. Lilian Weng. Llm-powered autonomous agents. lilianweng.github.io, Jun 2023. URL https://lilianweng. github.io/posts/2023-06-23-agent/. Ronald J Williams. Reinforcement learning and markov decision processes. CSG220, Spring, 2007. Jian Xie et al. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622, 2024. Zengzhuang Xu, Bingguang Hao, Zechuan Wang, Yuntao Wen, Xinyi Xu, Yang Liu, Long Chen, Dong Wang, Maolin Wang, Tong Zhao, Yicheng Chen, Cunyin Peng, Jinjie Gu, Leilei Gan, Xiangyu Zhao, Chenyi Zhuang, and Shi Gu. Funreason-mt technical report: Advanced data synthesis solution for real-world multi-turn tool-use. arXiv preprint arXiv:2510.24645, 2025a. Zhangchen Xu et al. Toucan: Synthesizing 1.5m tool-agentic data from real-world mcp environments. arXiv preprint arXiv:2510.01179, 2025b. Shunyu Yao et al. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. Fan Yin et al. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation. arXiv preprint arXiv:2503.07826, 2025. Yunpeng Zhai, Shuchang Tao, Cheng Chen, Anni Zou, Ziqian Chen, Qingxu Fu, Shinji Mai, Li Yu, Jiaji Deng, Zouying Cao, et al. Agentevolver: Towards efficient self-evolving agent system. arXiv preprint arXiv:2511.10395, 2025. Kangning Zhang et al. Looptool: Closing the data-training loop for robust llm tool calls. arXiv preprint arXiv:2511.09148, 2025.

11

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

A

Formulation of the Sequential Multi-turn Decision Process

We formalize the multi-turn tool-use problem within the framework of a Partially Observable Markov Decision Process (POMDP) (Williams, 2007). In this context, a single episode represents an entire user task, comprising a series of predefined sequential instructions, referred to as turns. Formally, we represent the sequence of user instructions as q1 , q2 , . . . , q N . The episode initiates with the initial observation o0 , which encapsulates the first instruction q1 alongside the documentation of accessible tools. To fulfill this instruction, a sequence of operations is executed. At each timestep t, an action at is sampled from a predefined action space A based on the policy πθ ( at |ot ). This action space consists of two primary categories: • Tool Invocation (atool t ): A formatted request to interact with one or multiple external APIs to acquire necessary context (e.g., <tool_call>...</tool_call>). Following execution, the environment yields a new observation containing the execution results. • Task Resolution (aanswer ): A conversational response directed at the user (e.g., <answer>...</answer>). t Emitting this action signifies the completion of the current sub-task, triggering the environment to provide the subsequent user instruction. For any given turn i, the interaction proceeds as an intermediate trajectory of tool invocations, culminating when a task resolution is output. Once aanswer is generated, the environment advances the state by embedding t the next instruction qi+1 into the subsequent observation ot+1 . This iterative process continues until all N instructions are completed. A complete episode yields a full trajectory τ = (o0 , a0 , o1 , a1 , . . . , o T ), which terminates at timestep T when the final response for the ultimate instruction q N is issued. Importantly, it is only at this terminal step T that a sparse, binary reward R T ∈ {0, 1} is assigned, reflecting the overall success or failure of the entire task. Such delayed and sparse feedback makes credit assignment and exploration during reinforcement learning particularly hard, a common obstacle in long-horizon scenarios. Therefore, the primary objective is to optimize the policy parameters θ to maximize the expected terminal reward: J (θ ) = Eτ ∼πθ ,P [ R T ] (2)

B

System Implementation and Hyperparameters

In this section, we detail the specific hyperparameters, engineering optimizations, and fault-tolerance mechanisms used in our system implementation. B.1 Hyperparameter Specifications Dataset Partitioning Thresholds. As introduced in Section 3.1, the dataset is partitioned into three zones. Empirically, we set the boundary thresholds as follows: • Boundary Zone: α− = 0.20 and α+ = 0.85. This captures tasks where the model exhibits partial but incomplete mastery, maximizing gradient variance. + • Mastered Zone: αretire = 0.95. − • Too Hard Zone: αretire = 0.20. Injection and Capacity Limits. To prevent distribution shock (Section 3.3), the per-epoch injection volume is strictly capped at 20% of the active pool size. Additionally, the active dataset is bounded by a maximum capacity Pmax = 400 generated items, triggering the priority-based eviction strategy when exceeded. The original seed dataset is strictly preserved and exempt from retirement. B.2 Data Injection Latency Analysis Variants generated during training step t may not enter the dataset until step t + ∆. Empirically, we observe a seed-to-variant synthesis delay averaging 1 step, and a total staging delay (awaiting the next epoch boundary) averaging 13 steps (roughly 1–2 full epochs). Within this asynchronous window, a data point’s pass@16 changes by less than 0.10 on average, ensuring it remains in its originally classified zone. As a result, freshly generated variants remain highly relevant to the evolving model capability upon injection, validating the asynchronous design.

12

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

B.3 System Engineering and Concurrency Correctness Guarantees: Simulation Environment Execution as Oracle. The Synthesis Simulation Environment is implemented as a faithful replica of the training simulation environment. If a variant executes successfully on the Simulation Environment during generation, it is mathematically guaranteed to execute without environment-level errors during training. This architectural choice eliminates the need for costly and error-prone LLM-based correctness judgments, replacing them with deterministic, reproducible data validation. Concurrency and Isolation. To prevent resource contention during online synthesis, each training environment instance maintains strictly isolated model and environment states. The asynchronous background generation module operates in a fully decoupled process space, utilizing filesystem-based inter-process communication with file-level locking. This design prevents shared-memory race conditions and allows the generation daemon to scale independently of the primary reinforcement learning loop. Prompt Engineering: <reason> vs. <think>. To prevent modern reasoning models (e.g., Qwen3, QwQ) from defaulting to unconstrained native thinking modes that disrupt structured parsing, our LLM agents use custom <reason> tags for Chain-of-Thought generation rather than standard reasoning tokens (e.g., <think>). Because <reason> is not mapped to a special token in the tokenizer’s vocabulary, it functions purely as a structural prompt directive, ensuring stable and parseable JSON/XML outputs. B.4 Crash Recovery and Fault Tolerance To support reliable, long-running RL experiments, the system is designed with full fault tolerance. PromptTracker state is serialized to tracker.json after each epoch boundary. Generated variants are written to immutable append-only logs (e.g., expanded_epoch_*.jsonl). On daemon or trainer restart, the SeedManager reconstructs the full synthesized dataset and training state by: 1. Loading the latest tracker.json to recover PromptTracker historical windows. 2. Replaying all expanded_*.jsonl files to rebuild the list of generated data. 3. Deterministically recomputing the data retirement logic from stored metrics. This guarantees zero data loss across restart boundaries and lets the curriculum resume where it was interrupted. B.5 Motivating Analysis: Gradient Variance at the Capability Boundary We present the analysis motivating our boundary-targeting design heuristic. Specifically, we establish the rationale for preferentially synthesizing new data from tasks where the model’s average progress reward is near the midpoint of its range (µ ≈ 0.5). This heuristic is grounded in the relationship between reward variance and gradient signal strength. In GRPO, for a given prompt x, the model generates K rollouts y1 , . . . , yK . The corresponding binary rewards (success = 1, failure = 0) are denoted as ri ∈ {0, 1}. While the Progress Reward (R P ) used in our training is a multi-valued dense signal rather than strictly binary, Popoviciu’s inequality states that the variance of any bounded variable X ∈ [0, 1] is upper-bounded by µ(1 − µ). This bound motivates our design heuristic: the gradient signal potential is highest near µ = 0.5, though the actual variance depends on the full reward distribution and may not saturate this bound. The policy gradient is estimated as:

∇θ J (θ ) ≈

1 K Âi ∇θ log πθ (yi | x ) K i∑ =1

(3)

r −µ

where Âi is the z-score normalized advantage: Âi = σi x +ϵx . Here, µ x is the empirical success rate (pass@K, p denoted as p), and σx = p(1 − p) is the standard deviation. Variance maximization in the binary case. For binary rewards, theq normalized advantage for successful q 1− p p + − =− (ri = 1) and failed (ri = 0) rollouts can be exactly computed as  = and  p 1− p , respectively. The total magnitude of the gradient signal (signal density) is proportional to the variance of the Bernoulli distribution: Signal Density ∝ p(1 − p) (4) When a task is too hard (p → 0) or already mastered (p → 1), the signal density vanishes because all rollouts yield identical rewards, resulting in zeroed-out advantages. For binary rewards, this signal density 13

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

is maximized at p = 0.5, where the model generates a balanced mix of successes and failures, allowing GRPO to clearly contrast correct and incorrect behaviors. For the continuous Progress Reward R P ∈ [0, 1], the Popoviciu bound provides an analogous (though not necessarily tight) guideline: targeting µ ≈ 0.5 maximizes the potential for high variance, making it a practical proxy for the learning frontier. Because GRPO inherently generates these K rollouts during the standard forward pass to compute the group advantage, we can identify these high-variance µ ≈ 0.5 tasks at zero additional inference cost.

C

Agent Prompt Design

This section provides the complete prompt templates used by the multi-agent synthesis pipeline. Each agent is implemented as an LLM-based component with carefully designed system and user prompts to ensure high-quality data generation. C.1 Planner Agent The Planner Agent is responsible for generating a novel function call sequence that matches the structural complexity of the boundary seed task while introducing variation in the specific functions and scenario. Planner Agent User Prompt # Task You are a function call planner for a multi-turn tool-calling benchmark. You are given a seed task with its user queries and ground truth function call sequence. Your goal is to select a function sequence from the available functions that tests SIMILAR capabilities as the seed – such as parameter extraction, multi-step reasoning, cross-turn dependency, etc. – but using potentially DIFFERENT functions. You may use both HIGH-LEVEL functions (which the system will automatically decompose into multiple bottom-level calls) and BOTTOM-LEVEL functions (executed directly). # Seed Task Classes: {classes_str} User queries: {queries_text} Ground truth function sequence: {gt_summary} # Available Functions {func_list}

# Guidelines 1. Select functions that test similar skills to the seed task (e.g., if the seed requires multi-step parameter passing, your plan should also require it) 2. HIGH-LEVEL functions are preferred when available – they produce richer, multi-step call sequences after decomposition 3. Each turn should have 1-3 functions from the SAME class 4. For multi-class seeds, alternate classes across turns (e.g., Turn 1: ClassA, Turn 2: ClassB, Turn 3: ClassA) 5. Output 2-5 turns total 6. Ensure the sequence is logically coherent (e.g., authenticate before posting, fill fuel before driving)

14

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

# Output Format First, analyze the seed task and plan your approach inside <reason></reason> tags. Then output a brief narrative scenario (2-3 sentences) inside <narrative> tags. Finally output each turn inside a <turn> tag. IMPORTANT: Output 2-5 turns total. Do NOT output more than 5 turns. IMPORTANT: ONLY use function names that appear in the Available Functions list above. Format: <reason> Your step-by-step analysis... </reason> <narrative> A user named [name] wants to [goal]... </narrative> <turn> ClassName: func1, func2, func3 </turn> <turn> ClassName: func4 </turn> Example 1 (single class – VehicleControlAPI): <turn> VehicleControlAPI: fillFuelTankWithLiter </turn> <turn> VehicleControlAPI: activateParkingBrake, pressBrakePedal, startEngine </turn> <turn> VehicleControlAPI: estimate_drive_feasibility_between_city </turn> Example 2 (multi-class – TwitterAPI + TravelAPI): <turn> TravelAPI: get_flight_cost_from_cityA_to_cityB </turn> <turn> TravelAPI: book_flight </turn> <turn> TwitterAPI: post_tweet_with_my_acount </turn> Example 3 (multi-class – GorillaFileSystem + MathAPI): <turn> GorillaFileSystem: find_cat </turn> <turn> GorillaFileSystem: grep_function_name </turn> <turn> MathAPI: mean </turn> Example 4 (single class – TradingBot, using high-level functions): <turn> TradingBot: update_market_status_with_current_time </turn> <turn> TradingBot: add_to_watchlist_with_company_name </turn> <turn> TradingBot: get_stock_info_with_company_name </turn> <turn> TradingBot: place_order_with_market_price </turn> Now generate a function plan for the seed task above.

C.2 Config Patch Agent (Error Critic) The Config Patch Agent analyzes execution failures and generates environment configuration patches to resolve state conflicts. This agent implements the Environment Patches mechanism described in Stage II of the synthesis pipeline. Config Patch Agent System Prompt System Prompt: You are an expert at diagnosing configuration issues in function-calling systems. You analyze why a function failed with a given config, and suggest minimal fixes.

Config Patch Agent User Prompt Input: A data generation pipeline failed with this error: Error type: {error_type} Failed function: {error_function} Detail: {error_detail} The initial_config used was: {config_str}

15

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Task: Analyze WHY this function failed with this config, and suggest the MINIMUM config change(s) to fix the issue. Common Fixes: • market_status “Closed” → “Open” for trading functions • authenticated: false → true for functions requiring login • Add pending orders for cancel_order • Ensure sufficient balance for transactions • fuelLevel too low → increase for driving functions

# Output Format First, analyze the error inside <reason></reason> tags. Then output each field change as a <patch> block: <reason> Your analysis... </reason> <patch> <class>ClassName</class> <field>field_name</field> <value>new_value</value> </patch> For nested fields, use dot notation: e.g., orders.12345.status Now analyze the error above and output the necessary patches.

C.3 Coherence Rewrite Agent The Coherence Rewrite Agent performs holistic semantic grounding (Stage III) by generating natural user queries that are semantically coherent across all turns, guided by the latent narrative. Coherence Rewrite Agent System Prompt System Prompt: You are a helpful assistant that writes natural user queries for function-calling conversations.

Coherence Rewrite Agent User Prompt # Task Rewrite user queries for a multi-turn function-calling conversation. # Scenario {narrative} # Ground Truth Function Calls Per Turn {turns_for_rewrite}

16

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

# Instructions Generate one natural user query for EACH turn that: 1. Accurately describes what the GT function calls do (without mentioning function names) 2. Sounds like a real user talking to an AI assistant 3. Is semantically coherent with the narrative scenario 4. Does NOT contain special characters that could cause parsing issues 5. For turns with multiple function calls, the query should naturally imply all of them

# Output Format Output each query inside <query> tags, one per turn: <query>Turn 1 user query here</query> <query>Turn 2 user query here</query> ... IMPORTANT: Output exactly {num_turns} queries, one for each turn.

C.4 Quality Judge Agent The Quality Judge validates synthesized trajectories against strict quality criteria to ensure data integrity before injection into the training pool. Quality Judge Agent System Prompt System Prompt: You are a strict quality judge for multi-turn function-calling data. You evaluate whether a generated sample meets quality standards.

Quality Judge Agent User Prompt Input: Evaluate this multi-turn function-calling sample against quality criteria. # Sample Data {sample_summary}

# Quality Criteria 1. Query-GT Alignment: Does each user query accurately describe what the GT function calls do? 2. State Consistency: Do parameter values reflect the actual environment config? 3. Cross-Turn Coherence: Is there logical state progression across turns? 4. Query Naturalness: Do queries sound like real user requests? 5. Structural Correctness: Are deliberate ambiguities (e.g., missing parameters) properly reflected?

17

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

# Automatic Rejection Patterns If the query contains any of these patterns, REJECT immediately: • “Thought Process”, “Construct Query”, “Step 1:”, “Step 2:” • Function names or parameter names mentioned explicitly • Technical jargon that a real user wouldn’t say

# Output Format First explain your reasoning inside <reason></reason> tags. Then output your decision: <reason> Your detailed analysis... </reason> <decision>accept</decision> or <decision>reject</decision> <fail_reason>Specific reason for rejection (if rejected)</fail_reason>

C.5 Refine Classify Agent When the Quality Judge rejects a sample, the Refine Classify Agent determines whether the issue can be fixed by rewriting the user query or if the ground truth itself is unfixable. Refine Classify Agent System Prompt System Prompt: You are a precise diagnostic assistant. Analyze the root cause of data quality issues.

Refine Classify Agent User Prompt Input: A Quality Judge rejected this multi-turn function-calling data sample. Rejection reason: {fail_reason} Data summary: {data_summary}

Task: Analyze the rejection reason and determine: • Is the problem in the USER QUERY (wrong wording, mentions wrong values, unnatural phrasing, format issues)? → These can be fixed by rewriting the query. • Is the problem in the GT function calls (wrong parameters, wrong function, wrong cross-turn state, calling non-existent resources)? → These CANNOT be fixed by rewriting the query.

# Output Format First explain your reasoning inside <reason></reason> tags, then output your answer. <reason> Your analysis of where the root cause is... </reason> <answer>query_fixable</answer> or <answer>gt_unfixable</answer>

C.6 Refine Rewrite Agent If the Refine Classify Agent determines the issue is query-fixable, the Refine Rewrite Agent performs a targeted rewrite of the problematic user query.

18

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Refine Rewrite Agent System Prompt System Prompt: You are a helpful assistant that rewrites user queries to be natural and accurate.

Refine Rewrite Agent User Prompt Input: A quality check found an issue with this user query in a function-calling dataset. Issue: {fail_reason} Original query: “{old_query}” Ground truth function calls for this turn: {gt_str}

Task: Rewrite the user query so that: 1. It naturally and accurately describes what the GT function calls actually do 2. It sounds like a real user talking to an AI assistant 3. It does NOT mention function names, parameter names, or technical details 4. It does NOT contain special characters that could cause parsing issues 5. It fixes the specific issue described above

# Output Format Output ONLY the rewritten query inside <answer> tags: <answer>your rewritten query here</answer>

D

Deterministic Execution Pipeline Internals

The Stage II execution pipeline instantiates the abstract plan within a Python-based sandbox environment that faithfully replicates the training environment’s API semantics. The pipeline processes each turn sequentially through the following stages: 1. Function Sampling: Given the Planner’s output specifying which functions to call per turn, the pipeline samples concrete function instances from the available catalog. HIGH-LEVEL functions are automatically decomposed into sequences of BOTTOM-LEVEL API calls. 2. Parameter Generation: For each selected function, an LLM generates concrete parameter values conditioned on the function schema, the current environment state Ct , and any dependencies from previous turns. 3. VM Execution: The parameterized function call is executed against the sandbox VM. The VM maintains a complete environment state (file systems, account balances, database entries, etc.) and returns execution results or error messages. 4. Query Generation: A per-class LLM prompt generates a natural user query that describes the function call’s intent without exposing function names or parameters. 5. Query Verification: A verification LLM checks whether the generated query’s semantics align with the GT function calls, rejecting queries that are misaligned or contain data generation artifacts. If any stage fails, a structured error is recorded with the error type, failing function, turn number, and diagnostic detail. This structured error feeds directly into the Config Patch Agent and the Planner reinvocation described in the main text.

19

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

E

Error Taxonomy for Feedback-Driven Re-synthesis

To provide structured feedback to the Error Critic and Planner Agent, the execution pipeline classifies failures into the following categories. Each error type triggers specific recovery strategies: 1. param_gen_failed: The LLM failed to generate valid parameters for a function call. The Config Patch Agent may adjust environment state to make parameter generation feasible. 2. decompose_failed: A HIGH-LEVEL function could not be decomposed into valid BOTTOM-LEVEL calls. The Planner is instructed to “Use BOTTOM-LEVEL functions only.” 3. func_sample_failed: No valid function could be sampled after multiple retries (e.g., all candidates require unavailable prerequisites). The Planner is told to “AVOID functions requiring authentication or specific prior state.” 4. vm_exec_failed: The function call executed but the VM returned an error (e.g., insufficient balance, closed market, missing resource). This is the primary trigger for Config Patching. 5. duplicate_func: The same function call appeared multiple times in the same turn. The pipeline rejects the sample. 6. query_gen_failed: The query generation LLM failed to produce a valid <query> tag after multiple retries. The Planner is told to “Use simpler function combinations.” 7. query_verify_failed: The generated query was rejected by the verification LLM as semantically misaligned with the GT. The Planner is told to “Use 1 function per turn to simplify.” 8. query_verify_no_tag: The verification LLM did not return a parseable verdict tag. 9. conversation_construct_failed: The multi-turn conversation assembly failed (e.g., cross-turn dependency resolution error). 10. no_prompts: No prompt template exists for the specified class. 11. no_pattern: No valid execution pattern could be derived for the given class order. 12. pipeline_exception: An uncaught exception during pipeline execution. Only the first four error types (param_gen_failed, decompose_failed, func_sample_failed, vm_exec_failed) trigger the Config Patch Agent, as these represent environment-level issues resolvable through state modification. Query-level errors trigger Planner re-invocation with action space constraints but do not invoke config patching.

F

Feedback Loop Implementation Details

This section provides detailed implementation of the feedback-driven re-synthesis mechanism. Dual-Feedback Loop. When the execution engine encounters an error (classified per Appendix E), recovery proceeds through a dual-feedback loop that simultaneously accumulates two types of corrective signals across Kmax = 3 pipeline attempts: 1. Environment Config Patching: If the error type is patchable (param_gen_failed, vm_exec_failed, func_sample_failed, or decompose_failed), the Config Patch Agent (Appendix C.2) analyzes the initial environment configuration and outputs structured XML patches to update the state. Patches are accumulated across retries via recursive deep-merge, where later patches override earlier ones for the same field but coexist for different fields. A safety mechanism prevents non-dict values from overwriting dict-structured fields (e.g., preventing a string summary from corrupting a file tree). 2. Action Space Pruning: Simultaneously, the names of all functions involved in failures are extracted and added to a cumulative blocklist. On re-invocation, the Planner Agent receives: (a) the full failure history with structured error descriptions, (b) a list of specifically blocked function names, and (c) errortype-specific guidance (e.g., “Use BOTTOM-LEVEL functions only” for decomposition failures, “AVOID functions requiring authentication” for sampling failures). The Planner is explicitly instructed to “Generate a COMPLETELY DIFFERENT plan using different functions.” Both corrective signals are applied jointly to the next pipeline attempt, progressively narrowing the search space until a valid instantiation is found. If all Kmax attempts fail, the seed is discarded.

G

Quality Judge and Refinement Loop

This section details the multi-tier validation pipeline and the iterative refinement mechanism. Rule-Based Validation (Gate 1–3). Before reaching the LLM Quality Judge, variants must pass three deterministic gates: 20

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

• VM Re-verification: All GT function calls are re-executed against a fresh VM instance initialized with the variant’s initial_config, ensuring execution correctness. • Tool Availability: Every function name in the GT must exist in the tools provided to the model. For miss_func variants, this check accounts for tools provided in recovery turns. • Parameter Complexity: List/tuple parameters are limited to ≤ 5 elements; string parameters to ≤ 200 characters. This ensures the model can realistically reproduce the GT during training (pass@16 > 0). LLM Quality Judge (Gate 4). The Quality Judge strictly evaluates synthesized trajectories against five semantic criteria: (1) Query–GT Alignment (ground truth must exactly match the query’s intent—no more, no less), (2) State Consistency (parameter values must reflect the actual environment config, not incorrect values stated by the user), (3) Cross-Turn Coherence (logical state progression across turns), (4) Query Naturalness (human-like conversational flow without data generation artifacts), and (5) Structural Correctness (e.g., ensuring deliberate ambiguity in missing-parameter scenarios). Automatic failure is triggered by prompt-leakage patterns (e.g., “Thought Process”, “Construct Query”, raw JSON tool definitions in non-recovery turns). The judge outputs a <reason> analysis followed by a <decision>accept</decision> or <decision>reject</decision> verdict. Full prompt is provided in Appendix C.4. Diagnostic Refinement Loop (max 1 cycle). If a trajectory is rejected, a Fail Classifier (Appendix C.5) analyzes the rejection reason and outputs one of two verdicts: • gt_unfixable: The problem lies in the GT function calls (e.g., wrong parameters, wrong function, state dependency violations). The sample is immediately dropped. • query_fixable: The problem is in the user query (e.g., unnatural wording, misaligned description). The sample is routed to a Refine Rewriter (Appendix C.6) which performs a one-shot targeted rewrite of the identified turn’s query. The rewritten variant is submitted to the Quality Judge for a second evaluation. Trajectories that fail this second check are permanently dropped. No recursive refinement is applied.

H

Reward Details

This section details the format reward Rformat design used in Stage 1 training, which evaluates the structural correctness of tool calls. Format Reward Formulation (Stage 1). In Stage 1, the agent is optimized purely for formatting and valid API execution syntax. We assign per-turn penalty codes: −3 for XML format errors, −2 for tool schema errors (e.g., invalid JSON), and −1 for valid syntax but execution failure. Valid executions receive 0 or 1. Let N be the total interaction rounds (turns), and n−k be the count of turns receiving code −k. The format and tool execution rewards are defined as: ( n   −1 if n−1 + n−2 > 0 N − n −3 , rtool = n−1 +n−2 rformat = max 0, (5) N 0 otherwise An indicator function 1tool = I[n−1 + n−2 > 0] ensures at least one tool call was attempted. The final Stage 1 reward is defined as Rfinal = 1tool · (rformat + rtool ) ∈ [0, 2]. In Stages 2 and 3, the Progress Reward (R P ) is computed simply as the fraction of successfully resolved turns.

I

Combination Experiment: RODS + EnvTuning

To analyze whether boundary-targeted synthesis (RODS) and environmental feedback augmentation (EnvTuning) provide complementary gradient signals, we plot their training progress rewards. As shown in Table 1, the static baseline without augmentation suffers from rapid reward saturation. While RODS maintains a continuously climbing progress reward by introducing boundary tasks, replacing static tasks alone still lacks granular execution hints. By combining both methods, the agent not only receives high-variance boundary tasks but also environment-aided corrections, leading to improved training stability and sample efficiency.

J

Training Details

This section lists the full training hyperparameters for the RL pipeline.

21

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Table 3: System Hyperparameters. Configuration details for RL training, data synthesis, and the dynamic replay buffer.

Category

Hyperparameter

RL Training (GRPO)

Actor Learning Rate KL Loss Coefficient (β) Number of Rollouts (K) PPO Mini-batch Size Total Training Epochs

1 × 10−6 0.01 16 512 5 (per stage)

Data Synthesis

LLM Backend Decoding Temperature Top-p Max Pipeline / Planner Retries

Qwen3-32B 1.0 0.7 3/3

Dynamic Buffer

K

Value

Boundary Lower Bound (α− ) Boundary Upper Bound (α+ ) Generated Pool Cap (Pmax ) Trial-Period Observation Count Trial-Period Eviction Threshold + Retirement Mastered Threshold (αretire ) − Retirement Hard Threshold (αretire )

0.20 0.85 400 1 0.20 0.95 0.20

Theoretical Extension to PPO

Although our empirical results focus on GRPO due to its memory efficiency, the RODS capability probe naturally extends to Proximal Policy Optimization (PPO). In PPO, the advantage function At is typically estimated via Generalized Advantage Estimation (GAE), driven by the TD-error δt = rt + γV (st+1 ) − V (st ). For binary or bounded task-level rewards (like our Progress Reward R P ), the variance of the TD-error across multiple trajectory samples from the same prompt remains tightly coupled to the variance of the final reward. When a task is fully mastered (R P → 1) or consistently failed (R P → 0), the Value network V (s) accurately predicts the outcome, leading to near-zero TD-errors (δt ≈ 0) and vanishing gradients. Conversely, at the capability boundary where outcomes are highly uncertain (p ≈ 0.5), the Value network’s prediction error is maximized, resulting in high-variance advantage estimates. Therefore, tracking the variance of the PPO advantage estimates Var( Â) or the raw reward variance inside the PPO rollout buffer serves identical functions to the GRPO variance probe, allowing RODS to dynamically identify and synthesize boundary tasks without architectural changes.

L

Out-of-Distribution Generalization Results

This section provides the detailed performance metrics for out-of-distribution (OOD) generalization across the BFCL V4, τ 2 -bench, and ACEBench Agent benchmarks. The results confirm that boundary-expanded training yields generalizable reasoning capabilities rather than mere in-distribution pattern matching. Table 4: OOD generalization performance on BFCL V4, τ 2 -bench, and ACEBench Agent benchmarks. All results are compared against the Llama-3.1-8B-Instruct base model. Models trained with RODS (rows in blue) show improvements on these OOD tasks. Scores for xLAM on the Retail and Airline domains are grayed out as it was trained on the original τ-bench, making them invalid for OOD evaluation. τ 2 -bench

BFCL V4

Model

ACEBench Agent

Avg. (%)

Web Search (%)

Memory (%)

Avg. (%)

Retail (%)

Airline (%)

Telecom (%)

Avg. (%)

Multi-turn (%)

Multi-step (%)

xLAM-2-8b-fc-r BitAgent-8B ToolACE-2-8B

9.17 7.41 15.75

5.00 4.50 8.90

13.33 10.32 22.60

36.67 10.00 10.00

57.50 7.50 7.50

40.00 15.00 15.00

12.50 7.50 7.50

9.15 8.65 11.00

13.30 12.00 15.00

5.00 5.30 7.00

Llama-3.1-8B-Instruct + EnvTuning + RODS (ours)

7.05 16.53 18.00

2.50 15.00 15.50

11.60 18.06 20.50

19.46 22.17 25.00

5.88 6.00 8.00

30.00 30.50 34.00

22.50 30.00 33.00

4.15 8.82 11.15

3.30 11.30 14.30

5.00 6.33 8.00

22

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

M

Synthesis LLM Robustness

A natural question is whether the performance gains of RODS are attributable to the framework design or to the specific generative capacity of the synthesis LLM. To investigate this, we replace the default synthesis backbone (Qwen3-32B) with GLM-4.5-Air—a generally stronger model—deployed via vLLM on 8×A100 GPUs, while keeping all other components—boundary detection, structural isomorphism, dynamic replay buffer, and training pipeline—strictly identical. The generated pool cap remains Pmax = 400. Table 5: Synthesis LLM robustness on BFCL V3 multi-turn. Replacing the synthesis backbone with a stronger model (GLM-4.5-Air) yields nearly identical overall performance (↓0.75%), confirming that RODS is model-agnostic: its gains derive from the framework design rather than from the specific synthesis LLM.

BFCL V3 Multi Turn

Synthesis LLM Qwen3-32B (default) GLM-4.5-Air

Avg.

Base

M. Func

M. Param

L. Ctxt

56.00 55.25

68.00 65.00

59.00 55.00

44.00 46.00

53.00 55.00

∆Avg. — ↓ 0.75

Analysis. The overall performance difference is 0.75% (56.00% → 55.25%). GLM-4.5-Air is a stronger model than Qwen3-32B, yet switching to it does not yield further gains. This suggests limited sensitivity to the synthesis backbone under our setup: schema-guided planning, deterministic VM validation, and multi-tier quality filtering together normalize the output regardless of the underlying LLM’s raw capability. Put differently, RODS’s performance is determined by the framework design (boundary detection, structural isomorphism, dynamic lifecycle), not by the generative capacity of the synthesis model. At the sub-split level, the two LLMs show mild distributional differences: GLM-4.5-Air scores higher on Missing Parameter (+2.00%) and Long Context (+2.00%), while Qwen3-32B leads on Base (+3.00%) and Missing Functions (+4.00%). These complementary biases suggest that different LLMs produce variants with subtly different structural characteristics, pointing to a potential direction of multi-backbone synthesis ensembles.

N

Synthesis Computational Cost

We report the computational overhead of the RODS synthesis pipeline to contextualize its cost relative to the RL training loop. Hardware allocation.

Training and synthesis run on separate GPU clusters in parallel:

• RL training: 8×A100 (80GB) GPUs running GRPO with the Qwen3-4B-Instruct policy. • Data synthesis: 8×A100 (80GB) GPUs hosting Qwen3-32B via vLLM, with 64 parallel worker threads dispatching synthesis requests. The two systems communicate asynchronously via filesystem-based queues (seed emission from the trainer, variant ingestion at epoch boundaries). The synthesis daemon does not block or slow the training loop. Wall-clock time. To reach the reported performance of 56.00% (at training step ∼600), the system runs for approximately 56 hours wall-clock. Since both clusters operate concurrently for the full duration, the total compute is: • Training: 8 × 56 = 448 GPU-hours. • Synthesis: 8 × 56 = 448 GPU-hours. • Total: ∼896 GPU-hours (16×A100 for ∼56 hours). The synthesis overhead is thus 1× the training cost in GPU-hours. However, the synthesis cluster runs a single vLLM instance with no gradient computation, so its actual FLOP consumption is lower than the training cluster. Per-variant synthesis cost. On the successful path, generating a single base variant requires approximately 9–15 LLM calls depending on the number of turns: 1 Planner call, 1 query generation + 1 query verification per turn (for a 3-turn variant: 6 calls), 1 coherence rewrite, and 1 quality judge evaluation. For miss_func and miss_param variants, 2–5 additional calls are needed for the adversarial transform and its verification. The

23

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

64-worker thread pool processes these calls concurrently across seeds, achieving high throughput despite the per-variant multi-stage pipeline. Synthesis latency. The measured seed-to-injection delay averages 1 training step (∼3.7 minutes). Since variant injection occurs at epoch boundaries and the daemon processes seeds continuously in the background, the synthesis pipeline does not introduce any idle time into the training loop. Variants generated during epoch n are staged and injected at the start of epoch n+1, as described in Section 3.3. Cost-efficiency perspective. While the synthesis overhead doubles the GPU footprint relative to standard GRPO training, the resulting 20× data efficiency gain (matching 17K-sample offline pipelines with an active training pool of ∼800 samples) represents a practical trade-off. Generating 17K high-quality multi-turn trajectories offline via similarly complex multi-agent simulation pipelines (such as APIGen-MT or FunReasonMT) typically incurs massive upstream computational costs before training even begins. By explicitly targeting only the high-variance capability boundary, RODS not only reduces the required data volume by an order of magnitude but also provides a favorable end-to-end data-compute trade-off relative to massive offline pipelines.

O

Benchmark and Evaluation Details

We evaluate our agents using several multi-turn tool-use benchmarks to assess both in-distribution learning and out-of-distribution (OOD) generalization. In-Distribution Evaluation (BFCL V3). The Berkeley Function Calling Leaderboard (BFCL) V3 (Patil et al., 2025) provides a reliable testbed for multi-turn scenarios. We utilize its 800-sample multi-turn subset, partitioned equally across four categories: • Base: Standard multi-turn tasks with straightforward dependencies. • Missing Function: Tasks requiring the agent to recognize missing capabilities and either gracefully decline or request alternative tools. • Missing Parameter: Tasks lacking necessary arguments, requiring the agent to ask the user for clarification before proceeding. • Long-Context: Scenarios involving extended conversations where context must be maintained across many turns. We adopt the exact 400/400 train/test split established by Lu et al. (2025) to ensure a fair comparison. Evaluation is performed via the official BFCL abstract syntax tree (AST) matching evaluator. Out-of-Distribution Evaluation. To verify that RODS induces generalizable reasoning rather than mere pattern matching, we evaluate on benchmarks featuring unseen APIs and interaction modalities: • BFCL V4: We test on the Web Search and Memory tracks, representing dynamic information retrieval and long-term state tracking not present in V3. • τ 2 -bench (Barres et al., 2025): A dual-control conversational benchmark set in the Retail, Airline, and Telecom domains, emphasizing highly constrained, real-world business logic. • ACEBench (Chen et al., 2025): We utilize the Multi-turn and Multi-step splits of the Agent track to test complex API topologies. All OOD evaluations strictly follow their respective official evaluation protocols and scoring scripts.

P

Synthesized Data Examples

This section presents seed-to-variant pairs for each data type, demonstrating how RODS preserves structural complexity while generating novel content. For each example, we show the original seed (left/top) and the synthesized variant (right/bottom) side by side. P.1 Base Type: VehicleControlAPI (Seed → Variant) Seed (multi_turn_base_63): A 3-turn task involving unit conversion, engine startup with safety checks, and distance estimation.

24

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Original Seed Turn 1: “I require assistance in determining the quantity of gasoline necessary for an extensive journey across California. I currently anticipate needing around 166 liters. How much is that in gallons?” Turn 2: “Prior to commencing the drive, kindly initiate the engine, ensuring all doors are securely closed and the parking brake is engaged.” Turn 3: “Could you provide me with the approximate distance between San Francisco and Rivermist? Will I be able to get there?”

Synthesized Variant: Same skill structure (unit conversion → vehicle status + fuel → multi-step distance estimation), but different functions and parameters. Synthesized Variant Turn 1 — Unit Conversion User: “I just checked my car’s fuel gauge and it shows 7.8 liters remaining. How many gallons is that?” GT: liter_to_gallon(liter=7.8)

Turn 2 — Vehicle Status + Fuel Operation User: “Please show me my current fuel level and add 4 gallons to the tank for the trip.” GT: displayCarStatus(option=’fuel’), fillFuelTank(fuelAmount=4.0)

Turn 3 — Multi-step Distance + Feasibility User: “I’m driving from Crescent Hollow to Autumnville – what’s the total distance, and can I complete the trip with 630 miles of fuel?” GT: get_zipcode_based_on_city(city=’Crescent Hollow’), get_zipcode_based_on_city(city=’Autumnville’), estimate_distance(cityA=’69238’, cityB=’51479’), estimate_drive_feasibility_by_mileage(distance=630.0)

Structural preservation: Both share the pattern unit conversion (1 call) → vehicle operation (2 calls) → distance planning (4 calls with dependency chain). The variant uses different city names, conversion direction (liters→gallons vs. gallons→liters), and vehicle operations (fuel display vs. engine startup), forcing the model to generalize the abstract reasoning pattern. P.2 Missing Function Type: GorillaFileSystem (Seed → Variant) Seed (multi_turn_miss_func_38): A file system task where rm is removed, requiring refusal and recovery. Original Seed Turn 1: “I’ve misplaced a vital document. Assist in locating a file named ‘findings_report’ within ‘SuperResearch’. Could you remove it and the directory.” Turn 2: [Function rm removed; agent must refuse] Turn 3: “What’s left in the current directory including the hidden files?”

Synthesized Variant: Different file operations, mkdir removed instead of rm, recovery in Turn 4. Synthesized Variant Turn 1: “I need to rename my JSON file ‘wqmmw.json’ to ‘data.csv’ for compatibility.” GT: mv(source=’wqmmw.json’, destination=’data.csv’)

25

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Turn 2: “Please check how many lines are in ‘data.csv’ so I can validate the data.” GT: wc(file_name=’data.csv’, mode=’l’)

Turn 3 — mkdir removed from tool list User: “I’d like to create a new folder called ‘docs’ to organize these files.” GT: []

(agent must refuse)

Turn 4 — Function restored; agent recovers User: [{“name”: “mkdir”, ...}] “Here’s a tool that might help.” GT: mkdir(dir_name=’docs’)

P.3 Missing Parameter Type: TradingBot + MathAPI (Seed → Variant) Seed (multi_turn_miss_param_144): A cross-class task where the user provides a vague computation request, requiring parameter clarification. Original Seed Turn 1: “After determining the current market status, retrieve the stock information for symbol ‘AAPL’.” Turn 2: “Using the current details of a stock, calculate the average of price, trading volume, MA5, and MA20.” [Parameters vague – which stock?] Turn 3: “The stock should be AAPL.” [User provides clarification]

Synthesized Variant: Different stock, different vague reference (“those two percentage changes”), same clarification pattern. Synthesized Variant Turn 1 — Stock lookup (TradingBot) User: “I’m looking at Synex Solutions’ stock – can you get their ticker symbol and the latest details?” GT: get_symbol_by_name(name=’Synex Solutions’), get_stock_info(symbol=’SYNX’)

Turn 2 — Vague query; concrete values omitted User: “I need the average of those two percentage changes we just saw.” GT: [] (agent must ask for clarification)

Turn 3 — User provides missing numerical values User: “They are −3.4 and −1.0.” GT: mean(numbers=[-3.4, -1.0])

Key observation: In both seed and variant, the ambiguity arises from a vague back-reference to prior tool output. The variant changes the specific stock, the nature of the computation (average of percentage changes vs. average of multiple metrics), and the exact missing values, while preserving the core skill: recognizing under-specified parameters and requesting clarification before executing.

26

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

Size

Overall

Base

Miss Func

Miss Param

Long Context

Closed-source Model Claude-Sonnet-4-5-20250929 Claude-Haiku-4-5-20251001 Gemini-3-Pro-Preview Gemini-2.5-Pro-Preview Gemini-2.5-Flash Grok-4-1-fast-reasoning Grok-4-1-fast-non-reasoning GPT-5.2-2025-12-11 GPT-4o-2024-11-20

-

61.38 53.63 60.75 28.75 16.75 58.88 46.75 28.13 42.50

69.00 63.50 64.50 32.00 14.50 70.50 58.00 36.50 55.50

65.00 42.50 60.00 29.00 16.50 59.50 39.50 18.00 34.50

52.50 52.50 54.50 22.00 17.50 43.00 37.50 27.50 29.00

59.00 56.00 64.00 32.00 18.50 62.50 52.00 30.50 51.00

Open-source Model Kimi-K2-Instruct DeepSeek-V3.2-Exp Llama-4-Maverick Qwen3-235B-A22B-Instruct Qwen3-32B Qwen3-30B-A3B-Thinking ToolACE-2-8B ToolACE-MT Nanbeige4-3B-Thinking-2511 xLAM-2-3b-fc-r

1043B 671B 400B 235B 32B 30B 8B 8B 3B 3B

50.63 44.88 20.25 44.63 47.88 30.00 38.38 40.25 51.12 58.38

62.00 55.00 27.00 54.00 56.00 43.50 49.00 57.50 58.50 71.50

41.00 49.00 22.00 42.50 52.50 10.50 28.00 31.50 54.00 59.00

44.50 27.00 14.00 31.50 40.00 25.00 30.50 34.00 45.00 57.50

55.00 48.50 18.00 50.50 43.00 41.00 46.00 38.00 47.00 45.50

7B 8B 4B -

7.00 36.92 (+29.92) 37.75 (+30.75) 40.25 (+33.25) 5.48 28.25 (+22.77) 28.38 (+22.90) 30.88 (+25.40) 22.13 50.00 (+27.87) 50.50 (+28.37) 56.00 (+33.87)

9.33 50.33 (+41.00) 51.50 (+42.17) 54.00 (+44.67) 6.15 28.20 (+22.05) 28.00 (+21.85) 32.00 (+25.85) 26.50 62.00 (+35.50) 64.00 (+37.50) 68.00 (+41.50)

9.33 40.33 (+31.00) 41.00 (+31.67) 43.50 (+34.17) 6.80 25.85 (+19.05) 25.50 (+18.70) 28.00 (+21.20) 21.00 51.00 (+30.00) 52.00 (+31.00) 59.00 (+38.00)

6.33 29.33 (+23.00) 30.50 (+24.17) 33.50 (+27.17) 3.20 22.15 (+18.95) 23.00 (+19.80) 24.50 (+21.30) 15.50 35.00 (+19.50) 35.00 (+19.50) 44.00 (+28.50)

3.00 27.67 (+24.67) 28.00 (+25.00) 30.00 (+27.00) 5.75 36.80 (+31.05) 37.00 (+31.25) 39.00 (+33.25) 25.50 52.00 (+26.50) 51.00 (+25.50) 53.00 (+27.50)

Model

Qwen2.5-7B-Instruct + Static dataset + EnvTuning + RODS (ours) Llama-3.1-8B-Instruct + Static dataset + EnvTuning + RODS (ours) Qwen3-4B-Instruct + Static dataset + EnvTuning + RODS (ours)

Table 6: Full in-distribution performance on BFCL V3 multi-turn (Tier 1: controlled RL comparisons). All RL methods share the same 400 training samples and GRPO setup; only the data/environment strategy differs. Red text indicates improvement over the base model. The best result within each model group is bolded.

Q

Full Benchmark Results

R

Ablation Configuration Details

This section provides detailed descriptions of each ablation condition in Table 2. (a) Boundary detection ablations. • w/ random seed selection: Instead of selecting seeds from the boundary region (Dboundary ), we sample seeds uniformly at random from the entire training pool regardless of their reward. This isolates the effect of boundary-targeted capability tracking. • w/ binary acc instead of progress reward: We replace the continuous Progress Reward R P with binary task accuracy (1 if all turns correct, 0 otherwise) for boundary detection. This tests whether the fine-grained partial credit of R P is necessary for accurate boundary identification. (b) Synthesis pipeline ablations. • w/o coherence rewrite: We skip Stage III (holistic semantic grounding). Per-turn queries are generated independently without the Rewrite Agent’s narrative-driven single-pass rendering. • w/o narrative planning: The Planner Agent generates a function sequence without an underlying narrative (N ). This removes the cross-turn thematic coherence that anchors all turns to a unified goal.

27

Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

• w/o feedback loop (blind retry): We remove the Error Critic and Config Patch Agent. When execution fails, the pipeline simply retries with a fresh random plan rather than accumulating corrective signals. (c) Lifecycle management ablations. • w/o retirement mechanism: All three retirement layers (L1–L3) are disabled. The pool only grows and is never pruned. • w/ static pool (no dynamic refresh): We generate variants once at the beginning of Stage 3 and freeze the pool thereafter. No new variants are synthesized as training progresses.

28