arXiv CSopen access

E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

E3 -TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning Weiyang Guo1 Zesheng Shi1 Liye Zhao2 Jiayuan Ma1 Zeen Zhu1 Junxian He3 Min Zhang1 Jing Li1# 1 Harbin Institute of Technology, Shenzhen, China 2 Huawei Technologies Co., Ltd. 3 Hong Kong University of Science and Technology [email protected] [email protected]

arXiv:2604.09455v1 [cs.AI] 10 Apr 2026

While Large Language Models (LLMs) have demonstrated significant potential in ToolIntegrated Reasoning (TIR), existing training paradigms face significant limitations: ZeroRL suffers from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. To address these challenges, we propose E3 -TIR (Enhanced Experience Exploitation), a warm-up paradigm for the early stages of agent training. Specifically, we formulate training as the dynamic integration of three experience types: Expert Prefixes, Expert Guided, and Self-Exploration. By executing diverse branching exploration around expert “anchors” and employing a mix policy optimization mechanism, we effectively mitigate distribution shifts and resolve optimization conflicts arising from shared prefixes. Our method dynamically adapts the model’s knowledge boundaries, effectively balancing exploration diversity with training efficiency. Experimental results demonstrate that E3 -TIR achieves a 6% performance improvement over traditional paradigms on tooluse tasks, while requiring less than 10% of the synthetic data. Furthermore, in terms of ROI—a comprehensive metric integrating performance, data cost, and training efficiency—we achieve a 1.46× gain compared to baselines. Code is available at https:// github.com/yuki-younai/E3-TIR.

Introduction

Large Language Models (LLMs) have demonstrated exceptional potential in complex reasoning and tool-assisted decision-making, which has driven the realization of advanced applications such as question answering, planning, and interactive agents (Zhang et al., 2025; Zhao et al., 2025). To # Corresponding author.

Average Test Accuracy

Abstract

Warm-Up

Post-RL Training

Fast Convergence

Entropy Collapse

SFT Start

Inefficient Exploration

Base Init

Overfitting

80 110 Training Step

Zero-RL SFT + RL E 3 -TIR (Ours) 140

170

Figure 1: (a) Zero-RL: Inefficient exploration and slow convergence. (b) SFT+RL: Vulnerable to overfitting and performance collapse after a high start. (c) E3 -TIR: Enables rapid adaptation via expert guidance with stable, continuous growth.

overcome the limitations of purely internal reasoning, LLMs are often deployed as Tool-Integrated Reasoning (TIR) agents (Dong et al., 2025a). A TIR agent performs reasoning and action through a multi-step, interleaved process that invokes external tools such as code interpreters (Xue et al., 2025) and search engines (Wang et al., 2025c). Recently, reinforcement Learning (RL) has shown significant promise in unlocking frontier LLM capabilities (Shao et al., 2024), methods such as Search-R1 (Jin et al., 2025a) and ToRL (Li et al., 2025b) train directly on the foundation model, utilizing self-exploration to acquire tool-use capabilities, but this compromises training efficiency. Conversely, approaches like Tool-Star (Dong et al., 2025b) and Tool-Light (Chen et al., 2025c) necessitate an initial stage of supervised fine-tuning with synthetic data before engaging in reinforcement learning. Reliance on synthetic data is not only computationally expensive but also introduces training redundancy. Our empirical analysis further reveals the deepseated limitations of existing paradigms. Zero-RL, lacking prior guidance, exhibits inefficient exploration and mode degradation: the model requires

Zero-RL Problem - Inefficient Exploration and Tool Overuse (a) Inefficient and Redundant Tool Usage

SFT+RL Problem - Low Entropy and Policy Restriction

(b) Compare of React Mode Rate

(a) Low-Entropy Collapse

(b) Diversity-constrained capability

Figure 2: Statistical analysis of the limitations of current training paradigms.

extensive trial-and-error to generate valid code at an early stage (Fig. 2 left,a) and often converges to a “react mode” (Yu et al., 2025) that over-relies on tools at the expense of reasoning (Fig. 2 left, b). Conversely, SFT-then-RL, while benefiting from training, leads to capability rigidity. This necessitates an adaptation period to overcome distribution shifts (Fig. 2 right,a) and frequently triggers low-entropy collapse in the later stages of training (Fig. 2 right,b), which severely restricts sample diversity and impedes further performance gains. To address these challenges, we propose E3 -TIR, a framework designed to balance exploration diversity with efficiency. In the sampling stage,we initiate branching sampling from high-entropy “anchors” within expert trajectories to explore diverse reasoning paths. Subsequently, we dynamically filter the mixed trajectories based on gradient validity and relative performance. For mix policy optimization, we employ hybrid advantage estimation to capture preference variances among branches sharing the same prefixes. Meanwhile, advantage blocking resolves gradient conflicts on shared prefixes, ensuring training stability. The main contributions of this study are as follows: • Through empirical analysis, we uncover the “react mode” inherent in Zero-RL tool call, alongside the phenomena of low-entropy collapse and capability rigidity that occur in the later stages of SFT+RL. • We propose E3 -TIR, which provides dynamic experience sources for an agent’s early training by integrating three distinct types of experience: expert prefixes, expert guidance, and self-exploration. This approach effectively balances the diversity and efficiency of earlystage exploration. • E3 -TIR outperforms SFT and RL baselines

using <10% of the synthetic data. With a minimal warm-up, it achieves a 1.46× gain in ROI—a composite metric of data, training, and performance efficiency.

Preliminaries

2.1

Multi-Turn Agent Framework

We consider an LLM agent performing a task x ∈ D through iterative interactions. At each step t, the policy πθ (·|H<t ) generates a thought τt and an action at , eliciting an observation ot from the environment. This yields a trajectory H = {(τt , at , ot )}Tt=1 . As the trajectory length grows linearly with the number of steps, optimizing πθ for complex tasks becomes increasingly challenging due to the accumulation of history. 2.2

Agentic Reinforcement Learning

Agentic RL typically adopts policy-gradient methods to optimize the agent policy πθ . We formulate the agentic RL training objective as: max Ex∼D,H∼πθ (·|x) [rϕ (x, H)]−βDKL [πθ (·)∥πref (·)] πθ

(1) where πθ represents the policy LLM, πref is the reference LLM, rϕ and DKL denotes the reward function and KL divergence respectively. To ensure proper credit assignment, we implement token masking (ot ). The loss is computed exclusively on the agent’s outputs (τt and at ), thereby excluding environmental feedback from gradient updates.

Related Work

Tool Integrated Reasoning. Tool-Integrated Reasoning (TIR) (Lin and Xu, 2025; Singh et al., 2025) empowers LLMs to overcome the inherent limitations of pure in-context reasoning by leveraging external utilities such as code interpreters and

The Overview of E^3-TIR Framwork Rollout Module Policy Model

Question

Multi-levels Experience

Mix Policy Optimization Output

Expert Prefix Experience

Expert Self

Expert Guided Experience Tool Environment

Self-Explored Experience

Reward

Reference Model

Expert Self

Reward Model

...

(a) Expert Experience-Based Rollout

(b) Mix Advantage and Policy Optimize

Prefix-Guided Branch Sampling

Advantage-Aware Gradient Detach

...

Top-k Anchor Selection

Expert Guided Rollout

Shared Expert Prefix

Global Group

Expert Self

Expert-Tree Group

...

Positive Gradient Flow

Final Objective Function Self-Explore Rollout

Dynamic Experience Filtering

gradient filter

Advantage

Gradient Detach

Hybrid Advantage Estimation

Self-Explored loss Expert Prefix loss

Self > Expert

Expert Guided loss Self < Expert Self-Explored Chains

Expert Tree

Figure 3: Illustration of our E3 -TIR framework. (a) Branching exploration from expert anchors and dynamic experience filtering. (b) Hybrid advantage estimation and advantage-aware gradient detachment.

search engines. For question-answer (QA) tasks, efforts such as Search-R1 (Jin et al., 2025a), R1Search (Song et al., 2025) , and Re-search (Chen et al., 2025a) primarily focus on utilizing search tools. Conversely, in mathematical tasks, TIR largely relies on agent-generated Python code to enhance reasoning accuracy and efficiency. Current methodologies for training LLMs in tool invocation present a distinct set of challenges. One path involves Zero-shot RL based online exploration, where models like ToRL (Li et al., 2025b), OTC (Wang et al., 2025b), and SimpleTIR (Xue et al., 2025) learn autonomous tool calls. However, this often leads to inefficient exploration and redundant tool invocation. The second path utilizes Supervised Fine-Tuning (SFT) for expert knowledge injection, with works like ReTool (Feng et al., 2025a), Tool-star (Dong et al., 2025b), focusing on synthesizing high-quality agent data to inject tool-use priors before RL training. Agent Reinforcement Learning. Reinforcement Learning (RL) plays a pivotal role in enabling LLMs agents to adapt effectively to dynamic and open-ended environments (Gao et al., 2025). Recent efforts have primarily focused on addressing the challenges of sparse rewards and complex decision-making in multi-turn interactions (Feng et al., 2025b; Singh et al., 2025). RAGEN (Wang et al., 2025d) integrates the reasoning state with the environmental interaction into turn-level responses and trains the agent using trajectory-level RL. To further alleviate the issue of reward sparsity, IGPO (Wang et al., 2025a) introduce more finegrained rewards, such as leveraging the agent’s self-

confidence. To overcome the difficulty of capturing complex multi-turn behaviors with trajectorylevel RL, Tree-GRPO (Ding and Ye, 2025) employs a Grouped Relative Policy Optimization based on tree search, allowing the model to effectively learn intermediate decisions. Furthermore, ARPO (Dong et al., 2025c) secures more diverse reasoning paths through high-entropy branch sampling, and AEPO (Dong et al., 2025a) advances this direction by proposing a policy optimization method based on entropy balancing.

Methodology

In our E3 -TIR framework, we divide the overall training flow into two main parts: 1) Expert Experience-Based Rollout. First, we utilize the current policy to conduct selfexploration and expert-guided exploration, respectively, performing branch sampling from the expert prefixes to form the experience pools Dexp and Dself (Section 4.1.1). Subsequently, we filter the experiences pools based on gradient validity and performance advantage to construct the final training set Dtrain (Section 4.1.2). 2) Mix Advantage and Policy Optimization. We categorize advantages into global advantage and expert-tree internal advantage to determine relative trajectory preference, processing different experiences separately (Section 4.2.1). To update the policy using mixed experiences, we introduce an off-policy shape and resolve gradient conflicts in shared prefixes via advantage-aware gradient blocking (Section 4.2.2).

4.1

Expert Experience-Based Rollout

4.1.1

Prefix-Guided Branch Sampling

Unlike self-exploration methods (Dong et al., 2025c,a), we propose expert-guided branch sampling, using expert prefixes as anchors to reduce search redundancy and ensure reasoning correctness (Theoretical Proof provided in Appendix E). Empirical analysis confirms this advantage: branching from expert prefixes yields significantly higher policy entropy and reward variance compared to self-generated trajectories (Figure 4). This demonstrates that expert anchors provide a more informative and gradient-valuable exploration space for subsequent policy optimization. Accordingly, for each problem x, we allocate a self-explore budget N and an expert-enhanced budget M . The sampling process is formalized as three stages: 1. Initialization. We initialize Dself with N starting states. For the expert trajectory H∗ , we measure exploration value using the step-wise policy entropy ht , conditioned on the prefix H<t : ht = −

πθ (y|H<t ) log πθ (y|H<t )

(2)

y∈Y

where Y denotes the generated token sequence. We construct Dexp by selecting the top-k high-entropy prefixes, assigning each a unique tree ID T . |H∗ |

Dexp ← Top-K({(Hnew = H<t , T new )}t=2 ) (3) 2. Sampling. To fully leverage expert experience, we sample non-terminated nodes from Dexp at the end of each round to spawn diverse branches. The sampling probability is determined by the exploration gain Pt : Pt =

(ht − ht−1 ) · I(|Dexp | < M ) n̂H

(4)

where α is a base sampling probability and n̂H represents the number of branches of this expert tree. If Pt exceeds a threshold γ and the budget M is not reached, a copy of the trajectory is created and inserted into Dexp : ( {Copy(H, T )}, Dexp ← Dexp ∪ ∅,

if Pt > γ otherwise (5)

3. Expansion. For all active states/trajectories H in DSelf and Dexp , we execute a single step of outward Rollout (expansion) using πθ . The experience pools are updated with the resulting longer trajectories Hnew : Dtotal ← Update({Hnew ∼ Rollout(H) if |H| < T }) (6) We iterate Steps 2 and 3 until termination or the turn limit T . This constrained rollout ensures parallel efficiency, avoiding the unbounded expansion of MCTS (Wang et al., 2025e). 4.1.2 Dynamic Experience Filtering To empower the agent to dynamically adjust its knowledge boundaries, we introduce a filtering mechanism based on gradient and performance. First, regarding gradient validity, we always retain Dself as the global advantage baseline. For the T i has zero reward expert pool: if an expert tree Dexp variance V, only one representative trajectory Hi is kept to reduce redundancy; otherwise, the full tree is retained: (S Ti i {H ∼ Dexp }, S Ti i Dexp ,

T )] = 0 if V[r(Dexp otherwise (7) Second, to prevent the model from regressing to suboptimal patterns, we construct the final training set Dtrain based on a performance criterion. Specifically, we compare the maximum rewards yielded by self-exploration against those from expert guidance: ( ′ ) Dself , if rϕmax (Dself ) ≥ rϕmax (Dexp Dtrain = ′ , otherwise Dself ∪ Dexp (8) This logic enforces an adaptive learning curriculum: expert trajectories are discarded when the agent matches or exceeds their performance, prioritizing the agent’s superior policy. ′ Dexp =

4.2

Mix Advantage and Policy Optimize

4.2.1 Hybrid Advantage Estimation Upon collecting the final experience pool Dtrain , the primary step is to determine the relative advantage for each trajectory H in Dtrain . This differentiation of relative advantage occurs at two critical levels: 1) Global Advantage (Âglobal ). This metric measures the trajectory’s performance relative to the average performance across the entire pool Dtrain .

(a) Distributions of Standard Deviations 60

40 30 20 10 0

Self-Exploration Sample Expert Prefixes Sample

0.6 Average Entropy

Percentage (%)

(b) Entropy Distribution after Tool Call 0.7

Self-Exploration Sample Expert Prefixes Sample

0.5

|Hi | X X

1 JHybrid (θ) = Z

0.4 0.3

Hi ∈Dself t=1

0.2

0.1 0.2 0.3 0.4 Standard Deviation of Reward

0.5

0.0

0.1 0.0

3 Tool Call

CLIP(ri,t (θ), Âself i , ϵ) }

Pure Self-Exploration (On-Policy)

Lprefix (Hk ) + Lsuffix (Hk )

Hk ∈Dexp

Figure 4: Distribution of Standard Deviation and Entropy under Different sampling strategies.

}

Expert prefixes and Expert guidance (Mix-Policy)

(12) We leverage the global advantage for baseline estimation, which is crucial for stabilizing training:

Âglobal (H ) =

train rϕ (Hi ) − mean(rϕ (Hj ))D j=1 train mean(rϕ (Hj ))D j=1

(9)

The mixed objective is divided into two parts: (1) the self-exploration part and (2) the mixed policy part. The mixed policy component, in turn, includes the prefix and suffix parts. Their specific forms are as follows: Lprefix

2) Expert-Tree Relative Advantage (Âexp-tree ). We compute an internal advantage within Dexp to measure variance among branches sharing the prefix T . This guides the model to capture the expert’s granular thought patterns.

Lprefix (Hk ) =

exp

CLIP(ρk,t (θ), Âk )

(13a)

t=1 |Hk |

Lsuffix (Hk ) =

exp

CLIP(rk,t (θ), Âk )

t=Lprefix +1

(13b) (T =T i )

Âtree (Hi ) =

Dexp rϕ (Hi ) − mean(rϕ (Hj ))j=1 (T =T i )

|Dexp

(T =T i )

exp | · mean(rϕ (Hj ))j=1

(10) Since different expert trees may share the same prefix, we normalize the advantage by dividing it by the number of trajectories within the current expert tree to prevent over-optimization. Consequently, the advantage estimation for trajectories within Dself is solely derived from the Âglobal (·). Conversely, for trajectories in Dexp , the advantage estimation combines both the Âglobal (·) and Âtree (·).

( Âself (Hi ) = Âglobal (Hi ) Âexp (Hi ) = Âglobal (Hi ) + Âtree (Hi ) 4.2.2

(11)

Mix Policy Optimization

Since distribution shifts in expert trajectories cause training instability, we adopt off-policy policy shape (Yan et al., 2025) to effectively utilize the mixed experience Dtrain . The policy object is:

Where the importance sampling ratio rk,t is defined at the token level t, and ρk,t is the policy reshaping function f (πθ ) = π/(π + λ) Branching sampling creates optimization conflicts where failed explorations (A < 0) incorrectly penalize valid shared prefixes. We address this via Advantage-Aware Gradient Detaching, which selectively blocks prefix gradients for failure paths (A < 0) while retaining full updates for successful ones (A > 0). The modified objective is: Lpre

Lprefix (Hk ) =

exp

CLIP(ρk,t (θ), Âk )·I(Âk > 0)

t=1

(14) Through this asymmetric optimization strategy, we theoretically eliminate the optimization ambiguity of the shared prefix under different branches.

Experiments

5.1

Experiment Setup

Tasks and Metrics. To comprehensively evaluate the effectiveness of our E3 -TIR algorithm in TIR, we conduct experiments on the following two types of tool-use tasks:

Table 1: Overall results on 10 challenging reasoning tasks. The top two results are highlighted in bold and underlined. The abbreviations of the datasets: Math (Math500), HQA (HotpotQA), 2Wiki. (2wikiMultiHopQA), MuSi. (MuSiQue), SimQA (SimpleQA) and Bamb (Bamboogle). Mathematical Reasoning

Knowledge-Intensive Reasoning

Method

Avg. RMR. AIME24 AIME25 AMC23 GSM8K MATH 2Wiki. Bamb. HQA MuSi. SimQA 4.5 6.0 11.2 12.0 19.0

8.6 10.8 16.5 13.3 20.8

38.3 35.6 51.8 48.7 56.0

65.3 58.3 69.1 71.9 72.3

81.0 81.0 85.1 85.4 85.2

19.2 30.3 54.2 46.7 52.3

27.5 35.2 47.3 50.8 50.3

11.5 35.8 50.5 48.8 51.4

6.3 16.3 25.3 23.2 26.9

5.9 19.5 30.6 31.1 32.6

26.8 32.9 44.2 43.2 46.7

7.4 6.1 7.9 6.2

Qwen2.5-7B-Instruct + Only SFT + SFT-then-RL + Zero-RL + E3 -TIR (Ours)

7.5 14.6 25.9 21.3 23.2

9.2 14.1 23.3 27.1 29.3

50.0 56.2 59.4 62.1 64.8

74.1 71.3 77.4 76.8 80.4

89.4 89.6 91.2 90.6 92.1

18.2 33.3 58.4 55.7 56.3

29.6 37.9 53.6 49.6 54.2

15.3 37.1 52.3 50.1 56.0

7.8 14.6 27.0 24.7 32.2

6.4 23.2 30.7 31.5 33.5

30.7 39.2 49.9 49.0 52.2

8.2 7.7 7.9 7.5

Llama3.1-8B-Instruct + Only SFT + SFT-then-RL + Zero-RL + E3 -TIR (Ours)

0 7.5 16.0 17.2 15.5

1.6 11.2 12.5 12.5 20.0

25.0 34.3 52.0 53.7 58.4

46.6 59.3 64.0 62.5 65.2

78.3 85.6 86.1 87.9 88.2

16.9 32.9 53.8 54.3 57.4

38.4 41.1 55.4 50.9 55.3

12.1 38.2 53.8 49.6 55.6

13.3 15.9 30.2 28.5 29.7

6.3 21.7 33.1 31.6 32.7

23.9 34.8 45.7 44.9 47.8

8.4 7.1 7.7 7.4

Reward

0.4 0.3 0.2

0.6

3.50

0.5

3.25

Turns Number

0.5

E3-TIR(Ours)

GRPO

0.4 0.3 0.2

0.1 0

Steps

(a) Entropy and Reward Curves on Warm-Up Stage

−

SFT-then-RL 0.56 0.54

Test Score

E3-TIR(Ours)

Entropy

−

Qwen2.5-3B-Instruct + Only SFT + SFT-then-RL + Zero-RL + E3 -TIR (Ours)

3.00 2.75

0.52 0.50 0.48

2.50 0

20 40 60 80 100 120

0.46

Steps

0 20 40 60 80 100 120

Steps

(b) Turns Number and Test Score on Post-RL Stage

Figure 5: Comparison of the training curves on Qwen2.5-3B-Instruct.

1) Mathematical Reasoning:. In this task, the model can automatically generate and execute code. We use metrics such as AIME24, AIME25, MATH500 (Hendrycks et al., 2021), AMC23, and GSM8K (Cobbe et al., 2021) to evaluate the model’s mathematical reasoning capabilities. 2) Knowledge-Intensive Reasoning: We evaluate on knowledge question benchmarks, including: HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), Musique (Trivedi et al., 2022), and Bamboogle (Press et al., 2023), SimpleQA (Press et al., 2023). Baselines. We compare the proposed E3 -TIR against the following baseline: (i) Training Pipeline.We compare against SFT-only, SFT-thenRL (via GRPO), Zero-RL (via GRPO), trained on the same dataset. (ii) LLM-based Search Agent. including Search-o1 (Li et al., 2025a), Search-R1 (Jin

et al., 2025b) and Tree-GRPO (Ding and Ye, 2025). (iii) LLM-based Math Agent. including ToRL (Li et al., 2025b) and SimpleTIR (Xue et al., 2025). (iv) Mult-Tool Integrated Agent. including ToolStar (Dong et al., 2025b), ReCall (Chen et al., 2025b) and ARPO (Dong et al., 2025c). Training Guideline. To ensure reproducibility, all training frameworks and datasets are sourced from open-access resources. Specifically, the expert trajectories data are from the Tool-Star SFT dataset. For the RL data, we also utilize the ToolStar RL dataset. 1. Initialization: To mitigate reward collapse during the initial RL training stage, the model is initialized using only a small amount of selfdistillation data.. 2. RL Stage: The RL phase adopts a two-stage training paradigm:

32.4 29.7

37.9

37.5

0.030

37.8

E3-TIR w/o detach w/o off-policy shape

0.025

35.4 35

0.020 0.015 0.010

Expert-Guided Number

Initial Expert Trajectory

5 4 3 2

0.005

E3-TIR w/o detach w/o off-policy shape

grad_norm

37.3

pg-clipfrac

Val. Score

37.1

Val. Score

37.6 35.6

Steps

Figure 6: Ablation analysis of different hyperparameters in Expert Branch Sample.

Figure 7: Ablation analysis of clip ratio and gradient norm on different mix policy optimization.

• E3-TIR (Warm Up). In this stage, the model undergoes mix policy training using the SFT dataset augmented with expert experience. • E3-TIR (Post-RL). In this stage, the model is trained using the standard RL dataset without any specific modifications. The detailed introduction of training setup are listed in Appendix C.

5.3

5.2

Main Results

Comparison with Training Paradigms. Table 1 presents our main experimental results. Compared to baseline methods, E3 -TIR consistently achieves superior performance. Whether applied to smaller 3B models or larger 7B/8B models, our mechanism yields a 6% average performance gain while simultaneously achieving higher tool-use efficiency. Comparison with SOTA Methods. Table 2 benchmarks E3 -TIR against current open-source SOTA methods. Results demonstrate that, under comparable parameter scales, our approach exhibits superior performance for both single-tool and multi-tool agents, maintaining a robust competitive edge across diverse benchmarks. Balancing Exploration Quality and Efficiency. As shown in Figure 5a, E3 -TIR achieves rapid reward ascent by leveraging expert anchors, in contrast to the cold start of Zero-RL. Simultaneously, the model maintains high policy entropy, indicating that it effectively exploits expert knowledge without loss diversity. Good Environmental Adaptability. As shown in Figure 5b, the SFT model struggles to adapt to the multi-turn interactions, leading to declining tool usage and an overfitting “fast rise, then drop” pattern in accuracy Conversely, E3 -TIR demonstrates robust exploration with steadily increasing tool usage and maintains a stable upward performance curve throughout the process.

Ablation Study

Component-Level Ablation. To isolate the contribution of each core design choice, we systematically remove individual components while keeping the same expert prefixes and training budget. We additionally compare against a simple weighted SFT+RL mixture baseline (Mix SFT+RL) that replaces our loss with a weighted combination following Lv et al. (2025). Results are reported over 3 independent runs. As shown in Table 4, removing AAGD leads to training collapse (accuracy drops to 20.6 / 14.7), confirming it is essential for resolving gradient conflicts on shared prefixes. Branching exploration contributes the largest performance gain among the remaining components: its removal causes a greater accuracy drop than removing hybrid advantage estimation, highlighting that diverse expert-anchored exploration is the primary driver of E3 -TIR’s effectiveness. The Mix SFT+RL baseline also collapses (27.6 / 32.3), demonstrating that standard SFT loss is ineffective for multi-turn, long-horizon tool-use tasks due to severe distribution shifts. Parameter Ablation for Expert Sample. We conducted an ablation study on two core parameters within the expert-guided sample module: 1) Balancing Self-Exploration (N ) and ExpertGuided (M ). We fixed the total sampling budget at 16 and varied M to evaluate model performance after the warm-up stage. As shown in Figure 6 (left), performance improves as M increases (for M < 8), demonstrating that expert-guided trajectories effectively accelerate early convergence. However, performance deteriorates as M increases further, indicating that retaining a sufficient proportion of self-generated trajectories is critical for stabilizing training. 2) Initial Expert Trajectories (K). With M fixed at 8, we varied the initial value of K. As illustrated in Figure 6 (right), performance peaks at

Table 2: Overall results on 10 challenging reasoning tasks. The top two results are highlighted in bold and underlined. All baseline methods use Qwen2.5-3B as the backbone. For baselines lacking a 3B Instruct version, we replicate them using the exact parameter settings from their open-source code. Computational Reasoning

Knowledge-Intensive Reasoning

Method

Avg. AIME24 AIME25 AMC23 GSM8K MATH 2Wiki. Bamb. HQA MuSi. SimQA

Qwen2.5-3B-Instruct

4.5

8.6

38.3

65.3

81.0

19.2

27.5

11.5

6.3

5.9

26.8

LLM-based Math Agent ToRL SimpleTIR

10.0 17.5

20.0 19.1

53.0 54.3

72.0 69.4

84.4 83.9

24.0 16.1

25.4 24.2

17.9 16.5

8.3 7.2

6.5 8.8

32.2 31.7

LLM-based Search Agent Search-o1 Search-R1 Tree-GRPO

8.6 3.3 9.2

10.3 10.0 16.6

36.5 31.2 32.5

58.6 58.0 52.3

74.2 60.5 56.5

27.4 44.9 44.1

38.5 44.2 43.7

29.5 50.1 50.5

12.5 22.2 26.7

10.8 31.4 29.0

30.7 35.6 36.1

Multi-Tool Integrated Agent ReCall Tool-Star ARPO E3 -TIR (Ours)

7.6 20.1 16.2 19.0

12.4 12.9 20.6 20.8

48.2 52.0 52.0 56.0

68.6 69.8 70.2 72.3

84.1 85.0 85.7 85.2

42.6 48.6 54.3 52.3

43.7 44.1 47.6 50.3

48.7 52.4 50.0 51.4

22.6 27.8 26.5 26.9

26.5 30.3 29.3 32.6

40.5 44.3 45.2 46.7

w/o Âglobal w/o Âtree

Musique.

ACM23

MATH

44.9

19.9

43.3

65.0

35.4(−9.5) 42.6(−2.3)

13.7(−6.2) 17.8(−2.1)

36.8(−6.5) 43.5(+0.2)

56.3(−8.7) 63.1(−1.9)

E3-TIR SFT Zero-RL

0.4 0.3 0.2

E3-TIR Zero-RL

100

solve-none

E3 -TIR (Warm up)

Computational.

HotQA

Code Failed Ratio

Knowledge.

Method

80 60 40

0.1 20 0

Table 3: Ablation on Hybrid Advantages Estimation.

Method E3 -TIR w/o Branch w/o AAGD w/o Hybrid Adv. Mix SFT+RL

Knowledge.

Computational.

HotQA

Musique.

AMC23

MATH

51.4

26.9

56.0

85.2

47.2(−4.2) 15.3(−36.1) 49.5(−1.9) 22.4(−29.0)

23.1(−3.8) 8.2(−18.7) 25.6(−1.3) 11.3(−15.6)

52.3(−3.7) 24.8(−31.2) 55.2(−0.8) 32.5(−23.5)

83.6(−1.6) 48.6(−36.6) 84.1(−1.1) 56.8(−28.4)

Table 4: Component ablation on Qwen2.5-3B-Instruct. w/o Branch: remove expert-guided branching; w/o AAGD: remove advantage-aware gradient detaching; w/o Hybrid Adv.: remove expert-tree advantage; Mix SFT+RL: weighted SFT+RL (Lv et al., 2025).

K = 3. Beyond this point, increasing K yields no significant performance variation. This is primarily attributed to the fact that the average number of interaction turns in expert trajectories typically ranges between 2 and 3. Effectiveness of Mix Policy Optimization. We validate the contribution of each component through ablation studies: 1) Mix Advantage Estimation: Table 3 shows that Atree alone causes training collapse, while the combined Atree +Aglobal significantly outperforms

Steps (Post RL)

100

Steps (Warm up)

Figure 8: Comparison of Code Failed Rate and SolveNone curves across different training stages.

Aglobal alone. This confirms that internal advantage captures fine-grained step differences, effectively enhancing sample efficiency. 2) Off-policy Reshaping: As shown in Figure 7 (left), direct updates with mixed data cause clip ratio spikes due to distributional disparity. In contrast, off-policy reshaping corrects these shifts, maintaining training stability. 3) Shared Prefix Gradient Detach: Figure 7 (right) demonstrates that without detachment, gradient norms oscillate due to optimization conflicts. AAGD ensures stable gradient flow, preserving the expert prefix by resolving these conflicts. 5.4

Analysis

Code Execution Failure Analysis. As shown in Figure 8 (left), we compared code failure rates during the Post-RL stage. Benefiting from the warmup, E3 -TIR exhibits a significantly lower initial error rate than Zero-RL. Notably, E3 -TIR consistently maintains a low level, demonstrating superior robustness in tool usage.

(a) Cost and Efficiency Analysis

Avg Calls

Fail Rate

0.6% 0.8% 0.5%

E3 -TIR

46.7

1.97

4.0%

0.6%

4.5% 3.8% 7.4%

2.24 1.82 2.52

Warm-up Steps (X)

32.9 44.2 43.2

Only SFT SFT-then-RL Zero-RL

Table 5: Tool-calling behavior audit on Qwen2.5-3BInstruct. Avg Calls: average tool invocations per query; Fail Rate: proportion of invalid tool calls; Redundancy: ratio of repeated calls.

Tool-Calling Behavior Audit. Beyond task accuracy, we audit tool-calling efficiency across training paradigms. As shown in Table 5, E3 -TIR achieves an average of 1.97 tool calls per query, substantially fewer than Zero-RL’s 2.52, indicating that expert anchors help establish efficient invocation patterns during the warm-up phase. The tool failure rate of E3 -TIR (4.0%) is nearly half that of ZeroRL (7.4%), confirming that expert-guided exploration effectively reduces invalid syntax errors and ill-formed queries. Redundancy rates remain consistently low across all methods, suggesting that repeated tool calls are primarily driven by unresolvable code errors rather than training paradigm differences. Extension of Capability Boundary. We tracked the proportion of effectively unsolvable problems during training. As illustrated in Figure 8 (right), compared to Zero-RL, E3 -TIR demonstrates a continuous decline in the Solve-None rate, stabilizing at a lower final level. This confirms that expert experience effectively expands the model’s knowledge boundary, enabling it to resolve problems previously beyond its reach. Warm-up Stage Cost and Efficiency Analysis. We evaluated training efficiency gains across different warm-up settings using the ROI metric (Appendix F.3). As shown in Figure 9a, ROI exhibits an “inverted-U” trend with increasing warm-up steps, attributed to the model’s diminishing reliance on experts. Meanwhile, gains from expert data size saturate after 4k samples, demonstrating that minimal expert data is sufficient for efficient startup. Comparison with Offline Fusion Baselines. We further benchmark our mixed strategy against prior offline fusion methods (Yan et al., 2025; Lv et al., 2025), as illustrated in Figure 9b. While E3 -TIR outperforms the off-policy free baseline by 19%, alternative methods like Luffy and HPT (utilizing

0.94

0.95

0.82

0.86

1.5 1.4

1.10

1.12

1.07

(b) Performance Comparison

1.6

Redundancy 0.93

1.3

ROI Score

Avg Score

Method

+19.0% 37.6

E3-TIR 28.5

LuFFY

1.2

1.29

1.46

1.32

1.19

1.1

17.3

HPT

1.0

1.35

1.42

1.30

1.34

10k

Expert Trajectories Size (Y)

0.9

31.6

Baseline

0.8

Average Test Score

Figure 9: (a) Cost-Benefit Heatmap during the Warm-up Stage. (b) Comparison with Other Mix Strategies. Method

ALFWorld

SCIWorld

Avg.

Only SFT SFT-then-RL Zero-RL

62.5 81.5 86.5

43.0 65.0 63.5

52.8 73.3 75.0

E3 -TIR

89.0

67.5

78.3

Table 6: Generalization on long-horizon agent tasks using Qwen2.5-3B-Instruct. Expert data: 2K trajectories from AgentGYM (?) per task.

SFT loss) result in decreased performance. This advantage stems from the task’s complexity: robust multi-turn TIR is difficult to achieve through sequence-level optimization alone. Generalization on Long-Horizon Tasks. To evaluate adaptability beyond math and QA, we conduct experiments on ALFWorld (?) and SCIWorld (?), two long-horizon agent exploration tasks requiring up to 30 interaction turns. As shown in Table 6, E3 -TIR achieves the best results on both tasks (89.0 and 67.5). Notably, Zero-RL (86.5) outperforms SFT-then-RL (81.5) on ALFWorld, suggesting that SFT-induced distribution shifts can be particularly harmful in stateful environments. E3 TIR further improves upon Zero-RL by effectively leveraging expert guidance without sacrificing exploration flexibility, demonstrating its generalizability to complex, multi-step agent settings.

Conclusion

In this paper, we introduce E3 -TIR, a framework that optimizes agent training by dynamically integrating different experience. By utilizing branching exploration around expert “anchors” alongside a mixed policy optimization mechanism , E3 -TIR balances exploration diversity with efficiency, transforming experience utilization into proactive acquisition and establishing a new paradigm for LLMs agent training.

Acknowledgements This work was supported in part by National Natural Science Foundation of China (62476070), Shenzhen Science and Technology Program (JCYJ2024 1202123503005, GXWD20231128103232001, Z DSYS20230626091203008, KQTD20240729102 154066), Department of Science and Technology of Guangdong (2024A1515011540) and National Key R&D Program of China (SQ2024YFE0200592).

Limitations In this section, we discuss the limitations identified in this study that point towards directions for future improvement. First, while E3 -TIR significantly mitigates the dependency on massive synthetic data, it remains sensitive to the quality of initial expert prefixes. Since the core mechanism relies on branching around these “anchors”, suboptimal or low-diversity seeds may inadvertently bias the search space, thereby constraining the model’s exploration efficiency in highly complex tasks. Furthermore, our current evaluation is primarily strictly within the domains of math and QA. Consequently, the framework’s generalizability to strictly more sophisticated, real-world scenarios—particularly those necessitating the simultaneous orchestration of multiple tools or maintaining state consistency over long-horizon interactions—remains to be fully validated.

References Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. 2025a. Research: Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470. Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. 2025b. Research: Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470. Yifei Chen, Guanting Dong, and Zhicheng Dou. 2025c. Toward effective tool-integrated reasoning via self-evolved preference learning. arXiv preprint arXiv:2509.23285. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.

2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Zheng Ding and Weirui Ye. 2025. Treegrpo: Treeadvantage grpo for online rl post-training of diffusion models. arXiv preprint arXiv:2512.08153. Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jinghan Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. 2025a. Agentic entropy-balanced policy optimization. arXiv preprint arXiv:2510.14545. Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. 2025b. Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning. arXiv preprint arXiv:2505.16410. Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. 2025c. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025a. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025b. Group-in-group policy optimization for llm agent training. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, and Xinzhe Juan. 2025. A survey of self-evolving agents: On path to artificial super intelligence. arXiv preprint arXiv:2507.21046. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 6609–6625. Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025a. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. In Proceedings of the Conference on Language Modeling (COLM).

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025b. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025a. Search-o1: Agentic searchenhanced large reasoning models. arXiv preprint arXiv:2501.05366. Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025b. Torl: Scaling tool-integrated RL. arXiv preprint arXiv:2503.23383. Heng Lin and Zhongwen Xu. 2025. Understanding tool-integrated reasoning. arXiv preprint arXiv:2508.19201. Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Lixuan He, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, Ning Ding, and Bowen Zhou. 2025. Towards a unified view of large language model post-training. arXiv preprint arXiv:2509.04419. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics (EMNLP), pages 5687–5711. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. 2025. Agentic reasoning and tool integration for llms via reinforcement learning. arXiv preprint arXiv:2505.01441. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and JiRong Wen. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics (TACL). Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying. 2025a. Information gain-based policy optimization: A simple and effective approach for multi-turn LLM agents. arXiv preprint arXiv:2510.14967. Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. 2025b. Acting less is reasoning more! teaching model to act efficiently. arXiv preprint arXiv:2504.14870.

Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, and Linjie Li. 2025c. RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, and Linjie Li. 2025d. RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073. Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. 2025e. Stepsearch: Igniting llms search ability via stepwise proximal policy optimization. arXiv preprint arXiv:2505.15107. Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. 2025. Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. arXiv preprint arXiv:2509.02479. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Zhaochen Yu, Ling Yang, Jiaru Zou, Shuicheng Yan, and Mengdi Wang. 2025. Demystifying reinforcement learning in agentic reasoning. arXiv preprint arXiv:2510.11701. Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, and Zaibin Zhang. 2025. The landscape of agentic reinforcement learning for llms: A survey. arXiv preprint arXiv:2509.02547. Bingxi Zhao, Lin Geng Foo, Ping Hu, Christian Theobalt, Hossein Rahmani, and Jun Liu. 2025. Llm-based agentic reasoning frameworks: A survey from methods to scenarios. arXiv preprint arXiv:2508.17692.

Appendix A The Use of Large Language Models

B Datasets and Benchmarks B.1 Mathematical Reasoning Benchmarks . . . . . . . . . . . . . . . B.2 Knowledge-Intensive Reasoning Benchmarks . . . . . . . . . . . . B.3 Training Datasets . . . . . . . . .

C Details of Experiment Setup C.1 Supervised Fine-Tuning. . . . . . C.2 Reinforcement Learning. . . . . . C.3 Self-distillation Initialization. . . . C.4 Training Pipeine. . . . . . . . . . C.5 Evaluation Setup. . . . . . . . . . C.6 Reward Score. . . . . . . . . . . . C.7 Tool Setup. . . . . . . . . . . . .

13 13 13 13 14 14 14 14

D The Algorithm Workflow Of E3 -TIR

E Theoretical Analysis E.1 Problem Formulation . . . . . . . E.2 Failure Analysis of Zero-RL . . . E.3 Success Guarantee of E 3 -TIR . . E.4 Numerical Illustration . . . . . . .

15 15 15 15 17

F Supplement Experimental Results F.1 Statistic Analyse of Problem in Current Training Paradigms. . . . F.2 Comparison of Expert Prefixes and Self-Exploration Sampling. . . . . F.3 Warm-up Stage Cost and Efficiency Analysis . . . . . . . . . . F.4 Comparison with Offline Fusion Baselines . . . . . . . . . . . . . F.5 Expert Trajectory Analysis during the Warm-up stage . . . . . . . .

G Instruction Template G.1 Instructions for Mathematical Reasoning Evaluation . . . . . . . . . G.2 Instructions for KnowledgeIntensive Reasoning Evaluation . . G.3 Instructions for Reinforcement Learning Training . . . . . . . . .

H Case Study

12 13 13

17 17 17 18 19

19 19 19

The Use of Large Language Models

Throughout the preparation of this manuscript, large language models were employed exclusively for light stylistic refinement and the occasional grammatical adjustment. Every conceptual insight, analytical thread, and interpretive conclusion emerged from the authors themselves; no algorithmic assistance was solicited for the framing, design, or substance of the work, and full scientific responsibility rests with the human contributors alone.

Datasets and Benchmarks

B.1

Mathematical Reasoning Benchmarks

1. AIME24 is a dataset in evaluating the mathematical reasoning ability of models. It consists of 30 challenging math problems. All of them are from the American Invitational Mathematics Examination. The problems in the AIME24 dataset cover a wide variety of mathematical fields such as algebraic equations and geometric puzzles. Due to the difficulty characteristics and the richness of question types, it has become a popular benchmark for evaluating the reasoning performance of models, and is widely used in multiple related research experiments. 2. AIME25 consists of 30 challenging math problems. It is directly composed of the real questions from the American Invitational Mathematics Examination newly released in February 2025. AIME25’s knowledge areas are extremely wide. It deeply covers core mathematical sections such as algebra, geometry, number theory, and combinatorial mathematics. This characteristic enables the AIME25 dataset to effectively distinguish the mathematical reasoning abilities of different models. 3. MATH500 (Hendrycks et al., 2021) is selected by OpenAI from the MATH evaluation dataset. It contains 500 high-difficulty math problems. These problems cover multiple mathematical fields such as algebra, geometry, calculus, and number theory. The difficulty is close to or exceeds the college level. In academic research, MATH500 dataset is often used to evaluate the performance of various reasoning models. 4. GSM8K (Cobbe et al., 2021) is an elementary school math problem dataset released by OpenAI. These problems require 2 to 8 steps to solve, mainly through a series of basic calculations to obtain the final answer. This dataset is primarily used to test

the logical and mathematical abilities of models and has been applied in multiple benchmark tests. 5. AMC23 is a benchmark dataset designed to evaluate the advanced mathematical reasoning capabilities of large language models (LLMs). This dataset is primarily composed of problems from the 2023 American Mathematics Competitions (AMC 10/12). B.2

Knowledge-Intensive Reasoning Benchmarks

HotPotQA (Yang et al., 2018) is a questionanswering dataset for multi-hop. All the documents are sourced from Wikipedia, which provides the dataset with a rich knowledge base and relatively structured information. It is an important benchmark for evaluating the ability of LLMs to understand complex search tasks. 2WikiMultihopQA (Ho et al., 2020) is a dataset specifically designed for the multi-hop questionanswering task. It aims to test and evaluate the ability of natural language processing models to answer questions that require multi-step reasoning and the integration of information from different documents. Musique (Trivedi et al., 2022) is a questionanswering dataset specifically designed for the multi-hop question-answering task. Musique aims to be a challenging benchmark for evaluating models’ multi-hop question-answering ability. It promotes the development of models from simple information retrieval to deeper semantic understanding and logical reasoning. Bamboogle (Press et al., 2023) is a diagnostic dataset designed to test the multi-hop reasoning capabilities of large language models, specifically focusing on questions that popular search engines (such as Google) cannot answer directly with a single search result. It consists of 125 manually curated questions that require “bridge” reasoning. B.3

Training Datasets

Tool-Star-SFT-54K (Dong et al., 2025b) is a supervised fine-tuning dataset designed for the "coldstart" phase of the Tool-Star framework. It contains 54,000 high-quality samples of tool-integrated reasoning, covering the complete process of autonomous tool selection, parameter invocation, and result parsing. By fine-tuning on this dataset, the model establishes foundational tool-use norms and

basic multi-step reasoning capabilities, providing a stable policy base for subsequent reinforcement learning and effectively mitigating issues like reward sparsity and excessive search space during early RL stages. Tool-Star-RL-10K (Dong et al., 2025b) is a training dataset specifically tailored for the "SelfCritic Reinforcement Learning" stage, consisting of approximately 10,000 task samples. This dataset integrates various challenging tasks, supporting end-to-end interactive training in multi-tool environments. Through reinforcement exploration on these 10,000 samples, the model leverages the Self-Critic mechanism to optimize reasoning paths based on real tool feedback, significantly enhancing its logical rigor and self-correction capabilities in complex, long-horizon tasks.

Details of Experiment Setup

C.1

Supervised Fine-Tuning.

In supervised fine-tuning phase, we train the model using the Llama Factory framework with a learning rate of 7 × 10−6 . We employ DeepSpeed ZeRO-3 and FlashAttention2 for optimization. The batch size is set to 128, with a weight decay of 0.1, and the model is trained for 3 epochs. We use BF16 mixed precision with a maximum input length of 4096 tokens. C.2

Reinforcement Learning.

we implement the Reinforcement Learning algorithm based on the VERL framework. As Table 7, We have provided configurations for different training methods. All of our experiments were conducted on the 8*A100. C.3

Self-distillation Initialization.

To address the model’s difficulty in adhering to the specific <python> invocation protocol, we constructed a small-scale format alignment dataset. We sampled a subset from the SFT data, prompted the model to generate tool-integrated responses, and selected instances containing valid tool calls. We then standardized the format by replacing Markdown code blocks (”’python ... ”’) with the target <python> ... </python> tags, resulting in 500 aligned examples. Since this process involved only format substitution without logical correctness verification—and the data originates from self-distillation—this stage has a negligible impact on downstream performance.

Config

Warm Up

Post-RL

Zero-RL

γ λ n m k optimizer learning rate training data total training steps training batch size PPO mini batch size max response length max observation length max tool-calls retriever

0.5 0.1 8 8 3 AdamW 1e-6 4000 50 128 16 8192 512 4 local wiki

0 0 16 0 0 AdamW 1e-6 10000 250 128 16 8192 512 4 local wiki

Table 7: Hyperparameters and configurations for different Training.

C.4

Training Pipeine.

E3 -TIR. In this setting, the model initially undergoes the warm-up training phase proposed in E3TIR using the expert trajectory dataset, followed by reinforcement learning on Tool-Star10k using the Post-RL configuration. SFT-then-RL. In this setting, the model first undergoes supervised fine-tuning (SFT) on the complete 52k dataset, followed by reinforcement learning on Tool-Star10k using the same Post-RL configuration. Zero-RL. In this setting, the model is trained using the Zero-RL configuration. To ensure a fair comparison with E3-TIR and eliminate discrepancies arising from data coverage, we align the training data by including both Tool-Star10k and the expert dataset used in E3-TIR. However, we explicitly exclude the expert trajectory supervision (i.e., using only the prompts) from the expert dataset to strictly maintain the zero-shot exploration setting. C.5

Evaluation Setup.

For evaluation, we extended the max_obs_length to 1k and utilized the F1 score for search tasks, while employing ground-truth matching for mathematical reasoning tasks. The specific configurations are as follows: top_p = 0.7, temperature = 1.0, n = 4 , do_sample = True.

C.6

Reward Score.

Reward signals serve as the optimization objective and directly guide the behavior of the policy model during training. Formally, the overall reward R is defined as follows:   max(Acc. + rM , Acc.) If format & Acc. > 0 R = rM If format & Acc. = 0   −1 Otherwise ( 0.1 If ∃(valid format) rM = 0 Otherwise (15) For the mathematical reasoning tasks, we score them based on the degree of match between the model’s returned answer and the actual answer. Specifically, we used the math_verify library for the matching process. For the search task, we use the F1 score. C.7

Tool Setup.

Search Engine. we use Wikipedia as the retrieval corpus and E5 as the retriever for four open-domain QA tasks. Code Interpreter. Code snippets generated by the language model are executed in a local environment, returning either the execution result or error messages based on correctness. Our Python interpreter implementation follows the design of ToRA , ensuring secure and accurate execution.

The Algorithm Workflow Of E3 -TIR

In this section, we provide a detailed flowchart of the E3 -TIR algorithm in diagram 1.

Theoretical Analysis

In this section, we provide a theoretical justification for the effectiveness of the E 3 -TIR framework. Unlike token-level generation tasks, ToolIntegrated Reasoning (TIR) is modeled as a discrete decision-making process over a sequence of nodes (interaction turns). We analyze how Expert-Guided Branching mitigates the exploration complexity in this multi-turn setting. E.1

Problem Formulation

Following the definitions in Section 2.1, we consider the reasoning process as a trajectory of nodes. Definition E.1 (Node and Trajectory). Let a reasoning task utilize a maximum of T interaction turns. A trajectory is a sequence of nodes H = {u1 , u2 , ..., uT }, where each node ut = (τt , at , ot ) represents the triplet of thought, action, and observation at turn t. The policy πθ generates the next node conditioned on the history of previous nodes: ut ∼ πθ (·|H<t ). Assumption E.2 (Step-wise Difficulty). For a complex reasoning task, obtaining a successful outcome requires a sequence of correct decisions. We assume the “student” policy πθ (before alignment) has a bounded probability p ≪ 1 of independently generating the optimal next node u∗t given the cor∗ : rect history H<t ∗ P (ut = u∗t |H<t , πθ ) ≤ p

(16)

Here, p represents the turn-level success rate, not token-level probability. E.2

Proof. The probability of generating a complete successful trajectory is the product of the success probabilities at each node. Assuming independence for the lower bound analysis:

Failure Analysis of Zero-RL

In the Zero-RL setting (or pure Self-Exploration), the agent must successfully navigate from the initial state u0 to a terminal success state within T turns. Theorem E.3 (Curse of Horizon in Zero-RL). For a task requiring T critical reasoning turns, the probability of a randomly initialized policy πθ generating a fully correct trajectory Hsucc is exponentially small with respect to the number of turns T.

P (Success|Zero-RL) =

T Y

∗ P (u∗t |H<t , πθ ) ≤ pT

t=1

(17) Since p < 1, as the required turns T increase, pT decays exponentially. For instance, if an agent has a 50% chance of making a correct tool call at each step (p = 0.5) and the task requires 10 steps (T = 10), the success rate is 0.510 ≈ 0.0009. This explains the “Inefficient Exploration” (Figure 1) where the agent fails to collect positive feedback during the cold-start phase. E.3

Success Guarantee of E 3 -TIR

E 3 -TIR addresses this challenge by utilizing Expert-Guided Branch Sampling (Section 4.1.1). We select an expert prefix containing k nodes, de∗ (where k < T ), and perform G indenoted as H<k pendent branch rollouts from this anchor node. Theorem E.4 (Efficiency of Node-level Branching). Let the agent branch from an expert anchor node at turn k. The effective horizon is reduced to Tef f = T − k turns. With G parallel branches, the probability of discovering a successful path is significantly amplified. Proof. Given the expert history of k nodes, the agent only needs to complete the remaining T − k turns correctly. 1. Single Branch Probability: The probability of a single rollout successfully completing the remaining turns is: Psingle = pT −k

(18)

Crucially, since we branch at deep nodes (e.g., k ≈ T /2 or k ≈ 3T /4), the remaining horizon T − k is small, implying pT −k ≫ pT . 2. Group Success Probability: With G indepen∗ ), dent branches sampled from the policy πθ (·|H<k the probability that at least one branch reaches the solution is: P (Success|E 3 -TIR) = 1 − (1 − pT −k )G (19)

Algorithm 1 Training Workflow of E3 -TIR (Enhanced Experience Exploitation) Require: Initial Policy πθ ; Reference Model πref ; Expert Dataset Dexpert ; Self-explore Budget N ; Expertguided Budget M Input: Query set Q from Dexpert 1: Initialize experience pools Dself ← ∅, Dexp ← ∅ 2: for each training iteration i = 1 . . . K do 3: // Phase 1: Expert Experience-Based Rollout 4: for each query q ∈ Q do 5: Step 1.1: Anchor Selection 6: Calculate step-wise entropy ht on expert trajectory H∗ 7: Select top-k high-entropy prefixes as Anchors Tanchor 8: Step 1.2: Hybrid Sampling 9: Sample N trajectories from scratch: Dself ← Dself ∪ Rollout(πθ , q, N ) 10: for each anchor t ∈ Tanchor do 11: Calculate exploration probability Pt ∝ (ht − ht−1 ) 12: if Pt > γ and |Dexp | < M then ∗ : D ∗ 13: Branch sample from prefix H<t exp ← Dexp ∪ Rollout(πθ , H<t ) 14: end if 15: end for 16: end for 17: // Phase 2: Dynamic Experience Filtering 18: Discard expert trajectories with zero reward variance: V[r] = 0 19: When the variance is 0, only one trajectory is retained. 20: if max(r(Dself )) ≥ max(r(Dexp )) then 21: Dtrain ← Dself ▷ Self-exploration is sufficient 22: else 23: Dtrain ← Dself ∪ Dexp ▷ Need expert guidance 24: end if 25: // Phase 3: Mix Policy Optimization 26: for each trajectory τ ∈ Dtrain do 27: Calculate Global Advantage: Âglobal ← (r(τ ) − µglobal )/σglobal 28: if τ ∈ Dexp then 29: Calculate Tree Advantage: Âtree ← (r(τ ) − µtree )/σtree 30: Final Advantage Â ← Âglobal + Âtree 31: else 32: Final Advantage Â ← Âglobal 33: end if 34: Advantage-Aware Gradient Detach: 35: if τ shares prefix with expert AND Â < 0 then 36: Detach gradients for the shared prefix steps (Block optimization on prefix) 37: end if 38: end for 39: Update πθ by maximizing JHybrid with off-policy correction 40: end for 41: return Optimized Policy πθ Mitigating Low-Entropy Collapse. Traditional SFT+RL often collapses into repetitive, simple patterns (Fast Mode) to minimize turn length. By forcing the agent to branch from expert nodes

∗ —which are by definition diverse and highH<k quality—we enforce exploration in the valid reasoning subspace. The condition for E 3 -TIR to outperform Zero-RL is simply (1 − pT −k )G > pT ,

which holds true for almost any k > 0 and G ≥ 1. E.4

response excluding tool generation segments. A higher RMR value implies that the agent tends to execute actions directly with minimal intermediate reasoning (Chain-of-Thought), indicating a degradation in reasoning capability.

Numerical Illustration

Consider a complex mathematical reasoning task requiring T = 8 tool-use turns. Let the base model’s turn-level correctness probability be p = 0.4. 0.48

• Zero-RL: Success probability ≈ ≈ 0.00065 (approx. 1 in 1500). The reward signal is extremely sparse. • E 3 -TIR: We use an expert prefix of k = 6 turns (leaving 2 turns to solve) and sample G = 8 branches. – Success rate per branch: 0.42 = 0.16. – Total success probability: 1 − (1 − 0.16)8 ≈ 1 − 0.25 ≈ 75%. This dramatic increase from < 0.1% to 75% demonstrates why anchoring at deep nodes allows E 3 -TIR to achieve “Fast Convergence” and break the exploration bottleneck.

Supplement Experimental Results

F.1

Statistic Analyse of Problem in Current Training Paradigms.

Zero-RL Exploration Efficiency and Mode Analysis. We analyzed the behavioral distribution of Zero-RL during Tool-Integrated Reasoning (TIR) tasks: • Tool Call Failure Statistics (Fig. 2 (left,a)): We tracked the ratio of valid tool calls and the distribution of consecutive code errors during the initial training phase (first 40 steps). A trajectory is considered valid only if it contains code blocks that are both syntactically correct and executable by the interpreter. • Reasoning Mode Degradation (Fig. 2 (left,b)): We monitored the prevalence of the direct action mode (i.e., invoking tools without intermediate reasoning). We utilize the React-Mode Ratio (RMR) as a metric, defined as: RMR = 10000 ∗

Ntool ∗ Lavg Ltext

SFT+RL Policy Diversity and Entropy Analysis. To investigate the phenomenon where SFT+RL starts strong but subsequently stagnates or collapses, we conducted the following analyses: • Low Entropy Collapse (Fig. 2 (right,a)): We tracked the policy entropy ht at each RL step. Results show a sharp decline in entropy as training progresses, indicating that the model collapses into repetitive, simplistic modes and loses the capacity to explore complex reasoning paths. • Capability Rigidity and Distribution Shift (Fig. 2 (right,b)): We visualized the representation of rollout sampling by embedding generated thought paths using Qwen-Embedding, followed by PCA dimensionality reduction and DBSCAN clustering. Comparing the trajectory diversity of Zero-RL and SFT models at the same training step, the results clearly show that the number of sample clusters for the SFT model decreases significantly in later stages. F.2

Comparison of Expert Prefixes and Self-Exploration Sampling.

To investigate the variations in reward standard deviation and entropy under different sampling paradigms, we conducted two distinct RL training sessions using Qwen2.5-3B-Instruct, with hyper parameters fixed at N = 8 and M = 8. The experiments differed in the source of the M branches: one utilized expert prefixes, while the other relied on self-generated trajectories. As this study serves as a preliminary analysis, we limited training to 50 steps and recorded: (1) The standard deviation of rewards within trajectory trees sharing a common prefix. (2) The response entropy arising from branching at different turns, calculated specifically over the first 20 generated tokens.

(20)

where Ntool denotes the total count of tool invocations, Ltext is the average response length and Ltext represents the token length of the

F.3

Warm-up Stage Cost and Efficiency Analysis

To quantitatively evaluate the trade-off between pre-training costs and the subsequent reinforce-

ment learning (RL) efficiency, we introduce a comprehensive metric, the Efficient Cost-Benefit Ratio (ROI). This metric assesses how the E3 -TIR paradigm, utilizing varying scales of expert data and warm-up durations, accelerates the convergence of the Post-RL stage while maintaining or enhancing final performance. We formulate the training process into two distinct stages: the Warm-up stage (utilizing E3 -TIR) and the Post-RL stage. The relevant variables are defined as follows: • X: The training duration (or steps) allocated to the Warm-up stage. • Y : The size of expert trajectories utilized during the Warm-up stage. • Z: The training duration (or steps) required to reach convergence in the Post-RL stage. • Zbase : The baseline time required for the model to reach convergence without any warm-up (i.e., Zero-RL setting). • V : The performance improvement coefficient, defined as the ratio of the final performance of the proposed method to the Zero-RL baseline (e.g., V = 1.12 implies a 12% improvement). The ROI is designed to reward methods that achieve higher performance gains (V ) and faster total convergence (Z + αX) while minimizing reliance on extensive expert data (Y ). The formula is defined as: ROI =

V |{z}

Performance

Zbase Y × 1− Z +α·X Yref | {z } | {z }

Time Efficiency

Data Efficiency

(21) where: • α represents the relative computational cost coefficient between the Warm-up stage and the Post-RL stage. Since the Warm-up stage (typically use Expert Prefixes) is computationally less intensive per step than RL (e.g., GRPO). Based on the update statistics of each step during the training process, we set α = 0.62 to normalize the warm-up cost into equivalent RL units. • Yref is the reference data scale constant (set to 52k in our experiments). This term normalizes the data efficiency, penalizing the excessive use of expert trajectories.

• Norm represents the operation of mapping this set of data to the range of 0 to 1. Interpretation. A higher ROI indicates that the E3-TIR paradigm successfully leverages a small amount of expert data and a short warm-up phase to significantly reduce the expensive Post-RL training time (Z ≪ Zbase ), thereby achieving a superior overall cost-benefit trade-off.

Data Size

Expert Trajectories

Warm up

10k

10 steps 30 steps 50 steps 70 steps

43.0 44.3 43.7 44.3

43.4 44.6 46.7 45.6

42.7 43.8 46.3 45.5

42.9 44.5 46.0 46.1

Table 8: Model performance under different expert trajectories size and Warm up steps on Qwen2.5-3BInstruct.

Data Size

Expert Trajectories

Warm up

10k

10 steps 30 steps 50 steps 70 steps

200 180 140 110

210 170 120 100

190 160 120 100

200 180 130 110

Table 9: Post-RL convergence steps under different expert trajectories size and Warm up steps on Qwen2.53B-Instruct.

F.4 Comparison with Offline Fusion Baselines We benchmark the mixed strategy of E3-TIR against other policy fusion paradigms by comparing it with two state-of-the-art baselines: • Luffy (Yan et al., 2025): This method formulates a dynamically adjustable objective that balances “teacher imitation” and “selfexploration.” Utilizing Group Relative Policy Optimization (GRPO), Luffy directly incorporates high-quality off-policy reasoning trajectories generated by a strong teacher (DeepSeek-R1) into the training batch, computing advantages in conjunction with the model’s own on-policy trajectories. • HPT (Lv et al., 2025): This approach implements a real-time gating mechanism based on the rollout accuracy P for each specific

query. It dynamically toggles between learning signals using a threshold γ: applying pure SFT with demonstrations when performance is low (P ≤ γ), and switching to pure GRPO for exploration when performance is high (P > γ). HPT achieves a minimalist fusion—characterized as “one formula, two signals, automatic switching”—without requiring manual weight tuning or auxiliary models.

F.5

Expert Trajectory Analysis during the Warm-up stage

We analyze the evolution of the mixed strategy during the warm-up phase, focusing on sampling ratios and relative rewards. Figure 10 (left) illustrates a decline in expert-guided samples concurrent with a rise in fully on-policy generation as training progresses. This trend reflects a reduced reliance on expert trajectories, validating E3 -TIR’s two-stage training design. Additionally, Figure 10 (right) demonstrates that the mixed strategy consistently yields higher rewards than the on-policy approach, indicating that expert prefixes effectively enhance the probability of generating optimal trajectories.

G.2

Instruction for Sampling and Reasoning Computational Datasets You are a helpful assistant that can solve the given question step by step with the help of the wikipedia search tool. Given a question, you need to first think about the reasoning process in the mind and then provide the answer. During thinking, you can invoke the wikipedia search tool to search for fact information about specific topics if needed. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags respectively, and the search query and result are enclosed within <search> </search> and <result> </result> tags respectively. For example, <think> This is the reasoning process. </think> <search> search query here </search> <result> search result here </result> <think> This is the reasoning process. </think> <answer> The final answer is \boxed{answer here} </answer>. In the last part of the answer, the final exact answer is enclosed within boxed with latex format. {question}

G.3

Instructions for Reinforcement Learning Training

Instruction for Sampling and Reasoning Computational Datasets You are a helpful assistant that can solve the given question step by step with the help of the wikipedia search tool and python interpreter tool. Given a question, you need to first think about the reasoning process in the mind and then provide the answer. During thinking, you can invoke the wikipedia search tool to search and python interpreter tool to calculate the math problem for fact information about specific topics if needed. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags respectively, and the search query and result are enclosed within <search> </search> and <result> </result> tags respectively. For example, <think> This is the reasoning process. </think> <search> search query here </search> <result> search result here </result> <think> This is the reasoning process. </think> <python> python code here </python> <result> python interpreter result here </result> <think> This is the reasoning process. </think> <answer> The final answer is \boxed{answer here} </answer>. In the last part of the answer, the final exact answer is enclosed within boxed with latex format.

Instruction Template

G.1 Instructions for Mathematical Reasoning Evaluation Instruction for Sampling and Reasoning Computational Datasets You are a helpful assistant that can solve the given question step by step with the help of the python interpreter tool. Given a question, you need to first think about the reasoning process in the mind and then provide the answer. During thinking, you can invoke the python interpreter tool to calculate the math problem for fact information about specific topics if needed. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags respectively. For example, <think> This is the reasoning process. </think> <python> python code here </python> <result> python interpreter result here </result> <think> This is the reasoning process. </think> <answer> The final answer is \boxed{answer here} </answer>. In the last part of the answer, the final exact answer is enclosed within boxed with latex format. {question}

Instructions for Knowledge-Intensive Reasoning Evaluation

Case Study

1000

0.8

Mix-Policy On-Policy

Rewards

Numbers

800 600 400

0.6 0.4 Mix-Policy On-Policy

0.2 0

Steps (Warm up)

Figure 10: Variation of the sample size and rewards in the preheating stage of the mixed strategy. Table 10: An example from E3 -TIR on the Musique dataset, with special symbols used in think content, search queries, returned results and final answer highlighted with purple box, green box and red box, respectively. Example from Musique Question: What country has the border troops of the country of the literature of the country of Karl Maron citizenship? Labeled Answer: East Germany Model Output by E3 -TIR (Qwen2.5-3B) <think> To solve this question, I need to find out the country whose border troops have the citizenship of the country where the literature of the country of Karl Maron is from. Step 1: Identify the country where Karl Maron is from. Step 2: Identify the country whose literature is from the country of Step 1. Step 3: Identify the citizenship of the country from Step 2. Step 4: Identify the country whose border troops have the citizenship from Step 3. Step 1: Identify the country where Karl Maron is from. I’ll use a query to search for this information:</think> <search> Karl Maron country of citizenship </search> <result> Doc 1 [id: 11966372] Title: Karl Maron Karl Maron (27 April 1903 – 2 February 1975) was a German politician, who served as the interior minister of East Germany. He also assumed different posts in East Germany’s government... Doc 2 [id: 9746062] Title: Karl Maron ...Maron was the step-father of author Monika Maron. Karl Maron married her mother in 1955. He died in 1975. Karl Maron Karl Maron (27 April 1903 – 2 February 1975) was a German politician, who served as the interior minister of East Germany... </result> <think> Step 2: Identify the country whose literature is from the country of Step 1. I’ll use a query to search for this information:</think> <search> country where the literature of East Germany is from </search> <result> Doc 1 [id: 5484532] Title: Literature of East Germany Literature of East Germany East German literature is the literature produced in East Germany from the time of the Soviet occupation in 1945 until the end of the communist government in 1990... Doc 2 [id: 12666639] Title: Literature of East Germany the literature produced in this period among the best of the entire GDR. The 1990s saw the reunification of East and West Germany and the abrupt demise of the dream of a German "socialist utopia."... </result> <think> Step 3: Identify the citizenship of the country from Step 2. I’ll use a query to search for this information:</think> <search> literature of East Germany country </search> <result> Doc 1 [id: 5484532] Title: Literature of East Germany Literature of East Germany East German literature is the literature produced in East Germany... Doc 3 [id: 16627150] Title: Culture of East Germany Culture of East Germany The culture of East Germany varied throughout the years due to the political and historical events that took place in the 20th century... </result> <think> This is last step and I need to give the final answer, which is the country whose border troops have the citizenship of the country where the literature of the country of Karl Maron is from.</think> <answer> The final answer is EastGermany </answer>

Table 11: An example from E3 -TIR on the AIME25 dataset, with special symbols used in think content. Example from Dataset Question: Find the number of ordered pairs (x, y), where both x and y are integers between −100 and 100 inclusive, such that 12x2 − xy − 6y 2 = 0. Labeled Answer: 117 Model Output by E3 -TIR (Qwen2.5-3B) <think> To solve the problem, we need to find the number of ordered pairs (x, y) such that both x and y are integers between −100 and 100 inclusive, and they satisfy the equation 12x2 − xy − 6y 2 = 0. First, let’s rearrange the equation to make it easier to solve: 12x2 − xy − 6y 2 = 0. This is a quadratic equation in terms of x. We can use the quadratic formula to solve for x: 12x2 − xy − 6y 2 = 0. The quadratic formula for ax2 + bx + c = 0 is given by: x=

−b ±

√

b2 − 4ac . 2a

In our equation, a = 12, b = −y, and c = −6y 2 . So the solutions for x are: x=

y±

y 2 + 24 · 6y 2 y± = 2 · 12

This gives us two solutions:

p p √ y 2 + 144y 2 y ± 145y 2 y ± y 145 = = . 24 24 24

√ y(1 − 145) . 24 √ √ √ For x to be an integer, y(1+24 145) and y(1−24 145) must be integers. Since 145 is not an integer, y must be a multiple of 24 x=

√

√ y(1 + 145) 24

and

√

for y(1+24 145) and y(1−24 145) to be integers. Let’s denote y = 24k, where k is an integer. Then the solutions for x become: √ √ √ √ 24k(1 + 145) 24k(1 − 145) x= = 1 + 145k and x = = 1 − 145k. 24 24 √ √ √ Since 145 is not √ an integer, k must √ be such that 1 + 145k and 1 − 145k are integers. This means k must be an integer that makes 1 + 145k and 1 − 145k integers. The only way this can happen is if k = 0, but k = 0 would make x = 1. However, we need to check if there are any other values of k that might work. Let’s check the possible values of k by iterating over the range of y from −100 to 100. We will use Python to check which pairs (x, y) satisfy the equation. Let’s write the Python code to find the number of such pairs (x, y). </think> <python> 1 2 3

# Define the range for x and y range_min = -100 range_max = 100

4 5 6

# Initialize a counter for the number of valid pairs count = 0

7 8 9 10 11 12

# Iterate over all possible values of x and y for x in range(range_min, range_max + 1): for y in range(range_min, range_max + 1): if 12*x**2 - x*y - 6*y**2 == 0: count += 1

13 14 15

# Output the result print(count) </python> <result>117</result> <think> The number of ordered pairs (x, y), where both x and y are integers between −100 and 100 inclusive, such that 12x2 − xy − 6y 2 = 0 is 117 . </think> <answer>The final answer is 117 </answer>

Table 12: An example from E3 -TIR on the AIME24 dataset, with special symbols used in think content. Example from Dataset Question: Let p be the least prime number for which there exists a positive integer n such that n4 + 1 is divisible by p2 . Find the least positive integer m such that m4 + 1 is divisible by p2 .Labeled Answer: 110 Model Output by E3 -TIR (Qwen2.5-3B) <think> To solve this problem, we need to follow these steps: 1. Identify the smallest prime number p for which there exists a positive integer n such that n4 + 1 is divisible by p2 . 2. Once we find this prime p, we need to find the smallest positive integer m such that m4 + 1 is divisible by p2 . Let’s start by iterating over prime numbers and checking the condition for n4 + 1 being divisible by p2 . We will use Python to find the smallest prime p and then the smallest m.First, we will write a function to check if a number is prime and then iterate over prime numbers to find the smallest p. Let’s write the code to achieve this. </think> <python> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

from sympy import isprime, primerange # Function to find the smallest prime p def find_smallest_prime_p(): p = 2 while True: found = False # Check for the first 1000 values of n to find if there exists an n # such that (n^4 + 1) is divisible by p^2 for n in range(1, 1000): if (n**4 + 1) % (p**2) == 0: found = True break if found: return p p = next_prime(p)

16 17 18 19

# Find the smallest prime p smallest_prime_p = find_smallest_prime_p() print((smallest_prime_p, smallest_m)) </python><result>Code execution failed: Traceback (most recent call last): File “<string>”, line 26, in <module> File “<string>”, line 15, in find_smallest_prime_p NameError: name ‘next_prime’ is not defined</result> <think> It seems that the previous approach had some issues. Let’s solve this problem step-by-step more methodically. 1. **Identify the smallest prime p such that there exists a positive integer n for which n4 + 1 is divisible by p2 .** 2. **Once we find p, determine the smallest integer m such that m4 + 1 is divisible by p2 .** We will start by iterating over prime numbers and check for each prime p whether there exists an n such that n4 + 1 ≡ 0 (mod p2 ). This means n4 ≡ −1 (mod p2 ). We will then check for the smallest m that satisfies this condition for p2 . Let’s write the corrected and more analytical code to solve this: 1. Generate a list of prime numbers. 2. For each prime p, check if there exists an n such that n4 ≡ −1 (mod p2 ). 3. Once we find the smallest such p, find the smallest m such that m4 ≡ −1 (mod p2 ). </think> <python>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

from sympy import isprime, primerange, next_prime # Function to find the smallest m for a given prime p def find_smallest_m_for_prime_p(p): m = 1 while True: if (pow(m, 4, p**2) + 1) % (p**2) == 0: return m m += 1 # Iterate over primes primes = list(primerange(2, 10000)) for p in primes: found = False # Check for an n such that n^4 + 1 is divisible by p^2 # The search range for n can be up to p^2 for n in range(1, 1000): if (pow(n, 4, p**2) + 1) % (p**2) == 0: found = True break if found: smallest_prime_p = p break smallest_m = find_smallest_m_for_prime_p(smallest_prime_p) print((smallest_prime_p, smallest_m)) </python><result>(17, 110)</result><think> The smallest prime number p for which there exists a positive integer n such that n4 + 1 is divisible by p2 is 17. The least positive integer m such that m4 + 1 is divisible by 172 = 289 is 110.Thus, the final answer is 110 . </think> <answer>The final answer is 110 </answer>

Record · ID 6000 · SHA-256 83a4af482cf2ee12
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.