arXiv CS open access

One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning Armin Dariani1,2 , Sifan Wu1,2 , Bang Liu1,2* , Entao Yang3* 1 DIRO, Université de Montréal 2 Mila - Quebec Artificial Intelligence Institute 3 Innovation Campus Delaware, Air Liquide armin.zolfagharidariani@mila.quebec, sifan.wu@umontreal.ca bang.liu@umontreal.ca, entao.yang@airliquide.com *

Corresponding authors

arXiv:2608.30952v1 [cs.LG] 31 Aug 2026

Abstract

Building such an agent reliably remains challenging for several reasons. First, chemistry tools are highly specialized and challenging for generalpurpose LLMs to reason. Modern chemistry agents interact with a large collection of tools covering molecular property prediction, reaction analysis, quantum chemistry, database retrieval, and materials simulation. Many tools expose similar interfaces while differing subtly in their underlying assumptions, input requirements, or prediction targets, making accurate tool selection a non-trivial reasoning problem rather than simple retrieval. Second, chemistry tools require structured inputs with strict scientific validity and computational precision. Their arguments often consist of formal chemical representations, such as SMILES strings, molecular identifiers, chemical formulas, or reaction specifications, where syntactic or semantic errors can invalidate execution or lead to scientifically incorrect results. Generating such arguments therefore requires precise structured prediction grounded in chemical knowledge. Third, chemistry problems often require long-horizon tool interaction instead of isolated function calls. Solving a task may involve retrieving molecular information, predicting intermediate properties, and using those outputs to parameterize subsequent computations. Because each tool call depends on the correctness of previous ones, early mistakes propagate through the execution trajectory and compound into downstream failures. The current state-of-the-art system on ChemToolBench, CheMatAgent’s Hierarchical Evolutionary Monte-Carlo Tree Search (HE-MCTS), meets this with machinery. One model plans tool selection while a second fills parameters; a Process Reward Model and an Outcome Reward Model are trained as search critics, the latter regressed onto a mixture of rule-based and GPT-assigned scores; the policy is then fine-tuned on trajectories the search collected. The results are strong and the pipeline

Chemistry questions often demand exact computation and database lookups that a language model cannot supply from its parameters, so it must reach for external tools. Tool use here is a three-part problem: select the right tool from a large pool, fill it with correctly typed arguments, and chain calls so that each consumes the outputs of the last. CheMatAgent, a previously published system, addresses this with hierarchical evolutionary MCTS: separate policy and execution models searching tool-call trees under two learned critics, one regressed partly onto GPT-assigned scores. We show that a single policy suffices. Our model interleaves reasoning, tool calls, and returns in one left-to-right generation, trained by a supervised warm-up and then outcome-level reinforcement learning against a programmatic reward read directly off the gold call chain, which leaves no learned critic and no judge in the training loop. On ChemToolBench multiple-tool comprehensive chemistry, on both backbones CheMatAgent use, we improve Tool F1 by 5.5% and Return F1 by 9.6% on Qwen-2.5-7B, and by 3.7% and 3.9% on Llama-3.1-8B, compared with their strongest search configuration, at one model invocation per question, against a search whose cost grows with the tree; we also lead answer Pass Rate on Qwen-2.5-7B.

Introduction

Large language models are increasingly capable on chemistry problems such as molecule generation and reaction prediction (Zhang et al., 2024), but their parameters cannot store every measured property, and they cannot carry out the exact calculations many questions require. The practical remedy is to let the model call external tools as it reasons: cheminformatics libraries, property predictors, database lookups, as in ChemCrow (Bran et al., 2024) and CACTUS (McNaughton et al., 2024). 1

(a) CheMatAgent: HE-MCTS — four models, search at training and test time Policy model p — proposes k calls per node

Question

Self-training loop HE-MCTS trajectories retrain p and train both critics

Execution model u — fills parameters, retries PRM — learned critic, scores each step

the number of model calls grows with the size of the tree

ORM — learned critic, scores the answer

Answer

one search iteration: select → expand → evaluate → simulate → back-propagate

(b) Ours: single-agent rollout with interleaved tool execution Policy LLM πθ Step 1

Advantage Step T

Step 2

··· Question Reward output dependence: a return becomes the next call’s argument

computed by program from the gold call chain; no learned critic

Chemistry tool server (real execution)

Figure 1: (a) CheMatAgent’s HE-MCTS (Wu et al., 2025) splits tool selection and parameter filling across two models and searches a tree under two learned critics — a Process Reward Model (PRM) over steps and an Outcome Reward Model (ORM) over answers, the latter trained on a GPT-4o assessment blended with a rule-based check. Four models are trained and deployed, and model calls grow with the tree. (b) One policy πθ interleaves reasoning with live tool execution in a single generation, so a question costs one model and one LLM call. Context accumulates until step T ends in an answer; faded rows are earlier context, and <tool_result> spans (dashed) come from the server rather than the policy. A program then scores the trajectory against the gold call chain — no learned critic, no search (§4.1).

is expensive: four models to train and coordinate, plus a per-query search cost the authors themselves note as a drawback. That leaves open whether any of it is necessary.

vary only how the model is trained. Contributions. (i) We recast chemistry tool learning as single-agent reinforcement learning, in which one policy interleaves reasoning with live tool execution in a single generation, a procedure we call our rollout protocol (§4.1). It is trained with GRPO against a verifiable reward, dispensing with the execution model, the PRM and ORM critics, and the tree search of the previous state of the art (Wu et al., 2025). (ii) We develop a systematic reward framework for multi-tool calling, introducing programmatic rewards that capture answer correctness, tool selection, tool-argument accuracy, and their combinations. We further analyze the trade-offs and failure modes induced by different reward designs, including tool spamming and non-termination. (iii) Our single-policy model consistently outperforms existing SOTA approaches across multiple metrics. It improves Tool F1 by 5.5% and Return F1 by 9.6% on Qwen-2.5-7B, and

Recent advances in reinforcement learning suggest the latter may be possible. Reinforcement learning with verifiable rewards can elicit strong reasoning in a single model (Guo et al., 2025), and the idea extends to tool use (Feng et al., 2026; Jin et al., 2025). Following that line, we train one policy that alternates thinking and tool calls inside a single generation and optimize it end to end with Group Relative Policy Optimization (Shao et al., 2024) against a reward computed directly from the gold call chain. The model reasons, calls a tool, reads the real execution result, and continues until it answers (Figure 1); there is no execution model, no learned critic, and no search. To keep the comparison controlled we reuse CheMatAgent’s benchmark, toolpool, and candidate-tool retrieval, and 2

by 3.7% and 3.9% on Llama-3.1-8B, respectively. To disentangle model improvements from evaluation effects, we further evaluate each untrained backbone under the same rollout protocol, as untrained performance reflects both the evaluation framework and the model capability. Compared with this baseline, supervised warm-up provides a larger contribution than reinforcement learning alone, while reinforcement learning primarily improves tool precision at the expense of recall.

Reinforcement learning for reasoning and tool use. DeepSeek-R1 (Guo et al., 2025) drew strong reasoning out of a single model almost entirely by reinforcement learning with verifiable rewards, using GRPO (Shao et al., 2024), which drops PPO’s learned critic and scores each sample against its group mean. The recipe then moved to tool use: ReTool (Feng et al., 2026) interleaves code execution with reasoning, and Search-R1 (Jin et al., 2025) does the same for search, masking retrieved tokens to keep training stable, a device we reuse. Closest to our reward design, ToolRL (Qian et al., 2026) decomposes the reward over tool names, parameter names and parameter values rather than matching the answer, Nemotron-Research-Tool-N1 (Zhang et al., 2026) uses a binary format-and-correctness reward on largely single-turn benchmarks, and MatchTIR (Qu et al., 2026) adds turn-level rewards from matched tool traces. We share this recipe and differ in setting and in signal. A chemistry question needs several dependent calls from a large specialized pool, each argument a SMILES string or formula, rather than one code block or one query, and our reward is read off the gold call chain, giving partial credit over tool names and their arguments rather than scoring the answer.

Related Work

Tool learning: data and methods. Toolformer (Schick et al., 2023) showed that a model can learn on its own when to call an API, and ToolLLM (Qin et al., 2024) scaled this to over 16,000 real APIs with the ToolBench benchmark. Later resources target the two hard parts, choosing a tool and filling its arguments: API-Bank (Li et al., 2023), SealTools (Wu et al., 2024), and ToolACE (Liu et al., 2025), which synthesises function-calling data with a multi-agent pipeline. The Berkeley Function Calling Leaderboard (Patil et al., 2025) is the standard evaluation, now extended to multi-turn, stateful settings. Most of this line teaches tool use by supervised fine-tuning alone. We use SFT only as a warm-up before online RL in which the model calls the tools during training.

Task and Benchmark

3.1

Problem Formulation

Given a chemistry question q in natural language, an agent must produce a final answer by calling tools from a fixed pool T = {t1 , . . . , tm }, each a Python function exposed through an OpenAIstyle JSON signature. Most questions cannot be answered with a single call: they require a chain in which a later call, and the arguments it takes, depend on the values earlier calls returned. The agent must therefore select the right tools and order dependent calls correctly, and it is judged both on how well its calls match a reference and on whether its final answer is correct. Every question is annotated with a validated ground-truth calling chain C = [(ti , argsi , ri )]L i=1 recording the tools, their arguments, and their returns; we use C both to build supervised targets and to compute rewards.

LLM agents for chemistry. Science suits toolaugmented agents because answers must be precise and checkable. ChemCrow (Bran et al., 2024) combines eighteen expert tools to plan organic syntheses, CACTUS (McNaughton et al., 2024) pairs open LLMs with cheminformatics packages such as RDKit, SciAgent (Ma et al., 2024) extends toolaugmented reasoning across scientific benchmarks, and ChemLLM (Zhang et al., 2024) instead builds chemical knowledge in through pretraining. Closest to us is CheMatAgent (Wu et al., 2025), the strongest of these, contributing a 137-tool chemistry pool, the ChemToolBench dataset, and HEMCTS: a policy model and an execution model are kept separate, a trained Process Reward Model and Outcome Reward Model guide tree search, and the policy is fine-tuned step by step on search trajectories. Accuracy is strong, but the pipeline is large and costly at inference, a point the authors themselves raise. We keep their benchmark and toolpool unchanged and ask whether much simpler training reaches the same level.

3.2

ChemToolBench

We use ChemToolBench, the tool-learning benchmark released with CheMatAgent (Wu et al., 2025). It has a chemistry and a materials split, each with single-tool and multi-tool questions par3

Source

separate one to fill their parameters, we use a single policy πθ that does both. The model talks to a tool server for at most T = 16 turns. On each turn it writes a short reasoning block inside <think>...</think> and emits one <tool_call> object of the form {"name": "<lib>/<func>", "arguments": {...}}. The server executes the call and appends its output as a <tool_result>...</tool_result> span, and the model opens its next <think> block using the value actually returned. An episode ends when the model writes a single-line Answer: or exhausts the turn and tool-call budget. Because <tool_result> spans are produced by the server rather than the policy, we mask them out of the training loss and update only on tokens the model generated itself. We call this procedure, one policy interleaving reasoning with live tool execution in a single generation, our rollout protocol. Every model we report, trained or untrained, is run under it, so a difference between rows is a difference in the model and not in the harness. Chains carry this dependency whenever they run, which is why we execute the real tools during training rather than simulating their outputs. Appendix C gives the prompt, the exact trajectory format and a complete rollout (Figure 2) in which a later call’s argument is a value an earlier call returned.

Amount 1

ChemCrow CACTUS2 chemlib3 pymatgen4 Chemistry Tools5 In total:

8 10 24 82 13 137

Table 1: The tool pool by source, counted after organization and rewriting, reproduced from Wu et al. (2025).

titioned into train, development, and test, and every question is paired with a validated gold calling chain. The chemistry split has 10,441 single-tool and 2,023 multi-tool questions, divided train/development/test as 8353/1044/1044 and 1623/200/200. The materials split is comparable in size (15,742 and 1,623). Our experiments target the harder multiple-tool chemistry split, whose questions chain between two and six dependent calls, 3.15 on average. We evaluate on all 200 test questions and train on the 1566 of 1623 training questions whose gold chains still reproduce against the current tools (Appendix A). We keep the benchmark and its tools otherwise unchanged and vary only how the model is trained, so any difference in performance is attributable to the training method rather than to the data or the tools. 3.3

Chemistry Tool Pool

4.2

The chemistry split is served by tools drawn from four libraries (Table 1) and rewritten by Wu et al. (2025) into a uniform, self-describing format. They span cheminformatics utilities, property and structure predictors, reaction and stoichiometry calculators, and drug-likeness scorers, and each is invoked by its library-qualified name (<lib>/<func>) with typed JSON arguments. Some run offline; others call live web services, which is what makes a small number of released gold return values go stale over time (Appendix A).

Method

4.1

Single-Agent Tool-Use Rollout

Supervised Fine-Tuning

Reinforcement learning from a cold base model is slow to acquire the tool-calling format, so we optionally warm the policy up with a short round of supervised fine-tuning. We linearise each gold calling chain into the rollout trajectory of §4.1, interleaving a reasoning block, the tool call and its returned result at every step, and set the target final answer to a complete natural-language sentence stating the quantities the question asks for. Training the policy to reproduce these trajectories teaches both the output format and a first approximation of correct tool selection. This checkpoint is the starting point for RL (SFT→RL); we also run RL directly from the base model (RL-only), and find the warm-up is worth at least as much as RL on its own (§7.1).

Figure 1 contrasts the two designs. Where HEMCTS uses one model to select tools and a 1 https://github.com/ur-whitelab/ chemcrow-public 2 https://github.com/pnnl/cactus 3 https://github.com/harirakul/chemlib 4 https://github.com/materialsproject/pymatgen 5 https://github.com/domdfcoding/chemistry_ tools

4.3

Programmatic Reward

Let C be the ground-truth calling chain, and let a response produce a set of calls Ĉ and a final answer â. Every reward below is computed by a program rather than a learned model, and each is mapped 4

linearly to [−1, 1]. This is a deliberate departure from CheMatAgent (Wu et al., 2025), whose search is guided by two learned critics, a Process Reward Model and an Outcome Reward Model, the latter regressed onto a weighted average of a rule-based score and a GPT-assigned one. Our reward needs no such critics and no LLM judge in the training loop. It depends only on C and is therefore cheap, deterministic, and reproducible. The main variants are the following.

only the warm-up; the backbones and baselines are listed in §5.

We adopt CheMatAgent’s benchmark, tool exposure and scoring rules throughout, so that a difference in the tables is attributable to training. 5.1

Benchmark and Backbones

We evaluate on the multiple-tool comprehensivechemistry split of ChemToolBench (Wu et al., 2025), using its full test set of 200 multi-step questions, the same evaluation set CheMatAgent use. A few released gold chains no longer reproduce, because some tools query live services, so we drop those records from the training and development data, leaving 1566 and 196 questions (Appendix A). We train the same two backbones CheMatAgent report, Qwen-2.5-7B-Instruct (Team, 2024) and Llama-3.1-8B-Instruct (Grattafiori et al., 2024), so that within each block of Table 2 only the training method varies, and add Qwen3-4B (Yang et al., 2025) as a scale check, which appears only in Table 3. Each untrained backbone is measured under the same rollout protocol (§4.1), which separates what training contributes from what the protocol itself does.

• Rans (value inclusion). The fraction of the ground-truth return values that appear verbatim, after normalisation, in the final answer â. It does not compare â with a reference answer, but asks only whether the values the tools produced were reported. Full credit requires all of them, and a partial match receives a negative reward. F 1 (tool set). Recall, or its F1 • Rtool and Rtool counterpart, over the set of tool names. F1 penalises extra calls, which discourages the shortcut of invoking everything in the pool. F 1 (tool and arguments). As • Rcall and Rcall Rtool , but a ground-truth call counts only when its arguments match too, with strings lowercased and floats rounded to six decimals. The F 1 variant treats calls as a list rather than Rcall a set, so repeats enlarge the denominator and lower precision, discouraging both spamming and padding.

5.2

Training and Inference

The two stages follow §4.2 (SFT warm-up) and §4.4 (GRPO), with SFT→RL continuing from the warm-up checkpoint and RL-only starting from the base model. The warm-up runs three epochs at learning rate 10−5 with batch size 64 over the 1394 training questions whose gold chain can be linearised into a supervised trajectory. Reinforcement learning then runs 100 rollout steps over all 1566, since a rollout is scored against the gold chain directly and needs no linearised target. At evaluation we decode greedily (τ =0), so no difference we report comes from sampling.

F 1 (hybrid). An equal mix of a • Rhyb and Rhyb tool reward and the answer reward, Rhyb = F 1 formed the 0.5 Rtool + 0.5 Rans , with Rhyb F 1 same way from Rtool . The policy is scored both on which calls it made and on whether it reported the values those calls returned.

4.4

Experimental Setup

Reinforcement Learning

We train πθ with GRPO (Shao et al., 2024). For each prompt we sample a group of n = 5 rollouts, standardize their rewards within the group to obtain advantages (dispensing with PPO’s value network), and take a clipped policy-gradient step at learning rate 1 × 10−6 . Training runs on the slime post-training stack (Zhu et al., 2025), where Megatron-LM updates the policy while an SGLang server produces the multi-turn rollouts and the two communicate through slime’s data buffer. Both starting points, RL-only and SFT→RL, use identical optimizer settings, so their difference reflects

5.3

Evaluation Metrics

We reimplement CheMatAgent’s metrics against their released scoring code so that our columns and theirs mean the same thing, micro-averaging over the whole test set as they do. Format is the fraction of emitted tool calls that parse. Tool matches tool names with list semantics, so a duplicated call scores as a false positive. Param awards partial credit per parameter value, so a call with two of 5

Tool

Param

Model

Format

GPT-4o-mini (CoT) Claude-3.5-S (CoT) ChemLLM∗ (SFT) GPT-4o-mini-M (HE-MCTS) Claude-3.5-S-M (HE-MCTS)

99.83 97.42 98.10 / /

83.22 83.64 76.84 85.06 89.80

78.86 85.37 88.08 83.57 86.27

80.98 84.50 82.07 84.31 88.00

Return F1

Pass F1

Rate

76.04 72.19 74.06 77.05 73.13 75.04 76.03 77.53 76.77 74.96 78.54 76.71 68.22 80.20 73.72 68.84 80.45 74.19 88.47 / / 75.97 74.64 75.30 86.36 / / 77.55 74.51 76.00

58.00 57.00 53.00 62.30 57.06

Base: Qwen-2.5-7B-Instruct CheMatAgent-I (no FT) 51.25 64.15 41.81 50.63 56.48 37.36 44.97 29.25 37.20 32.75 22.50 CheMatAgent-M3 (HE-MCTS) / 91.14 90.45 90.80 89.32 / / 81.41 80.79 81.10 67.32 Ours (SFT→RL) 100.00 96.30 95.23 95.76 88.35 87.36 87.85 89.39 88.39 88.89 68.50 Base: Llama-3.1-8B-Instruct CheMatAgent-I (no FT) 75.99 CheMatAgent-I∗ (SFT) 99.20 CheMatAgent-M0 (HE-MCTS) / CheMatAgent-M1 (HE-MCTS) / CheMatAgent-M2 (HE-MCTS) / CheMatAgent-M3 (HE-MCTS) / Ours (SFT→RL) 100.00

59.95 93.10 75.45 87.74 93.18 93.22 96.31

38.31 92.21 78.45 87.32 88.39 91.73 95.55

46.75 92.65 76.92 87.53 90.79 92.47 95.93

52.98 33.71 41.20 40.83 34.34 37.31 15.00 83.85 83.15 83.50 85.03 84.90 84.96 55.00 85.25 / / 65.09 67.68 66.36 29.19 85.92 / / 75.81 75.44 75.62 69.50 95.12 / / 88.64 84.07 86.36 72.30 92.36 / / 86.09 84.72 85.41 72.20 89.50 88.62 89.06 90.06 89.35 89.70 64.00

Table 2: Main results on the multiple-tool comprehensive-chemistry benchmark, grouped by backbone so that within a block only the training method varies. All rows use the same 200 test questions under CheMatAgent’s evaluation regime, and bold marks the best value within each shared-backbone block. On both backbones our single policy takes every Tool and Return column with no tree search, and on Qwen-2.5-7B it also takes Pass Rate. ∗ : fine-tuned on the comprehensive-chemistry split. “/”: value unavailable.

three arguments right scores 2/3. Return compares the value a call actually produced against the gold return. Pass Rate is answer-level, computed as CheMatAgent do with their grading prompt verbatim and GPT-4o-mini at temperature 0. The judge sees the question, the gold answer and the model’s answer and returns Yes or No, and we report the fraction of Yes over all 200 questions, a missing answer counting as No (Appendix D). None of these is involved in training, where the reward is a separate programmatic function (§4.3).

ence. Parameters and returns. Returns follow the same pattern as tools. We take all three Return columns on both backbones, 89.39/88.39/88.89 against -M3’s 81.41/80.79/81.10 on Qwen2.5-7B and 90.06/89.35/89.70 against -M2’s 88.64/84.07/86.36 on Llama-3.1-8B. Parameters are more mixed. Param recall and F1 are unavailable for the search rows, so precision is the only Param column on which those rows admit a comparison, and HE-MCTS keeps it on both backbones (89.32 and 95.12 against our 88.35 and 89.50). Against the supervised Llama I∗ model, which does report all three, we lead every Param column. Whatever the search buys here costs four trained models and a tree search at every question.

Results

Table 2 compares our models against CheMatAgent’s, grouped by backbone so that within a block the base model is held fixed and only the training varies.

Answer Pass Rate. Pass Rate is the metric that most directly reflects what a user receives, and it is where the two backbones part company. On Qwen2.5-7B a single pass reaches 68.50, ahead of -M3’s 67.32. On that backbone our model leads every column HE-MCTS reports comparably except Param precision, so almost everything the search procedure buys is recovered by training one policy differently. On Llama-3.1-8B we reach 64.00 against 72.30 for -M2, and search retains a clear advantage. Both scores beat every chain-of-thought baseline in the table, including GPT-4o-mini (58.00) and

Tool selection. The clearest result is on tool selection, the task the HE-MCTS policy model exists to perform. On both shared backbones our model takes Tool precision, recall, and F1 outright. On Qwen-2.5-7B we reach 95.76 Tool F1 against 90.80 for -M3, the strongest configuration they build on that base; on Llama-3.1-8B, 95.93 against 92.47 for -M3 and 92.65 for their supervised I∗ model. Because the comparison is within a backbone, the same weights, trained differently, select tools better without any search at infer6

Tool Training

Param

Return

Pass

Rate

86.17 93.16 89.98 95.23

85.62 93.54 92.11 95.76

75.98 85.98 85.86 88.35

76.40 85.25 82.72 87.36

76.19 85.61 84.26 87.85

76.45 87.50 88.67 89.39

77.42 86.80 84.58 88.39

76.94 87.15 86.57 88.89

54.00 65.50 60.00 68.50

Llama-3.1-8B-Instruct none (base) 36.70 84.90 SFT only 94.90 94.59 RL only 95.31 93.64 SFT→RL 96.31 95.55

51.25 94.75 94.47 95.93

20.82 87.76 86.86 89.50

54.63 87.64 85.39 88.62

30.16 87.70 86.12 89.06

25.50 88.36 87.86 90.06

58.98 88.08 86.33 89.35

35.60 88.22 87.09 89.70

24.50 69.50 62.50 64.00

Qwen3-4B none (base) SFT only RL only SFT→RL

80.21 93.41 84.45 95.68

80.84 84.59 87.90 88.35

67.56 84.83 72.47 87.36

73.60 84.71 79.45 87.85

83.14 86.83 89.27 90.02

69.79 86.96 74.09 88.87

75.89 86.89 80.97 89.44

53.00 65.50 57.00 68.00

Qwen-2.5-7B-Instruct none (base) 85.09 SFT only 93.91 RL only 94.33 SFT→RL 96.30

87.88 93.33 93.10 96.30

73.77 93.48 77.27 95.07

Table 3: Contribution of each training stage. All arms use our rollout protocol, the same 200 test questions and the F1 reward Rcall , so they differ only in training (§7.1).

SFT alone beats RL alone. SFT-only leads every F1 column and Pass Rate on all three backbones. On Tool F1 the margin is 1.43 on Qwen-2.5-7B, 0.28 on Llama-3.1-8B and 8.96 on Qwen3-4B, and on Pass Rate it is 5.50, 7.00 and 8.50. Every cell RL-only takes is a precision cell, the effect the next paragraph isolates. The two margins have different causes. On the tool columns both stages learn from the same gold call chains, but SFT sees them token by token while RL sees only one number per rollout summarising how well it matched, so the same information reaches the model through a much narrower channel. On Pass Rate the problem is not narrowness but absence. Supervised targets end in a complete natural language answer, so SFT is trained on how to report a result, while our reward reads the call chain alone and never inspects the answer (§4.3). RL has no signal at all for what Pass Rate measures.

Claude-3.5-Sonnet (57.00), both supervised models (ChemLLM 53.00, Llama-3.1-8B-I∗ 55.00), and the weaker search configurations (-M0 29.19, Claude-3.5-S-M 57.06). One asymmetry behind this comparison runs against us and is worth naming. Our reward is computed from the gold call chain and never inspects the final answer (§4.3), so nothing in training optimises Pass Rate. What the table reports is whatever answer quality follows from calling the right tools after a warm-up on complete answers. CheMatAgent’s Outcome Reward Model scores the final answer directly and is regressed in part onto GPT judgements, and it guides the search that selects the trajectories their policy is then fine-tuned on. Their Pass Rate is thus optimised, if indirectly, while ours is a by-product. That our Qwen model still leads suggests a tool-level objective carries most of the distance, while our Llama model shows it is not sufficient everywhere.

7.2

Analysis

7.1

What Each Training Stage Contributes

Reward Design and Reward Hacking

We ablate the reward with the warm-up removed (Table 4), so a pathology the reward permits appears undamped by an SFT prior. Rewarding recall alone is satisfied by calling everything, and the policy finds that solution. Rcall never charges for calls outside the gold chain and reaches the highest recall of any arm (98.25) by issuing 18.33 calls per question against a gold average of 3.15, half of them repeats, which collapses precision to 19.70 and leaves 97% of episodes exhausting the call budget without answering. One precision term prevents this and nothing further is needed: all

Table 3 runs all four arms (untrained backbone, SFT only, RL only, and the composition) through our rollout protocol on the same 200 questions, so they differ in training and nothing else. Both RL F 1 , and where arms use 100 rollout steps under Rcall a supervised stage is present it is the same threeepoch warm-up. Qwen3-4B is added as a scale check. 7

Tool Reward extra calls unpenalised Rcall (recall, name+args)

Param F1

Return F1

behaviour F1

calls

dup.

n-term

19.70 98.25 32.82 16.13 85.96 27.17 17.88 89.19 29.79 18.33 49.6% 97.0%

precision term added F1 Rtool (set F1, names) 95.53 95.23 95.38 82.87 83.57 83.22 85.33 85.06 85.19 F1 Rcall (list F1, +args) 94.33 89.98 92.11 85.86 82.72 84.26 88.67 84.58 86.57 F1 Rhyb (tool F1 + value incl.) 93.81 93.96 93.88 82.13 83.29 82.71 85.56 85.69 85.62

3.13 3.06 3.16

0.6% 1.1% 3.9%

0.0% 0.0% 0.0%

Table 4: Reward ablation on Qwen-2.5-7B, trained for 100 steps without a warm-up so the reward is the only signal. “calls” is the mean tool calls per question against a gold average of 3.15, “dup.” the share repeating a tool already used, and “n-term” the share of episodes exhausting the 16-call budget without answering (§7.2).

three F1 rewards stay within 0.1 of the gold call count, rarely repeat and always terminate, so the gap between 32.82 and 95.38 Tool F1 rests on that term alone. Their remaining differences are secondF 1 because it takes Param and order, and we use Rcall Return, the columns that check arguments. Scoring F 1 adds R , the answer as well does not help: Rhyb ans which demands each gold return value verbatim: only 57% of numeric returns match exactly where 92% are reported to within 1%, so it feeds noise F 1 is the least rather than signal into training, and Rhyb stable arm. 7.3

machinery is not necessary. A single policy that interleaves reasoning and tool calls in one pass, warmed up with a short round of SFT and then optimized with GRPO against a simple programmatic reward, matches or exceeds CheMatAgent’s HE-MCTS pipeline on tool, parameter, and return F1 on the same backbones, and is competitive on answer Pass Rate, all at one model invocation per question. Our analysis further shows where each ingredient helps: the supervised warm-up supplies tool recall, RL supplies precision, and list/F1 reward semantics are what keep the policy from hacking the tool score. Verifiable programmatic rewards are enough; no learned reward model or search is required to reach this level.

Self-Repair from Execution Feedback

Of the three problems in §1, output-dependent chaining carries a cost the other two problems do not. Because a later call consumes an earlier return, one bad value is inherited by every step after it, even when those steps select and type their own calls correctly. Our protocol executes tools for real in training as well as at test, so a misnamed or wrongly typed call returns an error string rather than a plausible value, and the policy reads it on the next turn. Our models act on this, re-issuing the call with the fault corrected instead of continuing as though it had succeeded. We call this self-repair from execution feedback, and supervision cannot be its source, since gold chains contain only calls that succeed. Figure 3 in Appendix E shows a case where one mistyped argument name would otherwise have cost both remaining values, however well those two calls were chosen.

Limitations Scope. We study one benchmark and one domain, the multiple-call comprehensive-chemistry split of ChemToolBench. To our knowledge no other chemistry benchmark pairs questions with multi-call gold chains, so in-domain generalisation cannot be tested, and whether the recipe transfers to other scientific-tool domains is untested. Its chains run two to six calls and our backbones span 4–8B parameters, so longer horizons and larger models are unevaluated.

Acknowledgments The authors gratefully acknowledge financial support from Air Liquide and the Mitacs Program (IT46383) for this research. We also acknowledge the computational resources provided by DeltaAI at the National Center for Supercomputing Applications through ACCESS allocation CIS260150 (E.Y., B.L.).

Conclusion

Chemistry tool learning has been approached with heavy, multi-part systems: a separate planner and executor, trained PRM and ORM critics, and inference-time tree search. We showed that this 8

References

of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 48371–48392. PMLR.

Andrés M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. 2024. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525– 535.

Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tur, Gokhan Tur, and Heng Ji. 2026. Toolrl: Reward is all tool learning needs. Advances in Neural Information Processing Systems, 38:105523–105553.

Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2026. Retool: Reinforcement learning for strategic tool use in llms. In International Conference on Learning Representations, volume 2026, pages 37909–37926. Aaron Grattafiori et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2024. Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations, volume 2024, pages 9695–9717.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633–638.

Changle Qu et al. 2026. MatchTIR: Fine-grained supervision for tool-integrated reasoning via bipartite matching. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL). ArXiv:2601.10712.

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, volume 36, pages 68539–68551. Curran Associates, Inc.

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116, Singapore. Association for Computational Linguistics.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, et al. 2025. Toolace: Winning the points of llm function calling. In International conference on learning representations, volume 2025, pages 41359–41381.

Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Mengsong Wu, YaFei Wang, Yidong Ming, Yuqi An, Yuwei Wan, Wenliang Chen, Binbin Lin, Yuqiang Li, Tong Xie, and Dongzhan Zhou. 2025. CheMatAgent: Enhancing LLMs for chemistry and materials science through tree-search based tool learning. arXiv preprint arXiv:2506.07551.

Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, and Aixin Sun. 2024. SciAgent: Toolaugmented language models for scientific reasoning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15701–15736, Miami, Florida, USA. Association for Computational Linguistics.

Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. 2024. Seal-tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 372–384. Springer.

Andrew D. McNaughton, Gautham Krishna Sankar Ramalaxmi, Agustin Kruel, Carter R. Knutson, Rohith A. Varikoti, and Neeraj Kumar. 2024. CACTUS: Chemistry agent connecting tool usage to science. ACS Omega, 9(46):46563–46573.

An Yang et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, Shufei Zhang, Mao Su, HanSen Zhong, Yuqiang Li, and Wanli Ouyang. 2024. Chemllm: A chemical large language model. ArXiv, abs/2402.06852.

Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Proceedings

below, the JSON signatures of the tools exposed for that query, one worked example, and the user turn.

Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. 2026. Nemotron-researchtool-n1: Exploring tool-using language models with reinforced reasoning. In International Conference on Learning Representations, volume 2026, pages 91437–91453.

System prompt. You are a chemistry assistant that solves problems by calling chemistry tools. Work strictly ONE step at a time: 1. Write a brief <think>...</think> block that reasons about ONLY the next single tool call. Do NOT plan multiple future steps in advance — you cannot know what a tool will return until it is actually called, so later steps must be decided after seeing real <tool_result> values. 2. Emit exactly one <tool_call>...</tool_call> JSON block. 3. Wait for the <tool_result>, then start the next <think> by incorporating what the tool actually returned. When you have enough information, give a COMPLETE final answer on a single line that states, in a full sentence, every quantity the question asked for, as: Answer: <your complete answer>.

Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. 2025. slime: An llm post-training framework for rl scaling. GitHub repository.

Refreshing stale gold returns

Some ChemToolBench tools query live services, so for a few records the return stored in the released gold chain is no longer what the tool returns today. A stale chain penalizes a model that calls the right tool with the right arguments and reports what it received, which is the behaviour the metric should reward. The failures are value mismatches in tools whose answer legitimately changes over time, led by chemcrow/PatentCheck, plus a few tools that now raise. A stale chain is a corrupt supervision target, so we drop the affected records from the data we train on, 57 of 1623 training and 4 of 200 development questions. We drop nothing from the test split.

The one-step-at-a-time instruction is what forces the interleaving of §4.1: the model cannot emit a plan for the whole chain, because each later argument may depend on a value it has not yet seen.

The chemistry tool pool

Tools and user turn. Tool signatures are inserted as JSON objects inside <tools></tools>, followed by the example, in a chat template whose role delimiters follow the backbone (Qwen or Llama). The user turn is the benchmark query with a fixed suffix instructing the model to emit <tool_call> blocks and finish with a single Answer: line.

The chemistry split is served by 55 tools across four libraries. Each is invoked by its library-qualified name with typed JSON arguments, and the model sees only the signature, never the implementation. Ten of the 55 call live web services at request time, which is why a small number of released gold returns no longer reproduce (Appendix A). Table 5 lists the calls the gold chains issue most often. Tool

Calls

chemistrytools/get_compound_CID chemistrytools/get_compound_ MolecularWeight_by_CID chem_lib/get_empirical_formula_ by_percent_composition chem_lib/calculate_compound_molar_mass

732

Rollout and supervision. An episode runs for at most T = 16 turns. The server executes each call and appends its output verbatim inside <tool_result></tool_result>; because those spans are produced by the tool server rather than the policy, they are masked out of the loss and only model-generated tokens are trained on. Episodes end at the first Answer: line or when the turn budget is exhausted. SFT targets are gold calling chains linearised into exactly this format (reasoning block, call, returned value, repeated) and closed with a complete natural-language answer, so the supervised and RL formats are identical.

616 446 438

Table 5: The most frequently called tools in the gold chains of the multiple-tool chemistry split, counted over train, development and test.

Prompts and rollout protocol

One prompt is used for RL rollouts, for SFT targets and for evaluation, so the policy never sees a format at test time that it was not trained on. It is assembled from four parts: the system prompt

Answer-level judging

Pass Rate is computed exactly as Wu et al. (2025) compute it, reusing their grading prompt verbatim: 10

Please evaluate whether the given answer to the question is correct according to the gold answer. Return "Yes" if it is correct, otherwise return "No". Question: {query} Gold Answer: {gold} Given Answer: {pred} Whether the given answer is correct:

Query (multiple_1991) Please retrieve the PubChem Compound Identifier (CID) for the chemical compound ’(5-Bromo-2methoxyphenyl)methanol’, and subsequently, obtain the molecular weight of this compound using the retrieved CID.

The judge is GPT-4o-mini at temperature 0. {gold} is their released natural-language gold answer for the question, not the raw tool returns, matching the reference their own response_generation.py supplies; {pred} is the text after the final Answer: marker in the rollout. Pass Rate is the fraction of Yes verdicts over all 200 test questions, with a missing or empty answer counted as No.

<think> We need to retrieve the PubChem Compound Identifier for ’(5-Bromo-2-methoxyphenyl)methanol’, so we call get_compound_CID with the compound name. </think> <tool_call> {"name": "chemistrytools/get_compound_CID", "arguments": {"compound": "(5-Bromo-2methoxyphenyl)methanol"}} </tool_call> <tool_result> from server 612298 </tool_result> <think> With the PubChem CID (612298) in hand, we call get_compound_MolecularWeight_by_CID to obtain its molecular weight. </think> <tool_call> {"name": "chemistrytools/get_compound_ MolecularWeight_by_CID", "arguments": {"compound_CID": 612298}} </tool_call> <tool_result> from server 217.06 </tool_result> Answer: The molecular weight of ’(5-Bromo-2methoxyphenyl)methanol’ is 217.06 g/mol.

A self-repair rollout

Figure 3 is the unedited training rollout discussed in §7.3, in which the model reads an execution error and re-issues the failed call corrected. Gold chains contain only calls that succeed, so this behaviour cannot have been imitated from supervision.

Figure 2: A complete, unedited rollout from our Qwen2.5-7B SFT→RL model. Shaded spans are returned by the tool server and masked from the loss; everything else is generated by the policy. The CID 612298 appears nowhere in the query and cannot be guessed, so the second call can only be written by reading it back out of the first call’s result. This rollout matches the gold chain exactly.

Query Please retrieve the Canonical SMILES representation for the chemical compound with PubChem Compound Identifier (CID) 258. Once you have the SMILES, calculate the log of the partition coefficient (LogP) and the Quantitative Estimate of Druglikeness (QED) for the compound. <tool_call> {"name": "chemistrytools/convert_compound_ CID_to_SMILES", "arguments": {"compound CID": 258}} </tool_call> <tool_result> from server Error: argument binding failed for chemistrytools/convert_compound_CID_to_ SMILES: got an unexpected keyword argument ’compound CID’ </tool_result> % self-repair: correct the argument name <tool_call> {"name": "chemistrytools/convert_compound_ CID_to_SMILES", "arguments": {"compound_CID": 258}} </tool_call> <tool_result> CN(CCC(CC(=O)NC1C=CC(OC1C(=O)O)N2C=CC (=NC2=O)N)N)C(=N)N </tool_result> % this value is the argument of both calls below <tool_call> {"name": "cactus/CalculateLogP", "arguments": {"compound_SMILES": <SMILES>}} </tool_call> <tool_result> -2.2187299999999963 </tool_result> <tool_call> {"name": "cactus/CalculateQED", "arguments": {"compound_SMILES": <SMILES>}} </tool_call> <tool_result> 0.150577431129479 </tool_result> Answer: The Canonical SMILES representation for the chemical compound with PubChem Compound Identifier (CID) 258 is CN(CCC(CC (=O)NC1C=CC(OC1C(=O)O)N2C=CC(=NC2=O)N)N) C(=N)N, its log of the partition coefficient (LogP) is -2.2187, and the Quantitative Estimate of Druglikeness (QED) is 0.1506.

Figure 3: Self-repair from execution feedback, in an unedited Qwen-2.5-7B RL-only training rollout; shaded spans are returned by the server. The model writes an argument name with a space, reads the binding error, and re-issues the call corrected. That recovered SMILES is the argument of both later calls, so one uncorrected error would have cost two more values. <SMILES> abbreviates the string, which the rollout repeats in full.

One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning

Analyze & explore related research in Conceptio

Related documents