Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents

Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents Haobin Li1 Zhenyu Huang1

Mouxing Yang1

1Sichuan University

arXiv:2609.38885v1 [cs.SE] 30 Sep 2026

Liang Jiang2 Xi Peng1

2Independent Researchers

Recently, coding agents have emerged as a dominant paradigm for real-world software engineering (SWE) scenarios, which solve complex tasks through multi-turn interactions with development environments. However, frequent interactions with environments would inevitably introduce substantial token overhead, leading to high usage costs and latency. Although recent studies have explored reducing token usage by context manipulation and interaction limits at inference time, these approaches focus on improving token efficiency while overlooking the risk of discarding task-relevant information, thus struggling to balance the trade-off between resolution rate and token efficiency. In this paper, we study a more general paradigm without suffering from the limitation, i.e., training token-efficient coding agents with promising resolution performance, which is a highly-practical yet less-explored problem. To this end, we reveal two core observations in SWE scenarios: i) Efficiency Variation: successful resolution could be achieved with fewer tokens; ii) Entropy Correlation: unproductive behaviors are associated with turn-level entropy. Motivated by observations, we propose a novel reinforcement learning framework, dubbed HERO. Specifically, HERO prioritizes task resolution over token efficiency during policy optimization and encourages efficient reasoning patterns at both trajectory and turn levels. Extensive experiments on SWE-bench Verified and SWE-bench Multilingual demonstrate that HERO achieves a favorable trade-off between resolution rate and token efficiency compared with state-of-the-art coding agents and reinforcement learning methods. GitHub

1

Hugging Face

I NTRODUCTION

Recently, coding agents (Yang et al., 2024; Zhang et al., 2024; Huang et al., 2023) powered by large language models (LLMs) (Chang et al., 2024; Minaee et al., 2024) have emerged as a dominant paradigm for real-world software engineering (SWE) (He et al., 2025; Jin et al., 2024; Li et al., 2026a) scenarios, demonstrating strong performance across various complex tasks, including but not limited to repository-level bug fixing, feature implementation, and end-to-end software development. Specifically, coding agents first interact with environments over multiple turns through different tools, e.g., search, editing, and command-line tools. By incorporating tool feedback from environments, coding agents could continually accumulate task-relevant information and thus facilitate the resolution of complex SWE tasks. Despite the promising performance of existing coding agents, the frequent interactions with environments would inevitably introduce substantial token overhead, leading to high usage costs and latency (Fan et al., 2025). To remedy this, recent studies have explored pruning (Wang et al., 2026c) or compressing (Liu et al., 2026a; Lodha et al., 2026) the input context and limiting interaction turns (Gao & Peng, 2025) at inference time. However, such inference-time approaches focus on improving token efficiency while overlooking the risk of discarding task-relevant information, thus struggling to balance the trade-off between resolution performance and token efficiency. Different from the above inference-time designs, an alternative paradigm is to internalize both effective and Emails: {haobinli.gm, zyhuang.gm, yangmouxing, pengx.gm}@gmail.com

1

Token Usage Unit

Resolved

F

T er ew

More

Coding Agent

ok

s en

ns Toke

Unresolved

Waste Tokens

(a) Motivations

(b) Observations

Figure 1: (a) Motivations: for a given SWE task, the coding agent could generate multiple reasoning trajectories with different token usages and resolution outcomes. Ideally, the goal of token-efficient coding agents is to achieve resolution with fewer tokens. (b) Observations: we compare token usage among successful trajectories within each task and analyze the correlation between turn-level entropy and unproductive behaviors. Note that token usage is normalized by the per-task mean over successful trajectories. As a result, one could observe that i) successful trajectories exhibit substantial variation in token usage, and ii) low-entropy turns are more prone to reasoning issues, i.e., repeated tool calls, excessive exploration, and erroneous commands. Together, these observations provide guidance for learning effective and efficient reasoning patterns.

efficient reasoning patterns into the agent policy through training-time designs, thereby reducing token usage without sacrificing the solution capability. To learn effective and efficient reasoning patterns, the most promising solution might be tokenefficient reinforcement learning (RL) (Arora & Zanette, 2026; Liu et al., 2026b; Xie et al., 2026), which encourages task success and penalizes token-intensive trajectories during policy optimization. However, existing methods are specifically designed for non-interactive reasoning or simple agentic tasks, which are inadequate for long-horizon SWE tasks due to the following two reasons. On the one hand, it remains under-explored whether coding agents could achieve a desirable trade-off between task success and token efficiency. On the other hand, simply penalizing token-intensive trajectories would discourage all interaction turns, including necessary reasoning steps, thereby hindering the acquisition of task-relevant information. To specifically develop a token-efficient RL method for training coding agents, we present two core observations as illustrated in Fig. 1. To be specific, i) Efficiency Variation: for a given task, it is possible to achieve successful resolution through a more efficient reasoning path with fewer tokens, paving the way to develop token-efficient coding agents; ii) Entropy Correlation: unproductive turns could be identified based on the turn-level entropy, providing a promising avenue to learn fine-grained efficient patterns through penalizing inefficient turns. Based on the above observations, we develop token-efficient coding agents by learning efficient reasoning patterns while preserving resolution capability. To this end, we propose a novel training framework, dubbed HiErarchical ReinfOrcement learning (HERO), which couples a hierarchical policy optimization framework with multi-granularity credit assignment. In brief, HERO consistently prioritizes task resolution over token efficiency and activates efficiency optimization once sufficient problem-solving capability is acquired. Thanks to such a hierarchical optimization objective, coding agents could achieve a favorable trade-off between resolution rate and token efficiency. Moreover, HERO adopts coarse-grained trajectory credit to encourage concise reasoning paths and fine-grained turn credit to penalize unproductive turns, thereby learning efficient reasoning patterns at multiple granularities. Experiments demonstrate that, using only 640 training tasks, HERO improves the average resolution rate from 37.4% to 42.2% across Qwen3.5 variants on SWE-bench series. Compared with GRPO, HERO yields better average improvement in resolution rate with 39.8% less token usage. In summary, the major contributions and novelties of this work are given as follows. 2

• Different from existing inference-time approaches, we aim to internalize both effective and efficient reasoning patterns into the agent policy through training-time designs, thus achieving token-efficient coding agents without sacrificing resolution performance. • We reveal two core observations regarding token-efficient coding agents in SWE scenarios, which not only demonstrate the potential to achieve a good trade-off between task success and token efficiency but also pave an avenue toward better efficiency through fine-grained optimization. • We propose a novel method termed HERO, which couples resolution-prioritized policy optimization with multi-granularity credit assignment at trajectory and turn levels, thereby improving token efficiency while preserving resolution performance. • Extensive experiments on SWE-bench Verified and SWE-bench Multilingual demonstrate that the proposed HERO achieves a favorable trade-off between resolution rate and token efficiency, significantly outperforming existing state-of-the-art coding agents and reinforcement learning methods.

2

R ELATED W ORK

In this section, we provide a brief review of two topics highly related to this work, including coding agents and token-efficient reinforcement learning. 2.1

C ODING AGENTS

Coding agents aim to solve real-world software engineering tasks by equipping LLMs with various tools to interact with environments. Based on how LLMs are endowed with tool-use capabilities, existing approaches could be broadly grouped into two categories: i) scaffold-based methods (Yang et al., 2024; Wang et al., 2025b; Xia et al., 2024), which design general-purpose tool commands and interaction workflows that could be applied to most widely-used LLMs; ii) training-based methods (Pan et al., 2024; Jain et al., 2025; Song et al., 2026), which enhance the ability to use the specific agent scaffold through supervised fine-tuning or reinforcement learning. Despite their promising performance, existing coding agents focus on improving resolution rate for complex SWE tasks, while paying limited attention to the substantial token usage incurred by longhorizon interactions. Different from them, this work aims to train token-efficient coding agents that simultaneously account for reasoning efficiency and resolution performance. 2.2

T OKEN -E FFICIENT R EINFORCEMENT L EARNING

Token-efficient reinforcement learning aims to reduce the token usage of LLMs through policy optimization. According to the application scenarios, existing approaches could be broadly divided into two categories: i) reasoning-oriented methods (Arora & Zanette, 2026; Shrivastava et al., 2026; Liu et al., 2026b), which encourage concise chain-of-thought reasoning by adopting explicit length constraints or length-based rollout filtering; ii) agent-oriented methods (Wang et al., 2025a; Xie et al., 2026), which penalize excessive cumulative token usage over multiple turns and are primarily adopted in simple agentic tasks, e.g., web search and multi-hop question answering. The major differences between existing token-efficient RL methods and this work are given below. On the one hand, existing methods overlook the non-negligible unproductive turns in long-horizon SWE tasks, whereas our approach could identify and discourage unproductive behaviors and thus facilitate token efficiency in multiple interactions. On the other hand, to the best of our knowledge, our approach is one of the first works to explore simultaneously preserving resolution rate and improving token efficiency in training coding agents.

3

M ETHOD

In this section, we introduce HiErarchical ReinfOrcement learning (HERO) for training efficient coding agents. In Section 3.1, we present the goal of training token-efficient coding agents. In Section3.2, we propose hierarchical policy optimization to preserve resolution performance during 3

efficiency optimization. In Section 3.3, we introduce multi-granularity credit assignment to learn efficient reasoning patterns at multiple granularities. 3.1

P ROBLEM F ORMULATION

Let xi = (ri , si ) denote a repository-level task in SWE scenarios, where ri is the repository environment and si is the corresponding issue report. For a given ri with si , the goal of coding agent A is to interact with environment ri for ni turns and then generate solution A(ri , si ) for si . To verify the effectiveness of the generated solution, we adopt the test-driven evaluation protocol and the corresponding resolution outcome could be derived as, yi = Pi (A(ri , si )) ∈ {0, 1},

(1)

where Pi (·) indicates whether the generated solution A(ri , si ) passes all task-specific tests. As discussed in the Introduction, beyond resolution performance, we further consider the cumulative token usage during multi-turn interactions. For a given task xi , the cumulative token usage is defined as, ni X ti,j , (2) ci = j=1 out in out where ti,j = tin i,j + ti,j denotes the token usage at the j-th turn, with ti,j and ti,j representing the

input and output tokens, respectively. With the above formulation, we define the objective of token-efficient coding agents as maximizing the Token-efficient Resolution Score (TRS), i.e., max TRS, θ

TRS = √

R , 1+C

(3)

PN PN 1 where R = 100 i=1 yi and C = N i=1 ci denote the average resolution rate and cumulative N token usage over N tasks, respectively. Note that C is measured in millions of tokens. Clearly, a higher TRS generally indicates a more favorable trade-off between resolution rate and token usage. However, existing coding agents are typically designed to improve resolution performance, while overlooking the substantial token overhead incurred by multi-turn interactions. As a remedy, we propose HiErarchical ReinfOrcement learning (HERO) as illustrated in Fig. 2, which couples hierarchical policy optimization over resolution and efficiency objectives with the multi-granularity efficiency credit assignment mechanism. In the following, we will elaborate on each of them. 3.2

H IERARCHICAL P OLICY O PTIMIZATION

As discussed in Introduction, existing token-efficient RL methods struggle to achieve a favorable trade-off between resolution rate and token efficiency. To address this, we propose a novel hierarchical policy optimization (HPO) mechanism that improves resolution performance while reducing token usage, i.e., h  i bi , clip (ρ (θ) , 1 − ϵ, 1 + ϵ) A bi , L(θ) = −Ei∈G min ρ (θ) A (4) eff bi = Γ(Ai ), A Ai = Ares i + wAi , where G denotes the rollout group for the same task, ρ(θ) denotes the importance ratio between the current and rollout policies, ϵ indicates the clipping threshold. In particular, Γ(·) denotes the eff resolution-first clipping strategy, Ares i and Ai represent the resolution and efficiency advantages of the i-th rollout in G, respectively. In the implementation, we derive Ares i using the standard grouprelative advantage estimator in GRPO (Shao et al., 2024), with the binary resolution outcome 0 or 1 as the reward. More specifically, we adopt an efficiency gating mechanism to balance the trade-off between resolution performance and token efficiency, i.e.,   1 X p−τ , 0, 1 , p = yi . (5) w = λ clip 1−τ |G| i∈G

4

Coding Agent

Turn

Low Entropy

High Entropy

Problem Issue

Traj. Token Usage

Resolved

Penalize

Hierarchical Policy Optimization

Coarse-grained Trajectory Credit

Step 1: 𝐴 = 𝐴!"# + 𝑤 𝐴"$$

Data leak in GBDT due to warm start...

Unresolved Trajectories

Token Usage

Token Usage

...

...

Multi-turn

Trajectories

Fine-grained Turn Credit Entropy Estimation Tokens

Encourage

Multi-granularity Efficiency Credit Assignment Resolved Trajectories

Codebase

Unresolved

𝒉𝒋

... 𝒉𝒌

Thre. Turn Entropy

Low Entropy

Resolved Rate

𝜏

0

1

Step 2: 𝐴" = Γ ( 𝐴) Resolution-first Clipping

Rank Turns across Trajectories

Mean

Efficiency Gating

Weight

𝜆

High Entropy

Clipped Adv.

Unresolved

Resolved

Initial Adv.

Figure 2: Overview of our method HERO. Given a swe task, the coding agent first samples multiple trajectories to form a rollout group and then computes multi-granularity advantages at trajectory and turn levels. Specifically, HERO encourages concise reasoning trajectories with the same outcome, while penalizing potentially unproductive behaviors with low turn entropy. After that, HERO adaptively activates efficiency optimization based on the resolution rate within the group and adopts resolution-first clipping to guarantee that resolved trajectories receive advantages no lower than unresolved ones. where λ controls the strength of efficiency optimization, p indicates the average resolution rate within the group G, and τ denotes the threshold that determines whether efficiency optimization is activated. Such a mechanism encourages coding agents to learn efficient reasoning patterns only when they demonstrate sufficient capability to resolve the given task, thereby prioritizing task resolution before optimizing token efficiency. Although the efficiency gating mechanism poses constraints on efficiency optimization, the efficiency signal might override the resolution signal, causing unresolved trajectories to receive higher advantages than resolved ones. To avoid this, we employ the resolution-first clipping strategy as follows,  max(Ai , 0), yi = 1 Γ(Ai ) = . (6) min(Ai , 0), yi = 0 Thanks to the resolution-first clipping strategy, advantages are non-negative for resolved trajectories and non-positive for unresolved ones, thereby further preserving the priority of task resolution during policy optimization. 3.3

M ULTI - GRANULARITY E FFICIENCY C REDIT A SSIGNMENT

To prevent inefficient behaviors from incurring substantial token overhead, we introduce a multigranularity credit assignment mechanism, which assigns efficiency credit at both the coarse-grained trajectory level and the fine-grained turn level. Specifically, the efficiency advantage is derived as follows, coarse Aeff + Afine (7) i,j = Ai i,j , where Acoarse denotes the coarse-grained advantage of the i-th rollout, Afine i i,j indicates the finegrained advantage of j-th turn in the corresponding rollout. In the following, we will elaborate on the two advantages in detail. 3.3.1

C OARSE - GRAINED T RAJECTORY A DVANTAGE

To facilitate the learning of efficient reasoning patterns, a straightforward approach is to penalize token-intensive trajectories while encouraging concise ones. However, recent studies (Fan et al., 2025; Majgaonkar et al., 2025) have demonstrated that unresolved trajectories typically consume more tokens than resolved ones. As a result, simply rewarding trajectories with lower token usage might conflate token efficiency with resolution rate rather than encouraging more efficient reasoning patterns. 5

To remedy this, we adopt the resolution-calibrated credit assignment strategy as follows, ui − ū Acoarse = , i σu (8)    c i yi − 1, 0 , B = Median {ck | k ∈ G, yk = yi } , ui = 1 − tanh max B yi where ui denotes the efficiency reward for the i-th rollout, Median{·} indicates the median token usage among rollouts in G with the same resolution outcome, and ū and σu represent the mean and standard deviation of the efficiency rewards within G, respectively. Intuitively, efficiency rewards are calibrated within each outcome subset, thus avoiding the conflation of token efficiency with resolution performance. After that, the calibrated rewards are standardized to obtain trajectorylevel advantages, facilitating the learning of efficient reasoning patterns under the same resolution outcome. 3.3.2

F INE - GRAINED T URN A DVANTAGE

Although the proposed coarse-grained advantage could reduce token usage to some extent, the trajectory-level signal treats all interaction turns equally, which overlooks that the unproductive turns are the primary source of substantial token overhead. As discussed in the Introduction, we reveal that low-entropy turns are more likely to exhibit unproductive behaviors. Motivated by this, we propose the following entropy-aware credit assignment mechanism to avoid inefficient behaviors, i.e.,    − tanh 1 − hi,j , h < q i,j Afine , q i,j =  (9) 0, hi,j ≥ q q = Percentile0.2 ({hi,j | i ∈ G, j ∈ {1, . . . , ni }}) , where hi,j denotes the average token entropy at the j-th turn of the i-th rollout. Following Wang et al. (2026a), Percentile0.2 (·) returns the entropy threshold corresponding to the lowest 0.2 fraction of turns within the group. Such a design would penalize low-entropy turns potentially associated with inefficient behaviors, thereby reducing the token overhead incurred by unproductive interactions.

4

E XPERIMENTS

In this section, we conduct extensive experiments on widely-used SWE benchmarks to verify the effectiveness of the proposed HERO. Due to space limitations, we present more experiments in Appendix B. 4.1

E XPERIMENT C ONFIGURATIONS

Benchmarks. To comprehensively evaluate the effectiveness of HERO, we evaluate our method on the most widely-used benchmarks in SWE scenarios, i.e., SWE-bench Verified and SWE-bench Multilingual (Jimenez et al., 2024). Specifically, SWE-bench Verified contains 500 high-quality issue-solving tasks constructed from real-world GitHub issues, covering bug fixing, feature implementation, code refactoring, and so on. While SWE-bench Multilingual extends the evaluation beyond Python, comprising 300 SWE tasks across nine programming languages. Implementation Details. We evaluate our proposed HERO using the widely-used SOTA model Qwen3.5 (Yang et al., 2025) and conduct comprehensive experiments on three variants with different model sizes, i.e., Qwen3.5-4B, Qwen3.5-9B, and Qwen3.5-35B-A3B. For a fair comparison, we adopt the most widely-acknowledged scaffold, i.e., Claude Code, as the unified framework for evaluating all coding agents and RL methods. As for comparisons between RL methods, we use the same training set of 640 multilingual SWE tasks sampled from SWE-Gym (Pan et al., 2024), MultiSWE-bench (Zan et al., 2026), and SWE-rebench (Badertdinov et al., 2026). See Appendix A.1 for more details. Regarding hyperparameter settings, the trade-off coefficient λ and threshold τ in Eq. 5 are set to 0.3 and 0.6, respectively. We repeat all experiments three times with different random seeds and report the mean results. 6

Table 1: Comparisons with state-of-the-art methods on SWE-bench Verified benchmark. Token usage is reported in millions (M). Method

Resolve Rate (%)

Token Usage (M)

TRS

2.3 2.4 4.6 2.4

19.5 23.4 18.0 26.2

Open-Source Coding Agents Gemma-4-26B-A4B Qwen3-Coder-30B-A3B-Instruct GLM-4.7-Flash-30B KAT-Dev-32B

35.4 43.2 42.6 48.4

Reinforcement Learning Qwen3.5-4B + GRPO + Our

32.8 37.4 40.0

4.5 6.8 4.5

14.0 13.4 17.1

Qwen3.5-9B + GRPO + Our

45.0 47.0 49.2

4.5 5.9 3.5

19.2 17.9 23.2

Qwen3.5-35B-A3B + GRPO + Our

57.2 59.2 60.6

2.9 5.2 2.7

29.0 23.8 31.5

Table 2: Comparisons with state-of-the-art methods on SWE-bench Multilingual benchmark. Method

Resolve Rate (%)

Token Usage (M)

TRS

3.5 2.1 4.0 2.3

12.9 11.7 11.3 15.2

Open-Source Coding Agents Gemma-4-26B-A4B Qwen3-Coder-30B-A3B-Instruct GLM-4.7-Flash-30B KAT-Dev-32B

27.3 20.6 25.3 27.6

Reinforcement Learning

4.2

Qwen3.5-4B + GRPO + Our

20.0 22.0 24.3

4.0 7.6 4.5

8.9 7.5 10.4

Qwen3.5-9B + GRPO + Our

29.3 31.0 34.0

4.3 6.0 4.1

12.7 11.7 15.1

Qwen3.5-35B-A3B + GRPO + Our

40.3 43.0 45.0

3.2 5.4 2.9

19.7 17.0 22.8

C OMPARISONS WITH S TATE - OF - THE -A RT M ETHODS

We compare our method with the widely-used reinforcement learning method, i.e., GRPO (Shao et al., 2024), using Qwen3.5 backbones of various model sizes. For more results on state-of-the-art token-efficient reinforcement learning methods, please refer to Appendix B.1. Besides, we compare against four state-of-the-art coding agents, i.e., Gemma-4-26B-A4B (Team et al., 2026), Qwen37

Figure 3: Dynamics of the training process.

Figure 4: Token usage distributions of resolved and unresolved trajectories.

Figure 5: Issue rate of unproductive behaviors at turn level.

Coder-30B-A3B-Instruct (Yang et al., 2025), GLM-4.7-Flash-30B (Zeng et al., 2026), and KATDev-32B (Zhan et al., 2025). From the results in Table 1-2, one could have the following observations and conclusions: i) stateof-the-art coding agents with strong resolution performance do not necessarily exhibit high token efficiency; ii) GRPO consistently improves resolution rates at the cost of substantially increased token usage, suggesting that GRPO could enhance issue-solving capabilities without necessarily encouraging efficient reasoning patterns; iii) in contrast, the proposed HERO prioritizes task resolution over token efficiency while reducing the token overhead incurred by unproductive interactions, achieving desirable resolution rates and token efficiency; iv) thanks to the proposed HERO, even small models such as Qwen3.5-4B and Qwen3.5-9B achieve competitive resolution performance against larger open-source SOTA coding agents. 4.3

A BLATION AND A NALYTIC S TUDY

In this section, we carry out a series of ablation studies and analytic experiments to investigate the effectiveness of HERO. Unless otherwise stated, experiments are conducted using Qwen3.5-4B on the SWE-bench Verified benchmark. Analytic Study on Training Dynamics. To investigate how HERO learns efficient reasoning patterns and improves resolution performance, we carry out an analytic study on the dynamics throughout the training process, including resolved rate, token usage, and TRS. As demonstrated in Fig. 3, HERO outperforms GRPO across the three metrics as training progresses, with more pronounced gains in later stages. Besides, we observe that token usage under HERO first rises until step 40 and then declines, whereas the resolution rate reaches a high level by this stage and remains consistently high thereafter. In other words, such a phenomenon suggests that HERO first develops issue-solving capabilities in the early stages and subsequently reduces token usage while preserving resolution performance in the later stage, thereby achieving a favorable trade-off between the two objectives. Analytic Study on Token Usage. To verify the effectiveness of HERO in improving token efficiency, we compare token usage distributions for resolved and unresolved trajectories under GRPO and the proposed HERO. From the results in Fig. 4, HERO achieves lower token usage for both resolved and unresolved trajectories compared with GRPO, suggesting that our method could learn efficient reasoning patterns and thus achieve resolution with fewer tokens. Analytic Study on Unproductive Turns. As pointed out in Introduction, low-entropy turns are more prone to inefficient behaviors. To verify the effectiveness of HERO in suppressing unproduc8

Table 3: Ablation studies on the designs in HERO, where “w/o HPO” denotes removing the hierarchical policy optimization. Variants

Resolve Rate

Token Usage

TRS

GRPO w/o HPO w/o Coarse w/o Fine

37.4 29.2 39.0 37.8

6.8 2.1 6.0 4.9

13.4 16.6 14.7 15.6

Default

40.0

4.5

17.1

Vanilla

Figure 6: Parameter analysis of efficiency coefficient λ in Eq. 5.

GRPO

Ours

Figure 7: Tool usage across various interaction turns.

tive turns, we analyze the frequencies of unproductive behaviors across interaction turns. As shown in Fig. 5, HERO achieves lower rates of repeated tool calls, overexploration, and tool errors than both the base model and GRPO, suggesting that penalizing low-entropy turns would discourage unproductive behaviors. Ablation Studies. To verify the importance of each design, we investigate the variants of HERO in Table 3, from which one could draw the following conclusions. On the one hand, removing hierarchical policy optimization would reduce token usage but degrade resolution performance, highlighting its role in preserving task resolution during efficiency optimization. On the other hand, both the coarse-grained and fine-grained efficiency advantages contribute to achieving lower token usage. Besides, we carry out parameter analysis of the efficiency coefficient λ in Eq. 5. As depicted in Fig. 6, HERO achieves stable performance improvement with low token consumption when λ ∈ [0.2, 0.4]. See Appendix B.8-B.9 for ablation studies on more parameters. Analytic Study on Reasoning Patterns. To further verify that HERO learns both effective and efficient reasoning patterns, we conduct an analytical study on the tool usage across interaction turns. From the results in Fig. 7, we observe that HERO shifts from inspection toward execution in later turns, mirroring the reasoning progression of GRPO while requiring fewer tokens. In other words, HERO internalizes more efficient reasoning patterns into the agent policy without undermining the necessary progression from code exploration to solution generation.

5

C ONCLUSION

In this paper, we study a highly-practical yet under-explored problem, i.e., training token-efficient coding agents while preserving resolution performance. To solve this problem, we propose a novel RL framework, i.e., HERO, which achieves efficient reasoning patterns through multi-granularity credit assignment and prioritizes task resolution over token efficiency through hierarchical policy optimization. Extensive experiments on SWE-bench series demonstrate that HERO significantly outperforms existing methods and achieves a favorable trade-off between resolution rate and token efficiency. In the future, we plan to extend HERO to more complex long-horizon agentic tasks, such as scientific data analysis and engineering design. 9

R EFERENCES Daman Arora and Andrea Zanette. Training language models to reason efficiently. In NeurIPS, 2026. Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents. In NeurIPS, 2026. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 2024. Zhiyu Fan, Kirill Vasilevski, Dayi Lin, Boyuan Chen, Yihao Chen, Zhiqing Zhong, Jie M Zhang, Pinjia He, and Ahmed E Hassan. Swe-effi: Re-evaluating software ai agent system effectiveness under resource constraints. arXiv preprint arXiv:2509.09853, 2025. Pengfei Gao and Chao Peng. More with less: An empirical study of turn-control strategies for efficient coding agents. arXiv preprint arXiv:2510.16786, 2025. Junda He, Christoph Treude, and David Lo. Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology, 2025. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2023. Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents. arXiv preprint arXiv:2504.07164, 2025. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In ICLR, 2024. Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. From llms to llmbased agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479, 2024. Haobin Li, Ping Deng, Weizhong Qian, Liang Jiang, Zhenyu Huang, Mouxing Yang, and Xi Peng. Active-swe: Benchmarking coding agents for proactive bug fixing without issue reports. arXiv preprint arXiv:2608.04682, 2026a. Haobin Li, Yutong Yang, Yijie Lin, Xiang Dai, Mouxing Yang, and Xi Peng. Reliable thinking with images. arXiv preprint arXiv:2602.12916, 2026b. Tobias Lindenbauer, Igor Slinko, Ludwig Felder, Egor Bogomolov, and Yaroslav Zharov. The complexity trap: Simple observation masking is as efficient as llm summarization for agent context management. arXiv preprint arXiv:2508.21433, 2025. Jiacheng Liu, Xiaohan Zhao, Xinyi Shang, and Zhiqiang Shen. Dive into claude code: The design space of today’s and future ai agent systems. arXiv preprint arXiv:2604.14228, 2026a. Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. Learn to reason efficiently with adaptive length-based reward shaping. In ICLR, 2026b. Abhilasha Lodha, Mahsa Pahlavikhah Varnosfaderani, Abir Chakraborty, and Abhinav Mithal. Less context, better agents: Efficient context engineering for long-horizon tool-using llm agents. arXiv preprint arXiv:2606.10209, 2026. 10

Oorja Majgaonkar, Zhiwei Fei, Xiang Li, Federica Sarro, and He Ye. Understanding code agent behaviour: An empirical study of success and failure trajectories. arXiv preprint arXiv:2511.00197, 2025. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Vaishnavi Shrivastava, Ahmed H Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning. In ICLR, 2026. Huatong Song, Lisheng Huang, Shuang Sun, Jinhao Jiang, Ran Le, Daixuan Cheng, Guoxin Chen, Yiwen Hu, Zongchao Chen, Yiming Jia, et al. Swe-master: Unleashing the potential of software engineering agents via post-training. arXiv preprint arXiv:2602.03411, 2026. Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026. Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Otc: Optimal tool calls via reinforcement learning. arXiv preprint arXiv:2504.14870, 2025a. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. In NeurIPS, 2026a. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In ICLR, 2025b. Yuhang Wang, Yuling Shi, Mo Yang, Rongrui Zhang, Shilin He, Heng Lian, Yuting Chen, Siyu Ye, Kai Cai, and Xiaodong Gu. Swe-pruner: Self-adaptive context pruning for coding agents. arXiv preprint arXiv:2601.16746, 2026b. Yuhang Wang, Yuling Shi, Shaoqiu Zhang, Jialiang Liang, Shilin He, Siyu Ye, Yuting Chen, Kai Cai, and Xiaodong Gu. Swe-pruner pro: The coder llm already knows what to prune. arXiv preprint arXiv:2607.18213, 2026c. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024. Zequn Xie, Junjie Wang, Dan Yang, Jie Feng, Yue Shen, Jian Wang, and Jinjie Gu. Slimsearcher: Training efficiency-aware web agents via adaptive reward gating. arXiv preprint arXiv:2606.07074, 2026. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In NeurIPS, 2024. 11

Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Li Aoyan, Lu Chen, Xiaojian Zhong, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. In NeurIPS, 2026. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Zizheng Zhan, Ken Deng, Jinghui Wang, Xiaojiang Zhang, Huaixi Tang, Minglei Zhang, Zhiyi Lai, Haoyang Huang, Wen Xiang, Kun Wu, et al. Kat-coder technical report. arXiv preprint arXiv:2510.18779, 2025. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In ACL, 2024.

12

A PPENDIX A More Implementation Details

14

A.1 Training Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

A.2 More Implementation Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

A.3 Entropy Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

A.4 Identification of Unproductive Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

B Additional Experimental Results

16

B.1 Comparisons with Token-Efficient Reinforcement Learning Methods . . . . . . . . . . . . .

16

B.2 Comparisons with Inference-Time Methods . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

B.3 Analytic Study on Prefix-Cache Reuse and Inference Cost . . . . . . . . . . . . . . . . . . .

17

B.4 Ablation Studies on SWE-bench Multilingual . . . . . . . . . . . . . . . . . . . . . . . . . .

17

B.5 Analytic Study on Token Usage Breakdown . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

B.6 Analytic Study on Scaffold Generalization . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

B.7 More Detailed Ablation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

B.8 Sensitivity to the Capability Threshold . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

B.9 Sensitivity to the Entropy Percentile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

B.10 Analytic Study on Fine-grained Turn Advantage . . . . . . . . . . . . . . . . . . . . . . . .

20

13

A

M ORE I MPLEMENTATION D ETAILS

In this section, we provide more implementation details about the training data, implementation settings, entropy estimation, and identification of unproductive behaviors. A.1

T RAINING DATA

As mentioned in Section 4, we employ 640 multilingual SWE tasks for training. Specifically, the training set contains 352 Python tasks, 38 Java tasks, and 50 tasks each in C/C++, JavaScript/TypeScript, PHP, Go, and Rust. Such a collection covers SWE tasks across multiple programming languages, endowing the training of coding agents with multilingual reasoning capabilities. A.2

M ORE I MPLEMENTATION S ETTINGS

We provide additional implementation details for training and evaluation. Specifically, the default tool configuration disables WebFetch, WebSearch, and Agent, while retaining all other scaffoldprovided tools. For training, each training rollout batch contains 8 tasks with 16 sampled trajectories per task, yielding 128 trajectories in total. Moreover, in Eq. 4, the derived turn-level advantage is assigned to every generated token within the turn. Note that, given the stronger issue-solving capability of Qwen3.5-35B-A3B, the trade-off coefficient λ and threshold τ in Eq. 5 are set to 0.1 and 0.8, respectively. For evaluation, we use the checkpoint from the final training iteration. Together, the default settings are summarized in Table 4. Table 4: Default implementation settings.

A.3

Category

Parameter

Setting

Agent Scaffold

Claude Code version

2.1.111

Training

Tasks per rollout batch Rollout group size Trajectories per rollout batch

8 16 128

Evaluation

Sampling temperature Top-p Maximum interaction turns Context length

0.6 0.95 200 131,072 tokens

E NTROPY E STIMATION

We provide additional details on the estimation of turn-level entropy. Following Holtzman et al. (2019); Li et al. (2026b), we estimate token-level entropy using the top-10 token probabilities and derive turn-level entropy as 1 X hi,j = Hk , (10) |Ti,j | k∈Ti,j

where Ti,j denotes the set of generated tokens in the j-th turn of rollout i, with Hk denotes the entropy of the k-th generated token. A.4

I DENTIFICATION OF U NPRODUCTIVE B EHAVIORS

In the manuscript, we analyze the association between turn-level entropy and unproductive behaviors in Fig. 1. Here, we provide the definition for repeated tool calls, excessive exploration, and erroneous commands in Table 5. Note that each behavior is counted at most once per turn. 14

Table 5: Identification rules for unproductive behaviors. Behavior

Identification Rule

Repetition

Repeats the same sequence of tool calls and arguments as the preceding turn.

Overexploration

Uses only Glob, Grep, Read, or NotebookRead for at least 15 consecutive turns.

Tool Error

Tool responses contain explicit tool-error markers, excluding failed tests or nonzero exit codes alone.

Any Issue

Exhibits at least one of the above behaviors.

15

B

A DDITIONAL E XPERIMENTAL R ESULTS

In this section, we present more experimental results of the proposed HERO. Unless otherwise specified, all experiments are conducted using Qwen3.5-4B. B.1

C OMPARISONS WITH T OKEN -E FFICIENT R EINFORCEMENT L EARNING M ETHODS

As discussed in Introduction, existing token-efficient RL methods are specifically designed for noninteractive reasoning or simple agentic tasks, which struggle to achieve a favorable trade-off between resolution rate and token efficiency in complex SWE scenarios. We compare HERO with four stateof-the-art token-efficient RL methods, i.e., Training Efficient (Arora & Zanette, 2026), LASER (Liu et al., 2026b), OTC-GRPO (Wang et al., 2025a), and SlimSearcher (Xie et al., 2026), on SWE-bench Verified and SWE-bench Multilingual. For fair comparison, we adopt the same training and evaluation settings across methods, except for method-specific configurations. Specifically, for Training Efficient, OTC-GRPO, and SlimSearcher, we follow the configurations reported in their respective papers. While for LASER, we use the fixed-threshold variant with a threshold of 8,192 total output tokens across interaction turns, estimated based on the generation lengths typical of SWE tasks. From the results in Table 6, one could have the following observations and conclusions. On the one hand, HERO achieves higher resolution rates with lower token usage than Training Efficient and SlimSearcher on both benchmarks. On the other hand, although LASER and OTC-GRPO consume fewer tokens, these token savings come at the cost of degrading resolution rates, highlighting the difficulty of balancing resolution performance and token efficiency for existing token-efficient RL methods. Overall, HERO significantly outperforms existing token-efficient RL methods in the TRS metric, demonstrating that HERO achieves a more favorable trade-off between the two objectives. Table 6: Comparisons with token-efficient RL methods on SWE-bench Verified and SWE-bench Multilingual. SWE-bench Verified Method Base GRPO Training Efficient LASER OTC-GRPO SlimSearcher Ours

B.2

SWE-bench Multilingual

Resolved Rate (%)

Tokens (M)

TRS

Resolved Rate (%)

Tokens (M)

TRS

32.8 37.4 37.4 33.2 32.4 37.6 40.0

4.5 6.8 6.4 3.4 4.0 6.3 4.5

14.0 13.4 13.7 15.8 14.5 13.9 17.1

20.0 22.0 21.3 21.7 17.0 21.0 24.3

4.0 7.6 6.8 4.8 3.2 6.4 4.5

8.9 7.5 7.6 9.0 8.3 7.7 10.4

C OMPARISONS WITH I NFERENCE -T IME M ETHODS

In the manuscript, we argue that existing inference-time approaches struggle to balance the trade-off between resolution performance and token efficiency. To verify the claim, we conduct additional experiments on four representative inference-time approaches, i.e., SWE-Pruner (Wang et al., 2026b), Pruning + Summ. (Lodha et al., 2026), Observation Masking (Lindenbauer et al., 2025), and Turn Limit (Gao & Peng, 2025). Specifically, we apply inference-time approaches to both Qwen3.5-4B and its HERO-trained counterpart, termed HERO-4B. From the results in Table 7, one could have the following observations and conclusions: i) most inference-time approaches reduce token usage without improving resolution performance over the base model, highlighting the difficulty of balancing resolution performance and token efficiency; ii) HERO-4B achieves higher TRS scores than inference-time approaches applied to the base model, suggesting that our training-time designs achieve a more favorable trade-off between resolution performance and token efficiency; iii) thanks to the training-time designs, HERO-4B could still benefit from inference-time approaches, suggesting that the proposed HERO could complement inference-time approaches. 16

Table 7: Comparisons with inference-time approaches on SWE-bench Verified and SWE-bench Multilingual. SWE-bench Verified Variants

B.3

SWE-bench Multilingual

Resolved Rate

Tokens

TRS

Resolved Rate

Tokens

TRS

Qwen3.5-4B + SWE-Pruner + Pruning + Summ. + Observation Masking + Turn Limit

32.8 33.0 31.4 30.0 29.0

4.5 4.3 3.2 3.8 3.0

14.0 14.3 15.3 13.7 14.5

20.0 19.7 18.7 18.0 16.7

4.0 3.8 3.3 3.5 3.1

8.9 9.0 9.0 8.5 8.2

HERO-4B + SWE-Pruner + Pruning + Summ. + Observation Masking + Turn Limit

40.0 40.4 36.4 39.4 36.0

4.5 4.3 3.4 3.8 3.1

17.1 17.5 17.4 18.0 17.8

24.3 23.7 21.7 23.4 21.0

4.5 4.4 3.6 3.8 3.1

10.4 10.2 10.1 10.7 10.4

A NALYTIC S TUDY ON P REFIX -C ACHE R EUSE AND I NFERENCE C OST

To further investigate efficiency in real-world applications, we conduct an analytic study on prefixcache reuse and relative inference cost on SWE-bench Verified. Following Lindenbauer et al. (2025), we estimate inference cost post hoc from recorded token usage. Specifically, we derive the tokenweighted cache-hit rate and per-instance cost as P cache i ti HitRate = 100 P , in (11) i ti
cache cache out Costi = tin − t + 0.1t + 4t , i i i i cache where tin , and tout denote the cumulative input, cached input, and output token counts for i , ti i instance i. We employ fixed cost weights of 1, 0.1, and 4 to uncached input, cached input, and output tokens, respectively. Relative inference cost is normalized by the average per-instance cost of the corresponding base model at each model size.

From the results in Table 8, one could have the following conclusions. On the one hand, although context manipulation methods, e.g., Pruning + Summ. and Observation Masking, could reduce token usage, these methods would undermine prefix-cache reuse, resulting in higher relative inference costs. On the other hand, interaction limits reduce both token usage and relative inference cost at the expense of a serious degradation in resolution performance. In contrast, HERO achieves competitive prefix-cache reuse and relative inference cost, supporting its effectiveness in practical applications. Table 8: Analytic study on prefix-cache reuse and relative inference cost on SWE-bench Verified. Variants Qwen3.5-4B + SWE-Pruner + Pruning + Summ. + Observation Masking + Turn Limit HERO-4B

B.4

Resolved Rate (%)

Cache Hit (%)

Relative Cost

32.8 33.0 31.4 30.0 29.0 40.0

92.4 86.3 71.9 80.3 89.2 91.9

1.0 1.4 1.3 1.3 0.7 1.0

A BLATION S TUDIES ON SWE- BENCH M ULTILINGUAL

In the manuscript, we have conducted ablation studies on SWE-bench Verified. Here, we extend the ablation studies to more complex multilingual SWE tasks, i.e., SWE-bench Multilingual. As shown in Table 9, removing hierarchical policy optimization (HPO) reduces token usage at the cost of 17

degrading resolution performance, while either coarse-grained or fine-grained efficiency advantages contribute to reducing token consumption and improving resolution performance. Table 9: Ablation studies of HERO on SWE-bench Multilingual.

B.5

Variants

Resolved Rate

Tokens

TRS

GRPO w/o HPO w/o Coarse w/o Fine

22.0 17.0 23.7 22.3

7.6 2.3 7.0 5.5

7.5 9.4 8.4 8.7

Default

24.3

4.5

10.4

A NALYTIC S TUDY ON T OKEN U SAGE B REAKDOWN

We conduct more in-depth analysis to investigate the overhead of multi-turn interactions. From the results in Tables 10 and 11, input tokens account for more than 99% of total consumption, highlighting the significant overhead of repeatedly processing accumulated context. Compared with GRPO, HERO achieves higher resolution rates with fewer interaction turns and lower token consumption on both benchmarks. Table 10: Token usage and interaction turns on SWE-bench Verified. “Input Ratio” denotes the proportion of input tokens in total token consumption. Method

Resolved Rate Total Tokens Input Ratio Avg. Turns

Qwen3.5-4B + GRPO + Ours

32.8 37.4 40.0

4.5 6.8 4.5

99.6 99.6 99.4

68.7 94.8 65.6

Table 11: Token usage and interaction turns on SWE-bench Multilingual. Method

Resolved Rate Total Tokens Input Ratio Avg. Turns

Qwen3.5-4B + GRPO + Ours

B.6

20.0 22.0 24.3

4.0 7.6 4.5

99.5 99.6 99.3

64.3 106.5 69.6

A NALYTIC S TUDY ON S CAFFOLD G ENERALIZATION

In the manuscript, we have evaluated the proposed HERO under the unified Claude Code scaffold for fairness. Here, we conduct additional experiments using widely-used mini-SWE-agent scaffold on SWE-bench Verified. As shown in Table 12, even training on the Claude Code scaffold, HERO achieves higher resolution rates with lower token consumption than GRPO under both scaffolds, indicating HERO learns efficient reasoning patterns that generalize across scaffolds. Table 12: Performance comparisons under different agent scaffolds on SWE-bench Verified. Claude Code Variants Qwen3.5-4B + GRPO + Our

mini-SWE-agent

Resolved Rate

Tokens

TRS

Resolved Rate

Tokens

TRS

32.8 37.4 40.0

4.5 6.8 4.5

14.0 13.4 17.1

39.5 41.2 43.0

4.0 5.2 4.0

17.7 16.5 19.2

18

B.7

M ORE D ETAILED A BLATION S TUDIES

To further verify the effectiveness of individual designs, we carry out ablation studies on the efficiency gating and resolution-first clipping in hierarchical policy optimization (HPO) and outcome conditioning in coarse-grained efficiency advantage. As demonstrated in Table 13, one could have the following conclusions: i) either efficiency gating or resolution-first clipping could help preserve resolution performance, which verifies the effectiveness of hierarchical policy optimization in prioritizing task resolution over token efficiency; ii) resolution-calibrated credit assignment could facilitate the learning of efficient reasoning patterns without conflating token efficiency with resolution performance, thus achieving a better trade-off between resolution rate and token efficiency.

Table 13: More detailed ablation studies on SWE-bench Verified. “w/o Calibration” denotes using shared threshold across all trajectories within the rollout group. Component Variants

B.8

Resolved Rate Tokens TRS

HPO

w/o Efficiency Gating w/o Clipping Default

36.0 37.8 40.0

3.8 4.1 4.5

16.4 16.7 17.1

Coarse

w/o Calibration Default

38.2 40.0

5.9 4.5

14.5 17.1

S ENSITIVITY TO THE C APABILITY T HRESHOLD

We further investigate the effect of the capability threshold τ in Eq. 5. As shown in Fig. 8, HERO achieves a desirable trade-off between resolution performance and token efficiency when τ ∈ [0.5, 0.6].

Figure 8: Parameter analysis of the capability threshold τ in Eq. 5.

B.9

S ENSITIVITY TO THE E NTROPY P ERCENTILE

We conduct additional parameter analysis of the entropy percentile in Eq. 9. As shown in Fig. 9, a larger percentile generally reduces token usage but also degrades resolution performance, suggesting that penalizing too many turns might suppress necessary reasoning. 19

Figure 9: Parameter analysis of the entropy percentile in Eq. 9. B.10

A NALYTIC S TUDY ON F INE - GRAINED T URN A DVANTAGE

To further investigate the effectiveness of the fine-grained turn advantage, we compare two alternative penalty strategies on SWE-bench Verified: i) Random: randomly selecting 20% of turns for penalization; ii) High Entropy: selecting the 20% of turns with the highest entropy for penalization. As demonstrated in Table 14, the default strategy of penalizing low-entropy turns achieves a higher resolution rate with lower token consumption than both alternatives, suggesting that penalizing lowentropy turns contributes to learning efficient reasoning patterns. Table 14: Analytic study of turn-level penalty strategies on SWE-bench Verified. Variants

Resolved Rate (%)

Tokens (M)

TRS

Random High Entropy

36.8 36.0

5.4 6.0

14.5 13.6

Default

40.0

4.5

17.1

20

Related documents

• Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents#1143486
• Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning#997438
• Understanding User Mental Models in AI-Driven Code Completion Tools: Insights from an Elicitation Study#987407
• ACEM: A Cost Estimation Model for Agentic Software Engineering#424031
• Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review#1063864