AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning
Haotian Zhao1∗
arXiv:2605.00425v1 [cs.AI] 1 May 2026
Stephen S.-T. Yau3 Yucheng Zeng1
Yuxin Zhang1∗
Wenyu Zhang1
Lun Tian1
Jingnan Gu1†
Songlin Zhou2∗ Tianshu Zhu1
Daxiang Dong1†
Yifeng Huang4 Jianmin Wu1†
1
{zhaohaotian02,zhangyuxin15,zhangwenyu08, zhutianshu,tianlun, zengyucheng,gujingnan,dongdaxiang,wujianmin}@baidu.com, Baidu 2 [email protected], Tsinghua University 3 [email protected], Tsinghua University 4 [email protected], Fudan University
Abstract Reinforcement learning (RL) has significantly advanced the ability of large language model (LLM) agents to interact with environments and solve multi-turn tasks. Yet effective training remains challenging, as sparse, outcome-only rewards make it difficult to assign credit to individual steps in an agent’s action trajectory. A common remedy is to introduce dense intermediate supervision, such as process reward models or auxiliary self-supervised signals, but this increases supervision and tuning complexity and often generalizes poorly across tasks and domains. This paper presents AEM, a supervision-free credit assignment method that adaptively modulates entropy dynamics during RL training to achieve a more effective exploration–exploitation trade-off. Theoretically, we elevate entropy analysis from the token-level to response-level to reduce token sampling variance and prove the entropy drift under natural gradients is controlled through the product of the advantage and the relative response surprisal intrinsically. Specifically, we derive a practical proxy to reshape training dynamics, enabling a natural transition from exploration to exploitation. Extensive experiments across various benchmarks and 1.5B–32B models demonstrate the effectiveness of AEM, with a particularly notable +1.4% gain when integrated into the state-of-the-art baseline on the highly challenging SWE-bench-Verified benchmark.
1
Introduction
Large language models (LLMs) are increasingly being deployed as interactive agents that solve complex tasks through multi-turn reasoning (Xu et al., 2025a; Zeng et al., 2025), tool use (Shen et al., 2024; Wu et al., 2024), and sustained interaction with external environments (Chen et al., 2026a; Fang et al., 2025). Beyond token-level generation and prediction, these agents act as sequential decisionmaking (Shinn et al., 2023; Erdogan et al., 2025) systems that reason, act, and adapt across long interaction trajectories, with applications ranging from autonomous software engineering (Yang et al., ∗ Equal contribution. † Corresponding author.
Preprint.
2024a, 2025) and embodied assistants (Yang et al., 2024b; Li et al., 2024) to GUI navigation (Yuan et al., 2026; Li et al., 2026). Reinforcement learning (RL) (Christiano et al., 2017; Schulman et al., 2017), with group-based methods such as GRPO (Shao et al., 2024), has emerged as a key paradigm for LLM post-training. However, extending these methods to agentic RL, a multi-turn setting where LLM-powered agents make consecutive actions in dynamic environments (Zhang et al., 2025), remains fundamentally challenging—sparse, outcome-level rewards provide little supervision for intermediate steps (Feng et al., 2025). As a result, exploratory, decisive, and redundant steps often receive indistinguishable updates, obscuring the distinction between uncertainty and confidence and ultimately limiting learning efficiency. Existing efforts for step-level credit assignment in agentic RL differ mainly in where and how credit signals are derived. Process reward models (Lightman et al., 2023), as representative reward shapingbased methods, provide dense credit supervision from external annotations but require additional model training. Self-supervised methods instead derive credit signals from the policy or its sampled trajectories without auxiliary supervision. Among these, GiGPO (Feng et al., 2025) and IGPO (Wang et al., 2026) infer step-level credit signals from inherent trajectory attributes, but remain prone to context inconsistency, grouping bias, and restrictive structural assumptions, limiting robustness and generalization. By contrast, Tree-GRPO (Ding and Ye, 2026) and ATPO (Cao et al., 2026) attempt to explicitly shape credit propagation through tree-structured optimization but incur high computational overhead in multi-turn settings. Collectively, these limitations call for a scalable, fine-grained credit assignment framework that does not rely on extra supervision, heavy computation, or restrictive structural assumptions. Specifically, we theoretically analyze entropy dynamics in agentic RL, establishing response-level entropy3 as an intrinsic signal for credit assignment, and showing that it provides a response-aligned measure of uncertainty, mitigates token-level sampling noise (Theorem 3.2.1), and induces entropy dynamics governed by the sampled-response advantage and relative surprisal (Theorem 3.2.2). Building on this analysis, we propose Adaptive Entropy Modulation (AEM), a credit assignment algorithm that leverages the shifting balance between positive and negative samples throughout training to adaptively regulate policy entropy, thereby inducing a natural transition from exploration to exploitation. Particularly, we derive a predictable proxy for response-level uncertainty, and use it to modulate step advantages without additional supervision, ultimately enabling adaptive entropy regulation and promoting policy exploration in early training and facilitating convergence to high-quality policies in later stages (Shen, 2026). Our contributions are three-fold. • We theoretically characterize response-level entropy dynamics, demonstrating that local entropy drift is governed by the interaction between sampled response advantage and relative surprisal, thereby revealing how response-level entropy can be used for fine-grained credit assignment during agentic RL. • We propose AEM, an adaptive, supervision-free credit assignment method that leverages the evolving balance between positive and negative samples under our entropy-dynamics formulation, enabling a natural transition from exploration to exploitation without explicit entropy regularization or hand-crafted schedules. • We conduct extensive experiments across ALFWorld, WebShop, and SWE-bench-Verified ranging from 1.5B to 32B. AEM consistently improves various baseline methods with peak gains of 8.8%, showing that entropy-aware response-level credit shaping is an effective inductive bias for multi-turn LLM agent optimization.
2
Related Work
From LLMs to Agentic RL. Large language models are increasingly explored as interactive agents that solve tasks through multi-turn reasoning, tool use, and closed-loop interaction with external environments. Representative works such as ReAct (Yao et al., 2023) and Toolformer (Schick et al., 3 In practice, a response usually combines reasoning and acting; in RL theory, it’s the "action" sampled from the policy. To
avoid ambiguity, we use the term "response."
2
2023) demonstrate that LLMs can interleave reasoning with actions and external tool invocation, shifting the role of LLMs from passive generators to interactive decision-makers. Training such agents increasingly relies on RL, where group-based methods such as RLOO (Ahmadian et al., 2024) and GRPO (Shao et al., 2024) have emerged as a dominant approach. Extending these methods from single-turn to multi-turn agentic settings exacerbates sparse rewards: feedback arrives only at the end, providing little guidance for intermediate decisions. The lack of step-level supervision yields high-variance gradients and ambiguous credit assignment, hindering the policy from identifying actions that drive success. Credit Assignment in Agentic RL. Credit assignment is a long-standing challenge in agentic RL with delayed and sparse rewards. Existing approaches can be broadly distinguished by the source and construction of their credit signals. Some rely on external signals, such as value functions or step-level supervision (Schulman et al., 2017; Lightman et al., 2023), but introduce additional modeling and scaling overhead. Others derive credit internally from sampled trajectories (Feng et al., 2025; Wang et al., 2026), avoiding auxiliary supervision; some methods infer credit implicitly from trajectory attributes, while others further refine credit through structured propagation (Cao et al., 2026; Ding and Ye, 2026) or explicit reward redistribution (Wang et al., 2025a), which improves granularity at the cost of additional computation in multi-turn settings. Despite their differences, these methods all overlook a more fundamental quantity: the policy’s own entropy. As entropy reflects model uncertainty at each decision, it provides a natural and intrinsic signal for distinguishing confident from exploratory steps—and it is precisely this signal that our method builds on for credit modulation. Entropy-Aware Policy Optimization. Entropy has been widely used in RL as a regularization signal to encourage exploration (Cui et al., 2025; Petrenko et al., 2026; Chen et al., 2026b) and stabilize optimization (Mnih et al., 2016). Recent works have explored entropy-aware objectives to modulate training, such as entropy-regularized policy optimization (Xu et al., 2025b) and entropyguided advantage scaling (Wang et al., 2025b). These methods demonstrate that policy entropy reflects model uncertainty and can serve as an informative signal beyond outcome-based rewards. However, existing approaches primarily rely on token-level entropy control or use entropy mainly to stabilize training, rather than shaping response-level credit assignment to adaptively regulate the optimization dynamics. In contrast, our work leverages response-level entropy as an intrinsic signal to modulate per-response advantages, thereby adaptively regulating policy entropy while enabling scalable credit assignment in agentic RL.
3
Theoretical Analysis
3.1
Preliminaries
We consider a multi-turn agentic RL setting, where an agent policy πθ (· | s) interacts with an environment over T steps. At each step t ∈ {1, . . . , T }, the agent observes a state st ∈ S (e.g., language messages, tool outputs, or webpage snapshots) and produces a textual action at ∈ V ≤n (e.g., free-form text, tool invocation with arguments, or interface selection), where V is the LLM vocabulary and n is the maximum output length. Given task and prompt s0 , an episode yields a QT −1 trajectory τ = {(s0 , a0 ), . . . , (sT −1 , aT −1 )}, sampled from Pθ (· | s0 ) = t=0 πθ (· | st ) under Markov Decision Process assumption with s0 . The policy is trained to maximize the expected trajectory return: J(θ) = Eτ ∼Pθ [R(τ )]. (1) Each sampled response at at state st is associated with an advantage A(at , st ) determined by the base advantage estimator. Hence, conditioning on a sampled pair (at , st ), the corresponding policy optimization surrogate objective is ℓat (π) = A(at , st ) log πθ (at | st ).
(2)
In agentic RL, the environment typically reacts after a complete response is generated, making the response the effective interaction unit, rather than an individual token. The objective ℓA at (π) is consistent with this granularity, assigning a single learning signal to the whole response. Accordingly, 3
we study response-level uncertainty, and define the response surprisal S(at | st ) := − log πθ (at | st ) = −
|at | X
log pθ (yℓ | st , y<ℓ ),
(3)
ℓ=1
with the response-level Shannon entropy X Hresp (st ) := − πθ (at | st ) log πθ (at | st ) = Eat ∼πθ (·|st ) [S(at | st )].
(4)
at ∈At
3.2
Response-Level Entropy Geometry
Theorem 3.2.1 (Relationship among token, response, and global entropy. Proved in Appendix C.1). Let at = (Y1 , .., YL ) ∼ πθ (·|st ) denote a sampled response spanned by tokens Yl ∼ pθ (· | Y<l ), and s0 denote the initial state in the dataset D. The token-level entropy Hℓ (at , st ) and the global entropy Htotal are respectively formulated by X Hℓ (at , st ) := E[− log pθ (Yl |st , Y<l )|st , Y<ℓ ] = − pθ (y|st , y<l ) log pθ (y|st , y<l ); (5) y∈V
|at | T −1 X X
Htotal = Es0 ∼D,τ ∼Pθ (·|s0 )
Hℓ (at , st ) .
(6)
t=0 ℓ=1
Then, the response-level entropy is the expectation of token-level entropy sum: X Hresp (st ) = Eat ∼πθ (·|st ) Hℓ (at , st )1{ℓ ≤ |at |}|st ,
(7)
ℓ≥1
and the global entropy is the expectation of response-level entropy sum: "T −1 # X Htotal = Es0 ∼D,τ ∼Pθ (·|s0 ) Hresp (st ) .
(8)
t=0
Therefore, response-level entropy provides a structurally faithful intermediate uncertainty measure: entropy modulation applied at the response level induces corresponding changes in global entropy, while being less sensitive to token-level sampling variation. To analyze how a sampled action and its advantage reshape the policy distribution from an informationtheoretic perspective, we formulate the policy given state s on the probability simplex ∆◦ (As ) equipped with the Fisher-Rao metric (Amari and Nagaoka (2000), Nielsen (2020)), this canonical information metric is the local quadratic form of KL divergence (Details in Appendix C.2). Within this geometry, the natural gradient Kakade (2001) induces parameterization-invariant policy updates. By analyzing response-level entropy dynamics and aggregating them over visited states, the following theorem shows that the entropy dynamics Htotal is governed by the advantage and relative surprisal of sampled responses. Theorem 3.2.2 (Entropy drift under fixed occupancy. Proved in Appendix C.3). Let gradF denote the natural gradient on the policy simplex ∆◦ (As ), then the directional derivative of Hresp along the update direction gradF ℓa (π) satisfies D E resp DRL (a; s) := gradF Hresp (π), gradF ℓa (π) = A(a, s) S(a | s) − Hresp (s . (9) Fisher-Rao
Assume a local policy update under a frozen rollout distribution, i.e., when differentiating the global entropy objective, we do not propagate gradients through the rollout distribution Pθ . Then the global entropy drift effected by a sampled response a equals the visitation-weighted expectation of the response-level entropy drift: D E DRL (a; s) := gradF Htotal (π), gradF ℓa (π) Fisher-Rao
=
T −1 X
Ps0 ∼D,τ ∼Pθ [st = s] A(a, s) S(a | s) − Hresp (s) .
t=0
4
(10)
Δ°(𝒜ₛ) = { π ∈ ℝ³ : π(𝑎) + π(𝑏) + π(𝑐) = 1, π > 0 } 𝑎
𝑎 grad ℓ! (π) 𝜃<90°
grad ℓ! (π)
grad ℋresp(π) 𝜃>90°
π
π
grad ℋresp(π) 𝑐
𝑏
𝑐 𝑏
Entropy Increases
Entropy Decreases
Figure 1: An example on a three-action policy simplex: entropy increases along the training direction when DRL (a; s) > 0 i.e., θ⟨gradF ℓa ,gradHresp ⟩ < 90◦ , and decreases otherwise. Therefore, the entropy dynamics in training is determined by advantage of sampled response A(a, s) and relative surprisal S(a | s) − Hresp (see Figure 1): sgn(A(a, s)(S(a | s) − Hresp )) > 0 =⇒ entropy increases; sgn(A(a, s)(S(a | s) − Hresp )) < 0 =⇒ entropy decreases.
(11)
Remark 3.2.3. In some practical agentic RL, the objective is not purely reward-driven: many methods also include entropy regularization or KL penalties. In Appendix C.3, we extend the theorem to the regularized objective: ℓa (πθ ) = A(a, s) log πθ + βψ(Hresp (πθ )) − γDKL (πθ ∥πref ),
(12)
where ψ is a positive increasing function and β, γ are regularization coefficients. It is demonstrated that, since these regularization terms act at the state level, they do not change the response-dependent modulation principle implemented by AEM. Theorem 3.2.2 characterizes how a sampled-response learning signal changes the entropy of the response distribution itself and enlightens AEM, whose design does not depend on the particular neural parameterization. For complement, Appendix C.5 discusses the parameter-space counterpart of this result to clarify the difference between the intrinsic policy-space mechanism and parameter updates. Therefore, the entropy dynamics can be modulated through advantage and relative surprisal of sampled responses . This provides a way to control entropy dynamics through response-level surprisal signals, without altering the underlying RL optimization backbone. Motivated by this observation, we next introduce AEM, which implements such intrinsic credit modulation in a practical form.
4
AEM: Adaptive Entropy Modulation
4.1
What is AEM?
AEM is a plug-in response-level advantage modulation applied on top of the advantage estimator, leveraging a proxy of relative surprisal as an intrinsic signal to control the dynamics of entropy. Let Abase i,t be the response-level advantage at t-th turn in i-th rollout Si produced by the base estimator. Here Si = {Si,1 , ..., Si,Ki }, with each Si,t = [begin_tokeni,t , end_tokeni,t ] corresponds to one completed response generated before the next environment transition. For each environment-reactive response span Si,t , AEM adaptively computes one scalar coefficient αi,t and applies it uniformly to all tokens in the span: AAEM = αi,t Abase i,t i,t . AEM only rescales the response-level advantage, aiming to inducing entropy-decreasing pressure on positive responses and entropy-increasing pressure on negative responses, ultimately achieves adaptive exploration-exploitation transition during RL training. 5
4.2
Modulation Mechanism
Since the state-specific baseline Hresp (st ) is not directly tractable during training, AEM does not explicitly reconstruct the exact gap. Instead, it converts the relative magnitude within the group of this proxy into a modulation coefficient α, so that α > 1 and α < 1 serve as practical indicators of relatively lower and higher surprising responses. Given the t-th response in a rollout, Theorem 3.2.2 shows that the sign of the local entropy drift is jointly governed by the relative surprisal S(at | st ) − Hresp (st ) and the response advantage A(at , st ). To reduce the sensitivity to the particular sampled tokens, we use the predictable proxy P|a| ℓ=1 Hℓ (at , st ) for S(at |st ) from Doob’s decomposition (see Appendix C.4 for details). With a length normalization to make the response-level entropy scale-free, we consider 1 X H̄i,t = Hℓ (at , st ), |Si,t |
(13)
ℓ∈Si,t
and apply a monotone decreasing map from H̄i,t to a response-uniform coefficient αi,t . Let G be a group as the set of all responses in the trajectories generated by a prompt. We normalize Hi,t within group min-max scaling to avoid numerical explosion: H̃i,t =
H̄i,t − min(j,n)∈G H̄j,n , max(j,n)∈G H̄j,n − min(j,n)∈G H̄j,n + ε
for (i, t) ∈ G.
(14)
When max(j,n)∈G H̄j,n −min(j,n)∈G H̄j,n < 0.1, we set αi,t = 1 to avoid sampling noise. Otherwise, we define the self-calibrated modulation weight with temperature λ: αi,t =
1 |G|
exp(−λH̃i,t ) , (j,n)∈G exp(−λH̃j,n ) + ε
P
for (i, t) ∈ G.
(15)
Hence AEM relatively upweights (α > 1) spans with lower relative surprisal proxy within the group, and downweights (α < 1) those with higher relative surprisal proxy, while preserving the overall modulation scale through self-calibration. 4.3
Exploration-Exploitation Transition
Analysis A shows that A(a, s) and α − 1 can indeed determine the pratical entropy dynamics base D̃RL (a; s): base sgn D̃RL (a; s) ≈ − sgn (A(a, s)(α − 1)) . (16) Generally, AEM systematically shifts the intrinsic entropy drift based purely on the sign of the advantage: AEM base sgn D̃RL − D̃RL = − sgn (α − 1)2 A(a, s) = − sgn A(a, s). (17) By Eq. (8) in Theorem 3.2.1, through modulating the entropy drift of relatively many responses, AEM induces a corresponding shift in the total entropy. As training progresses, it naturally induces an implicit transition from exploration to exploitation: Exploration. For negative responses A(a, s) < 0 which are relatively prevalent in early stage of RL training, AEM provides entropy-increasing pressure: ( base H̄i relatively large =⇒ α < 1, D̃RL < 0 =⇒ attenuate entropy-decreasing, (18) base H̄i relatively small =⇒ α > 1, D̃RL > 0 =⇒ amplify entropy-increasing. Exploitation. For positive responses A(a, s) > 0 which are relatively prevalent in late stage of RL training, AEM provides entropy-decreasing pressure: ( base H̄i relatively large =⇒ α < 1, D̃RL > 0 =⇒ attenuate entropy-increasing, (19) base H̄i relatively small =⇒ α > 1, D̃RL < 0 =⇒ amplify entropy-decreasing. Analysis B shows that AEM mitigates early entropy collapse, promotes more complete late-stage convergence, and improves final performance. 6
5
Experiments
5.1
Setup
Benchmarks. We evaluate AEM on three challenging multi-turn LLM agent benchmarks: ALFWorld (Shridhar et al., 2021), WebShop (Yao et al., 2022), and SWE-bench-Verified (Jimenez et al., 2024). ALFWorld is a text-based embodied benchmark for evaluating multi-turn decision-making in LLM agents. It comprises tasks spanning six categories of everyday household activities, including Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2). WebShop is a large-scale, web-based interactive benchmark for evaluating LLM agents in realistic online shopping tasks, featuring over 1.1M products and 12K natural-language instructions and a simulated HTML interface for search, navigation, and item selection. SWE-bench-Verified (Jimenez et al., 2024) is a curated subset of SWE-bench with expert annotation that removes noisy instances to ensure clear tasks, stable environments, and verifiable solutions, enabling more reliable evaluation of LLM agents on real-world software engineering tasks. Baselines. For ALFWorld and WebShop, we compare AEM against several competitive baselines, including: (1) closed-source LLMs: GPT-5.2-Pro (gpt, 2025) and Gemini-3-Pro (gem, 2025); (2) prompting-based methods: ReAct (Yao et al., 2023), which interleaves reasoning traces and executable actions to enable step-by-step decision-making in interactive environments; (3) reinforcement learning methods: PPO (Schulman et al., 2017), a representative actor–critic method requiring an additional value network; GRPO (Shao et al., 2024), a value-free approach that estimates advantages via group-based relative comparisons without a critic; DAPO (Yu et al., 2025), an RL algorithm built on GRPO, introduces Clip-Higher, Dynamic Sampling, Token-Level Policy Gradient Loss, and Overlong Reward Shaping to improve exploration, efficiency, and training stability; GSPO (Zheng et al., 2025) replaces GRPO’s token-level importance weighting with sequence-level ratio estimation and clipping, thereby improving the stability and efficiency of RL training. To further validate the generality of AEM in complex agentic RL scenarios, we integrate it into DeepSWE (Luo et al., 2025), a state-of-the-art RL framework for SWE-bench-Verified that extends GRPO to multi-turn software engineering agents with dynamic sampling, leave-one-out advantage estimation, length normalization, and trajectory filtering. For algorithmic details of these baselines, please see Appendix D.1. Implementation. We use rule-based outcome rewards across all benchmarks. In ALFWorld and WebShop, successful trajectories receive a reward of 10 and failed trajectories receive 0, with an additional penalty of −0.1 for invalid actions; in SWE task, we use binary rewards, assigning 1 for success and 0 for failure. Across all tasks, the rollout group size is fixed to N = 8. We use the verl-agent (Feng et al., 2025) training framework for ALFWorld and WebShop, and rLLM (Tan et al., 2025) for SWE-bench-Verified. All results are averaged over 3 random seeds Full implementation details are deferred to Appendix D.2. 5.2
Overall Performance
Performance on ALFWorld and WebShop. Table 1 reports the overall results of applying AEM to different baselines on ALFWorld and WebShop. Overall, AEM consistently improves group-based RL baselines across both benchmarks and model scales, and in several settings achieves performance competitive with strong closed-source models. These results validate adaptive entropy modulation as an effective plug-in mechanism for multi-turn agent training. In GRPO, the same group-based advantage is shared across the entire sampled response, which is often too coarse for multi-turn tasks where different responses may play different roles. By modulating advantages with response-level uncertainty, AEM provides finer credit assignment and yields consistent gains of 8.8% (5.7%) and 5.6% (4.6%) on ALFWorld and WebShop, respectively, using 1.5B (7B) models without any extra supervision. As discussed above, DAPO provides a stronger group-based optimization backbone than GRPO. Nevertheless, DAPO still benefits from AEM, achieving additional gains of up to 6.0%, which suggests that entropy-aware response modulation is complementary to stronger optimization backbones: DAPO improves how updates are performed, whereas AEM refines which responses should receive stronger learning signals during training. Moreover, AEM further improves GSPO by up to 5.4%, suggesting that entropy-aware credit assignment remains complementary even when applied on top of response-level optimization. The training curves are deferred to Appendix B.
7
Table 1: Performance comparison on ALFWorld and WebShop benchmarks. The results of ReAct and PPO are adopted from Feng et al. (2025). ALFWorld
Method Pick
Heat
Cool
WebShop
Look
Clean
Pick2
All
Score
Succ.
88.8 99.3
44.4 56.7
46.6 60.8
GPT-5.2-Pro Gemini-3-Pro
100 100
100 100
Closed-Source Model 100 61.3 87.0 100 96.8 100 100 100
ReAct PPO GRPO +AEM GSPO +AEM DAPO +AEM
17.4 64.8 78.2 88.6 75.4 75.5 100.0 97.3
20.5 40.5 49.9 67.6 54.2 56.5 70.3 90.3
Qwen2.5-1.5B-Instruct 15.7 6.2 7.7 2.0 57.1 60.6 46.4 47.4 70.5 72.0 75.0 39.2 76.4 60.9 76.7 69.9 64.6 70.0 74.6 30.0 78.1 75.0 70.2 46.7 90.6 91.3 86.6 82.9 98.8 98.4 90.9 89.5
12.8 54.4±3.1 68.0±0.8 76.8±1.8 66.7±5.3 71.9±8.4 88.5±1.2 94.5±1.4
40.1 73.8±3.0 83.6±0.2 86.4±2.1 75.1±7.1 76.3±3.8 86.5±0.9 88.0±1.0
11.3 51.5±2.9 65.0±0.6 70.6±2.4 61.5±4.5 66.9±3.2 75.9±2.9 78.5±1.0
ReAct PPO GRPO +AEM GSPO +AEM DAPO +AEM
48.5 92.3 91.3 98.9 95.1 88.9 100.0 99.0
35.4 64.0 91.5 78.6 66.9 56.8 96.3 91.7
34.3 92.5 79.9 89.4 73.9 92.6 100.0 100.0
Qwen2.5-7B-Instruct 13.2 18.2 17.6 89.5 80.3 68.8 76.9 75.2 44.3 84.1 79.5 65.7 80.0 79.8 69.7 85.2 84.8 78.3 94.7 90.3 94.3 96.3 95.2 93.2
31.2 80.4±2.7 78.7±1.6 84.4±3.1 80.7±2.3 83.4±3.1 96.1±2.1 96.6±0.7
46.2 81.4±3.1 84.1±2.5 86.9±1.4 80.4±1.9 81.9±1.0 93.7±0.5 94.5±1.0
19.5 68.7±5.1 75.9±3.4 80.5±2.1 71.6±4.6 72.1±3.0 86.7±1.4 88.9±0.9
Performance on SWE-bench-Verified. To further validate Table 2: SWE-bench-Verified rethe effectiveness of AEM in larger-scale models and more chal- sults with Qwen3-32B. lenging tasks, we evaluated it in the SWE-bench-Verified and compared it with DeepSWE. DeepSWE performs RL on Qwen3Method Succ. (%) 32B using the R2E dataset (Jain et al., 2024), and reports a 42.2% DeepSWE 42.3±0.3 success rate on SWE-bench-Verified at the time of release. In DeepSWE+AEM 43.7±0.4 our reproduction, DeepSWE achieves an average success rate of 42.3%, serving as a strong baseline for evaluating AEM. As shown in Figure 9, DeepSWE+AEM improves the final score to 43.7%, improving over DeepSWE by 1.4%. SWE-bench-Verified is substantially more challenging than ALFWorld and WebShop, with abstract objectives, large solution spaces, and open-ended software environments. Improvements on this benchmark suggest that AEM remains effective beyond controlled agent benchmarks, extending to realistic multi-turn settings that resemble production workloads. Analysis
Analysis A: Validating the transition of entropy. Figure 2 illustrates the entropy dynamics during the first 50 training steps under two gradient-masking strategies in GRPO on WebShop with Qwen2.5-1.5B. Masking the response groups such that (α > 1, A > 0) and (α < 1, A < 0), leads to
6
Entropy
5.3
w/o sgn(A(a)( Baseline w/o sgn(A(a)(
1)) > 0 1)) < 0
4 2
Masking sgn D̃RL = − sgn(A(a)(α − 1)) = −1, =⇒ entropy increases. whereas masking the responses with opposite sign leads to entropy decrease; the unmodified baseline remains in between. This empirical pattern is consistent with Eq. (16), suggesting that α − 1 provides an empirical partition of responses into the two sides of the entropy baseline, while the resulting entropy trend is jointly determined by A(a) and α − 1. 8
0 10
20
30
Training Step
40
50
Figure 2: The two masking strategies isolate entropy-increasing and entropydecreasing update groups, causing clearly diverging entropy trends.
Baseline+AEM
GRPO #2 GRPO #3
Baseline
Entropy
Success rate
1.2
1.4 1.2 1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4
0
30
60
90
Training step
120
150
0
30
60
90
120
150
80% 70% 60% 50% 40% 30% 20% 10% 0%
Success Rate
GRPO+AEM #3 GRPO #1
Entropy
Entropy
GRPO+AEM #1 GRPO+AEM #2
Training Step
(a) Entropy dynamics of all trajectories.
(b) Entropy and success rates of GRPO (+AEM).
Figure 3: (a): All GRPO (+AEM) training entropy curves, with statistical effect. (b): Entropy and success rate dynamics of one pair of the curve. Analysis B: AEM induces an implicit exploration-to-exploitation transition. Figure 3a shows the entropy dynamics on WebShop with Qwen2.5-1.5B. Across multiple runs, the baseline exhibits an abrupt entropy collapse at the beginning of training and then remains in a relatively flat entropy regime, indicating premature concentration and limited late-stage optimization. In contrast, AEM consistently preserves higher entropy in the early stage and gradually reduces it to a lower range later, suggesting a systematic transition from exploration to exploitation rather than an isolated run-specific effect. To better understand this transition, Figure 3b overlays entropy with success rate for a representative pair of runs. AEM maintains higher entropy early on, promoting response diversity. As the success rate increases by the train, the training batches contain a growing proportion of positive samples relative to negative ones, under which AEM gradually transitions from entropy-increasing to entropydecreasing dynamics adaptively. This enables the policy to exploit the diversity accumulated during early exploration and achieve a higher final success rate. In contrast, the baseline collapses entropy prematurely but shows limited further improvement, remaining in a locally suboptimal regime.
6
Computation Cost
This section investigates additional computaComponent share AEM A tion cost introduced by AEM, which is limited Rollout 45.9% 229.9s to lightweight response-level uncertainty esOld prob. 8.2% 41.2s timation and modulation, including responselevel entropy aggregation, group-wise normalRef prob. 8.6% 42.9s 36.0% 45.9% Total ization, and advantage rescaling. Importantly, 500.7s Update 36.0% 180.4s per step AEM does not require extra rollouts or addiA 0.2% 0.81s tional policy/reference model forward passes. 8.6% AEM 1.1% 5.62s 8.2% The entropy values used by AEM are obtained from the same recomputation pass used to compute the old-policy log-probabilities, incurring no additional model forward pass. Fig- Figure 4: Training time breakdown of GRPO+AEM. ure 4 reports a detailed per-iteration latency breakdown for training Qwen2.5-1.5B on ALFWorld using GRPO+AEM. The overall training time is dominated by rollout generation, model updates, and log-probability computation, which account for approximately 45.9%, 36.0%, and 16.8% of the total latency, respectively. In contrast, AEM-specific computations account for only 1.1%, indicating that AEM introduces negligible overhead in practice. base
base
7
Conclusions
This paper presents AEM, a supervision-free credit assignment framework for multi-turn agentic RL that uses response-level entropy as an intrinsic signal. Our analysis shows that entropy dynamics are governed by the interaction between advantage and relative response surprisal, which motivates an adaptive entropy modulation rule to regulate policy updates. By explicitly steering entropy dynamics during training, AEM enables a natural transition from exploration to exploitation in 9
a fully self-adaptive manner. As a lightweight plug-in to existing policy estimators, it improves credit assignment without auxiliary models, dense supervision, or restrictive structural assumptions. Across ALFWorld, WebShop, and SWE-bench-Verified, AEM consistently improves strong baselines, mitigates premature entropy collapse, and yields stronger final performance. These results highlight response-level entropy not only as a useful lens for understanding multi-turn agent training, but also as a practical mechanism for adaptive exploration–exploitation control.
Limitations In practice, Hresp (s) is not directly computable for open-ended LLM policies, as it would require summing over the entire response space. We therefore approximate the relative response surprisal with a group-based, length-normalized entropy proxy. While our experiments provide statistical evidence that this proxy is aligned with the desired entropy dynamics and improves training, it is still a heuristic surrogate rather than an exact estimator. Consequently, AEM does not guarantee optimal entropy modulation, and its behavior may depend on the quality and diversity of the sampled rollout group. Designing more accurate estimators of response-level relative surprisal is a promising direction for future work.
Acknowledgements We sincerely thank Peng Li from the Institute for AI Industry Research (AIR), Tsinghua University, for his valuable suggestions and insightful discussions, which helped improve the motivation, theoretical development, and presentation of this work. We also thank Mingzhe Lu from the University of Chinese Academy of Sciences for his valuable advice on refining the paper presentation. Songlin Zhou sincerely thanks Annan Li and Xiaomin Yuan from the Baidu FAMOU Institute for their generous encouragement and invaluable support in pursuing this project. Lastly, Haotian Zhao thanks Xiaofeng Wang for all the things.
References “Introducing Gemini 3 ", 2025. Available: https://blog.google/products-and-platforms/products/gemini/gemini3-collection/. “Introducing GPT-5.2 ", 2025. Available: https://openai.com/index/introducing-gpt-5-2/. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267, 2024. Shun-ichi Amari and Hiroshi Nagaoka. Methods of Information Geometry. American Mathematical Society and Oxford University Press, Providence, RI, 2000. Ruike Cao, Shaojie Bai, Fugen Yao, Liang Dong, Jian Xu, and Li Xiao. Atpo: Adaptive tree policy optimization for multi-turn medical dialogue. In The Fourteenth International Conference on Learning Representations, 2026. Arthur Chen, Zuxin Liu, Jianguo Zhang, Akshara Prabhakar, Zhiwei Liu, Shelby Heinecke, Silvio Savarese, Victor Zhong, and Caiming Xiong. Test-time adaptation for llm agents via environment interaction. In The Fourteenth International Conference on Learning Representations, 2026a. Kun Chen, Peng Shi, Fanfan Liu, Haibo Qiu, Zhixiong Zeng, Siqi Yang, and Wenji Mao. Flexible entropy control in rlvr with gradient-preserving perspective, 2026b. URL https://arxiv.org/abs/2602.09782. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025. Zheng Ding and Weirui Ye. Treegrpo: Tree-advantage grpo for online rl post-training of diffusion models. In The Fourteenth International Conference on Learning Representations, 2026.
10
Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks, 2025. URL https://arxiv.org/abs/2503.09572. Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. Webevolver: Enhancing web agent self-improvement with co-evolving world model. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8970–8986, 2025. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2e: Turning any github repository into a programming agent environment. In ICML 2024, 2024. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001. Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking llms for embodied decision making. In NeurIPS 2024, 2024. Ning Li, Xiangmou Qu, Jiamu Zhou, Jun Wang, Muning Wen, Kounianhua Du, Xingyu Lou, Qiuying Peng, and Weinan Zhang. Mobileuse: A hierarchical reflection-driven gui agent for autonomous mobile operation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The twelfth international conference on learning representations, 2023. Michael Luo, Naman Jain, Jaskirat Singh, Sijun Tan, Ameen Patel, Qingyang Wu, Alpay Ariyak, Colin Cai, Tarun Venkat, Shang Zhu, Ben Athiwaratkun, Manan Roongta, Ce Zhang, Li Erran Li, Raluca Ada Popa, Koushik Sen, and Ion Stoica. Deepswe: Training a state-of-the-art coding agent from scratch by scaling rl. N/A, 2025. Notion Blog. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928–1937, New York, New York, USA, 20–22 Jun 2016. Frank Nielsen. An Elementary Introduction to Information Geometry. Springer, Cham, 2020. Aleksei Petrenko, Ben Lipkin, Kevin Chen, Erik Wijmans, Marco Cusumano-Towner, Raja Giryes, and Philipp Krähenbühl. Entropy-preserving reinforcement learning, 2026. URL https://arxiv.org/abs/2603. 11682. Timo Schick, Jane Dwivedi-Yu, Roberto Dess ‘i, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS), 2023. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Han Shen. On entropy control in LLM-RL algorithms. In The Fourteenth International Conference on Learning Representations, 2026. Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. Advances in Neural Information Processing Systems, 37:4540–4574, 2024.
11
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/ 2010.03768. Sijun Tan, Michael Luo, Colin Cai, Tarun Venkat, Kyle Montgomery, Aaron Hao, Tianhao Wu, Arnav Balyan, Manan Roongta, Chenguang Wang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. rllm: A framework for post-training language agents. https://pretty-radio-b75.notion.site/ rLLM-A-Framework-for-Post-Training-Language-Agents-21b81902c146819db63cd98a54ba5f31, 2025. Notion Blog. Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying. Information gain-based policy optimization: A simple and effective approach for multi-turn llm agents. In International Conference on Learning Representations (ICLR), 2026. Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. Spa-rl: Reinforcing llm agents via stepwise progress attribution. arXiv preprint arXiv:2505.20732, 2025a. Jiawei Wang, Jiacai Liu, Yuqian Fu, Yingru Li, Xintao Wang, Yuan Lin, Yu Yue, Lin Zhang, Yang Wang, and Ke Wang. Harnessing uncertainty: Entropy-modulated policy gradients for long-horizon llm agents. arXiv preprint arXiv:2509.09265, 2025b. Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, Kaidi Cao, Vassilis N Ioannidis, Karthik Subbian, Jure Leskovec, and James Zou. Avatar: Optimizing llm agents for tool usage via contrastive reasoning. Advances in Neural Information Processing Systems, 37:25981–26010, 2024. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents. In Advances in Neural Information Processing Systems, 2025a. Wujiang Xu, Wentian Zhao, Zhenting Wang, Yu-Jhe Li, Can Jin, Mingyu Jin, Kai Mei, Kun Wan, and Dimitris N Metaxas. Epo: Entropy-regularized policy optimization for llm agents reinforcement learning. arXiv preprint arXiv:2509.22576, 2025b. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024a. John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. Embodied multi-modal agent trained by an llm from a parallel textworld. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26275–26285, 2024b. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 20744–20757, 2022. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Itamar Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. Se-gui: Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026.
12
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, Yang Katie Zhao, and Mingyi Hong. Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment. In ICML 2025 Workshop on Computer Use Agents, 2025. Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for llms: A survey. arXiv preprint arXiv:2509.02547, 2025. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025.
13
A
Pseudo-code Algorithm
The pseudo-code algorithm of AEM can be viewed in Algorithm 1. Algorithm 1 AEM Require: Sampled rollouts {τi }B i=1 in the current batch, all response groups G , t-th response (i, t) i in i-th trajectory, grouped by G ∈ G , entropy {Hℓ,t } of ℓ-th token in response (i, t), base base advantages {Ai,t }, temperature λ, stability constant ε. Ensure: Modulated AEM advantages {AAEM } i,t 1: Parse rollouts {τi } into environment-reactive agentic responses Si = {Si,1 , . . . , Si,Ki } 2: for all rollout i and response t ∈ {1, . . . , Ki } do P i 3: Compute response-level uncertainty proxy: H̄i,t ← |S1i,t | ℓ∈Si,t Hℓ,t 4: end for 5: for all G ∈ G do 6: Find group extrema: H̄Gmin ← min(j,n)∈G H̄j,n and H̄Gmax ← max(j,n)∈G H̄j,n 7: if H̄Gmax − H̄Gmin < 0.1 then 8: for all response (i, t) ∈ G do 9: Set the coefficient: αi,t ← 1 10: end for 11: else 12: for all response (i, t) ∈ G do 13: Min-max normalization: H̃i,t ← (H̄i,t − H̄Gmin ) / (H̄Gmax − H̄Gmin + ε) 14: Compute raw modulation weight: wi,t ← exp(−λH̃i,t ) 15: end for P 1 16: Compute group-average weight: w̄G ← |G| (j,n)∈G wj,n 17: for all response (i, t) ∈ G do 18: αi,t ← wi,t / (w̄G + ε) 19: end for 20: end if 21: end for 22: for all rollout i, response t do 23: Apply response-level uniform modulation: AAEM ← αi,t Abase i,t i,t 24: end for 25: return {AAEM } i,t
14
B
Experimental Training Curves
. 1.00
0.75 0.50 0.25 0.00
0
20
40
60
0.75 0.50 0.25 0.00
80 100 120 140
1.00
DAPO+AEM DAPO
Success Rate
GRPO+AEM GRPO
Success Rate
Success Rate
1.00
Training Step
0
20
40
60
0.50 0.25 0.00
80 100 120 140
GSPO + AEM GSPO
0.75
Training Step
0
20
40
60
80 100 120 140
Training Step
Figure 5: Training Curves of Qwen2.5-1.5B Model on ALFWorld. 1.00
0.75 0.50 0.25 0.00
0
20
40
60
0.75 0.50 0.25 0.00
80 100 120 140
1.00
DAPO + AEM DAPO
Success Rate
GRPO + AEM GRPO
Success Rate
Success Rate
1.00
Training Step
0
20
40
60
0.50 0.25 0.00
80 100 120 140
GSPO + AEM GSPO
0.75
Training Step
0
20
40
60
80 100 120 140
Training Step
Figure 6: Training Curves of Qwen2.5-1.5B Model on WebShop. 1.00
0.75 0.50 0.25 0.00
0
20
40
60
0.75 0.50 0.25 0.00
80 100 120 140
1.00
DAPO+AEM DAPO
Success Rate
GRPO+AEM GRPO
Success Rate
Success Rate
1.00
Training Step
0
20
40
60
0.50 0.25 0.00
80 100 120 140
GSPO + AEM GSPO
0.75
Training Step
0
20
40
60
80 100 120 140
Training Step
Figure 7: Training Curves of Qwen2.5-7B Model on ALFWorld. 1.00
0.75 0.50 0.25 0.00
0
20
40
60
80 100 120 140
Training Step
1.00
DAPO + AEM DAPO
0.75
Success Rate
GRPO + AEM GRPO
Success Rate
Success Rate
1.00
0.50 0.25 0.00
0
20
40
60
80 100 120 140
Training Step
GSPO + AEM GSPO
0.75 0.50 0.25 0.00
0
20
40
Figure 8: Training Curves of Qwen2.5-7B Model on WebShop.
15
60
80 100 120 140
Training Step
DeepSWE DeepSWE+AEM
Score
0.5 0.4 0.3 0.2 0
50
100
150
Training Step
200
250
Figure 9: Training reward curves of DeepSWE with and without AEM on the R2E dataset.
C
Theoretical Details and Proofs
In this section, we rigorously provide mathematical details, prove the theorems and properties related to algorithms listed in the main text. C.1
Proof of Theorem 3.2.1
Proof. Let B denote the set of prompts in the current training batch. For brevity, define X Z(at , st ) := Hℓ (at , st )1{ℓ ≤ |at |}.
(20)
ℓ≥1
Step 1: We show that the response-level entropy is the conditional expectation of the pathwise token-entropy sum. By definition, the response-level entropy is X Hresp (st ) = − πθ (at | st ) log πθ (at | st ).
(21)
at
Since the policy is autoregressive, for any response a = (y1 , . . . , y|a| ), log πθ (at | st ) =
|at | X
log pθ (yℓ | st , y<ℓ ).
(22)
ℓ=1
Therefore, Hresp (st ) = −
X
πθ (at | st )
at
=−
|at | X
log pθ (yℓ | st , y<ℓ )
ℓ=1
XX
πθ (at | st ) log pθ (Yℓ | st , Y<ℓ ) 1{ℓ ≤ |at |}
at ℓ≥1
=
X
Eat ∼πθ (·|st ) [− log pθ (Yℓ | st , Y<ℓ ) 1{ℓ ≤ |at |} | st ] .
(23)
ℓ≥1
Now apply the tower property. Since 1{ℓ ≤ |at |} is measurable with respect to the prefix (st , Y<ℓ ), X Hresp (st ) = Eat ∼πθ (·|st ) [− log pθ (Yℓ | st , Y<ℓ ) 1{ℓ ≤ |at |} | st ] ℓ≥1
=
X
Eat ∼πθ (·|st ) [1{ℓ ≤ |at |} E [− log pθ (Yℓ | st , Y<ℓ ) | st , Y<ℓ ] | st ]
ℓ≥1
=
X
Eat ∼πθ (·|st ) [Hℓ (at , st ) 1{ℓ ≤ |at |} | st ]
ℓ≥1
X Hℓ (at , st ) 1{ℓ ≤ |at |} st = Eat ∼πθ (·|st ) ℓ≥1
= Eat ∼πθ (·|st ) [Z(at , st ) | st ] . 16
(24)
Step 2: We show that the total token entropy is the expected sum of response-level entropies over on-policy visited states. Assume on-policy rollouts: s0 ∼ D,
τ ∼ Pθ (· | s0 ),
so that, at each visited state st , at ∼ πθ (· | st ). By the definition of Htotal , using the tower property and Step 1: "T −1 # X Htotal = Es0 ∼D,τ ∼Pθ (·|s0 ) Zθ (at , st ) . t=0
"T −1 X
= Es0 ∼D,τ ∼Pθ (·|s0 )
# E [Z(at , st ) | st ] .
t=0
"T −1 X
= Es0 ∼D,τ ∼Pθ (·|s0 )
# Hresp (st ) .
(25)
t=0
This proves that the total token entropy under on-policy rollouts is exactly the expected aggregation of response-level entropies over visited states. Combining Step 1 and Step 2 completes the proof. C.2
Policy Simplex
For a fixed state s, with finite action space |As | = m, the policy π = (πθ (a|s))a∈As is on the simplex ( ) X ∆◦ (As ) := π ∈ Rm : π(a) = 1, ∀a ∈ As , π(a) > 0 (26) a∈As
equipped with Fisher-Rao metric to become a Riemannian manifold: for any u, v ∈ Tπ ∆◦ (As ) gπ (u, v) :=
m X ua va a=1
πa
.
(27)
with tangent space is Tπ ∆◦ (As ) = {x ∈ Rm : 1⊤ x = 0}. (28) Fisher-Rao metric is the infinitesimal P quadratic form induced by the KL divergence. For any tangent perturbation δ ∈ Tπ ∆◦ (As ), i.e. a δa = 0, we have DKL (π + δ ∥ π) =
1 X δa2 1 + o(∥δ∥2 ) = gπ (δ, δ) + o(∥δ∥2 ). 2 πa 2 a∈As
Thus, the Fisher-Rao metric measures the local size of a policy update in the same units as a local KL trust region. C.3
State and Proof of the Generalized Version of Theorem 3.2.2
Theorem C.3.1 (Regularized Response-level entropy drift. Proved in Appendix C.3). Let gradF denote the natural gradient on the policy simplex, and consider the regularized local objective ℓa (π) = A(a, s) log π + β ψ(Hresp (π)) − γ DKL (π∥πref ). Then the directional derivative of Hresp along the update direction gradF ℓa (π) D E DRL (a; s) := gradF Htotal (π), gradF ℓa (π) Fisher-Rao
=
T −1 X
resp Ps0 ∼D,τ ∼Pθ (·|s0 ) [st = s]DRL (a; s)
t=0
17
(29) (30)
resp with DRL (a; s) defined by
resp DRL (a, s) = A(a, s) S(a | s) − Hresp (π) + (βψ ′ (Hresp (π)) + γ) Vara∼π(·|s) S(a | s) {z } | | {z } (I) reward-driven term
(II) entropy-expanding term
− γ Cova∼π(·|s) S(a | s), Sref (a | s) . | {z }
(31)
(III) reference-alignment term
If we let β = γ = 0, i.e., only reward objective is considered, then we obtain the Theorem 3.2.2. Remark C.3.2. The decomposition in (31) yields four immediate observations. • The entropy and KL regularization terms are state-level modulation terms: unlike the reward-driven term (I), they do not depend on the sampled action signal A(a, s). • Term (I) shows that the advantage and relative surprisal of sampled action can jointly determine entropy dynamics without entropy and KL regularization. ′ • The entropy regularizer contributes a positive force through βψ (Hresp ) Vara∼π(·|s) S(a | s) , which is consistent with its intended role. • The KL term contributes two parts: a positive variance term γ Vara∼π(·|s) S(a | s) , and a covariance term −γ Cova∼π(·|s) S(a | s), Sref (a | s) , whose sign is generally not fixed. Fig 1 demonstrates the entropy dynamics along updating in on three-action simplex.
Proof. Fix a state s. For brevity, write πb := π(b | s),
ρb := πref (b | s),
Aa := A(a, s),
and assume πb > 0 and ρb > 0 for all b ∈ As . Define Sbref := − log ρb ,
Sb := − log πb ,
H := Hresp (π) =
X
πb Sb .
b∈As
We also note Vara∼π(·|s) (S) :=
X
πb (Sb − H)2 ,
b∈As
and Cova∼π(·|s) (S, Sref ) :=
X
X πb (Sb − H) Sbref − Eπ [Sref ] = πb (Sb − H)Sbref ,
b∈As
b∈As
where the last equality follows from X
πb (Sb − H) = 0.
b∈As
Step 1. We first show Eq. (30): By Eq. (8) and definition, with the assumption that gradients are not propagated through the rollout distribution Pθ (τ | s0 )., since for st ̸= s, Hresp (st ) is a constant on the ∆◦ (As ), we then deduce: D E DRL (a; s) = gradF Htotal (π), gradF ℓa (π) * "T −1 #+ X F F = grad Es0 ∼D,τ ∼Pθ Hresp (π), grad ℓa (π) t=0
* =
F
grad Es0 ∼D,τ ∼Pθ
"T −1 X
#
+ F
1(st = s)Hresp (π) , grad ℓa (π)
t=0
=
T −1 X
D E Es0 ∼D,τ ∼Pθ [1(st = s)] gradF Hresp (π), gradF ℓa (π)
t=0
=
T −1 X
Ps0 ∼D,τ ∼Pθ [st = s]DRL (a; s),
t=0
18
(32)
which is exactly Eq. (30). Step 2. For any smooth function f : ∆◦ (As ) → R, its Fisher-Rao gradient is gradF f (π) = π ⊙ ∇π f − (π ⊤ ∇π f )1 .
(33)
where ⊙ denotes the Hadamard product, and 1 is the vector with all components to be one. Indeed, for any ξ ∈ Tπ ∆◦ (As ), since 1⊤ ξ = 0, X πb ∂πb f − π ⊤ ∇π f ξb ⊤ gπ π ⊙ ∇π f − (π ∇π f )1 , ξ = πb b∈As X X ∂πb f ξb − (π ⊤ ∇π f ) ξb = b∈As
b∈As ⊤
= ∇π f ξ = dfπ [ξ].
(34)
Thus (33) is the Riemannian gradient under the Fisher-Rao metric. Step 3. We compute the Fisher-Rao gradients of all terms in ℓa (π) = Aa log πa + β ψ(Hresp (π)) − γ DKL (π∥πref ). First, for the reward-driven term ℓA a (π) := Aa log πa , we have A ⊤ A gradF ℓA a (π) = π ⊙ ∇π ℓa (π) − (π ∇π ℓa (π))1 Aa =π⊙ ea − A a 1 πa = Aa (ea − π).
(35)
Next, for the response-level entropy, we have ∂πb Hresp (π) = −(1 + log πb ) = Sb − 1,
π ⊤ ∇π Hresp (π) =
X
πb (Sb − 1) = H − 1.
b∈As
Hence, gradF Hresp (π) = π ⊙ (Sb − 1)b∈As − (H − 1)1 = π ⊙ (Sb )b∈As − H1 .
(36)
For the entropy regularizer ℓE (π) := β ψ(Hresp (π)), the chain rule gives ∇π ℓE (π) = β ψ ′ (H)∇π Hresp (π). Since the Fisher-Rao projection is linear, we have gradF ℓE (π) = β ψ ′ (H) gradF Hresp (π) = β ψ ′ (H) π ⊙ (Sb )b∈As − H1 . Finally, consider the KL divergence term K(π) := DKL (π∥πref ) =
X b∈As
19
πb log
πb . ρb
(37)
we have gradF K(π) = π ⊙ ∇π K(π) − (π ⊤ ∇π K(π))1 ! πb +1 − (K(π) + 1)1 =π⊙ log ρb b∈As ! πb =π⊙ log − K(π)1 . ρb b∈As
(38)
Combining (35), (37), and (38), we obtain gradF ℓa (π) = Aa (ea − π) + β ψ ′ (H) π ⊙ (Sb )b∈As − H1 ! πb − K(π)1 . −γπ⊙ log ρb b∈As
(39)
Step 4. We compute resp DRL (a, s) = gπ gradF Hresp (π), gradF ℓa (π) . By (36) and (39), resp DRL (a, s) = gπ (π ⊙ (S − H1), Aa (ea − π))
+ β ψ ′ (H) gπ (π ⊙ (S − H1), π ⊙ (S − H1)) π . − γ gπ π ⊙ (S − H1), π ⊙ log − K(π)1 ρ
(40)
For the first term, gπ (π ⊙ (S − H1), Aa (ea − π)) = Aa
X πb (Sb − H) (ea )b − πb
πb
b∈As
"
#
= Aa Sa − H −
X
πb (Sb − H)
b∈As
= Aa (Sa − H).
(41)
For the second term, gπ (π ⊙ (S − H1), π ⊙ (S − H1)) =
X π 2 (Sb − H)2 b
b∈As
=
X
πb πb (Sb − H)2
b∈As
= Vara∼π(·|s) (S).
(42)
For the third term, since X
πb (Sb − H) = 0,
b∈As
we have π gπ π ⊙ (S − H1), π ⊙ log − K(π)1 ρ X πb = πb (Sb − H) log − K(π) ρb b∈As X πb = πb (Sb − H) log . ρb b∈As
20
(43)
Using log
πb = log πb − log ρb = −Sb + Sbref , ρb
we get X
πb (Sb − H) log
b∈As
X πb πb (Sb − H) −Sb + Sbref = ρb b∈As X X πb (Sb − H)Sbref πb (Sb − H)Sb + =− b∈As
b∈As
= − Vara∼π(·|s) (S) + Cova∼π(·|s) (S, Sref ).
(44)
Substituting (41), (42), and (44) into (40), we obtain resp DRL (a, s) = Aa (Sa − H) + β ψ ′ (H) Vara∼π(·|s) (S) + γ Vara∼π(·|s) (S) − γ Cova∼π(·|s) (S, Sref ) = A(a, s) S(a | s) − Hresp (π) + βψ ′ (Hresp (π)) + γ Vara∼π(·|s) S(a | s) − γ Cova∼π(·|s) S(a | s), Sref (a | s) . (45)
This proves the theorem. C.4
Doob decomposition of fixed-length response surprisal
Proposition C.4.1 (Doob decomposition of fixed-length response surprisal). Fix a state s, and let a = (Y1 , . . . , YL ) ∼ πθ (· | s) be a response sampled from the policy. Define the realized token surprisal Xℓ := − log pθ (Yℓ | s, Y<ℓ ), then the response surprisal admits the decomposition S(a | s) =
L X
Xℓ =
ℓ=1
where Mk := quently,
L X
Hℓ (a, s) + ML ,
ℓ=1
Pk
L ℓ=1 Xℓ − Hℓ (a, s), is a zero-mean martingale with respect to (Fk )k=0 . Conse-
S(a | s) − Hresp (s) =
L X
! Hℓ (a, s) − Hresp (s)
+ ML .
(46)
ℓ=1
Proof. For each ℓ, by definition, E[Xℓ | Fℓ−1 ] = E[− log pθ (Yℓ | s, Y<ℓ ) | s, Y<ℓ ] X =− pθ (y | s, Y<ℓ ) log pθ (y | s, Y<ℓ ) y∈V
= Hℓ (a, s).
(47)
Thus Hℓ (a, s) is Fℓ−1 -measurable and hence predictable.
E[Xℓ − Hℓ (a, s) | Fℓ−1 ] = E[Xℓ − Hℓ (a, s) | Fℓ−1 ] = E[Xℓ | Fℓ−1 ] − Hℓ (a, s) = 0. Therefore, Mk :=
Pk
ℓ=1 Xℓ − Hℓ (a, s) is a martingale.
21
(48)
With the definition of Xℓ − Hℓ (a, s), we obtain S(a | s) =
L X
Xℓ
ℓ=1
=
L X
(Hℓ (a, s) + Xℓ − Hℓ (a, s))
ℓ=1
=
L X
Hℓ (a, s) + ML .
(49)
ℓ=1
Finally, by the definition of response-level entropy over fixed-length responses, subtracting Hresp (s) from both sides of the Doob’s decomposition gives ! L X S(a | s) − Hresp (s) = Hℓ (a, s) − Hresp (s) + ML . ℓ=1
This completes the proof. C.5
Parametrized Version of Entropy Drift
In this section, we analyze how the parametrized response entropy varies along the sample-induced update direction in parameter space. The resulting entropy-drift formula is analogous in spirit to the main result in Theorem 3.2.2. However, once the policy is parameterized by θ, the drift additionally involves a kernel-weighted baseline term Bker (a;s). Theorem C.5.1 (Parametrized regularized response-level entropy drift). Fix a state s. Let πb := πθ (b | s),
ρb := πref (b | s),
Gb := ∇θ log πθ (b | s),
and Sb := − log πb ,
Sbref := − log ρb ,
H := Hresp (s) =
X
πb Sb .
b∈As
Define the policy-gradient kernel K(b, c; s) := ⟨Gb , Gc ⟩. Then the Euclidean parameter-space entropy drift satisfies θ DRL (a; s) = −A(a, s) [πθ (a | s)(H − Sa )K(a, a; s) + Bker (a; s)] + βψ ′ (H) + γ Vθ (S; s) − γ Cθ (S, Sref ; s),
(50)
where Bker (a; s) is the cross-response residual introduced by shared parameterization: Vθ (S; s) := Eb,c∼πθ (·|s) [(Sb − H)(Sc − H)K(b, c; s)] = ∥∇θ Hresp (s)∥22 , Cθ (S, Sref ; s) := Eb,c∼πθ (·|s) (Sb − H) Scref − Eπθ [Sref ] K(b, c; s) X Bker (a; s) := πθ (b | s)(H − Sb )K(b, a; s).
(51) (52) (53)
b̸=a
Proof of Theorem C.5.1 Proof. We first prove a general formula for an arbitrary smooth regularizer ℓR a (θ) = A(a, s) log πθ (a | s) + R(πθ (· | s)). Step 1. Gradient of response-level entropy. By definition, X ∇θ Hresp (s) = − ∇θ πb log πb b∈As
=−
X
(log πb + 1)∇θ πb
b∈As
=−
X b∈As
22
πb (log πb + 1)Gb .
(54)
Using the zero-score identity X
X
πb Gb =
∇θ πb = ∇θ
b∈As
b∈As
X
πb = 0,
b∈As
we have X
∇θ Hresp (s) = −
πb (log πb + H)Gb
b∈As
=
X
πb (Sb − H)Gb .
(55)
b∈As
Step 2. General regularizer. Let rc := ∂πc R(π),
X
r̄ :=
πc rc .
c∈As
By the chain rule, ∇θ R(πθ (· | s)) =
X
∂πc R(π)∇θ πc
c∈As
=
X
πc rc Gc
c∈As
=
X
πc (rc − r̄)Gc ,
(56)
c∈As
Therefore, ∇θ ℓR a = A(a, s)Ga +
X
πc (rc − r̄)Gc .
(57)
c∈As
Taking the inner product between (55) and (57), we obtain θ,R DRL (a; s) := ∇θ Hresp (s), ∇θ ℓR a X = A(a, s) πb (Sb − H)⟨Gb , Ga ⟩ b∈As
+
X
πb πc (Sb − H)(rc − r̄)⟨Gb , Gc ⟩
b,c∈As
= −A(a, s)Eb∼πθ (·|s) [(H − Sb )K(b, a; s)] + Eb,c∼πθ (·|s) [(Sb − H)(rc − r̄)K(b, c; s)] . This is the general regularized parameter-space entropy-drift identity. Step 3. Apply the general identity to entropy and KL regularization. Now take R(π) = βψ(Hresp (π)) − γDKL (π∥πref ), where DKL (π∥πref ) =
X
πc log
c∈As
πc , ρc
ρc := πref (c | s).
For the entropy term, ∂πc Hresp (π) = −(1 + log πc ) = Sc − 1. For the KL term, ∂πc DKL (π∥πref ) = log Therefore,
πc + 1. ρc
rc = ∂πc R(π) = βψ ′ (H)(Sc − 1) + γSc − γScref − γ.
Let S̄ref := Eπθ [Sref ] =
X c∈As
23
πc Scref .
(58)
we have r̄ =
X
πc rc = βψ ′ (H)(H − 1) + γH − γ S̄ref − γ.
(59)
c∈As
Hence rc − r̄ = βψ ′ (H)(Sc − H) + γ(Sc − H) − γ(Scref − S̄ref ) = βψ ′ (H) + γ (Sc − H) − γ(Scref − S̄ref ).
(60)
Substituting (60) into the second term of (58), we get Eb,c∼πθ [(Sb − H)(rc − r̄)K(b, c; s)] = βψ ′ (H) + γ Eb,c∼πθ [(Sb − H)(Sc − H)K(b, c; s)] − γEb,c∼πθ (Sb − H)(Scref − S̄ref )K(b, c; s) .
(61)
Define Vθ (S; s) := Eb,c∼πθ (·|s) [(Sb − H)(Sc − H)K(b, c; s)] , Cθ (S, Sref ; s) := Eb,c∼πθ (·|s) (Sb − H)(Scref − S̄ref )K(b, c; s) .
(62) (63)
Then θ DRL (a; s) = −A(a, s)Eb∼πθ (·|s) [(H − Sb )K(b, a; s)] + βψ ′ (H) + γ Vθ (S; s) − γCθ (S, Sref ; s).
It remains to verify
(64)
Vθ (S; s) = ∥∇θ Hresp (s)∥22 .
By (55), ∥∇θ Hresp (s)∥22 =
* X
+ πb (Sb − H)Gb ,
X
πc (Sc − H)Gc
c
b
=
X
πb πc (Sb − H)(Sc − H)⟨Gb , Gc ⟩
b,c
= Vθ (S; s).
(65)
Finally, separating the b = a term from the task-driven part, − A(a, s)Eb∼πθ (·|s) [(H − Sb )K(b, a; s)]
= −A(a, s) πθ (a | s)(H − Sa )K(a, a; s) +
X
πθ (b | s)(H − Sb )K(b, a; s) .
b̸=a
With Bker (a; s) :=
X
πθ (b | s)(H − Sb )K(b, a; s),
b̸=a
we obtain the split form. This completes the proof.
24
(66)
D
Experimental Details
D.1
Base RL Methods Used in Experiments
PPO. Proximal Policy Optimization (PPO) Schulman et al. (2017) is a representative actor-critic algorithm that stabilizes policy learning by constraining the update to remain close to the behavior policy. In LLM post-training, PPO typically treats each token as an action and estimates token-level advantages with a learned value function, usually via generalized advantage estimation (GAE). Its clipped surrogate objective is h i πθ (at | st ) JPPO (θ) = Et min ρt (θ)Ât , clip ρt (θ), 1 − ϵ, 1 + ϵ Ât , ρt (θ) = . (67) πθold (at | st ) PPO is stable and widely adopted, but it is relatively expensive for large language models because it requires an additional critic/value model to estimate Ât . GRPO. Group Relative Policy Optimization (GRPO) Shao et al. (2024) extends the group-based idea by replacing critic-based advantages with within-group relative rewards. Given a query q, GRPO samples a group of outputs {oi }G i=1 and computes a normalized group-based advantage ÂGRPO = i
Ri − mean({Rj }G j=1 ) , G std({Rj }j=1 ) + ϵ
(68)
which is shared across all tokens in output oi under the standard outcome-level setting. The policy is then updated by maximizing the clipped objective JGRPO (θ) = |oi | G X X 1 1 E min ρi,t (θ)ÂGRPO , clip ρi,t (θ), 1 − ϵ, 1 + ϵ ÂGRPO − λDKL πθ ∥ πref i i G i=1 |oi | t=1 (69) where ρi,t (θ) =
πθ (oi,t | q, oi,<t ) . πθold (oi,t | q, oi,<t )
(70)
GRPO preserves the stable clipped update of PPO while eliminating the critic, which makes it especially attractive for large-scale LLM reinforcement learning. DAPO. Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) Yu et al. (2025) is a GRPO-style estimator designed to improve GRPO-style training in long-form reasoning settings, especially long chain-of-thought trajectories. DAPO keeps the group-based advantage formulation ÂDAPO = i
Ri − mean({Rj }G j=1 ) , G std({Rj }j=1 ) + ϵ
(71)
but replaces the standard response-level averaging used in GRPO with a token-level aggregation over all tokens in the sampled group, which better balances updates across responses of different lengths: JDAPO (θ) = |oi | G X X 1 . E PG min ρi,t (θ)ÂDAPO , clip ρi,t (θ), 1 − ϵlow , 1 + ϵhigh ÂDAPO i i |o | i=1 i i=1 t=1
(72)
In the original formulation, DAPO further removes the explicit KL term and improves GRPOstyle training with four practical techniques: decoupled asymmetric clipping, dynamic sampling of informative groups, token-level policy-gradient loss, and overlong reward shaping. These refinements make policy optimization substantially more stable in long-CoT settings. 25
GSPO. Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025) is a GRPO-style groupbased RL method that moves importance weighting and clipping from token level to sequence level. For a given query q, GSPO samples a group of outputs {oi }G i=1 and uses the same normalized group-based advantage as GRPO: ÂGSPO = i
Ri − mean({Rj }G j=1 ) . G std({Rj }j=1 )
It then defines a length-normalized sequence-level importance ratio 1 |oi | X πθ (oi | q) |oi | 1 π (o | q, o ) θ i,t i,<t si (θ) = = exp log , πθold (oi | q) |oi | t=1 πθold (oi,t | q, oi,<t )
(73)
(74)
where the length normalization keeps the ratio scale comparable across responses of different lengths. The policy is optimized with the clipped sequence-level objective # " G 1 X JGSPO (θ) = E min si (θ)ÂGSPO , clip si (θ), 1 − ϵ, 1 + ϵ ÂGSPO . (75) i i G i=1 Compared with token-level clipping, GSPO aligns the optimization granularity with sequence-level rewards and improves training stability. D.2
Implementation Details
For ALFWorld and WebShop, we use rule-based outcome rewards: successful trajectories receive a reward of 10, unsuccessful trajectories receive 0, and invalid actions incur an additional penalty of −0.1. Across both benchmarks, the actor learning rate is set to 1 × 10−6 , the rollout temperature is 1.0, the validation temperature is 0.4, and the KL loss coefficient is fixed to 0.01. For all group-based RL methods, we use a rollout group size of N = 8 and sample 16 groups per rollout, yielding 128 environments in total. For all AEM experiments, we simply set λ = 1. ALFWorld uses a maximum prompt length of 2048 tokens and a maximum response length of 512 tokens, with each episode allowed to interact with the environment for at most 50 steps. WebShop uses a maximum prompt length of 4096 tokens and a maximum response length of 512 tokens, with each episode capped at 15 environment steps. For the SWE task, we use binary outcome rewards, assigning 1 to successful trajectories and 0 to unsuccessful ones. We train Qwen3-32B with a learning rate of 1 × 10−6 , a maximum prompt length of 4096 tokens, and a maximum response length of 65536 tokens. For group-based training, the rollout group size is fixed to N = 8, and the training batch size is 64. The sampling temperature is set to 1.0 during training and 0.6 during evaluation. In terms of compute, we train Qwen2.5-1.5B on 4×A800 GPUs and Qwen2.5-7B on 8×A800 GPUs for 150 steps on ALFWorld and WebShop. For SWE, we train Qwen3-32B on 64×H200 GPUs for 250 steps. The temperature λ controls the dynamic range of the modulation coefficient α: larger λ yields a sharper separation between high- and low-uncertainty spans. In our experiments, we simply set λ = 1. D.3
Prompts
Prompt Template for ALFWorld 1
2
You are an expert agent operating in the ALFRED embodied Environment. Your task is to: {task_description}. Prior to this step, you have already taken {step_count} step( s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history}. You are now at step {current_step} and your current observation is: {current_observation}. Your admissible actions of the current situation are: [{admissible_actions}]. Now it’s your turn to take an action. You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think>
26
tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
Prompt Template for WebShop 1 2
3
4
You are an expert autonomous agent operating in the WebShop e-commerce environment. Your task is to: {task_description}. Prior to this step, you have already taken { step_count} step(s). Below are the most recent {history_length} observations and the corresponding actions you took: {action_history}. You are now at step { current_step} and your current observation is: {current_observation}. Your admissible actions for the current situation are: [{available_actions}]. Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
Prompt Template for R2E Training 1
You are a programming agent who is provided a github issue and repository bash environment and is tasked to solve certain tasks (e.g., {task_types}) to resolve the issue.
2 3
We have access to the following functions:
4 5 6 7 8 9
-- BEGIN FUNCTION #1: file_editor -Description: {file_editor_description} Parameters: {file_editor_parameters} -- END FUNCTION #1 --
10 11 12 13 14 15
-- BEGIN FUNCTION #2: execute_bash -Description: {execute_bash_description} Parameters: {execute_bash_parameters} -- END FUNCTION #2 --
16 17 18 19 20 21
-- BEGIN FUNCTION #3: search -Description: {search_description} Parameters: {search_parameters} -- END FUNCTION #3 --
22 23 24 25 26 27
-- BEGIN FUNCTION #4: finish -Description: {finish_description} Parameters: {finish_parameters} -- END FUNCTION #4 --
28 29 30
If you choose to call a function ONLY reply in the following format with NO suffix: {function_call_format}
31 32 33 34
<IMPORTANT> {important_rules} </IMPORTANT>
27
Prompt Template for SWE-bench-Verified Eval 1
You are a programming agent who is provided a github issue and repository bash environment and is tasked to solve certain tasks (e.g., {task_types}) to resolve the issue.
2 3
We have access to the following functions:
4 5 6 7
-- BEGIN FUNCTION #1: file_editor -{file_editor_block} -- END FUNCTION #1 --
8 9 10 11
-- BEGIN FUNCTION #2: execute_bash -{execute_bash_block} -- END FUNCTION #2 --
12 13 14 15
-- BEGIN FUNCTION #3: search -{search_block} -- END FUNCTION #3 --
16 17 18 19
-- BEGIN FUNCTION #4: finish -{finish_block} -- END FUNCTION #4 --
20 21
If you choose to call a function ONLY reply in the following format with NO suffix:
22 23
{function_call_format}
24 25 26 27
<IMPORTANT> {important_rules} </IMPORTANT>
28